Huaijin
1f710bb22c
fix(self-profiles): build CPU self-profiles without jemalloc and tag them with the cluster name ( #16070 )
2026-10-30 12:03:14 +00:00
Hengfei Yang
1a2d94adac
chore: upgrade rust toolchain to nightly-2026-10-24 ( #15782 )
2026-10-26 01:23:11 +00:00
Hengfei Yang
610cd70a13
fix: generate and store the ext auth salt on a new install instead of refusing to start ( #15191 )
2026-10-08 09:56:01 +00:00
Ashish Kolhe and github-actions[bot]
46678ea8da
fix: tighten input validation, auth and outbound request checks ( #15054 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-08 00:54:05 +00:00
Anupam Mediratta and mmosarafO2
c3dca1f7ac
chore: secure statefulset.yaml (CWE-732) ( #15063 )
...
Co-authored-by: mmosarafO2 <mosraf@openobserve.ai >
2026-10-05 05:32:11 +00:00
Sangharsh Agarwal
74449c2332
fix: collapse apt-get layers and clean cache in Dockerfile and Dockerfile.amd64 ( #13633 )
...
## Summary
Fixes three Dockerfile best practice issues in `Dockerfile` and
`Dockerfile.amd64` runtime stages. `Dockerfile.pr` already follows the
correct pattern — this brings the other two files in line with it.
## Why
**Split `RUN` apt-get layers (cache invalidation bug):**
The runtime stage ran `apt-get update` and `apt-get install` in separate
`RUN` instructions. Docker caches each layer independently, so `apt-get
update` can be cached while `apt-get install` runs fresh — resulting in
stale package indexes and potentially inconsistent installs. This is
[explicitly called
out](https://docs.docker.com/build/building/best-practices/#apt-get ) in
Docker's official best practices.
**Missing `--no-install-recommends`:**
The second `apt-get install` was missing `--no-install-recommends`,
pulling in unnecessary recommended packages and bloating the image.
**apt cache left in image:**
No `rm -rf /var/lib/apt/lists/*` after the install, leaving the package
index cache in the final image unnecessarily.
## What changed
Both `Dockerfile` and `Dockerfile.amd64` runtime stages: collapsed three
separate `RUN` instructions into one, added `--no-install-recommends`,
and added `rm -rf /var/lib/apt/lists/*` to clean the apt cache.
**Before:**
```dockerfile
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates
RUN apt-get install -y curl htop iftop sysstat procps lsof net-tools
RUN update-ca-certificates
```
**After:**
```dockerfile
RUN apt-get update && \
apt-get install -y --no-install-recommends \
ca-certificates curl htop iftop sysstat procps lsof net-tools && \
update-ca-certificates && \
rm -rf /var/lib/apt/lists/*
```
No changes to installed packages, build stages, base images, or any
other files.
2026-08-07 06:46:46 +00:00
Hengfei Yang
8e7e9eb03e
ci: update docker token ( #13184 )
2026-07-14 20:26:15 +08:00
Hengfei Yang
39fc204061
chore(build): skip updating latest docker tags for pre-release versions ( #13166 )
2026-07-14 12:42:25 +08:00
Hengfei Yang
3e865a3e6a
ci: update rust ( #12394 )
2026-06-02 03:58:01 +00:00
Hengfei Yang
7ca2053026
ci: fix deps for vortex ( #10694 )
2026-02-27 23:28:36 +08:00
Hengfei Yang
cb911e6976
ci: disable sccache ( #10392 )
2026-02-05 21:38:39 +08:00
Hengfei Yang
b3c63ed35b
fix: docker build ( #9299 )
2025-11-26 13:46:23 +08:00
Hengfei Yang and Huaijin
157306eb3e
ci: update base image ( #9271 )
...
Co-authored-by: Huaijin <haohuaijin@gmail.com >
2025-11-24 23:28:13 +08:00
Hengfei Yang
5ffd64d65b
feat: add configurable batch size for file list deletion ( #9264 )
2025-11-24 16:42:12 +08:00
mmosarafO2
e93d0840f1
ci: udpated Dockerfile.pr ( #9260 )
2025-11-24 11:34:31 +05:30
Simha G
6ca371ec99
fix: Concurrency control for file locking has multiple anti-concurrent design patterns in the way ( #8523 )
2025-11-17 20:35:43 +08:00
Prabhat Sharma and mmosarafO2
3e92683350
fix: build manifest of both builds succeed plus more ( #8729 )
...
### **PR Type**
Enhancement
___
### **Description**
- Split PR image builds by architecture
- Unify Dockerfile for both builds
- Improve build cache isolation per arch
- Gate final job on both builds success
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["build_pr_image_amd64"] -- "produces amd64 image" --> C["build"]
B["build_pr_image_arm64"] -- "produces arm64 image" --> C
C -- "runs only if both succeed" --> D["post-build steps"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>build-pr-image.yml</strong><dd><code>Split and harden
PR image build workflow</code>
</dd></summary>
<hr>
.github/workflows/build-pr-image.yml
<ul><li>Rename jobs per arch: amd64/arm64<br> <li> Use single Dockerfile
path for both<br> <li> Add arch-scoped cache keys and prefixes<br> <li>
Make final job depend on both success</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8729/files#diff-ebeabab3ec6b3ad90cc66cb9c0d9f2ad72c7ec6116e30e9801241b46992742d6 ">+15/-22</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>Dockerfile.pr</strong><dd><code>Add unified runtime
Dockerfile for PR builds</code>
</dd></summary>
<hr>
deploy/build/Dockerfile.pr
<ul><li>Add Debian slim runtime base<br> <li> Install required runtime
packages<br> <li> Copy binary and init data dir<br> <li> Set default
command to run service</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8729/files#diff-0db44a1fcb127375b96718556fd43c738708efbcaf0f83a9bffe39d4bfa95a02 ">[link]</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: mmosarafO2 <mosraf@openobserve.ai >
2025-10-06 09:17:26 +05:30
Hengfei Yang
97d09eff1e
fix: support prefix star match_all ( #8451 )
2025-09-16 13:05:02 +08:00
udev
7cca606f16
feat: add metrics to count wal files in ingester ( #7898 )
...
### **PR Type**
Enhancement
other changes:
- change default of batch exporter to false
- update gitignore
___
### **Description**
- Add WAL file count gauge metric
- Implement and register collect_wal_metrics
- Invoke WAL metrics in job loop
- Disable batching exporter by default
___
### Diagram Walkthrough
```mermaid
flowchart LR
WRITERS["In-memory writers"]
IMMUTABLES["Rotated immutables"]
COLLECT["collect_wal_metrics()"]
METRIC["INGEST_WAL_FILES gauge"]
JOB["job::update_wal_metrics()"]
WRITERS -- "active WAL files" --> COLLECT
IMMUTABLES -- "rotated WAL files" --> COLLECT
COLLECT -- "set metric" --> METRIC
JOB -- "calls" --> COLLECT
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Disable pipeline batch
exporter by default</code>
</dd></summary>
<hr>
src/config/src/config.rs
- Changed `ZO_PIPELINE_BATCH_ENABLED` default to false
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7898/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62 ">+1/-1</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>metrics.rs</strong><dd><code>Add and register WAL files
metric</code>
</dd></summary>
<hr>
src/config/src/metrics.rs
<ul><li>Added <code>INGEST_WAL_FILES</code> gauge metric<br> <li>
Registered metric in registry</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7898/files#diff-950d37bdd77aa533539acd818a3fcdebd90ccb1f09acb5490b9c0f9b51cb2090 ">+15/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>immutable.rs</strong><dd><code>Expose get_key on
Immutable</code>
</dd></summary>
<hr>
src/ingester/src/immutable.rs
- Added `Immutable::get_key()` accessor
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7898/files#diff-eee941ce73df39e37edae4a98ba3ca679f2c61592ad3e5afa47eb7793c9b0410 ">+4/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>lib.rs</strong><dd><code>Expose collect_wal_metrics in
lib</code>
</dd></summary>
<hr>
src/ingester/src/lib.rs
- Re-exported `collect_wal_metrics` in public API
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7898/files#diff-603267e2f712eeb4bc46fdff981f1fad1f3a9d3b989a7db4b3c0984fc5b1ed43 ">+2/-2</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>writer.rs</strong><dd><code>Implement
collect_wal_metrics in writer</code>
</dd></summary>
<hr>
src/ingester/src/writer.rs
<ul><li>Implemented <code>collect_wal_metrics()</code> function<br> <li>
Counts active and rotated WAL files<br> <li> Updates
<code>INGEST_WAL_FILES</code> metric</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7898/files#diff-d1a9b1542cc2758afe4fa67abd454dd818718f4e1c7570303519af5a58d6301f ">+38/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>metrics.rs</strong><dd><code>Integrate WAL metrics in
job loop</code>
</dd></summary>
<hr>
src/job/metrics.rs
<ul><li>Added <code>update_wal_metrics()</code> function<br> <li>
Invoked WAL metrics in job loop when ingester</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7898/files#diff-ca98d90d1293682c98c33a777a653cf3c1466f6297969ae94a359bb81abca4b1 ">+13/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
2025-08-10 22:33:30 +05:30
mmosarafO2
5e3acceae9
chore: updated statefulset.yaml with o2cr.ai/openobserve ( #7749 )
...
### **User description**
updated statefulset.yaml with o2cr.ai/openobserve
___
### **PR Type**
Enhancement
___
### **Description**
- Switch OpenObserve container image to new registry
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>statefulset.yaml</strong><dd><code>Update OpenObserve
image registry path</code>
</dd></summary>
<hr>
deploy/k8s/statefulset.yaml
<ul><li>Replaced <code>public.ecr.aws/zinclabs/openobserve:latest</code>
image<br> reference <br>with new registry<br> <li> Now uses
<code>o2cr.ai/openobserve/openobserve:latest</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7749/files#diff-b7427b97c324a26e57eee30cfb5a659cf40e231f0299ac51bb70ddea5c24016e ">+1/-1</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
2025-07-31 00:22:00 +05:30
Hengfei Yang
14cfcf29dc
ci: update rust to nightly-2025-05-20 ( #7722 )
2025-07-28 11:07:36 +08:00
Hengfei Yang
0f5a31d0df
fix: cache file for alert querier ( #6573 )
...
We already checked `ConsistentHash` on the source node, so we don't need
check again, just cache the file send to the node.
2025-04-18 19:51:04 +08:00
Hengfei Yang
595490ea33
ci: update rust to nightly-2025-03-02 ( #6145 )
2025-03-02 17:27:10 +08:00
Hengfei Yang
3be0d3eda3
ci: add more debug tools ( #5601 )
2025-01-07 12:09:48 +08:00
Hengfei Yang
d6487eb9fe
ci: update rust version ( #5143 )
2024-11-25 19:20:07 +08:00
Hengfei Yang
df0a4a79ae
feat: refactor merge file on compactor ( #4971 )
2024-11-07 17:30:10 +08:00
Hengfei Yang
f78d013e54
ci: add docker build for Pi ( #4964 )
2024-11-01 22:14:49 +08:00
Ashish Kolhe and Hengfei Yang
335357f943
fix: updated cached enrichment table ( #4176 )
...
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced asynchronous functions for managing enrichment tables:
`notify_update`, `delete`, and `watch`.
- Added a notification mechanism to improve updates in the enrichment
data handling process.
- Enhanced deletion process for enrichment tables, reinforcing data
integrity.
- Introduced a new configuration option for maximum enrichment table
size to enhance user configurability.
- **Bug Fixes**
- Improved readability and debuggability in the monitoring process for
enrichment tables.
- **Chores**
- Updated the base image in the Dockerfile to a newer Debian version for
improved security and compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com >
2024-08-11 15:14:05 +05:30
Juan Calderon-Perez
3f6e48c199
chore: Bump PR debian image ( #4108 )
2024-08-02 15:23:48 +08:00
Juan Calderon-Perez
29b7ca1347
chore: Bump Node.js to v18.20.4 ( #4107 )
2024-08-02 15:23:14 +08:00
Hengfei Yang
03f1b192c2
feat: ugprade rust to 2024-07-07 ( #3922 )
2024-07-08 20:56:56 +08:00
Ankur Srivastava
5f329a461b
fix: ci change as -> AS in Dockerfiles ( #3856 )
...
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Dockerfiles to use consistent casing for `AS` keyword in
`FROM` statements.
- Aligned base images and staging names across various build
architectures to use the latest `bookworm` releases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-29 20:01:16 +02:00
Ankur Srivastava
0fecf9f49e
ci: pin the bookworm-slim image ( #3835 )
2024-06-26 18:36:47 +05:30
Hengfei Yang
611f2e1c29
perf: improve ingestion ( #3678 )
2024-06-12 22:01:03 +08:00
Hengfei Yang
d84726e5db
refactor: merge parquet files interface ( #3547 )
...
Prepare for vertical partition compact, change the interface first.
2024-05-20 15:31:25 +08:00
138ac5d8fe
fix: report cache and alert manager should install chrome ( #3348 )
...
Co-authored-by: Ankur Srivastava <best.ankur@gmail.com >
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com >
2024-04-26 09:24:32 +08:00
Hengfei Yang
7e457c3538
ci: add tokio-console build ( #3264 )
2024-04-17 15:20:38 +08:00
Hengfei Yang
72966bdb4a
ci: Profiling CPU for OpenObserve ( #3183 )
2024-04-06 22:53:50 +08:00
Hengfei Yang
e33ae00ea1
feat: merge dev into main ( #3016 )
2024-03-17 22:38:07 +08:00
Hengfei Yang
d745dace45
feat: search queue ( #2640 )
2024-02-02 18:31:51 +08:00
Hengfei Yang
c057bf40f3
ci: update prod docker images ( #2490 )
2024-01-16 20:39:36 +08:00
Ankur Srivastava
3b386ca8ad
feat: support building and publishing to registry via actions ( #2405 )
2024-01-08 11:24:42 +08:00
Hengfei Yang
f4c3423129
perf: improve ingester with memory ( #2382 )
2024-01-07 23:52:43 +08:00
Hengfei Yang
bb927db797
chore: delete migrate-file-list-from-dynamo ( #2361 )
2024-01-03 11:32:46 +08:00
Hengfei Yang
a4bae1a584
feat: Implement memtable + wal ( #2261 )
2023-12-28 07:40:22 +08:00
Juan Calderon-Perez
d475fc3cd1
chore: Bump Debian to 12 (Bookworm) ( #2299 )
2023-12-24 16:21:00 +08:00
Hengfei Yang
10bfe2c867
ci: update rust version ( #2301 )
2023-12-24 14:23:46 +08:00
Juan Calderon-Perez
0555c2f50f
chore: Bump Node.js to v18.19.0 ( #2278 )
2023-12-22 23:33:09 +08:00
Juan Calderon-Perez
cabee9c560
chore: Bump buildx to v0.12.0 ( #2279 )
2023-12-22 13:45:06 +08:00
Prabhat Sharma
9654427ec3
fix: add psql for debugging ( #2194 )
...
Adding psql to debug container for debugging
2023-12-15 10:15:06 +08:00
Huaijin
Hengfei Yang
Ashish Kolheandgithub-actions[bot]
Anupam MedirattaandmmosarafO2
Sangharsh Agarwal
Simha G
Prabhat SharmaandmmosarafO2
udev
Juan Calderon-Perez
Ankur Srivastava
