mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
Hengfei Yang
3fda083f51
Design at: [https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md](https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md) ## Summary - add an hourly Metadata time index per trace stream to resolve a trace ID to its full time range - add external time-range and trace-details APIs, including bounded locate/expand behavior and standard search responses - integrate index generation into trace ingestion and preserve query visibility through the synchronous Metadata write path - synchronize retention and deletion for associated Metadata streams - reuse the shared trace stream permission check across trace APIs ## Why Trace detail lookup currently depends on a caller-provided time range, which can omit spans outside that window. The time index provides a bounded, best-effort range lookup for `stream + trace_id` without scanning the full trace stream. Closes #13823 ## Environment variables | Variable | Default | Effect | | --- | --- | --- | | `ZO_TRACE_TIME_INDEX_ENABLED` | `true` | Enables per-trace-stream Metadata indexes, index writes during trace ingestion, and indexed time-range lookup. Only traces ingested after the feature is enabled are indexed; there is no historical backfill. | | `ZO_QUERY_TIMEOUT` | 600 seconds | Existing setting reused as the cumulative Trace Timer locate/expand deadline and as the default timeout for the trace-details search. | When enabled, each traces stream can create a related Metadata stream named `trace_time_index_{stream}`. Its retention settings follow the source stream, and it is removed with the source stream. ## API changes All timestamp parameters and returned time ranges use microseconds. ### Get a trace time range ```http GET /api/{org_id}/{stream_name}/traces/{trace_id}/time_range ``` Query parameters: - `hint_ts` (optional): timestamp hint used to start the index lookup Returns the source stream, trace ID, and the resolved `start_time` / `end_time`. The `range` field is `null` when the index cannot resolve the trace. ### Get trace details ```http GET /api/{org_id}/{stream_name}/traces/{trace_id}/details ``` Query parameters: - `start_time` and `end_time` (optional, but must be supplied together): caller-provided range - `hint_ts` (optional): timestamp hint for the time-index lookup - `timeout` (optional): trace-details search timeout in seconds The endpoint unions the caller range with the indexed range, so it never narrows a caller-provided range. If neither the caller range nor the index can provide a range, it returns `400`. The response uses the standard search response and includes the effective range in `new_start_time` and `new_end_time`. ## Validation - `cargo test -p openobserve-api-search --lib` — 145 passed - `cargo test -p openobserve-core traces::time_index::tests --lib` — 7 passed - `cargo test -p infra db::trace_time_index::tests::test_coverage_start_cache -- --exact --nocapture` — passed - `cargo check -p openobserve-api-search` - `cd web && npm run type-check:app` - end-to-end ingestion and query validation with the data generator ## Validation notes - targeted ESLint could not start because the local web dependencies are missing `@eslint/css` - enterprise-only compilation could not be completed from this OSS checkout because the local enterprise overlay is unavailable
openobserve-jobs
openobserve-jobs contains background-job orchestration that runs outside the request-serving API frontend.
Job implementations live under src/job and reuse application services from openobserve-core. Keeping jobs in a separate crate allows API-only changes to avoid recompiling the background-job frontend and vice versa.
This is an internal workspace crate and is not published independently. Reusable business logic should live in openobserve-core rather than being duplicated in a job.