* fix(query): stop encoding oversized dynamic filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): stop bounded encoding partway through an IN list
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add experimental Jev SQL filtering
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: gate Jev filtering behind an opt-in Cargo feature
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: verify Jev registration with default features
Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: clarify Jev concurrency scope and stabilization work
Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: enable renamed ai-functions feature by default
Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: remove extra AI feature-off checks
Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: rename AI feature to ai_functions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject duplicate region engine configurations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: reuse canonical engine names in config validation
Replace duplicated engine-name literals with the existing common-catalog constants so duplicate-config validation uses the shared engine names. Keep the TOML regression inputs independent to verify the public configuration tags.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: clarify zero-based duplicate engine config indices
State explicitly that duplicate region engine configuration indices are zero-based so users can map them to the order of TOML entries. Preserve the existing index values and error classification.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(telemetry): remove the shared lock from TraceLayer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(telemetry): stop collecting span data while tracing is disabled
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep value-field grouping labels out of matching filter propagation
#9202 propagates matching-label matchers to the scanned selector using the
planned operands' tag columns. For an aggregated operand those are its
grouping labels, and agg_modifier_to_col resolves by(...) names against the
input schema only, so a value field named in by(...) is reported as a tag.
A value field varies between the samples of one series, so lowering a
matcher on it below sample selection (PromInstantManipulate) can drop the
newest sample, promote a stale one from the lookback window, and fabricate
a match the un-rewritten query does not produce.
Track the by(...) labels that name value fields of the aggregated operand's
input in PromPlannerContext::aggregation_field_labels, and refuse to
propagate matchers on them while still requiring a matcher to name a
grouping label to cross an aggregation.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: sync matching_filter result fixture comment with the PR number
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): stream Prometheus HTTP responses incrementally
Replace try_collect + full-buffer conversion in the Stream branch of
from_query_result with incremental per-batch merging via the shared
merge_batch path, so record batches are dropped as soon as they are
consumed instead of all coexisting in memory.
- Add ColumnLayout to dedupe schema inference shared by both paths
- Add SeriesKeyLookup (Equivalent-based borrow lookup) so series keys
are looked up by borrowed label slices without materializing owned
keys per lookup
- Empty streams short-circuit to empty data before schema inference
- Add stream-side mixed histogram + Vector dual-path equivalence test
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(servers): address review comments on stream response tests
- Inline single-caller tags_capacity helper
- Drop timestamp column from empty-stream test so it pins the
short-circuit happens before schema inference
- Fix cross-batch merge test data so a series actually spans the
batch boundary (second batch carries earlier timestamps)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(servers): add lazy stream release assertion and pinned cross-batch content
- LazyDropCheckingStream creates one batch per poll and asserts via
Weak that the previous batch was released before the next poll,
protecting the core property that batches are dropped eagerly
- Pin series "a" merged samples to exact timestamps and values in
stream_result_matches_record_batches_result, so a regression in
shared merge_batch logic cannot make both paths agree on wrong output
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): insert MergeScan into nested scalar subqueries
DataFusion 55 keeps uncorrelated scalar subqueries as expression
subqueries (enable_physical_uncorrelated_scalar_subquery, default true)
instead of decorrelating them into joins, and executes them via the new
physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked
the plan with a plain TreeNode transform that does not descend into
expression subqueries, so MergeScan was only inserted for depth-1
subqueries. A scalar subquery nested inside another scalar subquery kept
a bare frontend DistTable TableScan and failed at execution with
"Unsupported operation: get stream from a distributed table".
Use the subquery-aware transform so handle_subquery (PlanRewriter /
MergeScan insertion) runs for subquery plans at every nesting depth.
Fixes#9260.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): strengthen nested scalar subquery regression coverage
Address review on #9261:
- Replace the ineffective 'no bare TableScan' string check with a real
subquery-aware plan walk (apply_with_subqueries); MergeScan hides its
remote input from traversal, so any TableScan the walk reaches was
genuinely left unwrapped.
- Add a distributed regression case on a range-partitioned table so the
nested inner aggregate must merge partial results across regions
(global AVG feeding an outer SUM filter).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): compare primary key ranges across schema versions
Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes.
Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): report invalid primary key ranges
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): scope primary key ranges to comparisons
Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts.
Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle.
Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): cache aligned primary key ranges per file
Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes.
Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): clarify primary key mapper schema snapshot
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): align primary key range fixtures with table contract
Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario.
Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): enable filesystem GC in release unit tests
Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito-codec): skip release value decoding in PK prefix counts
Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build.
Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito-codec): remove redundant prefix integration tests
Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: wait for MySQL to accept authenticated TCP queries
Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed.
Verify readiness with SELECT 1 over TCP rather than the initialization socket.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(prom): release the remote write v1 decode buffer before writing
remote_write_v1 kept the decoded builder alive until the handler returned,
so the decompressed request payload stayed resident across the downstream
write or pipeline await. With 8 concurrent large requests that is one extra
copy of every payload held for the whole write.
Rows and pipeline values own their data, so the builder can be dropped as
soon as the conversion is done.
Sustained-write A/B, 8 runs per side, 50M samples each: jemalloc allocated
median drops 7.8% (155.0-162.7 MiB -> 141.4-158.6 MiB); samples per CPU
second is unchanged (-0.6%, fully overlapping ranges).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(servers): move row values into SQL JSON responses
The format=json renderer cloned every serde_json::Value and kept the whole
row set alive while building the response. Move the values out instead, so
each row is released as soon as it is converted.
Slicing the row to the schema width keeps the panic on rows narrower than
the schema; a plain zip would silently truncate them. Duplicate column names
still resolve to the last value and extra row values are still ignored.
Isolated conversion measurements: live peak drops 33% on a 4096-row 16 KiB
string fixture and 36% on a nested-JSON fixture, with no measured slowdown.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(prom): release the compressed remote write body after decompression
Both the v1 and v2 decoders held the compressed Bytes until they returned,
which spans the whole protobuf decode and row conversion. Decompression
copies the payload into an independent buffer, so the body can go as soon
as it succeeds.
The compression fallback, decode errors and request counting are unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* revert(prom): keep the remote write v1 decode buffer until the write finishes
This reverts commit 8232c5b3bb.
The v1 decoder fabricates `&'static [u8]` pointing into its own decode buffer
(prom_remote_write/types.rs), so the compiler checks nothing about that
buffer's lifetime. Holding the builder until the handler returns is what keeps
the decoder safe by construction; releasing it early made that safety depend on
every consumer copying out of the buffer, which holds today but nothing
enforces.
Document the requirement at the binding instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(prom): release the remote write v1 decode buffer before writing
This reverts commit ca3af722d0, restoring
8232c5b3bb.
The decoder borrows the decompressed buffer while parsing, but copies
everything out when it builds rows: tag values through
`PromValidationMode::decode_string`, column names through `to_owned`, and the
only live borrows (`TableBuilder::col_indexes`) are dropped inside
`as_insert_requests`. The resulting `ContextReq` holds prost types with no
lifetime parameters, so it cannot reference the buffer.
Record that at the binding so the next reader does not have to re-derive it
from three files.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito2): split SWCS output files by size threshold
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): use resolved SWCS output size options
Read the SWCS output file size threshold from the resolved region options so database-level compaction settings are honored. Align the existing picker test with this configuration source.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: prefer JSON2 type hints for uncast read pushdown
* feat(query): materialize JSON get result types before planning
* fix(query): apply JSON2 type hints to parsed paths
* fix(query): apply JSON2 type hints before distributed planning
* chore: remove f
* refactor: the code style of json_get_type_hint
* chore: add sqlness case
* fix: add json expr planner, remove json get type hint
* chore: add sqlness test
* fix(query): respect JSON2 type hints in query planning
* chore: add more sqlness cases
* fix: cargo clippy
* fix: cargo clippy
* chore: update sqlness test result
* perf(otlp): share trace resource and scope attributes
Parsing an OTLP trace request copied the resource attributes and the
scope attributes once per span. A request with wide resource attributes
kept one full copy per span alive until the rows were built.
Spans and their group now share one `Arc` per resource and per scope.
Empty attributes stay `None`, so a resource or scope without attributes
costs no allocation and no refcounting. v0 takes ownership back when it
encodes, v1 clones attributes item by item instead of rebuilding a Vec,
and v2 borrows the span instead of cloning it for every row.
Column values, attribute order, the `service.name` lookup and the
auxiliary table writes are unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): stop copying v1 attribute keys
The v1 row writer rebuilds every column name as `{prefix}.{key}`, so the
key string it cloned from the shared resource and scope attributes was
dropped unused. `resource_attributes.service.name` was also cloned in
full before being skipped for the top level service name column.
Shared attributes are now read in place and only values that reach a row
are copied. Span attributes keep moving their values as before.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): cancel cache construction for incomplete scans
CacheBatchBuffer spawned the background concat task and dropped its join
handle. A scan that was cancelled or failed therefore left the task alive:
it kept compacting already queued batches, and while waiting for a range
result memory permit it held them, even though without a finish command
the result can never be put into the cache.
Keep the handle and abort it when the buffer is dropped. The handle is
cleared once the task owns the finish command, so a completed scan still
populates the cache after its stream is dropped. Abort does not preempt a
concat that is already running; it takes effect the next time the task is
polled.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito2): wait for the permit park before cancelling the buffer
An empty buffered_batches only proves the batches were enqueued, so the
cancellation test could abort a concat task that had never been polled.
Count acquisitions that find too few permits, a test-only signal, and drop
the buffer once the task has reached that wait. Nothing awaits between the
check and the parking, so an observed increment means the caller is about
to wait for permits the test holds for the rest of the case.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add config flag to control range index reads
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: disable range index builds when configured and default to off
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
CompactDispatcher acquired a borrowed permit inside the async wrapper and
dropped it as soon as the blocking task was submitted, so the semaphore
never bounded the compactions that actually ran. Acquire an owned permit
and move it into the blocking closure so it covers the queue wait and the
merge itself.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(prometheus): honor label matchers in __name__ values query
`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.
Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.
Only metric engine tables are covered; other engines share no column space
to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): batch-resolve metric names by table id
Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.
The catalog walk stays, but only to find the physical tables to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): read an absent label as the empty string
A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.
Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.
This is the Prometheus compatibility fix#8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): don't panic on pre-epoch label value bounds
`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.
Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): drop applicable_matchers, share the distinct scan
With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.
Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
discord9
jeremyhi
Weny Xu
dennis zhuang
Lei, HUANG
LFC
shuiyisongandLei, HUANG
discord9
Ning Sun
fys
Yingwen
Mohd Quamar Tyagianddennis zhuang
Dhruv VaishnavandWeny Xu
Sainath Singineedi