Rebase the GreptimeDB DataFusion fork from official 55.0.0 to 55.1.0.
DataFusion 55.1.0 is a patch release on branch-55 containing eleven
cherry-picked fixes (schema-adaptation struct filters, cast/projection
metadata propagation, nested-nullability aggregation adaptation,
UnnestExec batch_size, RightMark join ordering panic, empty-struct
ScalarValue, and FFI codec fixes). All twenty GreptimeDB fork patches
rebase onto it with no textual or semantic overlap; none of them is
absorbed upstream, so all are retained.
Fork pin moves to discord9/datafusion branch greptimedb-55.1.0,
commit 2aa87d52cdce7006af492330064738f33ed294c1 (55.1.0 +
20 patches).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): insert MergeScan into nested scalar subqueries
DataFusion 55 keeps uncorrelated scalar subqueries as expression
subqueries (enable_physical_uncorrelated_scalar_subquery, default true)
instead of decorrelating them into joins, and executes them via the new
physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked
the plan with a plain TreeNode transform that does not descend into
expression subqueries, so MergeScan was only inserted for depth-1
subqueries. A scalar subquery nested inside another scalar subquery kept
a bare frontend DistTable TableScan and failed at execution with
"Unsupported operation: get stream from a distributed table".
Use the subquery-aware transform so handle_subquery (PlanRewriter /
MergeScan insertion) runs for subquery plans at every nesting depth.
Fixes#9260.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): strengthen nested scalar subquery regression coverage
Address review on #9261:
- Replace the ineffective 'no bare TableScan' string check with a real
subquery-aware plan walk (apply_with_subqueries); MergeScan hides its
remote input from traversal, so any TableScan the walk reaches was
genuinely left unwrapped.
- Add a distributed regression case on a range-partitioned table so the
nested inner aggregate must merge partial results across regions
(global AVG feeding an outer SUM filter).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): compare primary key ranges across schema versions
Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes.
Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): report invalid primary key ranges
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): scope primary key ranges to comparisons
Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts.
Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle.
Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): cache aligned primary key ranges per file
Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes.
Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): clarify primary key mapper schema snapshot
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): align primary key range fixtures with table contract
Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario.
Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): enable filesystem GC in release unit tests
Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito-codec): skip release value decoding in PK prefix counts
Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build.
Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito-codec): remove redundant prefix integration tests
Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: wait for MySQL to accept authenticated TCP queries
Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed.
Verify readiness with SELECT 1 over TCP rather than the initialization socket.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(prom): release the remote write v1 decode buffer before writing
remote_write_v1 kept the decoded builder alive until the handler returned,
so the decompressed request payload stayed resident across the downstream
write or pipeline await. With 8 concurrent large requests that is one extra
copy of every payload held for the whole write.
Rows and pipeline values own their data, so the builder can be dropped as
soon as the conversion is done.
Sustained-write A/B, 8 runs per side, 50M samples each: jemalloc allocated
median drops 7.8% (155.0-162.7 MiB -> 141.4-158.6 MiB); samples per CPU
second is unchanged (-0.6%, fully overlapping ranges).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(servers): move row values into SQL JSON responses
The format=json renderer cloned every serde_json::Value and kept the whole
row set alive while building the response. Move the values out instead, so
each row is released as soon as it is converted.
Slicing the row to the schema width keeps the panic on rows narrower than
the schema; a plain zip would silently truncate them. Duplicate column names
still resolve to the last value and extra row values are still ignored.
Isolated conversion measurements: live peak drops 33% on a 4096-row 16 KiB
string fixture and 36% on a nested-JSON fixture, with no measured slowdown.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(prom): release the compressed remote write body after decompression
Both the v1 and v2 decoders held the compressed Bytes until they returned,
which spans the whole protobuf decode and row conversion. Decompression
copies the payload into an independent buffer, so the body can go as soon
as it succeeds.
The compression fallback, decode errors and request counting are unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* revert(prom): keep the remote write v1 decode buffer until the write finishes
This reverts commit 8232c5b3bb.
The v1 decoder fabricates `&'static [u8]` pointing into its own decode buffer
(prom_remote_write/types.rs), so the compiler checks nothing about that
buffer's lifetime. Holding the builder until the handler returns is what keeps
the decoder safe by construction; releasing it early made that safety depend on
every consumer copying out of the buffer, which holds today but nothing
enforces.
Document the requirement at the binding instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(prom): release the remote write v1 decode buffer before writing
This reverts commit ca3af722d0, restoring
8232c5b3bb.
The decoder borrows the decompressed buffer while parsing, but copies
everything out when it builds rows: tag values through
`PromValidationMode::decode_string`, column names through `to_owned`, and the
only live borrows (`TableBuilder::col_indexes`) are dropped inside
`as_insert_requests`. The resulting `ContextReq` holds prost types with no
lifetime parameters, so it cannot reference the buffer.
Record that at the binding so the next reader does not have to re-derive it
from three files.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito2): split SWCS output files by size threshold
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): use resolved SWCS output size options
Read the SWCS output file size threshold from the resolved region options so database-level compaction settings are honored. Align the existing picker test with this configuration source.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: prefer JSON2 type hints for uncast read pushdown
* feat(query): materialize JSON get result types before planning
* fix(query): apply JSON2 type hints to parsed paths
* fix(query): apply JSON2 type hints before distributed planning
* chore: remove f
* refactor: the code style of json_get_type_hint
* chore: add sqlness case
* fix: add json expr planner, remove json get type hint
* chore: add sqlness test
* fix(query): respect JSON2 type hints in query planning
* chore: add more sqlness cases
* fix: cargo clippy
* fix: cargo clippy
* chore: update sqlness test result
* perf(otlp): share trace resource and scope attributes
Parsing an OTLP trace request copied the resource attributes and the
scope attributes once per span. A request with wide resource attributes
kept one full copy per span alive until the rows were built.
Spans and their group now share one `Arc` per resource and per scope.
Empty attributes stay `None`, so a resource or scope without attributes
costs no allocation and no refcounting. v0 takes ownership back when it
encodes, v1 clones attributes item by item instead of rebuilding a Vec,
and v2 borrows the span instead of cloning it for every row.
Column values, attribute order, the `service.name` lookup and the
auxiliary table writes are unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): stop copying v1 attribute keys
The v1 row writer rebuilds every column name as `{prefix}.{key}`, so the
key string it cloned from the shared resource and scope attributes was
dropped unused. `resource_attributes.service.name` was also cloned in
full before being skipped for the top level service name column.
Shared attributes are now read in place and only values that reach a row
are copied. Span attributes keep moving their values as before.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): cancel cache construction for incomplete scans
CacheBatchBuffer spawned the background concat task and dropped its join
handle. A scan that was cancelled or failed therefore left the task alive:
it kept compacting already queued batches, and while waiting for a range
result memory permit it held them, even though without a finish command
the result can never be put into the cache.
Keep the handle and abort it when the buffer is dropped. The handle is
cleared once the task owns the finish command, so a completed scan still
populates the cache after its stream is dropped. Abort does not preempt a
concat that is already running; it takes effect the next time the task is
polled.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito2): wait for the permit park before cancelling the buffer
An empty buffered_batches only proves the batches were enqueued, so the
cancellation test could abort a concat task that had never been polled.
Count acquisitions that find too few permits, a test-only signal, and drop
the buffer once the task has reached that wait. Nothing awaits between the
check and the parking, so an observed increment means the caller is about
to wait for permits the test holds for the rest of the case.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add config flag to control range index reads
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: disable range index builds when configured and default to off
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
CompactDispatcher acquired a borrowed permit inside the async wrapper and
dropped it as soon as the blocking task was submitted, so the semaphore
never bounded the compactions that actually ran. Acquire an owned permit
and move it into the blocking closure so it covers the queue wait and the
merge itself.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(prometheus): honor label matchers in __name__ values query
`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.
Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.
Only metric engine tables are covered; other engines share no column space
to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): batch-resolve metric names by table id
Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.
The catalog walk stays, but only to find the physical tables to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): read an absent label as the empty string
A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.
Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.
This is the Prometheus compatibility fix#8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): don't panic on pre-epoch label value bounds
`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.
Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): drop applicable_matchers, share the distinct scan
With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.
Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Agents routinely check "This PR requires documentation updates" for
changes users never see. Say what the item means: the docs site repo,
not rustdoc or in-repo comments.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
crate-ci/typos renamed its default branch from master to main on
2026-09-18 (0a3d75e) and the master branch is gone, so every workflow
run since then fails at job setup with:
Unable to resolve action `crate-ci/typos@master`, unable to find version `master`
Pin to the latest release tag instead of tracking a branch. This matches
how the other actions in these two workflows are referenced and keeps the
check reproducible.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Incremental batching flows already merge sink state for scalar
aggregates and the HLL/UDDSketch/stddev state families. AVG now exposes
a mergeable Binary state (avg_state / avg_merge with
__avg_state_delta_merge), so admit those aggregates too instead of
forcing a full snapshot for flows whose only state column is an
average.
merge_op_for_aggregate_expr takes the aggregate input schema so the
avg_merge arm can require a Binary state argument; the state form is
the aggregate result persisted by the sink, so no extra coercion is
needed. Other input types keep being rejected.
Also cover avg_state, avg_merge and duplicate AVG projections in the
incremental plan analysis tests, extend the mixed state-family rewrite
test with an average column, and extend the standalone partitioned
state-merge SQLness case with avg_state/avg_merge compared against a
direct avg over the source.
Signed-off-by: discord9 <discord9@outlook.com>
Co-authored-by: discord9 <discord9@outlook.com>
* test(flow): wait for async source-table mirrors before FLUSH_FLOW
Streaming flows execute inline in the flownode insert handler since
#8976, and source-table inserts are mirrored from the frontend as
detached tasks. Tests that INSERT then ADMIN FLUSH_FLOW then SELECT
intermittently miss rows on slow CI runners because the flush no longer
implies the mirror reached the flownode (flow_no_aggr lost row 'l',
flow_advance_ttl lost row '23' after the post-restart reinsert).
Add SQLNESS SLEEP 3s between those mirror inserts and the flush, the
established pattern in the flow suite, so the assertions are stable.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): also guard first-section flushes in flow_advance_ttl
Local reproduction without sleeps failed in a window the previous
commit did not cover: the first INSERT (20,20,22) of each section is
followed immediately by ADMIN FLUSH_FLOW, and the distributed run
observed an empty sink on that SELECT (~1 in 8 iterations). Add the
same SLEEP 3s guard before those two flushes.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): propagate matching-label filters between binary operands
A one-to-one arithmetic binary expression inner-joins its operands on the
matching labels, so every row that survives the join already satisfies the
other operand's equality matchers on those labels. Copy those matchers to the
other operand so both scans drop non-joining series before execution instead of
feeding them to the join.
Both operands are planned before the rewrite: a selector matcher can also
constrain a value field, and only the planned contexts tell tags and fields
apart. A matcher is copied only when its name is a tag column on both sides, its
value is non-empty, and it is one of the matching labels. Operands are limited
to vector selectors, parentheses, and label-preserving range functions applied
directly to a matrix selector; `ignoring(...)`, group modifiers, fill values,
set and comparison operators, regex matchers and `or` matcher groups are left
alone.
Selector matchers are now deduplicated in place instead of through a `HashSet`,
so the generated scan filter keeps a stable order once a selector carries more
than one matcher.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): propagate ignoring, non-equality and aggregated matchers
Widens the matcher propagation added in the previous commit to the shapes it
was leaving on the table.
The join compares matching labels with plain column equality
(`normalized_match_key_expr` and its coalescing are confined to `or`), so any
predicate on a matching label is already enforced on both sides for every
surviving pair: it originates on one operand, and the join carries it to the
pairs it forms. Copying it to the other operand can only drop rows that had no
surviving partner. That argument does not depend on the matcher kind, so regular
expressions, negations and empty values now propagate too. `=~".*"` stays out:
it lowers to no filter at all, so copying it would only force a re-plan.
`ignoring(...)` is no longer rejected. A label is a join key exactly when it is a
tag on both sides and not named in `ignoring`, which is what
`binary_join_key_columns` computes and what the caller can now answer from the
two planned contexts.
Operands may now be aggregations that partition by their grouping labels, which
is the shape most real queries use. `agg_modifier_to_col` rewrites
`ctx.tag_columns` to the grouping labels, so a label found in an aggregated
operand's tags is a group key, and filtering the aggregate's input by it drops
exactly the corresponding output groups. `topk`, `bottomk` and `limitk` are
excluded because they select across a group and carry input labels through --
`prom_topk_bottomk_to_plan` leaves `ctx.tag_columns` alone, so the tag check
cannot catch them. `count_values` is excluded because it adds an output label
that does not exist in its input.
The rollup whitelist now covers every label-preserving range function the
planner implements, and finds the matrix argument by position so multi-argument
rollups such as `quantile_over_time` and `predict_linear` qualify.
`absent_over_time` stays out: it synthesizes a series from the matchers when its
input has none.
Re-running the sqlness case against an unmodified planner produces a
byte-identical `.result` for all 27 queries.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): pin that excluded matching labels stay on their own operand
`ignoring(device)` and `on(host)` both leave `device` out of the join keys, so a
`device` matcher must not reach the other operand. Neither case was covered end
to end: a regression there silently drops the left operand's `eth1` series
instead of returning them, which the two added queries now catch.
Also corrects the comment at the rewrite site, which still described re-planning
as touching a leaf selector. Aggregated operands are re-planned as a whole; what
holds is that the rewrite only ever adds matchers to a selector, so the enclosing
operand's table reference, time index and field columns are unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): log why matching-filter propagation was skipped
Splits the guard chain into `try_propagate`, whose `Err` names the reason the
rewrite does not apply, and logs it once in `propagate`. Debugging a query that
did not get the filter no longer means stepping through the guards.
The reasons also replace the comments that used to explain the same conditions,
and the remaining comments lose the parts that restated the code or repeated
each other.
Two of them were wrong rather than verbose. Saying `topk`'s input "must not be
filtered" reads as a claim about PromQL: `topk(1, m{host="x"})` is perfectly
legal, and what matters is that filtering before `topk` changes the candidate
set it ranks. Saying the subset-matching case "keeps its many-to-many result"
read as an endorsement of behaviour Prometheus rejects outright; it now records
that the query is a cross product here and points at #9209.
The propagation cannot change whether such a check would fire: series in one
match group agree on every matching label, so a matcher over one of those labels
keeps all of them or drops all of them.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep copied matchers out of the operand's selector metadata
Re-planning an operand with a copied matcher also rebuilt its
`PromPlannerContext::selector_matcher`, and that context is what an enclosing
expression reads. `create_absent_plan` turns the equality matchers found there
into the labels `absent()` reports, so
absent(counter_metric{host="missing"} / on(host, device) gauge_metric)
gained a `host="missing"` label that `main` does not produce. The inner
expression was empty either way; only the reported label set changed.
The copied matcher belongs to the scan, not to the operand's identity, so both
re-planned contexts now keep the matchers their operand was written with.
`selector_matcher` has one other reader, `create_table_scan_plan`, which
consumes it while the operand is being planned and is unaffected.
Reported by @discord9.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(log-store): add the object store WAL batch, catalog and I/O
Add the three modules between the object format and the store of the
object store WAL:
- batch: the open batch that accumulates admitted entries into the next
object and assigns entry ids at admission. Ids are object-sequence-major,
`(object_seq << 20) | position`, with positions starting at one per
region per object, so a batch that is rolled back and admitted again
under the same sequence hands out the same ids. The time the first entry
was admitted is kept for the store's age-based sealing; an empty
admission does not start it.
- catalog: the in-memory index over object footers, by sequence and per
region. Insertion is atomic and rejects an empty footer, duplicate
region segments, invalid entry ranges, an already indexed sequence and
entry ranges that are not strictly increasing across objects. The next
object sequence continues after the largest indexed one and is raised
above the largest entry id of any region, and is rejected once it no
longer fits an entry id.
- io: object store access under `<prefix>/objects/`: a conditional create
whose retry with identical content is a no-op and whose conflicting
content is rejected, whole and range reads, and a listing of well-formed
keys in sequence order. A prefix with a `.` or `..` component is
rejected. When the store reports that the object exists but the read
that compares its content fails, the read failure is returned with its
retry hint instead of the create failure.
The modules have no callers until the store lands, so they are declared
with `#[allow(dead_code)]`. Only `entry_id` is exported.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): drop comments that restate the batch test assertions
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): use pub(crate) in the object store WAL batch, catalog and I/O
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): keep entry_id crate-private and narrow the create collision check
Re-export `entry_id` as `pub(crate)`: nothing outside `log-store` uses it
yet, and exporting it would freeze the raw `(object_seq, position)`
encoding before the store owns id allocation.
Treat only `ConditionNotMatch` as the sign that a conditional create
collided with an existing object. A store may report `AlreadyExists` for
an unrelated path, for example when a parent of the object is a file, and
that failure must come back as the write failure rather than as the error
of the read that would compare content.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): rename the catalog's out_of_order helper to out_of_order_reason
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(function): add mergeable binary average states
Signed-off-by: discord9 <discord9@163.com>
* feat(function): expose avg_calc as an OSS scalar
Signed-off-by: discord9 <discord9@163.com>
* fix(function): borrow invalid AVG scalar argument type
Signed-off-by: discord9 <discord9@163.com>
* test(function): verify public AVG state SQL finalization
Signed-off-by: discord9 <discord9@163.com>
* fix(function): return canonical AVG1 state for empty window frames
DataFusion's plain-aggregate window executor bypasses the accumulator
and calls default_value() directly when a window frame contains no
rows. create_udaf's SimpleAggregateUDF derives default_value from the
return type, which yields SQL NULL for Binary output instead of the
canonical AVG1 empty state, so empty frames and frames over only NULL
inputs became observable differently (IS NULL, direct state saves,
state comparisons).
Register avg_state/avg_merge through a small AggregateUDFImpl (AvgUdaf)
that keeps the existing accumulator and declares the canonical AVG1
empty state as default_value, restoring the documented contract. Add a
unit test asserting default_value equals the empty accumulator's
evaluate() and an sqlness case covering the empty-frame window.
Signed-off-by: discord9 <discord9@outlook.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@outlook.com>
Co-authored-by: discord9 <discord9@outlook.com>
* perf: add flight coalesce regression case with high-cardinality aggregations
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf: add flight coalesce aggregations bench case
Add a direct_readable_sst case exercising grouped aggregation over
coalesced batches: 16 hosts x 4096 instances, 32 SSTs of 32768 rows,
timestamp-major series layout, three SQL queries (aggregation, topk,
count_by_host) each with a 10% max candidate latency regression
threshold.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations
The new revision (v0y4g3r/memcomparable main, PR #1) includes:
- Deserializer::read_bytes_into for reusable-buffer decoding
- read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain)
- bulk put_slice in both normal and reverse modes
- Error::Eof instead of panic on truncated input
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): extract sparse tag values lazily without full PK decode
Replace the full primary-key decode in two hot consumers of
SortField::deserialize with column-lazy extraction over
SparsePrimaryKeyView, eliminating the per-label owned values
(Vec<Value>, one heap allocation per string) for every distinct key:
- flat_format: DecodedPrimaryKeys keeps sparse keys encoded and
extracts each tag column directly from the raw dictionary bytes into
a reusable buffer. FlatConvertFormat (flush/compaction write path)
builds all projected tag columns in one pass per key, sharing offset
discovery between columns; the read path (file_range filters) keeps
per-column lazy extraction.
- series_index writer: builds rows from the reserved prefix and only
the indexed tags via a shared offsets cache and buffer, instead of
decoding all labels and copying out a few strings.
- mito-codec: add SparsePrimaryKeyView::reserved_value for typed
table_id/tsid extraction.
Persisted formats are unchanged; values pushed to column builders are
identical (NULL vs empty string, reserved column types, duplicate
dictionary entries and consecutive-run order are preserved).
Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion
30 samples): old = full decode + per-column builds, new = lazy
extraction (+ one-pass for multi-column projection).
| Workload | Before | After | Change |
| --- | ---: | ---: | ---: |
| 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% |
| 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% |
| 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% |
| 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% |
| 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% |
Also adds a decode_sparse benchmark group to mito-codec's
bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): share sparse tag scans across filters
Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics.
Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches.
Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): benchmark sparse tag predicate count scaling
Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(log-store): add the object store WAL object format
Add the byte format of a single object store WAL object: a header with
the GTWALOBJ magic, format version 1, the object sequence and the writer
instance id; one segment per region ordered by region id with entries
ordered by entry id; a footer that records each segment's region id,
entry id range, entry count, byte range and CRC32; and a fixed trailer
with the GTWALTRL magic, the footer location, the footer CRC32 and the
whole-object CRC32.
The module encodes objects deterministically and decodes the header,
trailer, footer and segments separately, with structural checks on
footer ranges and segment tiling. It has no callers yet; the store that
writes and reads objects follows in later changes.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): pin the object store WAL format with a byte fixture
Add a fixed version 1 object with two regions and five entries as a hex
literal. The test decodes it and checks the exact header, trailer,
footer entries and records, and checks that encoding the same records,
in either input order, reproduces the fixture byte for byte.
Round-trip tests alone pass when a refactor changes field order,
endianness or checksum coverage in both the encoder and the decoder.
The fixture bytes were derived from the documented layout rather than
from the encoder, so such a change now fails.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): reject an empty footer when decoding a WAL object
The encoder never writes an object without records, but decode_footer
accepted a footer that declares zero segments, and
verify_segment_ranges accepts an empty footer too. Only the test-only
decode_object rejected it, so a checksum-valid empty object would pass
the header, trailer and footer checks that recovery runs.
Reject a zero entry count in decode_footer and drop the now unreachable
check in decode_object. Add a test that builds a checksum-valid object
with an empty footer and checks that decode_footer and decode_object
reject it.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): use pub(crate) for the WAL object format API
Other log-store modules use pub(crate) for items shared across module
boundaries. Switch the format module from pub(super) to pub(crate) to
follow that convention. No behavior change.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat: use series indexes for SeriesScan candidate discovery
Signed-off-by: evenyag <realevenyag@gmail.com>
* perf: use range indexes in two-phase series reader
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): update series index test fixtures after rebase
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): move lazy range index searchers into file contexts
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: pin series index snapshot before data snapshot
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: preserve builder caching for index-covered SSTs
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
DataFusion 55 deprecated ExecutionPlan::partition_statistics in favor of
statistics_from_inputs with StatisticsContext::compute. Migrate the two
remaining GreptimeDB overrides, RegionScanExec and MergeScanExec, and
update the RegionScanExec test to the new API, dropping its
allow(deprecated).
Behavior is unchanged: RegionScanExec keeps the append-mode-only,
exact-source-rows statistics gate from #9154, and MergeScanExec keeps
reporting unknown statistics for per-partition requests.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): prototype ready-only Flight batch coalescing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(servers): bound ready Flight batch admission and cover lifecycle
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(servers): fix redundant error pattern assertion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(servers): simplify Flight coalescing with append-then-check accumulator
Remove the lookahead/pending machinery and coalesce with an
append-then-check soft budget. Extract the group state into a private
BatchAccumulator so the ordinary stream loop reads as a short control
flow: append each ready batch, flush when a budget is reached.
Budgets are flush thresholds, not memory limits, so a group may exceed
a budget by the final appended batch. First-batch direct send and
singleton passthrough for a batch that starts a group already at a
budget are unchanged.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(servers): split Flight stream into dispatcher and per-path units
FlightRecordBatchStream::flight_data_stream held two large side-by-side
branches (the verbose/ANALYZE metrics path and the plain/coalescing
path) inline, making the function hard to read. Split it into a small
dispatcher that keeps only the shared prologue (schema send, metrics
init) and the shared EOF final-metrics tail, and move each branch into a
self-contained unit:
- verbose path -> Self::verbose_metrics_stream
- plain/coalescing path -> private CoalescingBatcher (owns the
BatchAccumulator, sent_first_batch, and schema)
Each path returns whether it reached normal EOF; the dispatcher skips
the shared final-metrics tail on any early error/failed-send exit,
preserving the exact pre-split behavior (an early return previously
exited the whole function and bypassed the tail). No behavior, metrics
ordering, or coalescing semantics change.
Add a regression test that drives flight_data_stream directly and
asserts the verbose error path does not invoke the final-metrics tail
(producer-side metrics() call count), since the public message stream
hides the tail after an error.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(servers): describe Flight stream units by behavior, not refactor history
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): avoid copying over-budget batches and release merged inputs
Three review-driven improvements to Flight batch coalescing:
- After a successful merge, reuse the drained vector in place
(clear + push the merged batch) so the source batches are dropped
before the send loop instead of staying alive across a backpressured
send.
- A batch that is itself at/over a budget is forwarded as a singleton
even when encountered inside an accumulation group: the accumulated
group is flushed and sent first, then the over-budget batch. This
avoids copying a large batch into an aggregate just to merge the small
batches ahead of it. Under-budget append-then-check is unchanged
(600+600 still merges).
- Strengthen two tests: the first-batch test now drives the producer
directly and asserts the upstream poll count is exactly one at the
first-batch send; the dictionary test now decodes the merged
dictionary keys to assert the logical values, so a key-remapping
regression is caught.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): raise Flight coalesce row budget to 4096
mito2 commonly emits ~2000-row batches (~32-94KiB), so a 1024-row budget
marked every such batch oversized and the coalescer forwarded them
unbatched. Measured on a local distributed cluster (2M-row table):
- full selector: 1000 -> 337 batches, p50 latency 4133 -> 3316 ms (-20%)
- range scan: 501 -> 206 batches, 1466 -> 1162 ms (-21%)
- small/olap queries (single-series, top-k, group-by): unchanged
Larger budgets (8192 rows, or 1-4MiB bytes) coalesce more batches but do
not improve latency further and slightly regress it, so 4096 rows is the
sweet spot; MAX_BYTES stays at 256KiB and MAX_BATCHES at 16.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Ning Sun
Weny Xu
discord9
github-actions[bot]andgreptimedb-ci
Lei, HUANG
LFC
dennis zhuang
fys
jeremyhi
Yingwen
Mohd Quamar Tyagianddennis zhuang
Dhruv VaishnavandWeny Xu
Sainath Singineedi
shuiyisong