* feat(log-store): add the object store WAL object format
Add the byte format of a single object store WAL object: a header with
the GTWALOBJ magic, format version 1, the object sequence and the writer
instance id; one segment per region ordered by region id with entries
ordered by entry id; a footer that records each segment's region id,
entry id range, entry count, byte range and CRC32; and a fixed trailer
with the GTWALTRL magic, the footer location, the footer CRC32 and the
whole-object CRC32.
The module encodes objects deterministically and decodes the header,
trailer, footer and segments separately, with structural checks on
footer ranges and segment tiling. It has no callers yet; the store that
writes and reads objects follows in later changes.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): pin the object store WAL format with a byte fixture
Add a fixed version 1 object with two regions and five entries as a hex
literal. The test decodes it and checks the exact header, trailer,
footer entries and records, and checks that encoding the same records,
in either input order, reproduces the fixture byte for byte.
Round-trip tests alone pass when a refactor changes field order,
endianness or checksum coverage in both the encoder and the decoder.
The fixture bytes were derived from the documented layout rather than
from the encoder, so such a change now fails.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): reject an empty footer when decoding a WAL object
The encoder never writes an object without records, but decode_footer
accepted a footer that declares zero segments, and
verify_segment_ranges accepts an empty footer too. Only the test-only
decode_object rejected it, so a checksum-valid empty object would pass
the header, trailer and footer checks that recovery runs.
Reject a zero entry count in decode_footer and drop the now unreachable
check in decode_object. Add a test that builds a checksum-valid object
with an empty footer and checks that decode_footer and decode_object
reject it.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): use pub(crate) for the WAL object format API
Other log-store modules use pub(crate) for items shared across module
boundaries. Switch the format module from pub(super) to pub(crate) to
follow that convention. No behavior change.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat: use series indexes for SeriesScan candidate discovery
Signed-off-by: evenyag <realevenyag@gmail.com>
* perf: use range indexes in two-phase series reader
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): update series index test fixtures after rebase
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): move lazy range index searchers into file contexts
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: pin series index snapshot before data snapshot
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: preserve builder caching for index-covered SSTs
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
DataFusion 55 deprecated ExecutionPlan::partition_statistics in favor of
statistics_from_inputs with StatisticsContext::compute. Migrate the two
remaining GreptimeDB overrides, RegionScanExec and MergeScanExec, and
update the RegionScanExec test to the new API, dropping its
allow(deprecated).
Behavior is unchanged: RegionScanExec keeps the append-mode-only,
exact-source-rows statistics gate from #9154, and MergeScanExec keeps
reporting unknown statistics for per-partition requests.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): prototype ready-only Flight batch coalescing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(servers): bound ready Flight batch admission and cover lifecycle
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(servers): fix redundant error pattern assertion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(servers): simplify Flight coalescing with append-then-check accumulator
Remove the lookahead/pending machinery and coalesce with an
append-then-check soft budget. Extract the group state into a private
BatchAccumulator so the ordinary stream loop reads as a short control
flow: append each ready batch, flush when a budget is reached.
Budgets are flush thresholds, not memory limits, so a group may exceed
a budget by the final appended batch. First-batch direct send and
singleton passthrough for a batch that starts a group already at a
budget are unchanged.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(servers): split Flight stream into dispatcher and per-path units
FlightRecordBatchStream::flight_data_stream held two large side-by-side
branches (the verbose/ANALYZE metrics path and the plain/coalescing
path) inline, making the function hard to read. Split it into a small
dispatcher that keeps only the shared prologue (schema send, metrics
init) and the shared EOF final-metrics tail, and move each branch into a
self-contained unit:
- verbose path -> Self::verbose_metrics_stream
- plain/coalescing path -> private CoalescingBatcher (owns the
BatchAccumulator, sent_first_batch, and schema)
Each path returns whether it reached normal EOF; the dispatcher skips
the shared final-metrics tail on any early error/failed-send exit,
preserving the exact pre-split behavior (an early return previously
exited the whole function and bypassed the tail). No behavior, metrics
ordering, or coalescing semantics change.
Add a regression test that drives flight_data_stream directly and
asserts the verbose error path does not invoke the final-metrics tail
(producer-side metrics() call count), since the public message stream
hides the tail after an error.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(servers): describe Flight stream units by behavior, not refactor history
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): avoid copying over-budget batches and release merged inputs
Three review-driven improvements to Flight batch coalescing:
- After a successful merge, reuse the drained vector in place
(clear + push the merged batch) so the source batches are dropped
before the send loop instead of staying alive across a backpressured
send.
- A batch that is itself at/over a budget is forwarded as a singleton
even when encountered inside an accumulation group: the accumulated
group is flushed and sent first, then the over-budget batch. This
avoids copying a large batch into an aggregate just to merge the small
batches ahead of it. Under-budget append-then-check is unchanged
(600+600 still merges).
- Strengthen two tests: the first-batch test now drives the producer
directly and asserts the upstream poll count is exactly one at the
first-batch send; the dictionary test now decodes the merged
dictionary keys to assert the logical values, so a key-remapping
regression is caught.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): raise Flight coalesce row budget to 4096
mito2 commonly emits ~2000-row batches (~32-94KiB), so a 1024-row budget
marked every such batch oversized and the coalescer forwarded them
unbatched. Measured on a local distributed cluster (2M-row table):
- full selector: 1000 -> 337 batches, p50 latency 4133 -> 3316 ms (-20%)
- range scan: 501 -> 206 batches, 1466 -> 1162 ms (-21%)
- small/olap queries (single-series, top-k, group-by): unchanged
Larger budgets (8192 rows, or 1-4MiB bytes) coalesce more batches but do
not improve latency further and slightly regress it, so 4096 rows is the
sweet spot; MAX_BYTES stays at 256KiB and MAX_BATCHES at 16.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(wal): add the object store WAL provider identity and configuration
Add the identity and configuration of the experimental object store WAL
without the log store implementation:
- store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a
remote WAL.
- common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`)
with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and
`on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as
`object_store` with the key `wal.object_store.prefix`. The metasrv config
conversion rejects the new provider.
- common-meta, meta-srv, mito2: handle the new variants, map the region WAL
options to the provider and reject them on a Raft Engine or Kafka log store.
- datanode: validate the configuration and fail with a "not supported yet"
error until the log store lands.
- Example configs and the generated config docs.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(wal): accept object store WAL options when re-enabling WAL
Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(promql): reuse sliding min and max candidates
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(promql): simplify extrema benchmark parameters
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(promql): record baseline sliding extrema SQL results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): rescan windows that barely overlap
Reusing candidates loses to a plain scan when consecutive windows overlap
little: the deque bookkeeping then costs more than the rescan it replaces.
A local Criterion run on 4096 samples at width 240 / step 240 measured
10.79 -> 22.14 us for min and 12.83 -> 19.85 us for max.
Pick the evaluator once per batch from the first two windows. RangeManipulate
emits one window length and one step per batch, so that sample decides for all
of them, and both evaluators return identical bits, so a wrong pick costs time
only. Batches that do not qualify fold each window on its own.
Move the incremental state into SlidingExtrema so tests can drive it directly:
the exhaustive four-sample differential test cannot reach it through a UDF
call, because such a batch never qualifies for reuse.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* fix(promql): select the extrema evaluator from batch averages
Reading the window shape off the first two windows misreads the batch.
RangeManipulate starts a series at max(query start, first aligned sample),
so a series that begins inside the query range gets a first window covering
roughly one step, and a window covering no sample at all is emitted as
(0, 0). Either one closed the gate for the whole batch, including the
one-hour window at a 15s step that candidate reuse was written for.
Compare the batch averages instead: at least 32 samples per window, and a
step advancing at most a quarter of that. Uniform batches select exactly as
before, so the thresholds keep the meaning they were measured with.
The 32-sample rule had also moved most of the benchmark and query-regression
shapes onto the rescan, including the case built to measure reset and
rebuild. Widen those windows to 40 samples, add a step at the selection
boundary, and add an end-to-end case with 40-sample windows advancing 5.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* fix(promql): ignore empty windows when measuring batch advance
The advance was read from the first and last window offsets, but a window
covering no sample is emitted as (0, 0). A query whose last evaluation lands
exactly one window past the last sample ends on such a window, and its zero
offset made a batch of disjoint windows look like one that never moved, which
selected the evaluator built for overlap. Results stayed correct; the cost was
deque bookkeeping on the shape the scan fallback exists for.
Take the offset span over the windows that cover a sample. Empty windows stay
in the window count, where they only make both conditions stricter.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
Co-authored-by: Dennis Zhuang <xzhuang@greptime.com>
* fix(mito2): preserve last-non-null values during compaction
Use a dedicated LastNonNullPicker to close TWCS seeds over overlapping SSTs, merge connected seed groups, and defer closures with busy inputs. Keep LastRow and strict-window selection unchanged.
Index overlap candidates on the blocking compaction runtime and cover field promotion, transitive dependencies, expiration, and conservative key-range pruning with regression tests.
Refs #9146
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): guard last-non-null compaction against pending memtables
Track conservative memtable sequence lower bounds and capture them from the same version as compaction inputs. Defer unsafe SST closures and strict-window plans until pending writes no longer intersect their sequence span.
Cache minimum sequences during bulk conversion and expose a lightweight memtable getter to avoid rescanning rows or computing full statistics.
Fixes#9157
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): limit eligible last-non-null compaction outputs
Enumerate all TWCS seeds for LastNonNull while retaining bounded window planning concurrency. Apply the output budget after closure expansion and busy/memtable safety checks, preserving execution priority and ordinary TWCS behavior.
Cover rejected and absorbed seeds, priority ordering, and bounded and unlimited output counts with real-picker regression tests.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): include all candidates before last-non-null closure
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify last-non-null compaction closure
Extract transitive closure expansion from output scheduling and clarify seed handling. Document the dedicated picker rationale and schema-evolution risks in PK overlap checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover deferred non-seed compaction bridges
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): defer strict-window compaction with busy dependencies
Keep busy SSTs when building the LastNonNull strict-window dependency closure, then defer the entire rewrite plan if any selected input is busy. Preserve LastRow filtering and expired-file cleanup.\n\nCover requested, transitive and unrelated busy windows, including retry after the dependency is released.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document compaction overlap index fields
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: allow re-enabling WAL after disabling
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test: regenerate skip_wal sqlness result
The previous commit changed tests/cases/standalone/common/skip_wal.sql without
regenerating the matching .result, so every Sqlness suite failed on the
mismatch. tests/cases/distributed/common is a symlink to standalone/common, so
the single stale file accounted for all five failing variants.
Regenerated from a real run. The recorded output now covers the cases the .sql
added:
* A table created with skip_wal = 'true' has no real WAL provider, so enabling
WAL is refused with Unsupported rather than the previous InvalidArguments.
* A RaftEngine-backed table accepts the true -> false transition, and repeating
it is a no-op.
* SHOW CREATE TABLE reports skip_wal = 'false' after the transition, including
after a restart.
* A row written while WAL was skipped is absent after restart, while a row
written after WAL was restored survives.
Verified locally with nothing else running on the machine: sqlness skip_wal
passes in both the standalone and distributed environments, and the regenerated
file is byte-identical to the output of an independent earlier run.
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test: strengthen skip_wal provider coverage
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* Fix WAL provider handling during re-enable
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* Refactor WAL provider check
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
---------
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
verify_rows deep-cloned the whole HashMap<String, ColumnMetadata> of the
physical region on every put batch. On a datanode serving wide physical
tables this showed up as ~18% of total CPU in a CPU flame graph (HashMap
clone + RawTable/ColumnMetadata drop).
Wrap physical_columns in an Arc inside PhysicalRegionState and take a
cheap Arc snapshot instead; add_physical_columns now goes through
Arc::make_mut.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix(mysql): strip leading comments before the federated statement filter
JDBC clients prefix every statement with a comment. DataGrip sends
`/* ApplicationName=DataGrip <version> */` in front of each one, and every
pattern in the federated filter is anchored with `^`, so the prefix makes all
of them miss.
Two failures follow. `SET TRANSACTION READ WRITE` reaches the SQL parser and is
rejected. Worse, the DBeaver-specific entries such as
`^(/\* ApplicationName=(.*)SELECT @@(.*))` do match DataGrip's prefix, so
`SELECT @@GLOBAL.event_scheduler` and `SHOW VARIABLES LIKE ...` are absorbed
into a zero-column output, which the MySQL writer sends as an OK packet. The
JDBC driver reports that the statement returned no cursor.
Strip leading whitespace and comments before matching, and drop the
ApplicationName-specific entries the stripping makes redundant. The three that
had no unprefixed counterpart (`SHOW PLUGINS`, `SHOW ENGINES`, `SHOW @@...`)
keep their behaviour as plain patterns.
Also covers gaps found while probing the same path:
- `BEGIN` is absorbed like `START TRANSACTION`/`COMMIT`/`ROLLBACK`.
- `SHOW [GLOBAL|SESSION|LOCAL] VARIABLES|STATUS` parses; the scope is ignored
because GreptimeDB keeps no global/session split.
- `USER()`, `CURRENT_USER()`, `SYSTEM_USER()` and `SCHEMA()` are registered.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mysql): keep executable comments, reject multi-statement absorption
Review follow-up on the comment stripping, plus the remaining MySQL
compatibility gaps from #9155.
`/*!...*/` is an executable comment: mysqldump emits its initialization as
`/*!40101 SET NAMES ... */`, and the patterns match those verbatim. Stripping
it left an empty statement, so nothing matched and the original SQL reached the
parser, which rejects `SetNames` and `MultipleAssignments`. Leave executable
comments in place.
Absorbing a request also has to stop at a statement boundary, because every
pattern ends in `(.*)`. `BEGIN; INSERT INTO t VALUES (1)` used to fail on the
unsupported `BEGIN`; once `BEGIN` became absorbable the whole request would
report success and write nothing. A request is now scanned for a second
statement and handed to the query engine if it has one. The scan skips plain
comments and string literals so a `;` inside either is not a boundary, and
treats a `/*!...*/` that is not the request itself as a statement, since it
carries SQL.
`check()` now dispatches on the leading statement keyword, so INSERT, UPDATE,
CREATE and ordinary SELECTs run no regex at all — this replaces the narrower
hand-rolled INSERT shortcut. The statement scan runs only for a request the
patterns already matched, which is always a short one.
New compatibility surface:
- `SELECT CURRENT_USER|SESSION_USER|SYSTEM_USER|USER` without parentheses, and
`SELECT @var`, are answered here rather than in the parser. Both are anchored
to the whole statement, so `SELECT user FROM t` still reads the column.
- `information_schema.plugins`, `user_privileges` and `processlist` are
registered as empty tables, like the other MySQL-shape tables around them.
Sessions are reported through `information_schema.process_list` and
`SHOW PROCESSLIST`; `processlist` carries the column shape only.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito2): reconcile series indexes in background
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): clean up series indexes published during region drop
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(config): align series index examples with upstream enable flag
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): run series index tasks on compaction runtime
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): deliver complete WAL entries without waiting for next input
Entry completeness is self-contained via Entry::is_complete(); no
lookahead to the next entry is needed. Decode and yield complete entries
immediately in both the log-store reader and the entry distributor.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): simplify live WAL entry fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: use join_dir for series index config path
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: configure series indexes with an enable flag
Signed-off-by: evenyag <realevenyag@gmail.com>
* docs: omit experimental series index from example configs
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: preserve legacy cache cleanup path behavior
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: remove trivial path joining tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: isolate worker group WAL directories on Windows
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* perf(servers): group Prometheus response rows by label runs
Query output tends to be clustered by series, but a matrix response read
the same label values out of the tag columns once per row, and allocated
a key vector per row to look the series up.
Use `arrow::compute::partition` to find the runs of rows that share their
labels and build the series key once per run. The key buffer is hoisted
out of the row loop and handed to the map only when the series is new,
through the raw entry API so the key is hashed once either way.
Partitioning does not pay off when rows are not clustered, so a few
adjacent row pairs are probed first to pick between the run path and the
row-by-row path. Both paths produce the same series.
Drop the per-row "same labels as the previous row" check from #8815. Runs
cover the clustered case it was written for, and it now costs more than
it saves: 10% on a result with one row per series, 1-3% on clustered ones.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* refactor(servers): find label runs with cmp::distinct
`arrow::compute::partition` computes the same ranges on the same kernel,
but its contract takes lexicographically sorted columns, and query output
is not sorted: range queries run without the plan's output sort since
#9090, `sort`/`topk` order by value, and the tag column order in the
schema does not have to match any sort key. An implementation that
exploited the precondition would merge `a, b, a` into one run and
attribute one series' samples to another, without failing.
`cmp::distinct` is element-wise, so it holds for any row order, and its
null handling is the one a series key needs: a null label and an empty
one are distinct, two nulls are not. Building the ranges from the
boundary bitmask also folds away the `tag_columns.is_empty()` case, since
no columns means no boundaries means a single run.
Same kernel, so the benchmark does not move: -1.4% to +1.4% across shapes
with no consistent sign, against +-3% run-to-run drift.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
`PlanRewriter::should_expand` encoded every node it ascended past to
Substrait. Each call clones the plan, runs an analyzer over it and builds
a fresh `SessionState` with all default features, so a deep PromQL plan
pays that cost once per level.
Encode the root once in `PlanRewriter::new` and skip the per-node call
when it succeeds. `to_substrait_plan` recurses from the root to the
leaves, and `should_expand` only receives sub-trees that `f_down` pushed
on the stack untouched, so a root that encodes proves the whole descent
encodes. A root that does not encode falls back to the per-node check,
which is what locates the node that has to stay on the frontend.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): treat NULL field values as absent samples in range functions
Range functions read the value column through `Float64Array::values()`,
which returns the raw buffer and ignores the null bitmap. A NULL field
value means the series has no sample at that timestamp, so the padding
under a null slot (0.0 in practice) was counted as a real sample.
`rate`, `increase`, `delta`, `changes`, `resets`, `idelta`, `irate`,
`quantile_over_time` and `avg_over_time` now work on samples instead of
slots. `stddev_over_time` and `stdvar_over_time` used to panic on a NULL
slot, and tokio swallowed the panic so the query returned success with
that series missing.
Null handling is gated on `null_count() == 0` over the whole backing
array, checked once per batch, so tables without NULLs keep the existing
code path. `deriv` and `predict_linear` already had this guard through
`linear_regression_slices` and are untouched.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep other fields when one has no sample in the window
Review follow-up. Two things surfaced once range functions started
returning NULL for a window without samples.
The filter after a function call required every field column to be
non-NULL, so on a multi-field table one field with no samples in a
window would drop the other fields' results with it. It now keeps a row
when any field has a sample, which is the shape a selector already
emits. On a single field column the two predicates are identical.
`quantile_over_time` returned NaN rather than NULL for a window without
samples, so the row survived that filter. Prometheus returns an empty
vector there, so the emptiness check now sits in the range UDF; the
shared quantile kernel still yields NaN for an empty slice, matching
upstream's `quantile()` helper that `quantile_aggr` depends on.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(promql): correct two comments on the null handling
The planner one did not say why `preserve_any_value` is hardcoded at
that call site, which is the question a reader arrives with. The
`quantile_over_time` one described the empty-window behaviour while
sitting on the `has_nulls` line, and that behaviour had moved into
`window_quantile`.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): clamp extrapolation before snapping a counter to zero
Prometheus clamps `durationToStart` to half an average interval once the
first sample is past the extrapolation threshold, and only then lets the
counter zero-snap shorten it further, so the snap can never lengthen the
leading extrapolation. Running the snap first let it rescue a duration
the clamp should have cut, and `rate` and `increase` over-extrapolated
to the left. For samples 1@0s and 2@1s in a 4s window at 1s, upstream
gives 0.375 and this returned 0.5.
`extrapolation_matches_prometheus_on_seeded_windows` carries a
line-by-line port of upstream `extrapolatedRate` as an oracle and diffs
it against the UDF over seeded windows, so the order stays pinned.
`factor` also picked up upstream's guard against a zero sampled
interval, which previously divided by zero.
Two sqlness results move, both verified against the upstream algorithm.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): fold range_presence_null into null_samples
#9104 landed its own NULL-sample case whose data is the same series this
one already used: one host with interior NULLs, one host with nothing
but NULLs. Keeping both means two files asserting the same semantics on
the same rows.
The merged case keeps every query from both, so the presence functions
still cover the trailing-NULL window, the count of two, the empty
left-open window at t=7, and the all-NULL windows.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): look up row group stats by parquet leaf index for nested columns
On flat-format tables a logical column can expand to multiple parquet
leaf columns (e.g. a JSON2 struct stores the remainder and one leaf per
promoted path). ParquetFlat used the logical column index in the SST
schema directly as the leaf index when reading row group statistics, so
min/max/null stats of every column after a nested column were read from
wrong leaves.
When the misplaced leaf held order-compatible statistics (e.g. a small
Int64 JSON path vs. the timestamp window predicate), min-max pruning
dropped whole row groups by mistake. SWCS compaction reads inputs with a
time window predicate, so it silently lost all rows of such files; plain
queries with time-range predicates were affected as well.
Map each column to its first parquet leaf column and report NoStats for
columns with multiple leaves, which makes pruning conservative for them.
Add a unit test and a sqlness regression case that reproduces the data
loss on the unfixed binary.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): skip nested root stats and correct SWCS regression baseline
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve statistics leaves for primary-key SST readers
Resolve scalar roots against the actual Parquet schema in shared statistics helpers, covering both flat and primary-key readers. Remove flat-side translation to avoid mapping twice and align encoded primary-key statistics as well. Cover dense flat, legacy dense and sparse layouts with statistics and time-pruning regressions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): distinguish known null counts from unknown statistics
Assert validity before reading timestamp null counts and add a nullable scalar after the nested root with a known nonzero count. Exercise the assertions for flat and primary-key SST layouts.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): check JSON2 time pruning before SWCS compaction
Query the first time window immediately after FLUSH to cover predicate reads on flush-written SSTs independently of compaction outputs. Regenerate the sqlness expectation and retain the post-compaction checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): validate all business columns after JSON2 SWCS
Expand the final regression query to all eight business columns so the generated expectation verifies complete rows, including tags and scalar fields, after repeated compaction.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
jeremyhi
Yingwen
discord9
shuiyisong
Weny Xu
Lei, HUANG
Dhruv Vaishnav
fys
Lanqing Yang
LFC
dennis zhuang
Immanuel Tikhonov