mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-20 17:54:12 +00:00
ee9ad87fa16a1484cf1b5a22ea02ef17b47ca5e9
422
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ee9ad87fa1 |
Merge upstream/main into narthex (pre-merge rehearsal, not for push)
Rehearsal merge of main ( |
||
|
|
297a343fd4 |
fix(query): use current partition columns in prepared plans (#10244)
A prepared statement keeps its logical plan, and that plan holds the table as it was at prepare time. The analyzer reads the partition columns from that table while the regions to scan come from the current route. After a table's partition columns change, the analyzer can consider a query safe to push down, so each region runs the whole query and the frontend concatenates their results. The same happens to the source of a prepared INSERT ... SELECT, which then stores wrong values. Before executing a kept plan, compare each scanned table's partition columns with the current table found by id. Scans of a table whose columns changed get the current table when the schema is unchanged. Otherwise they keep their schema, because a scan's projection holds indices into it, and report the current columns as absent from it, so the query keeps merging on the frontend. Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
099059db10 |
perf(mito2): retain batch-backed labels for native samples (#10134)
Retain compacted metadata rows for new series candidates and materialize owned labels lazily. Preserve region-qualified identities, full physical-tag conflict checks and mixed-predicate fallback. Separate scan bindings from output labels and validate execution-time schemas. Validated cross-bucket retention and conflicts, filtered PromQL label binding, affected crate tests and strict Clippy. Sample-to-flat, merge and frame conversions remain. |
||
|
|
2714687ab7 |
feat(otlp): add v1 schema for auto-created OTLP log tables (#10207)
* feat(otlp): add v1 schema for auto-created OTLP log tables New tables created by direct OTLP logs ingestion get a v1 built-in schema: service_name tag from resource service.name ahead of scope_name in the primary key, a skipping index on trace_id, and event_name and observed_timestamp fields. The table is marked with table_data_model=greptime_log_v1. Existing tables without the marker keep the legacy schema. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(otlp): share resource service.name parsing between logs and traces Log tables now derive service_name with the same rules as trace spans, so both signals agree on the value used to correlate them. The integration test writes twice to cover the existing v1 table path. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
58a67a47c3 |
feat(query): execute native raw promql over real datanodes (#10150)
* feat(promql): serialize raw series source bindings through Substrait Signed-off-by: discord9 <discord9@163.com> * feat(query): transport interleaved raw series fragments Signed-off-by: discord9 <discord9@163.com> * feat(promql): reconstruct interleaved raw series at repartition Signed-off-by: discord9 <discord9@163.com> * feat(query): plan distributed native raw series Signed-off-by: discord9 <discord9@163.com> * chore: satisfy clippy for the raw series transport The workspace clippy run rejects a redundant rebinding, a constant-size `chunks_exact`, and an implicit `String` clone in the tests these commits carry. No behavior changes. Signed-off-by: discord9 <discord9@163.com> * feat(query): execute native raw promql over real datanodes Plan a whole native statement as one fused node over raw series sources when the frontend reaches regions over the wire, including single-selector statements, while keeping the in-process split and segment shapes unchanged. Validate the whole tree with the runtime constructor before committing to that plan, so unsupported shapes still fall back to the legacy route. Bound the region reads of that topology with a query-wide permit budget of target_partitions, held until each region body is drained; other plan shapes keep unbounded reads rather than risk a body budget they could deadlock. Acceptance on a real two-datanode cluster with TCP frontend clients: time-partitioned series are reconciled before the fused node and match the unpartitioned control, with the DoGet peak at the target bound; series that no region shares keep contiguous per-series order and reach the fused node without a reconciling router. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the raw series transport PR The raw series Substrait message lives in greptime-proto PR #349, which is not merged, so pin its head commit instead of carrying a local path patch. Move the rev to the merge commit once #349 lands. https://github.com/GreptimeTeam/greptime-proto/pull/349 Signed-off-by: discord9 <discord9@163.com> * fix(promql): reconcile a discovered union of raw series A nameless selector reads its matches as one `Union`, and `UnionExec` reports the frame metadata of its last child. A time-partitioned metric sorted before a host-partitioned one therefore looked contiguous, and the fused node skipped the reconciliation of the fragmented member's series, silently folding region fragments as separate series. Lower the declaration on the fused edge to the weakest ordering any member declares, before the fused node reads it. The union itself is unchanged; the fused node's existing interleaved-input path inserts the router. A projection carries the lowered declaration because the pinned physical planner ignores a union's own schema. Cover it with a planner regression for both member orders and an all-complete control, and with a real two-datanode acceptance case that mixes a time-partitioned and a host-partitioned metric and checks the extrapolated increase per series. Signed-off-by: discord9 <discord9@163.com> * test(promql): cover aggregates over discovered metrics A nameless selector's metric comes from its own frames, so a folding aggregate is a multi-input tree even with one selector: apply the same preflight and order-dependent refusals on the remote route. Cover `sum(increase(...))` and grouping by the metric name over the mixed discovery fixture. A grouping that names only `__name__` stays declined, since the output schema cannot name that group: here the metrics also differ by `site`, so grouping by the name and `site` exercises the two groups. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep remote multi-source refusals the split made Skipping the aggregate split and `segment_tree` over the wire must not drop their admission refusals. Two sources that fold groups by metric name would report groups that differ only by name under one label set, and an order-dependent operation over a discovered selector's union would answer in arrival order. Scope both refusals to the actual multi-source inputs rather than to the logical-table-id heuristic: a discovered selector reads several metrics through one selector, and a Mito table beside a metric one carries no logical id. Cover the Mito-plus-metric grouping refusal on a real two-datanode cluster, and cover a discovered `sum by (__name__)` over a raw selector (the name survives it) and a folding `sum(increase(...))`. Signed-off-by: discord9 <discord9@163.com> * docs: drop the local proto patch prerequisite The dependency is pinned to greptime-proto PR #349's commit, so the wire fixture no longer needs a local path patch. Signed-off-by: discord9 <discord9@163.com> * chore: finish the narthex rebase resolution Keep the upstream aggregate split (`FusedSeries::split_aggregate`) and pass the engine state through the native planner's callers, while the remote route still declines the split it cannot carry. Signed-off-by: discord9 <discord9@163.com> * test: pass the engine state to the native operand test The native operand planner now takes the engine state, which decides whether a statement is planned for remote regions. Keep the histogram operand contract test on that signature. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the split and segment plans over the wire Remote planning disabled the aggregate split and segmentation because the split's producer read a bare scan. The producer reads whatever the statement prepared, and over the wire that is the raw series source: the partial half runs per partition on the frontend, the final half folds the exported states, and no aggregate state crosses the wire. Keep the split and let segmentation build evaluated segments from the raw sources instead of bare scans, so per-series window work keeps the scan width. Admit the resulting fused chain to the query-wide read budget, since each fused node polls its input while it is drained. Remove the recovery-projection wrapper: the analyzer no longer inserts that projection, and the physical planner consumes the frame boundary as it is. Serialization is now only the node's encoding: the codec no longer decides placement, and the metric-name binding is checked against the declared frames rather than the node types below. Signed-off-by: discord9 <discord9@163.com> * refactor(query): drop the in-process encodability exception A split aggregate encodes now, so the analyzer no longer needs a special case that skipped the encoding gate for it. The gate itself stays: every plan a region receives still has to encode. Signed-off-by: discord9 <discord9@163.com> * test: cover a repeated selector over one partitioned table Two occurrences of one time-partitioned metric keep their own raw source, time bounds and identity over the wire, so a ratio of two windows and a binary expression over two identical ones stay native and reconcile each occurrence's fragments before the fused node reads them. Compare both against the unpartitioned control. Signed-off-by: discord9 <discord9@163.com> * fix(table): refuse a scan that plans interleaved ordering A scan's planned ordering is a guarantee its own partitions keep. Interleaved is the transport annotation for fragments several producers contribute, so a scanner that planned it has no raw-series read whose frame would honour that declaration. Reject it in the scan conversion, while the frame codec keeps accepting interleaved raw frames from the router that reconciles them. Signed-off-by: discord9 <discord9@163.com> * test(promql): assert the metric name beside the value `sum by (__name__)` over a discovered selector was asserted as a bare sorted list of values, so a plan that grouped by the wrong metric name - or dropped the name and folded everything into one series - could produce those same numbers. Read `__name__` and `val` by column name (the result puts `__name__` last, and a fold that drops the name drops the column rather than emptying it) and compare the `(name, value)` pairs against the fixture's own metric names. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the hybrid route for a partitioned pair A multi-selector tree over a partitioned table was admitted as one fused node for any pair of tables, not only for scans of one table. That preempted the frontend hybrid route, which copies the matcher's labels into each operand's scan and checks uniqueness above the join. `native_hybrid_binary_05_06_08` depends on exactly that plan: its partitioned RHS holds a duplicated `job="other"` group, so the whole-native binary reports the duplicate instead of the value the hybrid route returns. Refuse the whole-fused plan for that shape in process again; the remote raw-source admission is unchanged, since a remote multi-selector tree over a partitioned table is the point of this branch and the distributed acceptance tests read it. The hybrid route then has to behave the same way it does without native operands: `matching_filters::propagate` now runs for them too, and a rewritten operand is re-planned through `try_plan_native_operand` with its original `selector_matcher` preserved, falling back to the ordinary plan when the native route declines. The unpartitioned case, whose duplicate group the whole-native binary is expected to report, is unchanged. The two new re-plan call sites put the native route's large future inside the recursive planner frame at every nesting level (~178 KB became ~209 KB), which overflowed the 2 MiB test-thread stack in `test_parse_or_operator` even where the new branch never ran. Box the future in a non-async helper and move the re-plan into `replan_rewritten_operand`, which brings the frame back to the old size without changing the behaviour. The partitioned-pair unit test asserted the removed shape. It now asks for the remote engine state, where a cross-table pair really is one fused node over two raw sources, and asserts that the same pair in process keeps the hybrid route. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the merged raw series transport GreptimeTeam/greptime-proto#349 has been merged into `narthex-proto` as `f51cb5312bf7`. Move the rev from that PR's head commit to its merge commit, in `Cargo.toml` and `Cargo.lock` together, so the pin no longer points at an unmerged ref. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
8a38b90135 |
feat: add relational region engine skeleton and registration (#9953)
* feat: add relational region engine skeleton and registration Add a read-only relational RegionEngine (create/open/drop lifecycle) and register it as the 4th engine in build_store_engines alongside mito/metric/file. Add a datanode-level test asserting all 4 engines are registered. Signed-off-by: tian1220A <1573612521@qq.com> * feat: add relational connection and idempotent region lifecycle Move ConnectionInfo into relational-engine (DSN parsing + redacted rendering), place the RelationalOptions/RELATIONAL_META_KEY serialization contract in table::requests, make CREATE idempotent by comparing the persisted manifest definition against the incoming request (RegionExists on mismatch), and drop the empty EngineConfig placeholder. Signed-off-by: tian1220A <1573612521@qq.com> * refactor: address review comments on relational region engine - remove connection.rs (only used by its own tests); it will be reintroduced when connection handling is wired up - drop RelationalOptions::version (not validated or used to select a deserialization format) - validate an idempotent create against the existing region definition (region id / schema / primary key / options) and return Unexpected on mismatch, with a reason field on RegionExists - use serde.workspace and remove the empty [features] section - cargo fmt Signed-off-by: tian1220A <1573612521@qq.com> * fix: adapt relational engine to latest main - implement RegionEngine::set_region_role_with_lease (no-op lease, the engine takes no writes so no lease is needed) - pass node_id to build_object_store_manager in the registration test - drop stale url dependency from the relational-engine lock entry Signed-off-by: tian1220A <1573612521@qq.com> * ci: register relational-engine in scope rules Signed-off-by: tian1220A <1573612521@qq.com> * fix: remove table self-dependency from Cargo.lock Signed-off-by: tian1220A <1573612521@qq.com> --------- Signed-off-by: tian1220A <1573612521@qq.com> |
||
|
|
ec240ad7e6 |
feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions. Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): preserve reserved sparse key semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): clarify sparse errors and avoid panics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito): reduce sparse key validation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito-codec): reduce sparse validation cost Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(mito): remove ordinary sparse support Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
efed847ca6 |
fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce sparse primary key schema Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: fix Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
045cf74c3f |
perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task Compaction read, merged and encoded each output on a single task, so parquet encoding waited for the merge and the merge waited for encoding. The merge now runs on its own compaction runtime task and hands merged batches to the SST writer through a bounded channel. The channel also closes when that task panics, so the reading side awaits the task before reporting the end of the input. A panic fails the compaction instead of producing a truncated SST. One output of 4.8M rows took 1.28s instead of 1.65s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): parallelize decoding and encoding of compaction outputs Parquet encoding took about two thirds of a compaction output's time and ran on a single thread. When `experimental_compaction_output_parallelism` is above 1, the SST writer encodes the columns of each row group on up to that many blocking threads of the compaction runtime while the next row group is buffered, and the merge decodes up to that many input files ahead of it on compaction runtime tasks. The parallel writer feeds every column writer the same input as `ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group reach the object store writer one row group later, so `max_file_size` may split a file one row group later. The option defaults to 1, which keeps today's CPU usage. One output of 4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): cancel compaction producers before consumer polling Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): support per-table compaction output parallelism Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoding tasks before returning failures Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime. Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoders on SST write errors Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error. Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain compaction outputs on cancellation Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures. Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
b29a471575 |
fix(promql): repair native execution contracts and remove obsolete paths (#9980)
* fix(promql): restore native scan statistics and remove unused raw merger Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): repair native execution contracts and remove obsolete paths Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): leave fused frame output repairs to separate PR Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: restore native contracts and regenerate config reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: slice nullable sparse frame fields after span selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): intersect anchored timestamp selection windows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): name anchored timestamp boundary case type Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): align partitioned pair checks with hybrid operands Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> |
||
|
|
2e84833ac8 |
feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): reject partition column type changes and quote rule identifiers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(table): keep time index widening for partition columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(partition): replace PartitionFunction with IcebergHash Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(partition): describe iceberg_hash in the function module Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: quote partition rule identifiers in SHOW CREATE TABLE expectations Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(partition): check hash rules in information_schema and repartition target columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): coerce hash partition rules against the evaluation schema Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep quotes in partition column names when building filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
513b9cee58 |
feat(catalog): add public-global information schema scope (#9957)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
2b67d0d60c |
test(series): pin literal ordering declarations and decoder validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
62652ec62e |
fix(series): validate v2 ordering and repair scan binding fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
b20e22b426 |
refactor(series): unify raw and evaluated SeriesFrame codec
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
|
6157479d9a |
chore: integrate series-label naming with the recursive native POC
Apply only the type rename to the surviving recursive runtime and tests; preserve the removal of global final-label occupancy and speculative split-series admission guards. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
29ef60f6e8 |
test(promql): restore scan assignment coverage and update native guide
Signed-off-by: discord9 <discord9@163.com> |
||
|
|
8efec61743 |
fix(promql): keep the fused scan's parallelism
Signed-off-by: discord9 <discord9@163.com> |
||
|
|
5ff4f5b911 |
refactor(datatypes): name the PromQL series label set by what it is
`MetricName` held the whole PromQL-visible label set of a series -- the
`__name__` value plus its ordinary labels -- not one metric name. Rename it to
`SeriesLabels` so the type says what it holds, and record in its doc comment
that it is neither a canonical matching/grouping identity nor a storage
identity.
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit
|
||
|
|
2eff52c757 |
feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): guard point accounting and benchmark metrics ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove otlp_metric_compat logic Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): cache physical schema lookups during metric preflight Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: minor rename Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): batch metric rows and reuse schema validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: replace super in import path Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
7309800aef |
feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema A raw series frame written with key-encoded labels carries the primary key as opaque bytes; only the binding held by the scanner says how to turn those bytes back into a metric name and labels. A consumer that receives the frame without that binding cannot decode it, so the schema now describes its own label layout. The description is compact JSON under `greptime.series.label_decode`, carrying a version, the metric name, the primary-key encoding, the ordered key columns with their physical types, and the logical tags to expose -- exactly what decoding reads. It is derived from the binding, so there is one writer, and a reader that does not implement the version it finds rejects the frame instead of guessing. `unpack_raw_series` keeps its binding argument and now requires it to agree with the description the schema declares, so the two cannot drift. `PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key layout, so the metadata-backed and description-backed paths share their validation instead of growing a second decoder. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datatypes): compare label descriptions by what their order means The frame's declared label-decode description was compared to the binding's with plain equality, but the description has one part whose order carries no meaning: the decoder sorts logical tags by name when it binds them, so the same tags listed in another order denote the same labels and were being refused. `same_label_layout` canonicalises exactly that part and leaves `primary_key` in place, because for a dense key the array position is the encoding position: the same columns in another order are a different key, not the same key reordered. Both fields now say which order they mean. Three checks go with it. The binding stays for a stated reason -- it establishes identity, the physical region key layout a decoded identity came from -- rather than because a count of call sites kept it. The post-pack schema comparison in the series scan could not fire: the leaf already re-derives that schema and compares it, and every `SeriesSchema` field is covered by the checks between them. And two test groups re-asserted what the tracked v1_metadata fixture pins case by case. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): decode a raw frame from the frame alone `unpack_raw_series` took the label mode, the ordering and the caller's binding along with the stream, so a consumer that received a frame from a peer still needed a local copy of the binding to read it — the thing the frame's own description exists to avoid, and the reason the description was freezing a wire format for nobody. The stream is now the only input: the field schema, the label mode, the label decode description and the producer ordering all come from the frame's schema, and every frame is validated whether or not a caller asked for it. `expected` stays as an optional check on the schema that is declared — field names and their order, and no evaluation grid — and supplies nothing to the decoder, so a caller cannot smuggle in a layout the frame does not declare. Three call sites compared the caller's binding against the frame and all three were load-bearing: a peer's response is not trusted to be the region that was planned, and a native plan's child is an arbitrary execution plan. Those became `ensure_frame_matches_label_source`, which states what it compares: the label layout, not the region. Two regions with the same tag ids and names compare equal, and the doc now says so instead of claiming to establish identity. A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers `label_decode_spec()` instead; every consumer that decides whether to decode reads the description, so the placeholder never stands in for "these labels are complete". The merge contract is the description, not the binding: inputs are compared by the layout they declare, and a binding that could not be expressed on the wire — a `RegionMetadata` snapshot — no longer makes the merge stricter than the frames it merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a copy, because the caller's binding keeps its own order and the table scan compares producer tag lists positionally, and readers compare by equality. The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to cover. Rather than mirror all 25 datatype variants into a type this module owns, a frozen fixture records the text of every variant and an exhaustive match makes a new variant fail to compile until it is recorded. The version policy is written next to the constant: exact-text comparison across nodes, unknown versions rejected, any same-version change breaking — with the planning-time fallback stated as intended and marked not implemented. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(store-api): make every key-encoded raw stream declare its layout A raw series stream whose chunks carry key-encoded labels had to be decoded against a scan-bound `RawLabelSource`, and two consumers compensated for producers that stayed silent by deriving a description from that binding: `pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels` did not compensate, so an unframed producer stream reached it with no description and a chunk carrying no labels, and failed. The stream itself is the authority for its chunks. A key-encoded producer knows its own region metadata and can state the layout it writes, exactly as a frame states the layout it carries, so the split contract — "a producer declares its identity, a frame declares its layout" — is replaced by one rule: a stream whose chunks carry key-encoded labels declares the description, and a stream whose chunks carry complete labels answers `None`. mito2's `RawStream` caches the derivation at construction, so the declaration exists before the first poll and for an empty stream. The trait documentation states the rule and no longer exempts unframed producers. Both fallbacks are gone: `pack_raw_series` reads the stream's declaration only, and rejects a key-encoded stream that declares none rather than packing it as an explicit-label frame, which would describe chunks carrying no labels as if they carried all of them. The merge reads the declaration only. Test fixtures that model key-encoded producers now declare their layout too; fixtures that carry complete labels keep answering `None`. The new mito2 test decodes a real produced key through the declared description for both dense and sparse encodings, and checks the declaration is present before polling even for an empty stream. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(common-query): let the declaration decide the frame mode alone `pack_raw_series` rejected a key-encoded stream that declared no description outright, so a stream that stays silent could not be packed at all. That check was redundant: the declaration already decides the frame's label mode, and the first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor` with "raw chunk label mode does not match frame mode", so no such stream can pack data either way. What the check added was an early error for a producer that already breaks the contract, in exchange for reading the stream's scan binding here — in the one function whose rule is that the declaration is the only decode input. Drop it, so pack keeps deciding the mode from the declaration and nothing else. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(store-api): assert the fixture's declared label decode description The mock prepared source derives a `LabelDecodeSpec` from its label source and reports it through `RawSeriesStream::label_decode_spec`, but no assertion read it, so the derivation could drift from the chunks the fixture actually builds. Check it against the description the fixture's own key columns imply: an explicit source declares nothing, a primary-key source declares the metric, the dense encoding, the one key column and its tag. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): let the frame's declaration be the only label authority Two things still treated a caller-supplied binding as authoritative after the declaration became the decode input: - A merged stream forwarded its first input's scan-bound identity, though a merge is not a region scan and decodes from the declaration alone. It now reports no bound identity. - `KeyColumn.tag` repeated an invariant every key column is a tag. The check moves to the one place a decoder is bound, so a snapshot built without validation is still rejected instead of silently decoded. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
b3dc5674e9 |
feat(mito2)!: account SST decoding against shared scan memory limits (#9101)
* feat(mito2): add shared estimated SST scan memory budget Signed-off-by: evenyag <realevenyag@gmail.com> * feat: unify scan memory limits with configurable accounting modes Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse query runtime memory pool for series merges Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): wait for cancelled scan reservations before exhaustion check Signed-off-by: evenyag <realevenyag@gmail.com> * docs(mito2): explain parallel scan cancellation cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): deprecate waiting for scan memory quota Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(recordbatch): remove scan memory wait handling Signed-off-by: evenyag <realevenyag@gmail.com> * fix(table): initialize memory pool in scan test fixtures Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
200ed3ce36 |
feat(table): scan-bound typed raw series production via SeriesScanExec
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
5cfdc30a69 |
fix(table): stop claiming exact null and distinct counts for append scans (#9593)
RegionScanExec reported null_count = 0 and distinct_count = total_rows for every column of an append-only table. DataFusion answers count(col) and count(DISTINCT col) from these statistics, so both returned the row count. Keep the exact row count and an exact zero null count for non-nullable columns, so count(*) is still answered from statistics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
75bc4ce46d |
fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): only claim projected primary key prefix ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): keep physical key ordering for metric projections Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): compare unknown partition counts explicitly Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): model scoped internal TSID scan properties Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
2df33380a7 |
feat(catalog): declare information-schema table scopes (#9504)
* feat(catalog): declare information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: rename cluster to global Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: update sqlness result Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(catalog): require information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
28ec3ff5aa |
fix(table): require complete primary keys for hash partitioning (#9507)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
|
|
079ffec116 |
chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: remove unused RouteInfoCorrupted error Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
3c2aac0a55 |
feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(storage): route series index build requests Signed-off-by: evenyag <realevenyag@gmail.com> * feat(admin): add BUILD_SERIES_INDEX Signed-off-by: evenyag <realevenyag@gmail.com> * test: specify compaction type in series index fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * test: correct series index SQL fixtures and error assertions Signed-off-by: evenyag <realevenyag@gmail.com> * test(compat): preserve legacy index rebuild across upgrades Signed-off-by: evenyag <realevenyag@gmail.com> * test(sql): cover series index admin validation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): bound series index maintenance queue Signed-off-by: evenyag <realevenyag@gmail.com> * docs: remove series index how-to guide Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove index build upgrade compatibility case Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address series index reconciliation review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * fix: bound manual series index reconciliation admission Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto to merged index build options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
20619273ee |
chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates (#9303)
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates Move the workspace from the pinned nightly-2026-03-21 to stable 1.96.1 and drop all 23 '#![feature]' gates across 13 crates, rewriting the still-unstable API usages with stable equivalents: - try_blocks: closures / an async block (table, query, common-function, servers) - duration_constructors: Duration::from_secs(n * 86400) / (n * 60) - iterator_try_collect: collect::<Result<Vec<_>, _>>() - box_patterns: as_deref() + matches! chains (sql) - error_iter: error_chain_root() source-chain walker (common-error); sources() includes the error itself, so the walker never panics - int_roundings: div_floor -> div_euclid (equal for positive divisors) - iter_partition_in_place: stable sort_by_key partition helper (index) - hash_set_entry: HashSet::insert bool / contains+insert - trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin) - string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned() - never_type: Infallible (common-recordbatch) - debug_closure_helpers: closure-backed DebugFmt newtype (mito2) - binary_heap_pop_if: peek().is_some_and() + pop() - exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds - stmt_expr_attributes: stale gate, no usages Also fix release-dev-builder-images.yaml, which parsed rust-toolchain.toml with a date-only regex and would produce empty image versions with a stable channel; it now extracts the full channel token. Dev-builder images verified against stable 1.96.1 (image build, default-toolchain behavior, binstall/nextest, riscv64 and android targets, in-image cargo check). Validated on 1.96.1: cargo check --workspace --all-targets, clippy --workspace --all-targets --all-features -D warnings, cargo fmt --check, and nextest on all 13 affected crates (4586 passed). Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging first. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: update flake checksum * chore: use wild for linker in flake --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
b9f991502c |
refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
|
045441e3cc |
feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
16978cf6c2 |
feat(trace): support Semantic Graph for Trace V2 (follow-up to #9192) (#9278)
* feat(trace): support Semantic Graph for Trace V2 Signed-off-by: luofucong <luofc@foxmail.com> * fix: remove unused annotation context import Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
66d38e8e1c |
feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add `InsertLimitInterceptor` hook to `Inserter` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: attribute insert limit checks to the target table's database Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: unify write admission through metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce write admission for pending row batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: distinguish internal requests for ingestion metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: admit split ingestion requests once per database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: OpenTSDB throws error reason Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use meter crate main rev Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude database ingest rate limit from table options Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: reserve channel 255 for internal requests Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
aacf04cf6e |
feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
39e34c0b74 |
refactor: migrate partition_statistics to statistics_from_inputs (#9175)
DataFusion 55 deprecated ExecutionPlan::partition_statistics in favor of statistics_from_inputs with StatisticsContext::compute. Migrate the two remaining GreptimeDB overrides, RegionScanExec and MergeScanExec, and update the RegionScanExec test to the new API, dropping its allow(deprecated). Behavior is unchanged: RegionScanExec keeps the append-mode-only, exact-source-rows statistics gate from #9154, and MergeScanExec keeps reporting unknown statistics for per-partition requests. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
7e2a75f771 |
feat: allow re-enabling WAL after disabling (#9130)
* feat: allow re-enabling WAL after disabling Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: regenerate skip_wal sqlness result The previous commit changed tests/cases/standalone/common/skip_wal.sql without regenerating the matching .result, so every Sqlness suite failed on the mismatch. tests/cases/distributed/common is a symlink to standalone/common, so the single stale file accounted for all five failing variants. Regenerated from a real run. The recorded output now covers the cases the .sql added: * A table created with skip_wal = 'true' has no real WAL provider, so enabling WAL is refused with Unsupported rather than the previous InvalidArguments. * A RaftEngine-backed table accepts the true -> false transition, and repeating it is a no-op. * SHOW CREATE TABLE reports skip_wal = 'false' after the transition, including after a restart. * A row written while WAL was skipped is absent after restart, while a row written after WAL was restored survives. Verified locally with nothing else running on the machine: sqlness skip_wal passes in both the standalone and distributed environments, and the regenerated file is byte-identical to the output of an independent earlier run. Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: strengthen skip_wal provider coverage Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Fix WAL provider handling during re-enable Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Refactor WAL provider check Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
|
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
|
d9b97796a6 |
fix: preserve count correctness after repartition (#9154)
* fix: preserve count correctness after repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: restore full predicate guard for count statistics Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update count plans for full predicate guard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve count statistics for safe partition scans Signed-off-by: WenyXu <wenymedia@gmail.com> * test: retain repartition home guard and cover staging flush Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
ba0f7acd93 |
feat(mito2): add opt-in byte-stream-split encoding for float SST fields (#9069)
* feat(mito2): add opt-in byte stream split encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): correct float encoding checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): cover float SST encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile float encoding tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): release parquet test writer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): register float test primary key Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify BSS write lifecycles Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(metric-engine): verify BSS physical SST Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify bulk BSS lifecycle Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile bulk BSS test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding constructors Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float upgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float downgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding builder Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): add default versus BSS storage comparison Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): align BSS reader benchmarks with prior study Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): parse current read benchmark averages Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): retain default float encoding in direct SST fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): isolate BSS user SSTs and benchmark every file Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): record measured BSS storage and reader tradeoffs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): expose warm scan variability and evidence limits Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): clarify BSS baseline and storage measurement scope Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): model bounded mixed integer and fractional metric series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): report bounded mixed BSS measurements and query regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): qualify timings affected by concurrent host builds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): include float BSS comparison in default regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): omit unsupported float encoding option from baseline setup Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
adda50e03f |
feat: add repartition partition count hint (#9080)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
7cf84892d2 |
perf(table): filter decoded rows with dynamic predicates (#9004)
* perf(table): filter decoded rows with dynamic predicates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve unknown dynamic filter rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(table): use null guards for dynamic filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve null inputs during dynamic pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover frontend join dynamic filter transfer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): reset dynamic filters and scanner pruning state Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
bb9b7e8778 |
fix: re-scan stream-backed tables in recursive CTEs (#9039)
* fix: re-scan stream-backed tables in recursive CTEs A recursive CTE re-executes its recursive term on every iteration, but DfTableProviderAdapter hands StreamScanAdapter a single-use stream built at planning time. The second iteration failed with "Stream already exhausted" for every table served through DataSource::get_stream — information_schema, pg_catalog, the computed entity-graph tables and numbers. Keep that stream for the first execution and open a new one over the same scan request for later executions. Closes #9037 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: drop redundant binding in stream factory Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
b86da3d35f |
feat: support raw OTLP delta metrics (#8970)
* feat: support raw OTLP delta metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): update sqlness results for normalized label matching Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: derive temporality label from default column prefix Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): add analyze coverage for delta temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): scope label alignment to temporality marker Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: handle count-only histograms and vector broadcasts Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude temporality marker from entity descriptions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: use a fixed label for OTLP aggregation temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): preserve mixed-range semantics for raw delta Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
6f1dd0cb76 |
feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN Previously MODIFY COLUMN rejected the time index column outright. Now the time index unit can be widened (Second -> Milli -> Micro -> Nano), which is lossless for data that fits the target unit: historical data in old SSTs is cast to the new unit on read by the existing schema-compat layer, and compaction rewrites it lazily. Narrowing and non-timestamp targets remain rejected; tag columns keep being rejected. Widening is rejected if any SST's time range would overflow the target unit's i64 range (e.g. millisecond -> nanosecond beyond year 2262), since the cast would silently null those values. Read-path correctness for old-unit SSTs (verified by new engine e2e tests and sqlness WHERE queries): - row-group min/max pruning: parquet statistics of a timestamp column are raw integers in the file's unit; when the region metadata's type differs (also the case for altered field columns), stats are now interpreted in the file's type and converted to the expected type before pruning. Without this, a new-unit predicate silently pruned whole row groups of old-unit files (wrong results, rows missing). - SST-level simple filters are skipped for columns whose file type differs from the expected type; the predicate is applied by the query layer's residual filter above the region scan. Also drop the stale "timestamp columns cannot change type" debug_assert. - retry idempotency: a same-type ModifyColumnType on the time index validates as a no-op and `need_alter` returns false, so a retried alter procedure (region already altered before the previous attempt failed) converges instead of aborting forever. - add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to - relax ModifyColumnType validation in store-api and table metadata - tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests (flushed SST + new writes + reopen + retry + overflow + predicate scans, cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl. partitioned table and WHERE filters over old-unit data Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve parquet filter issue * test: provide sqlness tests * refactor: drop trivial test cases and shorten comments Review pass over the branch's additions: - datatypes: keep a representative subset of the widening-matrix asserts - table: collapse the three single-branch rejection blocks into one loop - mito2: drop the boundary gt_eq and post-compaction predicate asserts (covered by the exact-filter regression test and sqlness); drop the engine-level gt_eq/lt_eq casts (full operator matrix stays in the cast_timestamp_unit unit tests) - sqlness: drop a bare full scan already covered by the filter above it - shorten function doc comments across datatypes/store-api/table/mito2/ recordbatch to the essential semantics Signed-off-by: Ning Sun <sunning@greptime.com> * fix: drop physical prefilter for columns whose file type differs Follow-up to the review feedback on CompatBatch/prune reader/filter handling for widened time index units. Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext, which builds its physical expression against the FILE's schema while the predicate literals are in the expected (post-alter) unit. Evaluating them against an old-unit SST raised a cross-unit comparison error (Timestamp(ms) >= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates are best-effort pruning hints (the query layer re-applies them above the scan), so drop the prefilter when the column's file type differs from the expected type, mirroring the simple-filter strategy. Verified: Between and a non-rewritten (large) InList on old-unit data no longer error and filter exactly end-to-end (sqlness), and no matching rows are lost at the engine level (engine test). Signed-off-by: Ning Sun <sunning@greptime.com> * test: add direct unit tests for stats cast and prefilter drop The two-step stats cast (reinterpret raw Int64 stats in the file's timestamp type, then rescale to the expected type) and the physical prefilter drop on file/expected type mismatch were only covered end-to-end; add localized unit tests so a regression fails at the exact site: - stats.rs (previously no tests): RowGroupPruningStats min/max over a hand-built RowGroupMetaData — passthrough with no expected metadata, passthrough on same type, and rescale (1000ms -> 1_000_000us, not 1000us) on a widened expected unit - reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter when file and expected types match and drops it on unit mismatch Signed-off-by: Ning Sun <sunning@greptime.com> * fix: tolerate mixed time units range cache key coverage check * docs: flag mixed-unit hazard in the (unwired) series index The series index stores per-series min/max ts as raw Int64 in the unit of the region metadata at write time, and the searcher builds its range predicates from a single per-region metadata. After a time index unit widen, files of one region would carry mixed units, so a per-file unit (or an index rebuild on such alters) is required before this index is wired into scans. Leave notes at both sites. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover mixed-unit compaction for sparse encoding and strict windows Compaction-path audit follow-up. The compat cast and window math were already covered for dense regions; add the two remaining e2e scenarios: - sparse primary key encoding (used by metric-engine physical regions): widening then compacting mixed-unit files rewrites the old-unit time index correctly through the sparse compaction compat path - strict-window manual compaction: each window output trims rows with a predicate built in the region's new unit against an old-unit file; every instant must survive exactly once (no loss, no cross-window duplication), rescaled Also documents the audit finding that Regular ranged (manual) compaction never trims rows: TwcsPicker sets output_time_range to None and the request time range only selects candidate windows. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover time index unit change in FlatCompatBatch directly The compat layer's rescaling of a widened time index was only verified end-to-end; add direct unit tests for both paths: - dense: identical units skip compat entirely; a widened unit rescales the time index column (1000ms -> 1_000_000us, not reinterpreted) while other columns pass through and the output schema matches the expected metadata - compact sparse (the metric-engine compaction path): same rescaling Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: move timestamp unit division into common-time The exact unit division (UnitQuotient + div_mod_units) is time semantics, not filter logic; move it next to TimeUnit in common-time with a compact test covering representable/non-representable values, negative (floor) instants, and quotient overflow. The ScalarValue helpers stay in filter.rs since common-time has no datafusion dependency. Signed-off-by: Ning Sun <sunning@greptime.com> * test: compat rescales a widened time index and fills an added column together The realistic multi-alter sequence (widen at T1, add column at T2, read a T0 SST) exercises cast and default-fill in the same compute_index_and_fields pass; assert both in one output batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preflight time index widening overflow before any region alters Address review feedback on the overflow guard: - preflight: when the frontend operator receives a widening alter on the time index, run an existence scan (ts outside the target unit's i64 range, LIMIT 1, via the query engine so it covers every region of the table in both standalone and distributed modes) BEFORE any DDL task is submitted. A region that fits can no longer commit the new schema while another region rejects the alter with a non-retryable error. File and row-group pruning keep the scan cheap when nothing overflows. The per-region check in mito2 stays as the final guard for data written after the preflight (the remaining race window); without a validate-only wire field (region.proto lives in the external greptime-proto repo) a fully atomic two-phase validate/commit is out of scope here. - fast path: cast_timestamp_unit returns the filter unchanged when the literal is already in the target unit, skipping the div-mod rebuild. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address auto review comments * fix: remove time index widening overflow preflight Overflow needs timestamps beyond the target unit's i64 range (~year 2262 for nanoseconds), which real workloads never write, so the two existence scans before every widening alter are not worth the cost. Region validation already rejects the alter when an SST's time range overflows the target unit; it now logs the rejection (with the offending file) and returns a deterministic client-facing message. Signed-off-by: Ning Sun <sunning@greptime.com> * test: make sqlness test stable * fix: log instead of rejecting time index widening overflow Overflowing values cast to NULL on read but do not otherwise affect reads or writes, so the alter is allowed; the region-level check now only logs (with the offending file) when an SST's time range exceeds the target unit's i64 range. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |