Commit Graph
388 Commits
Author SHA1 Message Date
discord9 bf97de2ad1 feat(promql): compile subqueries with deferred grid materialization (#9835)
Compile range functions over explicit/default-step, offset, anchored, and nested subqueries using the shared range-evaluator grid calculation. Retain complete sampled coverage and reject empty child grids explicitly.

Keep compilation allocation proportional to plan structure, not timestamp count: validate deferred root and child grids; preserve eager EvalGrid::try_new; materialize shared timestamps fallibly only when a consumer requests them. Migrate Rust timestamp-buffer callers to Result without changing wire or persisted formats.

Regression coverage includes grid phase/coverage, rounded sampled endpoints, offset overflow, huge and nested compile-only grids, sparse gathering, sharing, and allocation-error propagation. Deferred compilation does not bound execution memory. No public Fused entrypoint or Scalar runtime is added here.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-22 16:48:41 +08:00
discord9 7309800aef feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema

A raw series frame written with key-encoded labels carries the primary key
as opaque bytes; only the binding held by the scanner says how to turn
those bytes back into a metric name and labels. A consumer that receives
the frame without that binding cannot decode it, so the schema now
describes its own label layout.

The description is compact JSON under `greptime.series.label_decode`,
carrying a version, the metric name, the primary-key encoding, the ordered
key columns with their physical types, and the logical tags to expose
-- exactly what decoding reads. It is derived from the binding, so there
is one writer, and a reader that does not implement the version it finds
rejects the frame instead of guessing.

`unpack_raw_series` keeps its binding argument and now requires it to
agree with the description the schema declares, so the two cannot drift.
`PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key
layout, so the metadata-backed and description-backed paths share their
validation instead of growing a second decoder.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datatypes): compare label descriptions by what their order means

The frame's declared label-decode description was compared to the binding's with
plain equality, but the description has one part whose order carries no meaning:
the decoder sorts logical tags by name when it binds them, so the same tags listed
in another order denote the same labels and were being refused.

`same_label_layout` canonicalises exactly that part and leaves `primary_key` in
place, because for a dense key the array position is the encoding position: the same
columns in another order are a different key, not the same key reordered. Both
fields now say which order they mean.

Three checks go with it. The binding stays for a stated reason -- it establishes
identity, the physical region key layout a decoded identity came from -- rather than
because a count of call sites kept it. The post-pack schema comparison in the series
scan could not fire: the leaf already re-derives that schema and compares it, and
every `SeriesSchema` field is covered by the checks between them. And two test
groups re-asserted what the tracked v1_metadata fixture pins case by case.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(promql): decode a raw frame from the frame alone

`unpack_raw_series` took the label mode, the ordering and the caller's binding
along with the stream, so a consumer that received a frame from a peer still
needed a local copy of the binding to read it — the thing the frame's own
description exists to avoid, and the reason the description was freezing a wire
format for nobody.

The stream is now the only input: the field schema, the label mode, the label
decode description and the producer ordering all come from the frame's schema,
and every frame is validated whether or not a caller asked for it. `expected`
stays as an optional check on the schema that is declared — field names and their
order, and no evaluation grid — and supplies nothing to the decoder, so a caller
cannot smuggle in a layout the frame does not declare.

Three call sites compared the caller's binding against the frame and all three
were load-bearing: a peer's response is not trusted to be the region that was
planned, and a native plan's child is an arbitrary execution plan. Those became
`ensure_frame_matches_label_source`, which states what it compares: the label
layout, not the region. Two regions with the same tag ids and names compare equal,
and the doc now says so instead of claiming to establish identity.

A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers
`label_decode_spec()` instead; every consumer that decides whether to decode reads
the description, so the placeholder never stands in for "these labels are
complete".

The merge contract is the description, not the binding: inputs are compared by
the layout they declare, and a binding that could not be expressed on the wire — a
`RegionMetadata` snapshot — no longer makes the merge stricter than the frames it
merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a
copy, because the caller's binding keeps its own order and the table scan compares
producer tag lists positionally, and readers compare by equality.

The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to
cover. Rather than mirror all 25 datatype variants into a type this module owns, a
frozen fixture records the text of every variant and an exhaustive match makes a
new variant fail to compile until it is recorded. The version policy is written
next to the constant: exact-text comparison across nodes, unknown versions
rejected, any same-version change breaking — with the planning-time fallback
stated as intended and marked not implemented.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(store-api): make every key-encoded raw stream declare its layout

A raw series stream whose chunks carry key-encoded labels had to be decoded
against a scan-bound `RawLabelSource`, and two consumers compensated for
producers that stayed silent by deriving a description from that binding:
`pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels`
did not compensate, so an unframed producer stream reached it with no
description and a chunk carrying no labels, and failed.

The stream itself is the authority for its chunks. A key-encoded producer
knows its own region metadata and can state the layout it writes, exactly as a
frame states the layout it carries, so the split contract — "a producer
declares its identity, a frame declares its layout" — is replaced by one rule:
a stream whose chunks carry key-encoded labels declares the description, and a
stream whose chunks carry complete labels answers `None`.

mito2's `RawStream` caches the derivation at construction, so the declaration
exists before the first poll and for an empty stream. The trait documentation
states the rule and no longer exempts unframed producers. Both fallbacks are
gone: `pack_raw_series` reads the stream's declaration only, and rejects a
key-encoded stream that declares none rather than packing it as an
explicit-label frame, which would describe chunks carrying no labels as if they
carried all of them. The merge reads the declaration only.

Test fixtures that model key-encoded producers now declare their layout too;
fixtures that carry complete labels keep answering `None`. The new mito2 test
decodes a real produced key through the declared description for both dense and
sparse encodings, and checks the declaration is present before polling even for
an empty stream.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(common-query): let the declaration decide the frame mode alone

`pack_raw_series` rejected a key-encoded stream that declared no description
outright, so a stream that stays silent could not be packed at all. That check
was redundant: the declaration already decides the frame's label mode, and the
first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor`
with "raw chunk label mode does not match frame mode", so no such stream can
pack data either way.

What the check added was an early error for a producer that already breaks the
contract, in exchange for reading the stream's scan binding here — in the one
function whose rule is that the declaration is the only decode input. Drop it,
so pack keeps deciding the mode from the declaration and nothing else.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(store-api): assert the fixture's declared label decode description

The mock prepared source derives a `LabelDecodeSpec` from its label source and
reports it through `RawSeriesStream::label_decode_spec`, but no assertion read
it, so the derivation could drift from the chunks the fixture actually builds.
Check it against the description the fixture's own key columns imply: an
explicit source declares nothing, a primary-key source declares the metric, the
dense encoding, the one key column and its tag.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): let the frame's declaration be the only label authority

Two things still treated a caller-supplied binding as authoritative after the
declaration became the decode input:

- A merged stream forwarded its first input's scan-bound identity, though a
  merge is not a region scan and decodes from the declaration alone. It now
  reports no bound identity.
- `KeyColumn.tag` repeated an invariant every key column is a tag. The check
  moves to the one place a decoder is bound, so a snapshot built without
  validation is still rejected instead of silently decoded.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-21 19:11:24 +08:00
discord9 82c7d145b7 feat(datatypes): support sparse evaluation-grid axes (#9762)
* feat(datatypes): support sparse evaluation-grid axes

A series that holds a value at only some grid positions had no way to say so:
its time axis was one contiguous range, and an explicit timestamp list is not
the same thing -- it loses the evaluation grid the series is validated
against, and cannot answer which grid position a row belongs to.

`TimeAxis` already owns positions, length, timestamp access and slicing, and
already has a chunked form that keeps several Arrow buffers unflattened. This
gives grids the same treatment. `GridSpan` is one run of absolute positions;
`GridRanges` is an ordered, non-overlapping list of spans on one shared
`EvalGrid`; `TimeAxis::GridRanges` carries it beside the contiguous
`TimeAxis::Grid(GridRange)` fast path, which keeps its contract including the
zero-copy `timestamps()` slice.

The coordinates are fixed by the axis and documented on it. `len()` counts
carried rows, not the distance from the first position to the last;
`timestamp`, `slice` and `row_of` speak compact rows; `positions()` walks
`(absolute position, compact row)` pairs in one pass with one cursor per span;
`intersect_runs` merges two run lists without forming their product;
`union_positions` merges several axes. Spans are validated at construction --
ordered, non-overlapping, inside the grid, arithmetic checked -- and an axis
with no rows still owns its grid, so nothing unwraps a first span to find it.
`row_of` is a binary search and `slice` rebuilds through the checked
constructor. `Samples` gains a forward `RowCursor` so a walk reads rows in
order without scanning chunk descriptors per row.

`Series::try_new` accepts the new form under the same rule as the contiguous
one: its grid must match `SeriesSchema::eval_grid`, and an explicit axis still
may not carry an evaluation grid.

Nothing produces a fragmented axis yet. The type and its invariants land
first, and every existing reader keeps its behaviour: the places that require
a contiguous axis already return an error for anything else, so they reject
the new form rather than misreading it. Wiring the readers changes what those
places must do with their results, which is why it follows separately.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datatypes): check the shared grid, keep one constructor, and stop a cursor from poisoning itself

`intersect_runs` and `union_positions` walk two axes together but never said
they must be addressed on one grid; on an explicit axis every position is 0, so
the pairs were silent nonsense. Both now state the precondition and assert it in
debug builds.

`GridRanges::try_new` accepted a single span, which is not normalized into the
one run a `TimeAxis` fast path expects, so a public caller could build a range
that reports `is_contiguous()` and still has no grid range. It is now
`pub(crate)`; `TimeAxis::from_grid_spans` stays the public way in and keeps
normalizing a single span.

`Samples::RowCursor::at` drained its iterator on an out-of-range row, so one bad
call made every later call answer `None`. It now bound-checks first and leaves
the cursor where it was.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-20 15:53:33 +08:00
discord9 ba88f817cd fix(datatypes): keep frame flush guard clippy clean
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-16 17:44:58 +08:00
discord9 3406a53345 perf(datatypes): encode bounded multirow series frames
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-16 16:32:50 +08:00
discord9 2c04075baa chore: synchronize shared series contract parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

# Conflicts:
#	src/datatypes/src/series/raw.rs
#	src/store-api/src/region_engine.rs
#	src/store-api/src/region_engine/prepared_series_tests.rs
2026-10-16 16:04:07 +08:00
discord9 650ee58c22 feat(datatypes): share series ordering and expose source label binding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-16 15:45:42 +08:00
discord9 1284c043e5 feat(storage): separate prepared series selection from raw readers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-16 15:00:20 +08:00
discord9 d639f1569e feat(storage): separate prepared series selection from raw readers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-16 15:00:04 +08:00
discord9 35ba224c18 chore: synchronize reviewed raw time axis contract
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 21:17:46 +08:00
discord9 5fa4fed582 fix(datatypes): share raw axis validation and expose explicit buffer
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 20:53:35 +08:00
discord9 d4a3f67f65 fix(datatypes): preserve series frame field names
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 20:37:16 +08:00
discord9 489c9453cd chore: synchronize series frame stack base
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 18:22:41 +08:00
discord9 380fde19df chore: synchronize merged histogram foundations
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 18:21:36 +08:00
discord9 ad1d1ee0fb refactor(datatypes): share series time validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 18:15:01 +08:00
discord9 44b3022a73 chore: synchronize merged series foundation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 18:08:39 +08:00
discord9 bdc12034e5 fix(datatypes): address series foundation review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 17:56:57 +08:00
discord9 3717cc2e2a fix(datatypes): retain histogram table views across frame densification
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 16:31:56 +08:00
discord9 cdca81ed9e chore: sync reviewed series foundation fixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 15:37:25 +08:00
discord9 69c73740ec feat(datatypes): add raw series frame codec
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 15:35:59 +08:00
discord9 4f0cc3244a chore: sync histogram-only densification fix
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 14:47:37 +08:00
discord9 cb0a75f2c0 fix(datatypes): skip float buffers for histogram-only densification
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 14:45:36 +08:00
discord9 bb84af91a8 chore: sync reviewed series test cleanup
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 14:34:16 +08:00
discord9 1bfef429e7 chore: sync reviewed foundation test cleanup
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 14:30:55 +08:00
discord9 10dd49564a test(series): assert densification values without table layout
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 14:29:58 +08:00
discord9 78e94ced92 test(series): clarify histogram nullable-item schema coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 14:29:45 +08:00
discord9 8d0278876d Merge commit '971e1893973233332b51201c67bfcdab8b556dea' into feat/narthex-raw-contract
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 13:38:50 +08:00
discord9 4ca29e88b1 fix(datatypes): retain shared histogram tables when densifying
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 13:36:48 +08:00
discord9 a8bb74cbf9 feat(storage): define raw series and optional scanner contract
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 13:32:32 +08:00
discord9 7e5acfb9a9 fix(datatypes): size dense sample buffers to the local result
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:51:54 +08:00
discord9 d632c062f4 Merge branch 'feat/narthex-series-foundation' into feat/narthex-series-samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:47:45 +08:00
discord9 bee536ff27 feat(datatypes): add shared native series samples and fields
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:47:45 +08:00
discord9 a6cdbf966b fix(datatypes): avoid oversized default histogram buffers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:41:47 +08:00
discord9 e8f76480d8 test(datatypes): remove duplicate absent-label lookup assertion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:22:56 +08:00
discord9 ab1ee752d6 refactor(datatypes): use standard series test module layout
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:19:28 +08:00
discord9 fbf5f0c544 feat(datatypes): add Narthex shared series identity and time foundations
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:07:03 +08:00
dennis zhuang 2a70194930 feat: add per-column COMPRESSION option for SST columns (#9426)
* feat: add per-column COMPRESSION option for SST columns

CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N,
page_size = '<size>') on a column. The options are stored in the column
metadata, carried through gRPC column options, shown by SHOW CREATE
TABLE, and applied by the mito SST writer and bulk part encoder as
parquet column properties. Unset fields keep the SST defaults. Nested
types are rejected.

A page size below the dictionary page limit also lowers the column's
dictionary page limit: parquet sizes write batches by the dictionary
budget and keeps that size after falling back to plain encoding, which
would otherwise ignore the smaller data page limit.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: validate column compression options before persistence

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: reject compression overrides on metric logical tables

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-13 08:03:49 +00:00
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
dennis zhuang b9f991502c refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 05:38:54 +00:00
fys 5a78e07be6 fix(json2): restrict JSON2 type hints (#9316)
* fix(sql): restrict JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): allow equivalent 64-bit aliases in JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): support UInt64 conversion and preserve JSON2 hint aliases

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(sql): remove redundant JSON2 type hint normalization

Signed-off-by: fys <fengys1996@gmail.com>

* fix(datatypes): restrict JSON2 hint types in JsonSettings::try_new

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-23 09:19:18 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
Ning Sun 0f118bc5ba fix: serialize struct to json in postgres (#9170)
* feat: serialize struct to json in postgres

* fix: support view scalars and preserve null structs in scalar-to-value conversion

Address PR review:
- Utf8View/BinaryView ScalarValues now convert like their non-view forms
  instead of failing row extraction for struct columns
- a null struct scalar converts to Value::Null so a null struct inside a
  list stays null in the serialized JSON

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: return errors instead of panics for unsupported arrow field types

Struct-typed query results with arrow field types greptimedb cannot
represent (e.g. Decimal256) used to panic during schema conversion and
row extraction, dropping the client connection. They now surface as
query errors:

- ConcreteDataType::try_from builds struct types fallibly via the new
  StructType::try_from_arrow_fields
- Value::try_from(ScalarValue::Struct) uses the same fallible path
- new try_value_from_array converts an arrow element to Value with
  error propagation, used by the postgres struct encoding

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-23 01:55:46 +00:00
fys 045441e3cc feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-22 12:56:26 +00:00
fys 5b631fd8eb feat(json2)!: remove nullable and default type hint options (#9213)
feat(json2): remove nullable and default type hint options
2026-09-18 07:08:35 +00:00
LFC 697cc5fa29 fix(json): fix JSONPath panic with jsonb 0.5.6 (follow-up to #9192) (#9228)
* fix(json): upgrade jsonb to fix unterminated JSONPath panic

Signed-off-by: luofucong <luofc@foxmail.com>

* fix(json): align integer extraction with JSON2 conversions

Signed-off-by: luofucong <luofc@foxmail.com>

* style: format jsonb dependency declaration

Signed-off-by: luofucong <luofc@foxmail.com>

* fix(otlp): report concrete unsupported JSONB type names

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-18 04:55:02 +00:00
LFC aacf04cf6e feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-17 10:08:45 +00:00
Weny Xu 809283e6b4 feat: add prepared batch write primitives (#9186)
* refactor: expose shared operator test fixtures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: convert prepared table rows to Arrow batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: write prepared table batches through bulk insert

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: define prepared batch submission interface

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: validate prepared batch values and clarify write contracts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reuse JSON2 array schema alignment for prepared writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: refresh compatibility window for v1.2.1

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-16 14:34:22 +00:00
Weny Xu e0915a8380 refactor: reuse common batching components in Prom ingestion (#9114)
* refactor: compose Prom ingestion with common batcher components

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: share bulk insert IPC encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: centralize batcher worker channel creation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: share timestamp extraction for batched writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: preserve sharded batcher worker lookup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker lookup and configuration validation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: validate batcher limits inside fallible constructors

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-16 09:21:49 +00:00
fys 1c01b541ec refactor(json2): JSON2 parquet projection and schema alignment (#9137)
* refactor(mito2): make JSON schema alignment targets explicit

Signed-off-by: fys <fengys1996@gmail.com>

* fix: cr

* fix: cargo fmt

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-16 04:44:32 +00:00
discord9andNing Sun 94d7e2c7fc feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: migrate DataFusion 55 APIs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve table function planning behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve PostgreSQL query compatibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve distributed execution plan behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover DataFusion 55 behavior regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update DataFusion 55 SQLness expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DataFusion 55 test API migration

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address DataFusion 55 CI regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address remaining DataFusion 55 regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: adapt latest base code to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: normalize environment-specific DataFusion 55 plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align final DataFusion 55 expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: isolate DataFusion 55 regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve empty result schema in timestamp widening

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve JSON source column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: use released DataFusion 55 integrations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: adapt latest execution plan mock to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: pin DataFusion recursive schema and date repairs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): align dictionary temporality match keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: retain Greptime DataFusion fork behaviors on version 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore ordinary function error expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh distributed count compatibility plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt last-row cast hint to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh instant last-row empty results for Arrow 59

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: simplify DataFusion expression visitor imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore sorting and PostgreSQL column-order assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): restore primitive numeric coercion signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(function): share geo integer signature types

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover timestamp widening overflow boundaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix decimal coercion regression imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): preserve scalar count_hash NULL state semantics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: simplify decimal clamp case type inference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: retain historical count_hash wrapper result

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restore timestamp widening equality and IN pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream aggregate dynamic filter correctness fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream null and predicate simplification fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore baseline JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore histogram JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh empty PromQL range result schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align native timestamp plan with DF55 decimal display

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh native timestamp SQLness results for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regenerate NULL sample empty result headers for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use DF55 child replacement API in timestamp regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: expose pushed scan dynamic filters to DF55 producers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: encode string-backed PostgreSQL OID aliases in binary results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify REGPROC binary and text over PostgreSQL protocol

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: register real PostgreSQL catalogs in server fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DF55 expression inventories for custom query plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: correct RangeSelect expression fixture and column identities

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* ci: wait for Kafka WAL helper deployment rollout

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update custom storage empty result headers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: require exact row counts in scan statistics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: suppress deprecated partition_statistics warning in test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-09-15 11:42:38 +00:00