mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
eeca48bc43c4f4752bbda45ca756a40ea8301fc5
1148
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
2b67d0d60c |
test(series): pin literal ordering declarations and decoder validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
62652ec62e |
fix(series): validate v2 ordering and repair scan binding fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
b20e22b426 |
refactor(series): unify raw and evaluated SeriesFrame codec
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
discord9
|
6eff9f5804 |
test(promql): verify scalar routes and four-partition broadcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
78be1b1138 |
test(promql): regenerate independent matcher and range results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
8dbf08d252 |
feat(promql): execute independent selector scans in fused binary trees
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
c3dd11bf48 |
test(promql): record shared-selector SQL regression results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
2c3dace6b2 |
test(promql): align frame ordering and SQL result sorting
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
4748081579 |
feat(promql): fan out a shared raw scan to bound binary inputs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 jeremyhi
|
a0a7ec02b5 |
ci: compare several images in one object store WAL S3 benchmark run (#9691)
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit
|
||
|
jeremyhi
|
b2b606115b |
ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark (#9632)
* ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark
Extend the object store WAL S3 benchmark beyond one load per configuration:
- Inputs for a full image reference, the configurations to run (R0 adds Raft
Engine with the standalone default sync_write = false), extra environment
for every server container, a restart after each load, and a JSON list of
named cell groups that override those inputs.
- Restart cells SIGKILL the loaded server, restart it on the same prefix and
data home, and time /health, the first correct count and the log store
recovery; min_objects fails a cell that left too few objects indexed.
- Crash groups write striped 1000-row batches from several writers, kill the
server at a random point, restart it and check every acknowledged batch
against the table (durable: present exactly once; enqueued: no duplicate,
no partial batch, losses counted). A pinned variant keeps one row of a
second table unflushed and samples the indexed objects while it runs.
- Idle windows scrape the server before and after a quiet period.
- A load group may split the loaded table into equal event_time ranges, so
every batch writes to every region, and checks that each region took rows.
- Drop groups write a table, drop it, kill and restart the server, then
write and flush another table and wait for the dropped table's objects to
be collected.
- The summary adds WAL traffic, objects per second, mean object size,
entries and appends per object, S3 requests per operation, server and host
CPU, the loader's own request latency, rows per region, crash cycles, and
each restart's replay per region from the server log.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: report the object sequences a replay spans instead of objects replayed
The width of a region's replayed entry range counts every object sequence in it, including objects that hold no entry of the region and are skipped, so the summary labels it as a span.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a crash cycle that loses a batch an earlier restart recovered
The oracle accepted every missing acknowledged batch in the enqueued mode,
so a batch that one restart recovered and a later restart no longer found
passed. Every cycle keeps what its restart found, and a cycle now fails in
either mode unless every batch an earlier restart recovered is still
present, so only the current cycle's backlog that was not durable at its
kill may be missing.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: keep the pinned WAL pinned and label OpenDAL operations as such
The pin table inherited the engine's 10-minute auto flush, so a longer
pinned cycle could flush its row and release the WAL before the kill. It now
sets its own auto_flush_interval of one day, longer than any crash group can
run, and a pinned cycle fails unless its indexed objects never fall while
it runs.
The summary's request counts come from opendal_operation_duration_seconds,
which counts OpenDAL calls rather than HTTP requests (a paginated list or a
multipart write is one operation), so the fields and columns now say S3
operations.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check a pinned cycle's retention by the collected-object counter
Net growth of the indexed objects can hide collection, and short cycles take no periodic sample, so a pinned cycle now fails unless deleted_objects_total is the same in the scrape before its writers start and in the one before the kill, both of the same server process. The operation columns name OpenDAL.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let a pinned cell collect only the objects below its pin
The first write after a start lets the store collect the empty start object asynchronously, so a deletion after a cycle's baseline does not mean the pin let go. A pinned cell now scrapes the server before it writes the pin row and fails only once its servers together have collected more objects than were indexed then, counting the first server from that scrape and every restarted one from its start.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: test that a pinned cell's allowance spans its server restarts
Two servers that each collect fewer objects than lie below the pin but more together now fail the cell in the second cycle.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit
|
||
 Yingwen
|
6a40c0b811 |
feat(mito): implement columnar samples format (#9887)
* docs(mito): specify columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): implement columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): address columnar samples review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * style(mito): use crate-rooted samples imports Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 discord9
|
7309800aef |
feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema A raw series frame written with key-encoded labels carries the primary key as opaque bytes; only the binding held by the scanner says how to turn those bytes back into a metric name and labels. A consumer that receives the frame without that binding cannot decode it, so the schema now describes its own label layout. The description is compact JSON under `greptime.series.label_decode`, carrying a version, the metric name, the primary-key encoding, the ordered key columns with their physical types, and the logical tags to expose -- exactly what decoding reads. It is derived from the binding, so there is one writer, and a reader that does not implement the version it finds rejects the frame instead of guessing. `unpack_raw_series` keeps its binding argument and now requires it to agree with the description the schema declares, so the two cannot drift. `PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key layout, so the metadata-backed and description-backed paths share their validation instead of growing a second decoder. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datatypes): compare label descriptions by what their order means The frame's declared label-decode description was compared to the binding's with plain equality, but the description has one part whose order carries no meaning: the decoder sorts logical tags by name when it binds them, so the same tags listed in another order denote the same labels and were being refused. `same_label_layout` canonicalises exactly that part and leaves `primary_key` in place, because for a dense key the array position is the encoding position: the same columns in another order are a different key, not the same key reordered. Both fields now say which order they mean. Three checks go with it. The binding stays for a stated reason -- it establishes identity, the physical region key layout a decoded identity came from -- rather than because a count of call sites kept it. The post-pack schema comparison in the series scan could not fire: the leaf already re-derives that schema and compares it, and every `SeriesSchema` field is covered by the checks between them. And two test groups re-asserted what the tracked v1_metadata fixture pins case by case. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): decode a raw frame from the frame alone `unpack_raw_series` took the label mode, the ordering and the caller's binding along with the stream, so a consumer that received a frame from a peer still needed a local copy of the binding to read it — the thing the frame's own description exists to avoid, and the reason the description was freezing a wire format for nobody. The stream is now the only input: the field schema, the label mode, the label decode description and the producer ordering all come from the frame's schema, and every frame is validated whether or not a caller asked for it. `expected` stays as an optional check on the schema that is declared — field names and their order, and no evaluation grid — and supplies nothing to the decoder, so a caller cannot smuggle in a layout the frame does not declare. Three call sites compared the caller's binding against the frame and all three were load-bearing: a peer's response is not trusted to be the region that was planned, and a native plan's child is an arbitrary execution plan. Those became `ensure_frame_matches_label_source`, which states what it compares: the label layout, not the region. Two regions with the same tag ids and names compare equal, and the doc now says so instead of claiming to establish identity. A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers `label_decode_spec()` instead; every consumer that decides whether to decode reads the description, so the placeholder never stands in for "these labels are complete". The merge contract is the description, not the binding: inputs are compared by the layout they declare, and a binding that could not be expressed on the wire — a `RegionMetadata` snapshot — no longer makes the merge stricter than the frames it merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a copy, because the caller's binding keeps its own order and the table scan compares producer tag lists positionally, and readers compare by equality. The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to cover. Rather than mirror all 25 datatype variants into a type this module owns, a frozen fixture records the text of every variant and an exhaustive match makes a new variant fail to compile until it is recorded. The version policy is written next to the constant: exact-text comparison across nodes, unknown versions rejected, any same-version change breaking — with the planning-time fallback stated as intended and marked not implemented. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(store-api): make every key-encoded raw stream declare its layout A raw series stream whose chunks carry key-encoded labels had to be decoded against a scan-bound `RawLabelSource`, and two consumers compensated for producers that stayed silent by deriving a description from that binding: `pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels` did not compensate, so an unframed producer stream reached it with no description and a chunk carrying no labels, and failed. The stream itself is the authority for its chunks. A key-encoded producer knows its own region metadata and can state the layout it writes, exactly as a frame states the layout it carries, so the split contract — "a producer declares its identity, a frame declares its layout" — is replaced by one rule: a stream whose chunks carry key-encoded labels declares the description, and a stream whose chunks carry complete labels answers `None`. mito2's `RawStream` caches the derivation at construction, so the declaration exists before the first poll and for an empty stream. The trait documentation states the rule and no longer exempts unframed producers. Both fallbacks are gone: `pack_raw_series` reads the stream's declaration only, and rejects a key-encoded stream that declares none rather than packing it as an explicit-label frame, which would describe chunks carrying no labels as if they carried all of them. The merge reads the declaration only. Test fixtures that model key-encoded producers now declare their layout too; fixtures that carry complete labels keep answering `None`. The new mito2 test decodes a real produced key through the declared description for both dense and sparse encodings, and checks the declaration is present before polling even for an empty stream. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(common-query): let the declaration decide the frame mode alone `pack_raw_series` rejected a key-encoded stream that declared no description outright, so a stream that stays silent could not be packed at all. That check was redundant: the declaration already decides the frame's label mode, and the first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor` with "raw chunk label mode does not match frame mode", so no such stream can pack data either way. What the check added was an early error for a producer that already breaks the contract, in exchange for reading the stream's scan binding here — in the one function whose rule is that the declaration is the only decode input. Drop it, so pack keeps deciding the mode from the declaration and nothing else. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(store-api): assert the fixture's declared label decode description The mock prepared source derives a `LabelDecodeSpec` from its label source and reports it through `RawSeriesStream::label_decode_spec`, but no assertion read it, so the derivation could drift from the chunks the fixture actually builds. Check it against the description the fixture's own key columns imply: an explicit source declares nothing, a primary-key source declares the metric, the dense encoding, the one key column and its tag. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): let the frame's declaration be the only label authority Two things still treated a caller-supplied binding as authoritative after the declaration became the decode input: - A merged stream forwarded its first input's scan-bound identity, though a merge is not a region scan and decodes from the declaration alone. It now reports no bound identity. - `KeyColumn.tag` repeated an invariant every key column is a tag. The check moves to the one place a decoder is bound, so a snapshot built without validation is still rejected instead of silently decoded. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
c879564671 |
chore(common-query): drop the raw-series request extension transport (#9815)
* chore(query): drop the prepared raw-series RPC skeleton The raw-series preparation RPC was merged into `narthex` for a plan that never landed: no server or frontend dispatches `greptime.prepare_raw_series.v1`, and the three `Datanode` methods, `bind_raw_series_query` and the `QueryEngine` trio have no call sites and no implementors. `PreparedNamedRawSeries` had no user outside them. The wire envelope never shipped -- no tag contains the commit that added it -- so the compatibility fixtures go with it. The harness drives `wire/` fixtures from the Rust tests that own their context conversion rather than from SQLness discovery, and has no retirement mechanism to record instead. Left in place: `RawSeriesRequest` and its extension key, `raw_series_frame_ordering`, `unpack_raw_series` and `RawLabelSource`, all of which the surviving series-frame path still uses. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(common-query): drop the raw-series request extension transport The prepared raw-series RPC carried a `RawSeriesRequest` through a query-context extension, and with that RPC gone (`607de914464`) nothing in `src/` writes or reads the extension any more. What remains is the binding the table scan actually uses: `RawSeriesRequest`, `RawSeriesTag`, `bind`, `validate` and `same_encoding_identity`, unchanged. Deleted: the extension key, `take_raw_series_request`, and the two serialization methods that only existed to encode and decode that transport, plus the wire fixture whose only reader was the test for them. The fixture never shipped — the commit that introduced it is in no tag. The tests that covered `validate` are kept, with their rejections now asserted directly instead of through a JSON round-trip: an unsupported version, an empty field list, a duplicate field, and a non-field column id are all still refused. The binding test is untouched, including its order, identity and `Arc` assertions. `pub` items in a library crate never trip `dead_code`, so the compiler could not have found this; the greps above are the evidence. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1156cfbe0c |
chore(query): drop the prepared raw-series RPC skeleton (#9797)
The raw-series preparation RPC was merged into `narthex` for a plan that never landed: no server or frontend dispatches `greptime.prepare_raw_series.v1`, and the three `Datanode` methods, `bind_raw_series_query` and the `QueryEngine` trio have no call sites and no implementors. `PreparedNamedRawSeries` had no user outside them. The wire envelope never shipped -- no tag contains the commit that added it -- so the compatibility fixtures go with it. The harness drives `wire/` fixtures from the Rust tests that own their context conversion rather than from SQLness discovery, and has no retirement mechanism to record instead. Left in place: `RawSeriesRequest` and its extension key, `raw_series_frame_ordering`, `unpack_raw_series` and `RawLabelSource`, all of which the surviving series-frame path still uses. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Yingwen
|
8543cbf401 |
feat(mito): scaffold series format writers and artifact lifecycle (#9788)
* feat(mito): add pluggable series artifact reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): prepare independent series reads from SST artifacts Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): prepare mixed sidecar SST and memtable series reads Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): make prepared series sources replaceable Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): narrow series formats to artifact lifecycle integration Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito): populate artifact regression buckets above build threshold Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
3ef71b5702 |
feat(common-query): add raw series request binding and preflight types
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
88a1e684a0 |
feat(common-recordbatch): recognize SeriesScanExec in metrics and preserve typed errors (#9656)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Independent PR. No dependencies. ## What's changed and what's your intention? Recognize `SeriesScanExec` in query metrics and preserve typed errors through the record batch adapter: - `PlanMetrics`: recognizes `SeriesScanExec` nodes so region-scoped stats attribution works. - `RecordBatchStreamAdapter`: preserves the original typed error via `BoxedError` instead of wrapping into a generic execution error. This lets downstream recover the specific status code (e.g., `InvalidArguments` instead of `EngineExecuteQuery`). **2 files, +203/-9 production + 1 sqlness expected result.** ### Error code change The typed error preservation changes the error code for the `semantic_graph` upper-bound-only query from `3001(EngineExecuteQuery)` to `1004(InvalidArguments)`. The error message text is unchanged. The new code is correct: the original error was an argument validation error, not a generic execution failure. The sqlness expected result has been updated. ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-recordbatch` → **65/65 passed** ✅ - sqlness expected result updated for the typed error code change ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
55e59f6c91 |
fix: update semantic_graph expected error code for typed error preservation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
997cbf87f7 |
feat(common-query): declare raw series frame ordering in schema metadata
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 dennis zhuang
|
5c4d9eb4f8 |
fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE The bloom fulltext index stored tokens that did not follow matches_term's word boundaries (`_`, ASCII next to Han, Han substrings), so pruning dropped rows matches_term accepts. Tokens and query probes now share the character classes of matches_term, Han runs are indexed as bigrams, and LIKE patterns use the same probes. The bloom blob is written under a new key; blobs from older versions are no longer probed. Tantivy no longer serves matches_term. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: drop unused lazy_static from index and avoid typo-flagged test tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: link like_probes to tokenize instead of the downstream ScriptTokenizer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name probe extraction with verbs and document CharClass Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: rebuild legacy fulltext bloom indexes and probe lower(col) The manifest records the fulltext bloom version per column, so ADMIN BUILD_INDEX now rebuilds SSTs whose bloom index predates the current tokens, and ssts_index_meta shows the version. Case-insensitive bloom indexes also store the tokens of the whole lowercased text for rows where it can differ (U+0130, U+212A, sigma), so matches_term(lower(col), ..) and lower(col) LIKE .. use the index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover legacy bloom index rewrite by compaction in the compat case Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: log outdated fulltext bloom indexes in the consistency check Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: explain the unrecorded fulltext bloom version Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
00aa65de7a |
fix(query): guard deep expressions during query planning (#9564)
* fix(query): avoid speculative clones in constant normalization Signed-off-by: evenyag <realevenyag@gmail.com> * fix(query): reject deeply nested expressions before optimization Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
  NightG1owZandNightG1owZ
|
caefcfd012 |
test: add inverted index compatibility case (#9519)
* test: add inverted index compatibility case (#5188) Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(compat): assert the newly indexed column stays queryable (#9535) Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(compat): narrow to_range to versions carrying #9545 The filtered read added in the previous commit only returns rows once #9545 is present. That fix is on main only (no backport-* labels), so it ships in the v1.4.0 line. Narrow to_range to >=v1.4.0-alpha.0 so the documented downgrade invocation (--to-version v1.1.4) is not broken. Also add the missing trailing newline to the three case files. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> --------- Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> |
||
|
dennis zhuang
|
2a70194930 |
feat: add per-column COMPRESSION option for SST columns (#9426)
* feat: add per-column COMPRESSION option for SST columns CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N, page_size = '<size>') on a column. The options are stored in the column metadata, carried through gRPC column options, shown by SHOW CREATE TABLE, and applied by the mito SST writer and bulk part encoder as parquet column properties. Unset fields keep the SST defaults. Nested types are rejected. A page size below the dictionary page limit also lowers the column's dictionary page limit: parquet sizes write batches by the dictionary budget and keeps that size after falling back to plain encoding, which would otherwise ignore the smaller data page limit. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: validate column compression options before persistence Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject compression overrides on metric logical tables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Weny Xu
|
287ced2de2 |
fix(mito2): ignore missing inverted indexes in old SSTs (#9545)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
c1a0ca9595 |
ci: benchmark the object store WAL against S3 from an EC2 runner (#9536)
* ci: benchmark the object store WAL against S3 from an EC2 runner Add a workflow_dispatch workflow that loads the o11ybench agent observability dataset into a fresh GreptimeDB container per cell and compares three WAL configurations that differ only in [wal]: Raft Engine with sync_write on the instance store, and the object store WAL in durable and enqueued mode, with the data files on S3 in the runner's region for all three. Rounds alternate the configuration order. A summary reports rows per second, D/R and E/R as the geometric mean over rounds, append acknowledgement p50 and p99, and the object store WAL counters. A smoke mode runs the same chain on ubuntu-latest against the MinIO fixture. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: scrape WAL metrics on failed loads and keep logic out of the benchmark YAML Scrape the object store WAL metrics before stopping a cell's server whether the load succeeded or failed, so failed cells keep their stall, poison and create failure counters. Build the summary from the plan so that cells which never ran are listed and leave their ratios undefined. Move budget, region, runtime image, manifest and credential scan logic from the workflow into the helper scripts, and drive the cell lifecycle in the tests through fake docker and curl binaries. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: drop redundant required flags from benchmark inputs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: fail a benchmark cell whose metrics scrape fails and zero untouched WAL counters A load that succeeds without its metrics scrape now fails the cell with exit code 4 and records metrics_scraped in cell.json. The summary reports an event counter that never registered as zero when the cell is an object store WAL cell whose scrape succeeded and created objects, and as unknown otherwise. Planned cells that never ran are read through the same path as recorded ones. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: register the S3 benchmark workflow for dispatch from its branch A workflow_dispatch workflow can only be dispatched once GitHub has registered it, which happens on the default branch or when one of its triggers fires. A push trigger limited to this branch registers it; validate runs only for workflow_dispatch, so a push run skips every job. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: delete the benchmark S3 prefix one object at a time boto3 now sends a CRC checksum instead of Content-MD5 with DeleteObjects, which the MinIO fixture of the smoke run rejects. DeleteObject needs neither and the prefix of one run holds at most a few thousand objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: default the S3 benchmark to c6id.2xlarge c7id is not offered in the runner region ap-southeast-1, where DescribeInstanceTypes rejects it. c6id is the compute optimized x86 family with a local NVMe instance store that the region offers, with the same 8 vCPU and 16 GiB at 2xlarge. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: drop the registration trigger of the S3 benchmark workflow The workflow is registered now, so dispatches no longer need a push trigger on its branch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: fail an S3 benchmark run on EC2 without a local instance store Without an instance store the Raft Engine WAL of the R cells silently stayed on the root EBS volume, so the run measured a different baseline. Only a smoke run may keep it on the root volume now. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: let the instance store check fail its workflow step The step piped wal-disk into tee without pipefail, so a rejected host still passed. wal-disk now writes its record itself and the step runs it alone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 shuiyisong
|
2df33380a7 |
feat(catalog): declare information-schema table scopes (#9504)
* feat(catalog): declare information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: rename cluster to global Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: update sqlness result Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(catalog): require information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
f3a0841f4f |
fix(promql): follow IEEE 754 float semantics in comparisons and aggregations (#9479)
* fix(promql): follow IEEE 754 float semantics in operators and aggregations - Comparisons: Arrow orders floats totally (NaN == NaN, NaN > +Inf); PromQL comparisons with NaN are false except !=. - min/max: skip NaN unless the whole group is NaN. DataFusion's grouped min/max let NaN replace and be replaced by neighbours, so the result depended on input order. - topk/bottomk rank NaN last. - round() rounds ties up and returns NaN for a zero step, like Prometheus. - sqrt() of a negative number is NaN instead of an error. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): round() without a step rounds to integers The planner passed a zero step for single-argument round(), which now yields NaN like Prometheus; pass 1 instead. Found by the compatibility harness. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): apply the sqrt guard to builtin functions and register min/max for plan decoding sqrt resolves through the session's builtin functions, not the UDF branch. Distributed plans are decoded with the serializer's session, which did not know prom_min/prom_max. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover NaN and rounding semantics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep the nested count rewrite for PromQL min/max count(min(x) by (l)) was rewritten by count_nest_aggr only for the builtin min/max names. prom_min/prom_max return NULL for the same groups, so the rewrite stays valid. Update plan snapshots for the IEEE comparison, sqrt, topk and min/max changes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): register prom_min/prom_max in the function registry Distributed plans split an aggregation into state and merge steps only for functions whose state function exists in FUNCTION_REGISTRY; prom_min and prom_max lived in the promql crate, so min/max stopped being pushed down to datanodes. Move them to common-function and register them before the state functions are derived. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: update sqlness results for PromQL float semantics Plans and column names now show prom_min/prom_max, the NaN guards of comparisons and sqrt, and the NaN sort key of topk. round() results follow Prometheus' formula. Distributed min/max keep the state/merge split. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep column names and NULLs in IEEE comparisons and sqrt Rewriting comparisons as `a op b AND NOT (isnan(a) OR isnan(b))` changed TQL output column names and turned NULL operands into true for `!=` and false for the bool forms. Implement comparisons and sqrt as PromQL UDFs that propagate NULL and report the column name of the expression they replace; prom_min/prom_max report min/max column names the same way. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: restore column names in sqlness results for PromQL float semantics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): share the prom_min/prom_max groups accumulator with tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep distinct names for prom_min/prom_max Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
845fc0af9c |
fix(promql): default a stepless subquery to a one-minute step (#9481)
* fix(promql): default a stepless subquery to a one-minute step `x[5m:]` evaluated the inner expression at the query step, which is 1s for instant queries. Prometheus uses the evaluation interval, one minute by default. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover the default subquery step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover the default subquery step in a range query Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
28ec3ff5aa |
fix(table): require complete primary keys for hash partitioning (#9507)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
32e8467cda |
fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): use current DataFusion plan downcasts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): match null rendering in join regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh union plan for unknown scan partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh shared plans after distribution rule removal Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover semi joins with remote dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen partitioned merge scan regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove unused merge scan test helpers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): measure merge scan hash repartition cost Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt merge scan deserialization to two-arg constructor Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): adapt nested merge scan integration constructor Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 fys
|
9a57e1195d |
feat(json2): json_get respect type hints (#9349)
Signed-off-by: fys <fengys1996@gmail.com> |
||
|
dennis zhuang
|
b913be2af7 |
fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries An instant query of a range selector or subquery (e.g. `up[5m]`) returned the internal range arrays, which panicked when the frontend decoded them as dictionaries. Plan such queries as the selected samples with their own timestamps, and reject range-vector expressions in range queries like Prometheus does. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): keep encode_substrait on a valid range query A range query of a range-vector expression now fails like in Prometheus. Wrap the subquery in max_over_time so the case still encodes both RangeManipulate nodes through substrait. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): plan wrapped range-vector roots and reject a zero subquery step TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the range-vector path did not unwrap. A zero subquery step divided by zero. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover wrapped range-vector roots and a zero subquery step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): default subquery step and reject range vectors in range queries A root subquery without a step uses the one-minute default instead of the query step. The HTTP range query endpoint rejects range-vector expressions even when start equals end. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover root subquery default step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): align root subqueries with subquery_child_window Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): return millisecond timestamps for range-vector results Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): keep the time index timezone and name its table in errors Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
178a0021b8 |
fix: unify unknown flow option rejection across OSS and enterprise builds (#9319)
* fix(operator): unify unknown flow option rejection across OSS and enterprise builds Remove the enterprise-gated passthrough that let unknown flow options through SQL-layer validation. Both OSS and enterprise binaries now reject unknown flow options with a single 1004(InvalidArguments) error at the SQL layer, producing identical sqlness output. The metasrv-side validate_flow_options remains as a defensive check. Signed-off-by: discord9 <discord9@163.com> * fix(flow): use DISTINCT output columns as sink key for streaming flows When a streaming flow's source table has no primary key, sink key inference produced an empty key. For SELECT DISTINCT v, the DISTINCT establishes v as the output key, but the inference only looked at source primary keys. Combined with the zero-valued placeholder timestamp, all output rows shared the same storage identity and were deduplicated into a single row. Use all non-time-index output columns as the sink key when the plan is a DISTINCT and no source primary key was inherited. Signed-off-by: discord9 <discord9@163.com> * fix(flow): preserve extension options and complete DISTINCT sink identity Keep enterprise CREATE FLOW options available to the existing handler and report unknown ordinary options as InvalidArguments with the OSS SQL error text. Infer DISTINCT sink keys from every selected non-time-index output column, including when source key columns survive. Wait for asynchronous mirrors in show_create_flow and exercise 1000 distinct sink rows in the SQL regression. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): align meta unknown option status assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): use valid aliases in DISTINCT sink regression SQL Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): record complete DISTINCT sink SQL results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
a8e293fd79 |
fix(promql): accept evaluation times before the Unix epoch (#9480)
* fix(promql): accept evaluation times before the Unix epoch Prometheus accepts negative timestamps in query parameters; GreptimeDB rejected them while parsing and would panic converting them in the planner. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: update flow_tql error for a negative TQL end time A negative end now parses and fails the end >= start check instead of the timestamp conversion. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): reject evaluation times outside the millisecond range The signed millisecond conversion cast u128 to i64 unchecked, so a far-past time wrapped into the future. Reuse the checked conversion from label_values and return an error instead. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: drop an unused import Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): round query times to milliseconds like Prometheus Numeric times round the fraction of a second to milliseconds, and pre-epoch times floor sub-millisecond parts, matching Prometheus' parseTime and timestamp.FromTime. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): round the float fraction before converting to nanoseconds Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: skip the overflow case where SystemTime cannot hold it Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Weny Xu
|
6e1a137411 |
ci: enhance benchmark workflows with version comparisons and diagnostics (#9383)
* ci: update Agent and Traces benchmark runtime Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: use unrestricted benchmark runtimes by default Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: retry benchmark image pulls before generating datasets Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: increase tracesbench defaults to 8 CPU and 32 GiB Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: configure tracesbench query timeout and update runtime Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: update tracesbench runtime and configure import scan workers Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: use shared parallel tracesbench import preparation Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: update tracesbench runtime to preserve Tempo trace requests Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: summarize tracesbench query results and lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: show benchmark resource usage and update parallel loader runtime Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): resolve shared summary module in existing test harness Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: select tracesbench engines and GreptimeDB image version Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: update benchmark runtime with parallel Agent preparation Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: select benchmark runtime registry by cloud provider Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: retry long-range benchmark image pulls Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: compare baseline and candidate GreptimeDB benchmarks Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: pin published dual-version benchmark runtime Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: prepare agent benchmark indexes with updated runtime Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: validate agent admission and render traces version comparison Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: use runtime with traces index preparation Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: use consolidated runtime for benchmark version reports Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add license header to benchmark comparison tests Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): read legacy Tempo ingestion metrics in traces summaries Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: require tracesbench query plan artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: use optimized tracesbench report runtime Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: update tracesbench runtime to merged report validation Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
b3c756a7dd |
perf(promql): reduce timestamp payload allocations and use narrow cursor scans (#9417)
* perf(promql): zero-copy timestamp payload for millisecond zero-offset ranges RangeManipulate rebuilt the timestamp range payload per sample through wide-precision i128 conversion even for millisecond input with zero offset, where the input is already millisecond and nothing changes. That turned a previously zero-copy path into a per-sample loop plus a Vec allocation, regressing range-function queries. Reuse the input timestamp array directly when the time index is already millisecond and the offset is zero. Other units and non-zero offsets keep the wide-precision conversion (required so ns/us data near the native type limits stays correct under large offsets). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> * perf(promql): vectorize timestamp payload with plain i64 RangeManipulate rebuilt the timestamp range payload per sample through wide-precision i128 conversion, turning a hot path into a scalar loop with a Vec allocation. Realistic timestamps and offsets stay far inside i64, so the wide path is unnecessary. - millisecond + zero offset reuses the input array with zero copy; - other units / offsets shift in one vectorized i64 pass (no i128, no per-sample Option, no overflow branch). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> * perf(promql): check timestamp payload overflow via arrow try_unary Replace the silent wrapping shift with try_unary + checked_mul/checked_add. checked arithmetic lowers to a per-lane overflow flag so the single pass still vectorizes over the dense i64 buffer; any overflow short-circuits to an error instead of silently wrapping to a wrong timestamp. No i128, no per-sample Option. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> * perf(promql): vectorize cursor scans with i64 fast path RangeManipulate::calculate_range and InstantManipulate::manipulate compared every sample through i128 wide-precision arithmetic. For realistic timestamps and offsets the shifted values stay far inside i64, so the wide path was pure overhead. Split both scans into an i64 fast path (checked arithmetic precomputed once per batch) and an i128 exact fallback. Any input that would overflow i64 falls back to the i128 scan, keeping window membership semantics identical. Verified: 47/47 range_manipulate + instant_manipulate tests, 275/275 promql crate tests, 1612 randomized cases byte-identical to i128 scan. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> * refactor(promql): generalize cursor scans and share endpoint precheck Address review comments on PR #9417: - Generalize scan_ranges/scan_instants over Nanoseconds (i64/i128), eliminating the duplicated scan bodies. - Extract shared narrow_scan precheck used by both RangeManipulate and InstantManipulate, with identical checked endpoint logic. - Trim doc comments to one line; fix factual errors about fallback trigger (2262 for ms/s, not native type limits). Semantics unchanged: same cursor algorithm, same overflow fallback, same i64-vs-i128 selection, same stale/null/tie handling. Verified: 47/47 range_manipulate + instant_manipulate tests, clippy clean. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> * perf(promql): vectorize payload with try_unary and add scan carrier bench Replace the per-sample i128 payload loop with a single arrow try_unary pass: i128 intermediate holds any native timestamp shift, only the final millisecond result must fit i64. This is semantically equivalent to the original i128 implementation (no silent wrap, no per-sample branching). Add cursor_scan_carriers A/B benchmark: i64 fast path vs i128 exact fallback on four workloads. Add five deterministic differential tests pinning scan_ranges::<i64> and scan_ranges::<i128> to be pointwise equivalent on boundary shapes (sub-ms samples, fallback triggers, nanosecond extremes, seeded precision matrix). Bench (N=4096, 15s cadence): - payload wall_time 20x15s: 74.94us -> 48.98us (-35%) - payload wall_time 240x15s: 102.44us - scan i64 fast path: 10.97us vs i128 19.84us (1.81x) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): restore zero-copy payload for ms+offset=0 and fix comment Restore the zero-copy fast path for millisecond timestamps with zero offset, guarded by the input column's exact data type (Timestamp (Millisecond, None)) rather than self.time_unit. The old time_unit guard would accept a tz-carrying millisecond column and produce a dictionary value type disagreeing with the output schema; the datatype guard is the correct check. Also fix the comment: arrow's try_unary is a fallible kernel that LLVM cannot auto-vectorize; the win over the hand-written loop comes from eliminating the per-sample Option/Vec and branches, not SIMD. Add a regression test with a timezone-carrying millisecond index to prove it falls back to the try_unary path. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): replace try_unary with unary + Cell overflow flag Arrow's try_unary is a fallible kernel that LLVM cannot auto-vectorize. Replace it with the infallible unary kernel: the closure is a plain i64 -> i64 mapping that sets a Cell<bool> on overflow instead of returning Err per sample. The flag is checked once after the loop. This removes the per-lane Result construction and lets LLVM vectorize the shift+truncate sequence. Bench (N=4096, 15s cadence): - window4x15s: 42.12us -> 28.77us (-32%) - window20x15s: 48.98us -> 27.68us (-43%) - window240x15s: 102.44us -> 27.61us (-73%) - regular_sparse: 90.04us -> 29.20us (-68%) All cases now run at ~27-29us regardless of window size; the zero-copy path dominates for ms+offset=0. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): dispatch payload conversion on null_count; drop wall-time bench Null-free batches (dominant PromQL case) take arrow's vectorizable unary kernel with a Cell<bool> overflow flag; nullable batches take try_unary so a NULL slot's arbitrary underlying value cannot raise a false batch-wide overflow error. Add a regression test exercising both paths. Remove bench_range_manipulate_wall_time (all-ms zero-offset cases only exercised the zero-copy path, measuring nothing about the conversion) and its orphaned helpers; restore the cursor-scan bench helpers it shared. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): keep nullable timestamp cursor scans on i128 NULL slots may contain unordered arbitrary values, so the endpoint-only narrow-scan proof cannot bound their interior timestamp arithmetic. Both range and instant scans now attempt i64 only for null-free timestamp arrays and otherwise retain the existing exact i128 path. Add focused interior-NULL regressions through both active entrypoints, checking concrete selections and output against the existing wide scan. The null-free fast path and timestamp payload conversion are unchanged. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> |
||
 Niladri Adhikary
|
0dde3a7b70 |
fix(promql): return no point for zero-interval irate/idelta (#9455)
fix(promql): irate/idelta return no point for a zero-ms sample gap Signed-off-by: Niladri Adhikary <niladrix719@gmail.com> |
||
 fengyue-xve
|
764ab8bc1a |
test(promql): cover label_replace normalization with distinct series (#9485)
The duplicate-label-set rejection must not fire when a label rewrite keeps every series distinct through another label. Normalizing mixed histogram boundary representations (`10.0` -> `10`) while `instance` still differs is the false positive reported in #9444. The check already accounts for it since it became a per-timestamp row count instead of a planning-time rejection, but no fixture covered the positive case. Add one, together with the aggregation variant that produces a genuine duplicate and must still fail. Signed-off-by: fengyue-xve <1759519137@qq.com> |
||
|
dennis zhuang
|
6da5352c53 |
fix(promql): keep vector labels when a computed scalar is on the right (#9478)
* fix(promql): keep vector labels when a computed scalar is on the right `metric + time()` or `metric * scalar(other)` planned the join with the scalar side's context, which has no tags, so every series collapsed into one unlabeled series. Use the vector side's context when the right operand is scalar-typed. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): update results for labels kept with a computed scalar on the right v * pi() and v + time() now keep the vector's labels and time index name, as in Prometheus. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): qualify columns of an unnamed operand in vector joins A comparison filter's projection has no table name, so after the join its time index resolved unqualified and was ambiguous when the other operand had the same column. Also keep the empty-metric branch unchanged for scalar left operands. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover comparison filters joined with a scalar Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
0d2cb1aa7e |
fix(promql): align subqueries to the absolute evaluation grid (#9400)
* fix(promql): align subqueries to the absolute evaluation grid Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): streamline subquery comments Remove redundant arithmetic narration and restated test comments. Keep grid, fallback, validation and overflow contracts, and preserve all regression tests and assertions. Regenerate SQLness output only for comment echoes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
fys
|
fe8a647bc8 |
fix(json2): support settings in ALTER TABLE ADD COLUMN (#9472)
* fix(json2): support settings in ALTER TABLE ADD COLUMN Signed-off-by: fys <fengys1996@gmail.com> * refactor(sql): reuse Column for ADD COLUMN definitions Signed-off-by: fys <fengys1996@gmail.com> * chore: better method name * refactor(sql): share column parsing between CREATE TABLE and ADD COLUMN Signed-off-by: fys <fengys1996@gmail.com> * chore: simplify tests Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): accept trailing commas in ALTER TABLE ADD COLUMN Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): preserve ALTER TABLE ADD COLUMN compatibility Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
fys
|
c46fbde9ac |
fix(json2): preserve metadata when adding columns via alter table (#8696)
* fix(json2): support adding columns via alter table Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): scope missing-column null handling to JSON2 --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
dennis zhuang
|
526dd48bda |
fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: verify compound filter semantics and last-row selection Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: consolidate label filter regression coverage Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: separate filter evaluation and timestamp adaptation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: inline predicate and timestamp tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: keep direct IN predicates outside simple filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: coerce partition bounds before simple filtering Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover partition bound overflow after timestamp widening Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
b78dc4b33b |
test(sqlness): wait for mirrored inserts before flushing streaming flows (#9424)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
97913b4b3a |
fix: coalesce equal histogram quantile bucket boundaries (#9451)
* fix: coalesce equal histogram quantile bucket boundaries Merge numerically equivalent bucket boundaries before monotonicity repair, preserving the allocation-free path for unique boundaries. Share coalescing with histogram_fraction and cover duplicate boundaries with unit and SQL regressions. Fixes #9443 Signed-off-by: evenyag <realevenyag@gmail.com> * fix: keep duplicate infinity buckets in one histogram group Confirm classic histogram completion at a series/timestamp boundary or EOF instead of the first positive infinity. Retain lookahead across batches and reject incomplete optimistic groups so equivalent infinity labels reach numeric coalescing together. Add stream and SQL regressions for duplicate infinity labels, including established bucket counts and EOF. Correct older fixtures to use distinct keys for distinct histogram groups. Signed-off-by: evenyag <realevenyag@gmail.com> * fix: avoid buffering histogram groups with invalid trailing bounds Discover candidate bucket counts at the first group boundary and let optimistic validation fall back to safe mode for malformed bounds. Document discovery and cover trailing invalid and null labels with a focused streaming regression. Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
3a4c24a863 |
fix: temporarily disable v2 series scan by default (#9436)
* fix: disable v2 series scan by default temporarily Signed-off-by: evenyag <realevenyag@gmail.com> * test: explicitly enable v2 series scan in sqlness configs Signed-off-by: evenyag <realevenyag@gmail.com> * test: regenerate sqlness results for legacy series scan default Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
78a7b93292 |
fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics - timestamp() over a selector reports the selected sample's timestamp, without adding the offset. - timestamp() over any other expression reports the evaluation time instead of the input value. - label_join that overwrites an existing label rejects duplicate label sets at runtime. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): align label_replace and label_join edge cases with Prometheus - label_replace leaves non-matching series unchanged instead of copying the source value into the destination label. - label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join. - label_join with no source labels removes the destination label, and an empty source label name is rejected. - Empty label values produced by these functions are NULL, the same as an absent label. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): join absent labels as empty strings in label_join concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): skip rows without a sample in timestamp() timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
218000e21b |
fix(promql): resolve dotted column names as unqualified columns (#9391)
* fix(promql): resolve dotted column names as unqualified columns col(), From<&str>/From<String> for Column and string join keys go through Column::from_qualified_name, which splits `service.name` into relation `service` and column `name` and lowercases unquoted identifiers. Build PromQL column references with Column::from_name / ident() instead. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): resolve remote read matcher labels as unqualified columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover same-name columns differing in case and without() Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |