mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-29 15:06:59 +00:00
180e2716fc5fb3f192a6f6cd57eac308892ccdcc
990
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
984f80e277 |
fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit- or timezone-changing time casts A cast on the time index column keeps the windowed-sort optimization eligible only when it preserves the underlying timestamp values. Casts that change the time unit (e.g. ms -> s) or that add a timezone to a naive timestamp re-interpret the values, so the working ranges derived from the time index no longer describe the sort column and the scan fails with a working-range error or silently wrong order. Reject both cast shapes so such queries fall back to the normal SortExec path. The integration fixture uses two rows 1000 seconds apart so the cast output stays distinct and the ordering assertions are not vacuous; the descending arm gets its own expectation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit-changing timestamp wrappers to_timestamp_seconds/millis/... cast their argument to their own unit inside the function, so the hidden cast bypassed the explicit-cast guard and kept the windowed-sort optimization eligible on a sort column whose values no longer match the working ranges. The wrapper branch now applies the same unit- and value-preserving predicate as the cast branch, with the predicate extracted into preserves_timestamp_values and shared by both. The integration regression gains to_timestamp_seconds ASC/DESC arms and a to_timestamp_millis value-preserving control; unit tests cover unit-changing, tz-adding, and non-timestamp wrapper inputs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regen windowed sort sqlness after wrapper guard The windowed_sort_nanos and windowed_sort_advance cases sort by to_timestamp_nanos over a millisecond index and to_timestamp_millis over a nanosecond index. Both wrappers change the time unit inside the function, so the windowed-sort rule no longer applies and the plans render a plain SortExec. Query results are unchanged; only the EXPLAIN plan line differs. Regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): simplify windowed sort cast regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 dennis zhuang
|
2e84833ac8 |
feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): reject partition column type changes and quote rule identifiers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(table): keep time index widening for partition columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(partition): replace PartitionFunction with IcebergHash Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(partition): describe iceberg_hash in the function module Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: quote partition rule identifiers in SHOW CREATE TABLE expectations Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(partition): check hash rules in information_schema and repartition target columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): coerce hash partition rules against the evaluation schema Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep quotes in partition column names when building filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Chenglong Zheng
|
2117f8b059 |
feat: Support procedure_state() using the last procedure ID (#9876)
* feat: Support procedure_state() using the last procedure ID Signed-off-by: hahaha <a310608318@gmail.com> * feat: Fix and add tests according to review comments Signed-off-by: hahaha <a310608318@gmail.com> * fix: the zero-argument signature in ADMIN swallows explicit arguments Signed-off-by: hahaha <a310608318@gmail.com> * fix: fix the CI failures caused by SQL version issues Signed-off-by: hahaha <a310608318@gmail.com> * fix: apply the changes based on the review Signed-off-by: hahaha <a310608318@gmail.com> --------- Signed-off-by: hahaha <a310608318@gmail.com> |
||
 shuiyisong
|
5e9bb3c689 |
feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: align native histogram ingestion with remote read validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Yingwen
|
371c4e9004 |
feat: support query concurrency limits in standalone (#9958)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Ning Sun
|
80df1c397b |
feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support * feat: add postgres copy-in framework * feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher Follow-up to #9302: begin_copy_in now uses the batching-aware new_query_context, so copy-in batches are submitted to the pending-rows batcher like SQL INSERTs when PostgreSQL batching is enabled for the connection, and keep the direct write path when it is not. The batcher acknowledges synchronously, so the CommandComplete row count stays exact in both modes. Adds test_postgres_copy_in_with_batching asserting the batcher flush metric increments only when batching is enabled. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options The WITH (...) option list is now open instead of restricted to sqlparser's COPY grammar: - COPY ... FROM STDIN statements are parsed textually (new copy_in::parse module, tokenizer-based) so arbitrary KEY value entries reach the codecs; sqlparser would reject options it does not know before we ever see them. WITH is optional, matching PostgreSQL. - CopyOptionSet becomes an open ordered map of KEY -> String | Number | Ident | Boolean | Flag with typed take_* accessors (take_string, take_single_byte, take_flag, raw take_value) and case-insensitive keys; duplicate options are rejected. - Each format codec takes the options it understands and rejects the leftovers with one message naming them ('COPY option(s) X are not supported by format Y'), so a typo cannot be silently ignored. The text format now also rejects HEADER/QUOTE/ESCAPE instead of ignoring them, matching PostgreSQL. A future format (e.g. influxdb line protocol with WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec and a registry entry; no parser changes. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing Replace the hand-rolled CSV record parser and per-value conversion with arrow-csv's incremental Decoder — the same engine backing COPY ... FROM 'file.csv' in common-datasource — so both COPY routes accept identical data. The decoder handles record splitting (quoting, escaped quotes, custom delimiters, CRLF, headers, chunk boundaries) and typed conversion against the targeted columns' schema in one pass; decoded batches become insert rows via the existing api::helper::vectors_to_rows and datatypes Helper::try_into_vectors, and the decode schema comes from the greptime Schema's arrow projection. The codec framework now has parsers emit complete insertable rows (api Row) instead of raw byte fields: create_parser takes the targeted columns and is fallible; text converts with the shared text-representation helpers, csv with the arrow decoder. Documented consequences of sharing the engine: JSON and Binary columns and bare epoch-integer timestamps are rejected for csv, and an empty field — quoted or not — is NULL; a custom NULL string is rejected accordingly. Signed-off-by: Ning Sun <sunning@greptime.com> * docs(postgres): make the copy-in framework explicitly format-open Reframe the codec framework docs so nothing assumes csv is the only other text format: the shared pieces (Field, convert_text_field, check_text_representable_type) are described as building blocks for any format whose records carry values as text, while formats with their own typed decoding (csv today) build rows themselves. State explicitly that formats are not limited to PostgreSQL's own — GreptimeDB-specific formats (e.g. a line protocol) register like any other and define their own options through the open WITH clause. No code changes beyond a test rename and a doc clarifier. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): driver-level coverage for CSV copy-in options Add an end-to-end test through tokio-postgres exercising HEADER, custom DELIMITER, quoted fields with embedded newlines, escaped quotes and CRLF record separators, with read-back verification through the real insert path. WITH is omitted, matching PostgreSQL. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): drop the raw-socket copy-in tests The simple-protocol raw client duplicated what tokio-postgres and the integration tests already cover; keep the driver-based tests only. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode - text feed: take the longest complete line prefix in one drain and iterate lines over the slice, instead of draining per line (each drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of short lines) - csv check_column_type: delegate to the shared check_text_representable_type and additionally reject Binary/Json - decode_bytea: use the hex crate instead of a hand-rolled decoder - semantic_type_of: use ColumnSchema::is_time_index() like the flow adapter's column_schemas_to_proto, instead of index lookups - parse: drop the unused expect_ident parameter; document the identifier-case decision - PostgresServer:🆕 document that query_handler and copy_in_handler are the same frontend instance Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove unrelated files * feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format Remove the hand-rolled PostgreSQL text-format codec (backslash escapes, octal, \N nulls) that arrow-csv cannot express, keeping arrow-csv-based csv as the only implemented format. - COPY FROM STDIN without FORMAT now fails with guidance ('requires an explicit format in this version; use WITH (FORMAT csv)') rather than silently defaulting to the removed format - FORMAT text/txt and FORMAT binary report 'not supported yet' via registry stubs instead of unknown-format errors - codec.rs shrinks to the trait framework: the text-representation helpers (Field, convert_text_field, bytea hex, timestamp parsing) are removed with the codec; the module docs note where to find their shape should a future text-serializing format need them - csv implements check_column_type directly (no shared default) Breaking for clients relying on the PostgreSQL default text format; explicit WITH (FORMAT csv) is required. Signed-off-by: Ning Sun <sunning@greptime.com> * perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics - COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher each flush pays the full insert pipeline (catalog lookup, schema check, partition split, region write round trip), so 4x larger batches cut round trips 4x at linearly-bounded memory; the constant also sizes the arrow decoder batch. Comment documents the trade-off and points high-rate users at the pending-rows batcher, which turns flushes into enqueues. - rows buffer pre-allocates COPY_IN_FLUSH_ROWS. - flush_copy_in warns on unexpected output variants instead of silently counting 0 rows. - batching wire test feeds 70000 rows to stay above the new threshold. Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv) sends FORMAT csv and succeeds; default \copy (text) gets the 'requires an explicit format' guidance; COPY TO STDOUT round-trips. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): reset copy-in state on successful CopyDone on_copy_done left the CopyInState in place (inert but holding buffers) until the next COPY overwrote it; take it like on_copy_fail does, so buffers are freed eagerly and a stray post-done CopyData gets the protocol-violation error instead of a finished parser. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass typo check * chore: format change * fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries The csv feed loop flushed the arrow-csv decoder at every CopyData chunk end — including partial batches at record boundaries — but arrow-csv's contract permits flush() only after decode() returned 0 (a full batch of complete records). Flushing outside that point corrupts subsequent decoding: fields mis-parse shifted by the size of the record prefix buffered so far ('000001,host-001,...' parses as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse, silently writing wrong values. Which chunks trigger it depends on where record boundaries land, so it appears intermittent with random-length data (psql 8KB / pgx ~64KB chunking both hit it). Adopt the strict decoder protocol: feed decodes until input is exhausted and flushes only when decode returns 0 (batch full); trailing partial batches surface at the EOF signal in finish(). With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is unchanged; sub-threshold streams now emit their rows at CopyDone instead of per chunk, which is semantically identical. Adds test_csv_chunked_feeds_are_alignment_safe: the same payload fed at chunk sizes 1..65531 (plus shifted starts) with every row's content verified — reproduces the corruption on the old loop at chunk sizes 7, 8, 13, 64 and 1000. Verified with the reporter's Go reproducer (pgx, content-checked): 10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the previously failing benchmark sizes 500/1000/2000/200/800 all round- trip exactly. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass spell check * chore: bypass spell check * fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter Three review findings (shuiyisong, PR #9297): [P1] Enforce the catalog restriction on COPY targets. copy_in_table now applies validate_catalog_and_schema when QueryOptions:: disallow_cross_catalog_query is set, mirroring SQL INSERT's check_permission path, BEFORE begin_copy_in switches the query context to the target catalog (which previously made the restriction unenforceable downstream). Cross-catalog COPY is now rejected with 'Illegal access to catalog' like INSERT. [P2] Canonicalize unquoted identifiers like the SQL parser (ParserContext::canonicalize_identifier): unquoted table-name parts and columns fold to lowercase, quoted ones keep their case. COPY Metrics (Host, Ts) now addresses the table CREATE TABLE Metrics created, and no longer silently misroutes into a coexisting quoted "Metrics". [P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal tab while still rejecting newlines and other control characters, and the option parser now also accepts E'...' escape-string constants (DELIMITER E'\t'), processing PostgreSQL's backslash escapes. Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round- trip into the right tables, cross-catalog rejection asserted in a new frontend unit test (QueryOptions plugin + second catalog), and both DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure Two review findings (shuiyisong, PR #9297): [P2] Boolean columns round-trip between the COPY directions. arrow-csv only parses true/false while PostgreSQL COPY OUT emits t/f, so boolean tables failed to re-import their own export (22P02). Boolean columns now decode as Utf8 in the arrow schema and flush_batch converts them with PostgreSQL's boolean input grammar (true/false, t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact bytes are fed back via COPY FROM STDIN and verified. [P2] Drop the COPY state on every server-side failure. on_copy_data and on_copy_done now take ownership of the state up front, so parse, conversion or insert errors drop the buffered data instead of retaining it on long-lived or pooled connections until the next COPY or connection close (the reviewer measured ~17 MiB retained after a failed 8 MiB row); on_copy_data success puts the state back. An e2e test drives a failed COPY followed by a successful one on the same connection. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): interpret offset-free COPY timestamps in the session timezone Review finding (shuiyisong, PR #9297): the arrow decode schema carried Timestamp fields with no timezone, so the CSV decoder parsed offset-free timestamp strings as UTC while SQL INSERT parses them in the session timezone — the same input string silently stored a different instant depending on the ingestion method (8h apart in Asia/Shanghai). CopyInCodec::create_parser now receives the session timezone and the csv decode schema attaches it to Timestamp fields, so naive strings parse like INSERT; explicit offsets in the input (Z, +08:00) still take precedence, and UTC sessions are unchanged. Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z) and the end-to-end test_postgres_copy_in_session_timezone_parity, which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the same string, and asserts both CAST(ts AS BIGINT) readbacks agree on 1704038400000. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): don't unescape E-string COPY options twice Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already decodes PostgreSQL escape-string constants before returning Token::EscapedStringLiteral, so the second unescape_pg_string pass altered the decoded value — DELIMITER E'\\t' (a literal backslash followed by t, two characters) was turned into a tab and wrongly accepted as a single-byte delimiter. Use the EscapedStringLiteral payload as-is, like SingleQuotedString, and delete unescape_pg_string. Regression coverage distinguishes the three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t' -> rejected as a multi-character delimiter. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN Two review findings (shuiyisong, PR #9297): [P2] Preserve timestamp instants across COPY OUT/IN in non-UTC sessions. COPY OUT rendered naive UTC text while COPY IN (since the session-tz fix) interprets offset-free strings as local time, so re-importing an export shifted instants by the zone offset. COPY OUT now renders timestamps with an explicit session-timezone offset (2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the session tz (arrow-pg converts to a tz-aware DateTime) and the pg field type becomes TIMESTAMPTZ so the offset-bearing text format applies. Embedded offsets win on re-import regardless of session, keeping the OUT -> IN round trip exact; UTC sessions are unchanged. [P2] Preserve quoted empty strings. The shared arrow-csv decoder nulls every empty field, so a quoted empty string ("" — an empty STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner runs over the same bytes with the decoder's own splitting rules and records per field whether it was quoted; flush_batch restores quoted empties to empty strings for string columns, while unquoted empty fields stay NULL. Masks track records across chunk boundaries and align with decoder batches. Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip (Asia/Shanghai export asserted to carry +08, exact-epoch re-import comparison) and test_postgres_copy_in_quoted_empty_string (NULL, quoted empty and non-empty in one import), plus unit tests including a chunk-boundary sweep for the quoted-empty mask alignment. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns Two review findings (fengjiachun, PR #9297): [P2] Doubled quotes desynchronized the shadow quote-mask scanner. In QuoteInQuoted the doubled-quote branch skipped the second quote AND the following byte, so a closing quote or delimiter directly after an escaped quote was swallowed and later fields lost their masks -- import semantics depended on CopyData chunk boundaries. The branch now consumes only the escaped quote and returns to the in-quoted state, mirroring the decoder. Regression: "a""b","" ... verified across chunk sizes 1-64. [P2] A NOT NULL string column rejected quoted empty strings before restoration could run: the decode schema preserved the target nullability, and arrow-csv's batch validation rejects the temporary NULL a quoted empty decodes to. String fields now decode as nullable (safe: quoted-vs-unquoted is exactly the restoration's job), and the target NOT NULL constraint is enforced in flush_batch after restoration, so "" imports as an empty string while an unquoted empty field fails with a not-null-constraint error. Regressions at unit and e2e level: NOT NULL column accepting "" and rejecting unquoted NULL. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators Two review findings (fengjiachun, PR #9297): [P2] Blank lines emitted a spurious empty mask. csv-core skips blank lines, but the scanner queued one mask per blank record, so every subsequent decoded row consumed the wrong mask (a leading blank line turned a quoted empty into NULL and an unquoted empty into an empty string, and tripped NOT NULL on the wrong row). A blank line before the header also shifted the header expectation. end_record now returns no mask for a completely blank record without consuming the header expectation. [P2] Standalone CR was not a record terminator. The decoder accepts CR, CRLF and LF alike; the scanner only recognized CR before LF or at EOF, so CR-separated input desynchronized masks and flush_batch failed with 'quote masks out of sync'. is_record_end now treats CR alone as a terminator, consuming one byte (two for CRLF), while a CR at a chunk boundary still defers via carry so CRLF split across CopyData chunks stays one terminator. Regressions: blank lines before/inside records and before a header (NULL/empty preservation plus NOT NULL behavior), and mixed CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing CR-only record. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY The shadow quote-mask scanner — kept to preserve PostgreSQL's quoted-empty-string vs unquoted-NULL distinction — mirrored the decoder's record-splitting grammar by hand and required a fix for every rule it drifted on (doubled quotes, blank lines, CR terminators, NOT NULL interplay). Per maintainers' decision, that complexity is not worth the divergence: COPY FROM STDIN now behaves EXACTLY like the file-based COPY ... FROM 'file.csv' — an empty field, quoted or not, imports as NULL (arrow-csv's engine semantics, identical in both routes). Removed: QuoteMaskScanner and helpers, the always-nullable string decode schema, the restoration + NOT NULL enforcement in flush_batch, set_string_at, and the scanner-era unit tests. The decode schema again preserves target nullability, so the decoder enforces NOT NULL itself, same as the file route. Kept (small, schema-level, no second grammar): boolean spellings (t/f) and session-timezone timestamp decoding, both addressing separate review findings. E2E: test_postgres_copy_in_empty_field_is_null pins the parity — quoted empty, unquoted empty and non-empty in one import. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove empty file * chore: lint issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
shuiyisong
|
2eff52c757 |
feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): guard point accounting and benchmark metrics ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove otlp_metric_compat logic Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): cache physical schema lookups during metric preflight Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: minor rename Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): batch metric rows and reuse schema validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: replace super in import path Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Ning Sun
|
af52dfafd8 |
chore(deps): upgrade sqlx to 0.9 to pick up webpki-roots 1.0.10 (#9862)
sqlx 0.8 pins the unmaintained webpki-roots 0.26 line for its rustls trust store, which still contains the recently compromised e-Tugra CA root. Upgrading sqlx to 0.9 switches its dependency to webpki-roots 1.x, and the lockfile now resolves to 1.0.10 where the distrusted root has been removed from the compiled-in Mozilla trust store. sqlx 0.9 requires dynamic SQL strings to be explicitly audited via the new SqlSafeStr trait: only &'static str literals are accepted as-is. All affected call sites (MySQL KV backend, MySQL election, integration tests and fuzz targets) now wrap their internally generated SQL templates in AssertSqlSafe; the wrapped SQL is built from operator config or test-local identifiers only, and dynamic data keeps flowing through bind parameters. The mysql-rsa feature is enabled alongside mysql to preserve 0.8 authentication behavior: sqlx 0.9 gates non-TLS RSA password auth (caching_sha2_password / sha256_password) behind it. Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Chenglong Zheng
|
9f9b35c270 |
feat: support wait and timeout options for TRUNCATE TABLE (#9753)
* feat: support wait and timeout options for TRUNCATE TABLE * fix: address review feedback for truncate DDL options * fix: address review feedback for truncate DDL options |
||
|
Ning Sun
|
a7230f7a92 |
fix(partition): coerce partition exprs against the batch schema before routing (#9752)
* fix(partition): coerce partition exprs against the batch schema before routing Batch partition routing (MultiDimPartitionRule::split_record_batch) builds physical expressions via PartitionExpr::try_as_physical_expr, which did no type coercion. After widening the time index unit (e.g. ms -> ns via ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition bounds keep their original unit, so evaluating the bounds against the batch failed with: Invalid comparison operation: Timestamp(ns) < Timestamp(ms) Row inserts were unaffected because find_region compares Values and Timestamp::cmp converts units; scans were unaffected because PredicateGroup::new coerces the region partition expr against the region schema first. Coerce the logical partition expression against the evaluation schema in try_as_physical_expr before creating the physical expression, the same way PredicateGroup does. This also covers the region partition filter built for SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this conversion. Fixes #9747 Signed-off-by: Ning Sun <sunning@greptime.com> * test: regenerate change_col_type result after comment update The ts_widen_part comment in change_col_type.sql was reworded after the result file was last generated; sqlness echoes SQL comments into the result, so the stale comment made the case fail. Regenerate it. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Yingwen
|
3e2751d71d |
feat: improve SQL analyze scan diagnostics (#9565)
* feat: improve SQL analyze scan diagnostics Signed-off-by: evenyag <realevenyag@gmail.com> * test: redact dynamic filters in sqlness results Signed-off-by: evenyag <realevenyag@gmail.com> * test: stabilize analyze snapshots for merge queue Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove obsolete scan_details assertion Signed-off-by: evenyag <realevenyag@gmail.com> * test: refresh explain snapshots after partition filter changes Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
b3dc5674e9 |
feat(mito2)!: account SST decoding against shared scan memory limits (#9101)
* feat(mito2): add shared estimated SST scan memory budget Signed-off-by: evenyag <realevenyag@gmail.com> * feat: unify scan memory limits with configurable accounting modes Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse query runtime memory pool for series merges Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): wait for cancelled scan reservations before exhaustion check Signed-off-by: evenyag <realevenyag@gmail.com> * docs(mito2): explain parallel scan cancellation cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): deprecate waiting for scan memory quota Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(recordbatch): remove scan memory wait handling Signed-off-by: evenyag <realevenyag@gmail.com> * fix(table): initialize memory pool in scan test fixtures Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 fys
|
4e8bc6df86 |
fix(prom): align remote read label matcher semantics (#9633)
* fix(prom): align remote read label matcher semantics Signed-off-by: fys <fengys1996@gmail.com> * chore: optimize comment of validate_matchers * fix(prom): preserve existing binary label regex handling Signed-off-by: fys <fengys1996@gmail.com> * refactor: condition * test(prom): simplify remote read matcher tests and clarify comments * fix(prom): return binary remote read labels as UTF-8 * fix(prom): reject binary labels in remote read * refactor(prom): align remote read null label handling with PromQL Signed-off-by: fys <fengys1996@gmail.com> * refactor(prom): simplify null handling in label matcher branches Signed-off-by: fys <fengys1996@gmail.com> * fix(prom): restrict remote read labels to string types Signed-off-by: fys <fengys1996@gmail.com> * test(prom): use string label in default value column test Signed-off-by: fys <fengys1996@gmail.com> * fix: code review --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
dennis zhuang
|
8b0a155702 |
perf(otlp): chunk trace spans across resource groups (#9621)
* perf(otlp): chunk trace spans across resource groups Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(otlp): plan spans of a mixed binary/JSON chunk individually Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(otlp): describe cross-group trace chunking Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
672e94f61d |
feat: add column encoding option and use it for trace span ids (#9629)
* feat: add column encoding option and set it for trace span ids COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'. Either disables the dictionary for the column; delta_binary_packed is limited to integer and timestamp columns. OTLP trace v1/v2 tables now create span_id and parent_span_id with plain encoding and 64KiB pages, and timestamp_end with delta_binary_packed. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add compatibility cases for column encoding options Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: move column encoding downgrade checks into downgrade_compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 YingNing
|
ce428bb74d |
feat(servers): accept application/x-json-stream for log ingestion (#9581)
* feat(servers): accept application/x-json-stream for log ingestion
Support fluentbit's json_stream output format on the HTTP log ingestion
endpoints by accepting `application/x-json-stream` as a payload content
type.
The payload is parsed as concatenated JSON -- multiple JSON values placed
back-to-back without a separator, e.g. `{"a":1}{"b":2}` -- reusing the
streaming value iterator already used for `application/json`, so the
behavior for concatenated or mixed payloads is identical.
Aimed at #5404 (msgpack support is left as a possible follow-up).
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
* chore(servers): remove redundant comments per review feedback
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
---------
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
|
||
 Weny Xu
|
4859e57e0b |
feat(meta): apply affinity to failover and repartition (#9604)
* feat(meta): apply affinity to failover and repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): share repartition affinity until group allocation Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
fys
|
cf0f6c308a |
fix(prom): return empty remote read results for missing metrics (#9613)
* fix: return empty remote read results for missing metrics Signed-off-by: fys <fengys1996@gmail.com> * test(prom): cover empty reads for missing databases Signed-off-by: fys <fengys1996@gmail.com> * test(prom): simplify remote read regression cases Signed-off-by: fys <fengys1996@gmail.com> * fix(prom): log missing tables during remote read Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
shuiyisong
|
ad823db020 |
fix(pipeline)!: disable VRL host access by default (#9610)
fix(pipeline): disable VRL host access by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Weny Xu
|
cbfbad469f |
feat(meta): add affinity hooks (#9596)
* feat(meta): add affinity hooks Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): cover load-based affinity fallback queries Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): borrow affinity requirements during selection Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): remove unused affinity error variant Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
cf16c7313b |
fix(jaeger): return the most recent traces when a search hits its limit (#9595)
* fix(jaeger): return the most recent traces when a search hits its limit Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search matching more traces than the limit returned an arbitrary subset that changed between identical requests. Order traces by their latest matching span before applying the limit. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(jaeger): name the trace dataframe builder with a verb and document execution Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Ning Sun
|
40770ddb72 |
test: only assert genuinely invalid read-preference values at request ingresses (#9576)
The flow and nested-MergeScan ingress tests hardcoded "follower" among the read-preference values that must be rejected, coupling them to the current leader-only ReadPreference enum. Assert only genuinely invalid values there, matching the wire-values test which already limits itself to ["", "unknown"]. This keeps the ingress tests valid for builds that extend ReadPreference with more variants. Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Yingwen
|
00aa65de7a |
fix(query): guard deep expressions during query planning (#9564)
* fix(query): avoid speculative clones in constant normalization Signed-off-by: evenyag <realevenyag@gmail.com> * fix(query): reject deeply nested expressions before optimization Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
0e40281262 |
fix(promql): stop claiming scalar calculate preserves order (#9523)
* test: cover scalar calculate order contract Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): stop claiming scalar calculate preserves order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(test): describe scalar ordering regression as pre-fix behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
75bc4ce46d |
fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): only claim projected primary key prefix ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): keep physical key ordering for metric projections Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): compare unknown partition counts explicitly Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): model scoped internal TSID scan properties Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
863eebdecb |
feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve retry policy through error hints Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): restore sparse partition tags when rebuilding writes Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): stop detached write retries when callers cancel Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(frontend): configure retries across distributed write paths Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage. Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
2039f508fc |
fix(promql): stop fabricating histogram fold hash partitioning (#9509)
* fix(promql): stop fabricating histogram fold hash partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover histogram fraction binary join missing rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): normalize dictionary labels in histogram regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover histogram joins on logical metric tables Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: specify insert columns for logical metric fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use supported histogram child replacement API Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): keep histogram fold fixtures truthful Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 jeremyhi
|
83e62a7725 |
feat: remove the server-side Metric export gate (#9547)
* feat: remove the server-side Metric export gate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused export capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused CLI capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: import the packed export error selector Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
fc5831043f |
feat(cli): simplify packed metric exports with a boolean flag (#9531)
* feat(cli): select packed metric export with the layout option Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cli): inspect the packed capability error cause Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cli): follow the standard capability error source Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(cli): clarify the metric export gate helper name Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(cli): use a boolean flag for packed metric exports Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
18726c7602 |
fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(snapshot): bound writes and reduce logical export overhead Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: clarify metric snapshot acceptance criteria Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): split batches only for merged readers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(snapshot): cover supported types in both layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: consolidate metric snapshot experiment conclusions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): honor series statistics when splitting L0 scans Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito): explain memory risk of splitting short L0 series Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address snapshot export review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): accept source errors in lazy snapshot contexts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
483d1e49a3 |
fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep filter fetch when rewriting matches term Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
2d99a5cf69 |
feat(jaeger): add query endpoints for Jaeger UI v2 (#9409)
* feat(jaeger): support v3 HTTP query APIs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(jaeger): preserve full traces with v3 time hints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(jaeger): clarify validation of v3 lookup hints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(jaeger): bound v3 searches and preserve v1 binary attributes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(jaeger): match resource attributes in v0 tag filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
32e8467cda |
fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): use current DataFusion plan downcasts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): match null rendering in join regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh union plan for unknown scan partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh shared plans after distribution rule removal Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover semi joins with remote dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen partitioned merge scan regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove unused merge scan test helpers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): measure merge scan hash repartition cost Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt merge scan deserialization to two-arg constructor Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): adapt nested merge scan integration constructor Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
b913be2af7 |
fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries An instant query of a range selector or subquery (e.g. `up[5m]`) returned the internal range arrays, which panicked when the frontend decoded them as dictionaries. Plan such queries as the selected samples with their own timestamps, and reject range-vector expressions in range queries like Prometheus does. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): keep encode_substrait on a valid range query A range query of a range-vector expression now fails like in Prometheus. Wrap the subquery in max_over_time so the case still encodes both RangeManipulate nodes through substrait. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): plan wrapped range-vector roots and reject a zero subquery step TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the range-vector path did not unwrap. A zero subquery step divided by zero. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover wrapped range-vector roots and a zero subquery step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): default subquery step and reject range vectors in range queries A root subquery without a step uses the one-minute default instead of the query step. The HTTP range query endpoint rejects range-vector expressions even when start equals end. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover root subquery default step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): align root subqueries with subquery_child_window Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): return millisecond timestamps for range-vector results Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): keep the time index timezone and name its table in errors Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
shuiyisong
|
a907cbfbe4 |
perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues for tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: bound asynchronous trace auxiliary writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf: bound trace auxiliary cache by estimated memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: size trace auxiliary cache from host or pod memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add metrics for cache hit and miss Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: unify frontend cache metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: merge and reduce Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add init for the metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
be91fb5ad2 |
test: support metasrv plugins in integration cluster builder (#9423)
* test: support plugins and flownodes in integration cluster builder Allow the meta-srv mock and the tests-integration distributed cluster builder to install metasrv Plugins and to start flownodes that register with the mock metasrv, so integration tests can exercise plugin-provided DDL handlers and flows against a real multi-datanode cluster. The default path is unchanged: mock()/mock_with_client_channel_config() keep their signatures, and GreptimeDbClusterBuilder builds the metasrv plugin-free and starts no flownode unless with_plugins/with_flownodes is called. Signed-off-by: discord9 <discord9@163.com> * test: narrow cluster plugins to metasrv injection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: remove unnecessary plugin container mutability Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
e4872ec1a2 |
feat(log-store): collect object store WAL objects below a persisted reclaim boundary (#9471)
* feat(log-store): collect object store WAL objects below a persisted reclaim boundary Every object header carries a reclaim boundary: every object below it was fully flushed when the object was sealed. The writer computes the boundary from the catalog and the obsolete watermarks after every obsolete and every indexed object, and stamps it on every sealed object and on the start object. The boundary of the last indexed object permits deletion: every present object below it, indexed, listed by recovery off the chain, or left by a failed batch, is deleted in the background, at most four at a time. Recovery judges every candidate with its own boundary: a link to a predecessor below it holds whether or not the predecessor is present, and the chain stops there. A read that listed an object before it was collected skips it instead of failing. The header grows from 46 to 54 bytes; the format version stays 1. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): scan the reclaim boundary from the previous boundary The boundary was recomputed by scanning every region's indexed objects from the first one, so the flushed objects of busy regions were scanned again on every indexed object while another region held the boundary. Scan the objects in sequence order from the previous boundary instead and stop at the first one holding a live segment. Watermarks never move down and a region first appears above the boundary, so the result is the same and an object is not scanned again once the boundary passed it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): name the parked operation helpers after both their uses Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
6265c31e4a |
feat: batch logical table DDL during import-v2 (#9467)
* feat: batch logical table DDL during import-v2 Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve one-shot batch DDL transport and context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: check schema-only import state at its own path Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve oversized import DDL compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: reuse import DDL encoding and metric constants Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Ning Sun
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
|
jeremyhi
|
c82da8308b |
feat: add bounded Metric logical-table batch DDL (#9427)
* feat: add bounded Metric logical-table batch DDL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: enforce logical batch DDL interceptor boundaries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: bound rewritten DDL size through shared SQL formatting Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve logical DDL error diagnostics Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: verify logical DDL error logging Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
f9ab3c296b |
feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region A segment that does not decode, because its checksum does not match or its content disagrees with its footer entry, is fetched once more. If it still does not decode, the read follows `on_corrupted_segment`: `skip` records a WAL hole for the region with the object key, sequence and the footer's entry id range, increments `greptime_logstore_object_store_wal_skipped_segments_total`, logs a warning and continues with the next object; `fail` fails the read. A failed fetch fails the read in both modes. Holes are kept in memory per region and can be listed from the store. The standalone test builder allocates WAL options through `build_standalone_wal_provider`, and integration tests run a standalone instance on the object store WAL across restarts, on S3 when the `GT_S3_*` variables are set. The standalone shutdown runs every step and returns the first error. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cmd): cover the best-effort standalone stop Build a standalone instance whose leader services fail to stop and whose datanode servers fail afterwards, and check that `Instance::stop` still stops the object store WAL and returns the leader services error. Rename `skip_segment` to `record_skipped_segment` and the file store integration test to `test_standalone_object_store_wal_survives_restarts_on_file`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Yingwen
|
3a4c24a863 |
fix: temporarily disable v2 series scan by default (#9436)
* fix: disable v2 series scan by default temporarily Signed-off-by: evenyag <realevenyag@gmail.com> * test: explicitly enable v2 series scan in sqlness configs Signed-off-by: evenyag <realevenyag@gmail.com> * test: regenerate sqlness results for legacy series scan default Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Dragon RoarandDennis Zhuang
|
d7f5331876 |
feat: add CORS support for gRPC-Web on the frontend gRPC server (#9374)
* feat: add CORS support for gRPC-Web on the frontend gRPC server A browser-based gRPC-Web client cannot read a cross-origin response. The request carries a non-simple content type, so the browser sends an `OPTIONS` preflight first, and it drops the call when the response has no `Access-Control-Allow-Origin`. Add `enable_cors` and `cors_allowed_origins` to `[grpc]`, threaded from `GrpcOptions` to `GrpcServerConfig` the same way `tls` is. The CORS layer sits outside `tonic_web::GrpcWebLayer`, so it answers the preflight before the request reaches the gRPC routes. It exposes `grpc-status`, `grpc-message` and `grpc-status-details-bin`, which a browser cannot read otherwise. An empty `cors_allowed_origins` allows any origin. `enable_cors` defaults to false, unlike `[http]`. `GrpcOptions` is shared by the public `[grpc]` section and the internal `[internal_grpc]` one, and serde cannot tell them apart, so a true default would also turn CORS on for the internal gRPC listeners: the frontend internal gRPC server, and the datanode and flownode gRPC servers. None of them authenticate callers, and a browser on the host or in the cluster network can reach all of them. Set `enable_cors = true` to turn it on. Tests start a real server on an ephemeral port and cover the preflight, a custom origin list, and the disabled case. Update the example TOMLs and regenerate config/config.md. Signed-off-by: lczllx <2181719471@qq.com> * fix: drop the redundant OPTIONS from the gRPC CORS allow_methods Signed-off-by: lczllx <2181719471@qq.com> * test: assert the allow-headers and allow-methods headers in the gRPC CORS preflight The preflight answers with `access-control-allow-headers: *` from `AllowHeaders::any()`, and with `access-control-allow-methods: post` from the POST-only `allow_methods` list. Pin both, the way the HTTP CORS test pins its own headers: a missing allow-headers header would let the preflight pass the origin check and still have the browser block every gRPC-Web call, which a non-browser client would never notice. Signed-off-by: lczllx <2181719471@qq.com> * docs(config): stop the example configs from setting the new gRPC CORS keys Signed-off-by: lczllx <2181719471@qq.com> * test(servers): cover the exposed gRPC CORS headers and pin the option plumbing Signed-off-by: lczllx <2181719471@qq.com> * docs(config): document the gRPC CORS origin example with `#+` Signed-off-by: lczllx <2181719471@qq.com> * fix(frontend): keep CORS off the internal gRPC server Signed-off-by: lczllx <2181719471@qq.com> * docs(grpc): document that the internal gRPC server never serves CORS Signed-off-by: lczllx <2181719471@qq.com> * Update src/servers/src/grpc.rs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: trim redundant comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: share CORS origin parsing and simplify gRPC-Web CORS tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: enable gRPC CORS only on the frontend public server Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(config): clarify the gRPC CORS options Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: lczllx <2181719471@qq.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
78eefbe4c4 |
perf: batch schema export requests (#9408)
* perf: batch schema export requests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: propagate legacy schema export failures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep credentials out of SQL response errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use a typo-safe dotted catalog name Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(cli): address schema export review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
78a7b93292 |
fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics - timestamp() over a selector reports the selected sample's timestamp, without adding the offset. - timestamp() over any other expression reports the evaluation time instead of the input value. - label_join that overwrites an existing label rejects duplicate label sets at runtime. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): align label_replace and label_join edge cases with Prometheus - label_replace leaves non-matching series unchanged instead of copying the source value into the destination label. - label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join. - label_join with no source labels removes the destination label, and an empty source label name is rejected. - Empty label values produced by these functions are NULL, the same as an absent label. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): join absent labels as empty strings in label_join concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): skip rows without a sample in timestamp() timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Ning Sun
|
3131cdbcb6 |
fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests The request-memory accounting only bounds and charges the encoded bytes on the wire, but a compressed body can expand far beyond that during decompression, so tiny requests could allocate disproportionate frontend memory before any protobuf validation or quota charge. - Handler-level decompression (Prometheus remote read/write v1+v2, Loki) now enforces a hard 512 MiB decoded-size cap, checked before any output buffer is allocated, and charges the decoded bytes to the shared ServerMemoryLimiter, holding the permits for the lifetime of the decompressed buffer. - gRPC requests with transport compression reserve the configured max_recv_message_size before tonic decompresses, so the decoding phase is admitted against max_in_flight_write_bytes; the later per-message charge is skipped to avoid double accounting. - The HTTP memory-limit middleware keeps its upfront Content-Length charge but now also accounts the bytes actually streamed beyond it, so chunked requests and understated Content-Length headers no longer bypass the aggregate quota. - Routes that decompress request bodies via RequestDecompressionLayer (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards) now charge the decompressed bytes as handlers consume them: the global middleware marks Content-Encoding requests, and a route-local accounting layer inside the decompression layer charges the decoded stream. Plain requests are skipped to avoid double-counting. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on memory admission guard lifetimes and 429 mapping - Retain the gRPC pre-decode reservation for the whole request: the extensions holding the guard were dropped when the request was consumed (into_inner), releasing the reservation while per-message charges stayed skipped. Both the unary and streaming handlers now clone and hold the reservation marker for the duration of request handling. - Retain the HTTP body permits across the handler: the AccountedBody wrapper and the request extensions are dropped once the extractors finish collecting the body, before the handler is done with the decoded data. Both accounting middlewares now keep their own accounting handle alive across next.run(req).await. - Return a ChargedBuffer from the Loki snappy decompressor so the reservation outlives the decompressed bytes through the caller's protobuf decoding, matching the Prometheus path. - Map mid-stream quota exhaustion to 429 instead of the generic body error (400): the accounting flags exhaustion and the middlewares rewrite the extractor rejection, so clients can distinguish backpressure from malformed input. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve the in-flight body acquisition before trying a new one A parked acquisition in AccountedBody::charge always belongs to the currently buffered frame, so it must be resolved before any new acquisition is tried. Letting the fast-path try_acquire succeed while a waiter is parked left the waiter alive, and its late completion would then be credited with a later frame's byte count, under-reserving memory relative to what was marked charged. Adds a regression test that reproduces the misattribution: with the buggy ordering the test delivers a body the quota cannot cover; with the fix the over-quota frame waits for its own acquisition and times out. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
discord9
|
abf1396c28 |
fix(client): yield Flight batches and affected rows without waiting for next message (#8918)
* fix(client): avoid Flight metrics lookahead stalls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(client): extract trailing Flight metrics task Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(client): synchronize trailing metrics and bound cancellation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Lei, HUANG
|
5ccfcd4644 |
feat: support automatic column addition for Flight bulk inserts (#9285)
* fix: auto-add columns when initializing bulk insert streams Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: reject nested columns in bulk schema auto-add Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: reject unknown columns in non-empty bulk inserts Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
3c0e2a8d55 |
feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject empty packed export time ranges Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover cancellation during packed export I/O Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |