mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-29 07:06:20 +00:00
91a59b82fbff9d1443968024dc5df1c96e661d4e
1516
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Yingwen
|
371c4e9004 |
feat: support query concurrency limits in standalone (#9958)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Ning Sun
|
80df1c397b |
feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support * feat: add postgres copy-in framework * feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher Follow-up to #9302: begin_copy_in now uses the batching-aware new_query_context, so copy-in batches are submitted to the pending-rows batcher like SQL INSERTs when PostgreSQL batching is enabled for the connection, and keep the direct write path when it is not. The batcher acknowledges synchronously, so the CommandComplete row count stays exact in both modes. Adds test_postgres_copy_in_with_batching asserting the batcher flush metric increments only when batching is enabled. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options The WITH (...) option list is now open instead of restricted to sqlparser's COPY grammar: - COPY ... FROM STDIN statements are parsed textually (new copy_in::parse module, tokenizer-based) so arbitrary KEY value entries reach the codecs; sqlparser would reject options it does not know before we ever see them. WITH is optional, matching PostgreSQL. - CopyOptionSet becomes an open ordered map of KEY -> String | Number | Ident | Boolean | Flag with typed take_* accessors (take_string, take_single_byte, take_flag, raw take_value) and case-insensitive keys; duplicate options are rejected. - Each format codec takes the options it understands and rejects the leftovers with one message naming them ('COPY option(s) X are not supported by format Y'), so a typo cannot be silently ignored. The text format now also rejects HEADER/QUOTE/ESCAPE instead of ignoring them, matching PostgreSQL. A future format (e.g. influxdb line protocol with WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec and a registry entry; no parser changes. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing Replace the hand-rolled CSV record parser and per-value conversion with arrow-csv's incremental Decoder — the same engine backing COPY ... FROM 'file.csv' in common-datasource — so both COPY routes accept identical data. The decoder handles record splitting (quoting, escaped quotes, custom delimiters, CRLF, headers, chunk boundaries) and typed conversion against the targeted columns' schema in one pass; decoded batches become insert rows via the existing api::helper::vectors_to_rows and datatypes Helper::try_into_vectors, and the decode schema comes from the greptime Schema's arrow projection. The codec framework now has parsers emit complete insertable rows (api Row) instead of raw byte fields: create_parser takes the targeted columns and is fallible; text converts with the shared text-representation helpers, csv with the arrow decoder. Documented consequences of sharing the engine: JSON and Binary columns and bare epoch-integer timestamps are rejected for csv, and an empty field — quoted or not — is NULL; a custom NULL string is rejected accordingly. Signed-off-by: Ning Sun <sunning@greptime.com> * docs(postgres): make the copy-in framework explicitly format-open Reframe the codec framework docs so nothing assumes csv is the only other text format: the shared pieces (Field, convert_text_field, check_text_representable_type) are described as building blocks for any format whose records carry values as text, while formats with their own typed decoding (csv today) build rows themselves. State explicitly that formats are not limited to PostgreSQL's own — GreptimeDB-specific formats (e.g. a line protocol) register like any other and define their own options through the open WITH clause. No code changes beyond a test rename and a doc clarifier. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): driver-level coverage for CSV copy-in options Add an end-to-end test through tokio-postgres exercising HEADER, custom DELIMITER, quoted fields with embedded newlines, escaped quotes and CRLF record separators, with read-back verification through the real insert path. WITH is omitted, matching PostgreSQL. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): drop the raw-socket copy-in tests The simple-protocol raw client duplicated what tokio-postgres and the integration tests already cover; keep the driver-based tests only. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode - text feed: take the longest complete line prefix in one drain and iterate lines over the slice, instead of draining per line (each drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of short lines) - csv check_column_type: delegate to the shared check_text_representable_type and additionally reject Binary/Json - decode_bytea: use the hex crate instead of a hand-rolled decoder - semantic_type_of: use ColumnSchema::is_time_index() like the flow adapter's column_schemas_to_proto, instead of index lookups - parse: drop the unused expect_ident parameter; document the identifier-case decision - PostgresServer:🆕 document that query_handler and copy_in_handler are the same frontend instance Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove unrelated files * feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format Remove the hand-rolled PostgreSQL text-format codec (backslash escapes, octal, \N nulls) that arrow-csv cannot express, keeping arrow-csv-based csv as the only implemented format. - COPY FROM STDIN without FORMAT now fails with guidance ('requires an explicit format in this version; use WITH (FORMAT csv)') rather than silently defaulting to the removed format - FORMAT text/txt and FORMAT binary report 'not supported yet' via registry stubs instead of unknown-format errors - codec.rs shrinks to the trait framework: the text-representation helpers (Field, convert_text_field, bytea hex, timestamp parsing) are removed with the codec; the module docs note where to find their shape should a future text-serializing format need them - csv implements check_column_type directly (no shared default) Breaking for clients relying on the PostgreSQL default text format; explicit WITH (FORMAT csv) is required. Signed-off-by: Ning Sun <sunning@greptime.com> * perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics - COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher each flush pays the full insert pipeline (catalog lookup, schema check, partition split, region write round trip), so 4x larger batches cut round trips 4x at linearly-bounded memory; the constant also sizes the arrow decoder batch. Comment documents the trade-off and points high-rate users at the pending-rows batcher, which turns flushes into enqueues. - rows buffer pre-allocates COPY_IN_FLUSH_ROWS. - flush_copy_in warns on unexpected output variants instead of silently counting 0 rows. - batching wire test feeds 70000 rows to stay above the new threshold. Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv) sends FORMAT csv and succeeds; default \copy (text) gets the 'requires an explicit format' guidance; COPY TO STDOUT round-trips. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): reset copy-in state on successful CopyDone on_copy_done left the CopyInState in place (inert but holding buffers) until the next COPY overwrote it; take it like on_copy_fail does, so buffers are freed eagerly and a stray post-done CopyData gets the protocol-violation error instead of a finished parser. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass typo check * chore: format change * fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries The csv feed loop flushed the arrow-csv decoder at every CopyData chunk end — including partial batches at record boundaries — but arrow-csv's contract permits flush() only after decode() returned 0 (a full batch of complete records). Flushing outside that point corrupts subsequent decoding: fields mis-parse shifted by the size of the record prefix buffered so far ('000001,host-001,...' parses as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse, silently writing wrong values. Which chunks trigger it depends on where record boundaries land, so it appears intermittent with random-length data (psql 8KB / pgx ~64KB chunking both hit it). Adopt the strict decoder protocol: feed decodes until input is exhausted and flushes only when decode returns 0 (batch full); trailing partial batches surface at the EOF signal in finish(). With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is unchanged; sub-threshold streams now emit their rows at CopyDone instead of per chunk, which is semantically identical. Adds test_csv_chunked_feeds_are_alignment_safe: the same payload fed at chunk sizes 1..65531 (plus shifted starts) with every row's content verified — reproduces the corruption on the old loop at chunk sizes 7, 8, 13, 64 and 1000. Verified with the reporter's Go reproducer (pgx, content-checked): 10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the previously failing benchmark sizes 500/1000/2000/200/800 all round- trip exactly. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass spell check * chore: bypass spell check * fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter Three review findings (shuiyisong, PR #9297): [P1] Enforce the catalog restriction on COPY targets. copy_in_table now applies validate_catalog_and_schema when QueryOptions:: disallow_cross_catalog_query is set, mirroring SQL INSERT's check_permission path, BEFORE begin_copy_in switches the query context to the target catalog (which previously made the restriction unenforceable downstream). Cross-catalog COPY is now rejected with 'Illegal access to catalog' like INSERT. [P2] Canonicalize unquoted identifiers like the SQL parser (ParserContext::canonicalize_identifier): unquoted table-name parts and columns fold to lowercase, quoted ones keep their case. COPY Metrics (Host, Ts) now addresses the table CREATE TABLE Metrics created, and no longer silently misroutes into a coexisting quoted "Metrics". [P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal tab while still rejecting newlines and other control characters, and the option parser now also accepts E'...' escape-string constants (DELIMITER E'\t'), processing PostgreSQL's backslash escapes. Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round- trip into the right tables, cross-catalog rejection asserted in a new frontend unit test (QueryOptions plugin + second catalog), and both DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure Two review findings (shuiyisong, PR #9297): [P2] Boolean columns round-trip between the COPY directions. arrow-csv only parses true/false while PostgreSQL COPY OUT emits t/f, so boolean tables failed to re-import their own export (22P02). Boolean columns now decode as Utf8 in the arrow schema and flush_batch converts them with PostgreSQL's boolean input grammar (true/false, t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact bytes are fed back via COPY FROM STDIN and verified. [P2] Drop the COPY state on every server-side failure. on_copy_data and on_copy_done now take ownership of the state up front, so parse, conversion or insert errors drop the buffered data instead of retaining it on long-lived or pooled connections until the next COPY or connection close (the reviewer measured ~17 MiB retained after a failed 8 MiB row); on_copy_data success puts the state back. An e2e test drives a failed COPY followed by a successful one on the same connection. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): interpret offset-free COPY timestamps in the session timezone Review finding (shuiyisong, PR #9297): the arrow decode schema carried Timestamp fields with no timezone, so the CSV decoder parsed offset-free timestamp strings as UTC while SQL INSERT parses them in the session timezone — the same input string silently stored a different instant depending on the ingestion method (8h apart in Asia/Shanghai). CopyInCodec::create_parser now receives the session timezone and the csv decode schema attaches it to Timestamp fields, so naive strings parse like INSERT; explicit offsets in the input (Z, +08:00) still take precedence, and UTC sessions are unchanged. Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z) and the end-to-end test_postgres_copy_in_session_timezone_parity, which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the same string, and asserts both CAST(ts AS BIGINT) readbacks agree on 1704038400000. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): don't unescape E-string COPY options twice Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already decodes PostgreSQL escape-string constants before returning Token::EscapedStringLiteral, so the second unescape_pg_string pass altered the decoded value — DELIMITER E'\\t' (a literal backslash followed by t, two characters) was turned into a tab and wrongly accepted as a single-byte delimiter. Use the EscapedStringLiteral payload as-is, like SingleQuotedString, and delete unescape_pg_string. Regression coverage distinguishes the three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t' -> rejected as a multi-character delimiter. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN Two review findings (shuiyisong, PR #9297): [P2] Preserve timestamp instants across COPY OUT/IN in non-UTC sessions. COPY OUT rendered naive UTC text while COPY IN (since the session-tz fix) interprets offset-free strings as local time, so re-importing an export shifted instants by the zone offset. COPY OUT now renders timestamps with an explicit session-timezone offset (2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the session tz (arrow-pg converts to a tz-aware DateTime) and the pg field type becomes TIMESTAMPTZ so the offset-bearing text format applies. Embedded offsets win on re-import regardless of session, keeping the OUT -> IN round trip exact; UTC sessions are unchanged. [P2] Preserve quoted empty strings. The shared arrow-csv decoder nulls every empty field, so a quoted empty string ("" — an empty STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner runs over the same bytes with the decoder's own splitting rules and records per field whether it was quoted; flush_batch restores quoted empties to empty strings for string columns, while unquoted empty fields stay NULL. Masks track records across chunk boundaries and align with decoder batches. Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip (Asia/Shanghai export asserted to carry +08, exact-epoch re-import comparison) and test_postgres_copy_in_quoted_empty_string (NULL, quoted empty and non-empty in one import), plus unit tests including a chunk-boundary sweep for the quoted-empty mask alignment. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns Two review findings (fengjiachun, PR #9297): [P2] Doubled quotes desynchronized the shadow quote-mask scanner. In QuoteInQuoted the doubled-quote branch skipped the second quote AND the following byte, so a closing quote or delimiter directly after an escaped quote was swallowed and later fields lost their masks -- import semantics depended on CopyData chunk boundaries. The branch now consumes only the escaped quote and returns to the in-quoted state, mirroring the decoder. Regression: "a""b","" ... verified across chunk sizes 1-64. [P2] A NOT NULL string column rejected quoted empty strings before restoration could run: the decode schema preserved the target nullability, and arrow-csv's batch validation rejects the temporary NULL a quoted empty decodes to. String fields now decode as nullable (safe: quoted-vs-unquoted is exactly the restoration's job), and the target NOT NULL constraint is enforced in flush_batch after restoration, so "" imports as an empty string while an unquoted empty field fails with a not-null-constraint error. Regressions at unit and e2e level: NOT NULL column accepting "" and rejecting unquoted NULL. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators Two review findings (fengjiachun, PR #9297): [P2] Blank lines emitted a spurious empty mask. csv-core skips blank lines, but the scanner queued one mask per blank record, so every subsequent decoded row consumed the wrong mask (a leading blank line turned a quoted empty into NULL and an unquoted empty into an empty string, and tripped NOT NULL on the wrong row). A blank line before the header also shifted the header expectation. end_record now returns no mask for a completely blank record without consuming the header expectation. [P2] Standalone CR was not a record terminator. The decoder accepts CR, CRLF and LF alike; the scanner only recognized CR before LF or at EOF, so CR-separated input desynchronized masks and flush_batch failed with 'quote masks out of sync'. is_record_end now treats CR alone as a terminator, consuming one byte (two for CRLF), while a CR at a chunk boundary still defers via carry so CRLF split across CopyData chunks stays one terminator. Regressions: blank lines before/inside records and before a header (NULL/empty preservation plus NOT NULL behavior), and mixed CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing CR-only record. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY The shadow quote-mask scanner — kept to preserve PostgreSQL's quoted-empty-string vs unquoted-NULL distinction — mirrored the decoder's record-splitting grammar by hand and required a fix for every rule it drifted on (doubled quotes, blank lines, CR terminators, NOT NULL interplay). Per maintainers' decision, that complexity is not worth the divergence: COPY FROM STDIN now behaves EXACTLY like the file-based COPY ... FROM 'file.csv' — an empty field, quoted or not, imports as NULL (arrow-csv's engine semantics, identical in both routes). Removed: QuoteMaskScanner and helpers, the always-nullable string decode schema, the restoration + NOT NULL enforcement in flush_batch, set_string_at, and the scanner-era unit tests. The decode schema again preserves target nullability, so the decoder enforces NOT NULL itself, same as the file route. Kept (small, schema-level, no second grammar): boolean spellings (t/f) and session-timezone timestamp decoding, both addressing separate review findings. E2E: test_postgres_copy_in_empty_field_is_null pins the parity — quoted empty, unquoted empty and non-empty in one import. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove empty file * chore: lint issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 shuiyisong
|
2eff52c757 |
feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): guard point accounting and benchmark metrics ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove otlp_metric_compat logic Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): cache physical schema lookups during metric preflight Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: minor rename Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): batch metric rows and reuse schema validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: replace super in import path Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 fys
|
f35057d9c2 |
feat(json2): support inverted index (#9324)
feat: build inverted indexes for JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Ning Sun
|
af52dfafd8 |
chore(deps): upgrade sqlx to 0.9 to pick up webpki-roots 1.0.10 (#9862)
sqlx 0.8 pins the unmaintained webpki-roots 0.26 line for its rustls trust store, which still contains the recently compromised e-Tugra CA root. Upgrading sqlx to 0.9 switches its dependency to webpki-roots 1.x, and the lockfile now resolves to 1.0.10 where the distrusted root has been removed from the compiled-in Mozilla trust store. sqlx 0.9 requires dynamic SQL strings to be explicitly audited via the new SqlSafeStr trait: only &'static str literals are accepted as-is. All affected call sites (MySQL KV backend, MySQL election, integration tests and fuzz targets) now wrap their internally generated SQL templates in AssertSqlSafe; the wrapped SQL is built from operator config or test-local identifiers only, and dynamic data keeps flowing through bind parameters. The mysql-rsa feature is enabled alongside mysql to preserve 0.8 authentication behavior: sqlx 0.9 gates non-TLS RSA password auth (caching_sha2_password / sha256_password) behind it. Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Ning Sun
|
ec4ef45f40 |
feat: use pgwire auth mechanisms API to simplify startuphandler (#9634)
* feat: use pgwire auth mechanisms API to simplify startuphandler * chore: update pgwire * chore: update pgwire revision to merged one * refactor: model based on upstream api * refactor: simplify implementation by introducing authsource * refactor: update to latest pgwire apis * refactor: adopt pgwire api changes * chore: switch to released pgwire * test: harden pg scram mixed-auth integration test and reject channel-binding clients - Fix MixedAuthProvider to return a SCRAM verifier for the denied user, so the test exercises rejection by authorization after successful SCRAM verification; record the selected mechanism from the returned variant and assert authorize ran exactly once per verified user. - Assert the denied connection fails with 28P01 before any terminal message, protecting the completion-ordering introduced by pgwire's authentication gate. - Add a raw-protocol test proving a client-first with a p= channel binding flag is rejected (RFC 5802 section 6), while n-flag clients (tokio_postgres) continue normally. Bump pgwire to 0.42.1, which enforces RFC 5802 channel binding negotiation and rejects short/overlong SCRAM proofs on the verifier path (sunng87/pgwire#476). Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 Weny Xu
|
f866ef2bd9 |
feat: return per-entry region write failures (#9728)
* feat: return per-entry region write failures Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve retry hints for single-entry region writes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve region write error compatibility Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify region partial result handling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align partial failure summaries with input order Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate region write response handling Signed-off-by: WenyXu <wenymedia@gmail.com> * test: fix partial result expectations and region map guard Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: limit lockfile changes to greptime proto Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: make partial region results an explicit client opt-in Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin greptime-proto to merged partial-result changes Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
a7230f7a92 |
fix(partition): coerce partition exprs against the batch schema before routing (#9752)
* fix(partition): coerce partition exprs against the batch schema before routing Batch partition routing (MultiDimPartitionRule::split_record_batch) builds physical expressions via PartitionExpr::try_as_physical_expr, which did no type coercion. After widening the time index unit (e.g. ms -> ns via ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition bounds keep their original unit, so evaluating the bounds against the batch failed with: Invalid comparison operation: Timestamp(ns) < Timestamp(ms) Row inserts were unaffected because find_region compares Values and Timestamp::cmp converts units; scans were unaffected because PredicateGroup::new coerces the region partition expr against the region schema first. Coerce the logical partition expression against the evaluation schema in try_as_physical_expr before creating the physical expression, the same way PredicateGroup does. This also covers the region partition filter built for SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this conversion. Fixes #9747 Signed-off-by: Ning Sun <sunning@greptime.com> * test: regenerate change_col_type result after comment update The ts_widen_part comment in change_col_type.sql was reworded after the result file was last generated; sqlness echoes SQL comments into the result, so the stale comment made the case fail. Regenerate it. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 fengyue-xve
|
1e29e8d4b6 |
fix(meta): validate persisted partition bounds before altering regions (#9487)
* fix(meta): validate persisted partition bounds before altering regions
A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.
Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.
Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.
Fixes #9454
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix: access ColumnSchema fields directly and use plain format args
ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: access ColumnSchema data_type field in alter table tests
Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop borrows now that data_type is read as a field
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop useless format! in nested expression test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: cover the partition tail in change_col_type sqlness case
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop spurious blank line in change_col_type result
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): compare partition bounds with the scan comparison coercion
The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.
Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: update the sqlness fixture for the new incompatibility message
The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency
common-meta gained a datafusion-expr dependency in
|
||
|
Yingwen
|
b3dc5674e9 |
feat(mito2)!: account SST decoding against shared scan memory limits (#9101)
* feat(mito2): add shared estimated SST scan memory budget Signed-off-by: evenyag <realevenyag@gmail.com> * feat: unify scan memory limits with configurable accounting modes Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse query runtime memory pool for series merges Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): wait for cancelled scan reservations before exhaustion check Signed-off-by: evenyag <realevenyag@gmail.com> * docs(mito2): explain parallel scan cancellation cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): deprecate waiting for scan memory quota Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(recordbatch): remove scan memory wait handling Signed-off-by: evenyag <realevenyag@gmail.com> * fix(table): initialize memory pool in scan test fixtures Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Lei, HUANG
|
f663569cf0 |
feat(auth): expose public authentication config (#9626)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 discord9
|
c3469fad51 |
fix(flow): retain tracked write completion across transient failures (#9525)
* fix(flow): retain tracked write completion across transient failures Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths. Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities. Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): preserve upstream dependencies after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): fail fast on definitive tracked-write open rejection A tracked write treats every application-level error from the frontend as an ambiguous transport failure and retries the same pinned frontend forever. During a rolling upgrade an older frontend does not recognize the tracked-write command extension and processes Open as an ordinary empty SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal endpoint rejects the extension outright. Both are definitive rejections that no same-frontend retry can ever resolve, so the flow stalls. Classify the Open-phase errors by status code: a definitive rejection (Unsupported / InvalidArguments / PermissionDenied) fails fast with a terminal pre-write error, while genuinely ambiguous transport failures (Unavailable, timeout, connection reset, ...) keep the same-capability retry. Write/Close rejections are deliberately NOT failed fast: once a write may have been admitted, a same-code rejection (e.g. WriterLost, or UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome ambiguous, and failing fast would let the caller replay and duplicate rows. Add a regression test that a legacy empty-SQL Open rejection fails fast with a single attempt and surfaces as Error::External carrying a TerminalError. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): preflight tracked writes without submitting data Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): retain tracked command extensions while decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Lei, HUANG
|
33879b07c8 |
chore: bump rustls to 0.23.46 (#9614)
rustls 0.23.46 fixes rustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout. |
||
|
shuiyisong
|
ad823db020 |
fix(pipeline)!: disable VRL host access by default (#9610)
fix(pipeline): disable VRL host access by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
0dc81777e3 |
fix(pprof): initialize framehop before signal sampling (#9612)
fix(pprof): pin upstream framehop initialization fix Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 jeremyhi
|
60655a933b |
fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases A region's release lived only in memory: a restart after DROP TABLE and before the next object never re-established the dropped region's watermark, so its segments counted as live and pinned garbage collection of the whole prefix. A release now writes a release record (region id and the entry id the region is released through) into an object sealed at once, and returns once that object is durable and indexed, in both acknowledgement modes. Recovery seeds the obsolete watermarks from the release records of the indexed objects before it computes any reclaim boundary. obsolete_all releases in full; the new LogStore::release_region, a no-op by default, releases a dropped region through the largest entry id the store handed out for it, and Mito calls it after obsolete when it drops a region. Objects with release records use format version 2, the version 1 layout with the records at the end of the footer; every other object is still written as version 1. A counter of release records made durable is added. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe persisted object store WAL region releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): carry a failed region release into the next object A release whose batch failed left its record only in an orphan object, so a drop that failed there could still leave the region unreleased after a restart. The store now keeps the release records of failed batches and seals them with its next batch, on a timer tick when no entry is admitted, so the release becomes durable once a create succeeds again. The version 1 and version 2 object fixtures are frozen under tests/compatibility/wal/object_store and loaded by the format tests, and the RFC no longer says that state is persisted only in headers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): complete a region drop whose WAL release fails A drop request is not sent again once the region was removed, so a drop whose WAL release failed used to leave the region half dropped. The drop now completes, cleaning up the region's runtime state and scheduling its files for deletion, and reports the release failure. In the durable mode the object store WAL now creates one more object at stop with the release records of failed and dropped batches, so a graceful stop does not lose a release that failed. The release record counter is described as counting the records of indexed objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe the stop-time release create of the WAL actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): state the accepted window of unreleased dropped regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe release waiters in the WAL actor's outcome table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail region releases while the object store WAL is failing A failing store keeps its backlog and rejects appends until a create succeeds, so a release would otherwise wait for an upload that is not coming. A release that arrives in that state now fails at once with the retryable error, and one whose batch is still pending when the store enters it fails as well. Neither poisons the store: the record of the first waits for the next sealed batch, like that of a batch that failed, and the record of the second stays in the backlog, so both become durable once a create succeeds. Rejected releases count in rejected_appends_total. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe region releases while the object store WAL is failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the drop crash window in the drop procedure The crash window between the deletion of a table's metadata and a durable release record is a gap of the drop procedure, not of the WAL. Describe it as far as the code shows, name the remedy that belongs in the procedure, cleaning up offline a region a resumed region drop finds not open, and list it under future work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep rejected release records coalesced while failing A release rejected while the store is failing kept its record for the next sealed batch, but every timer tick then sealed that record alone behind the refused backlog, so a release repeated across ticks grew the backlog without bound. A failing store now keeps such records coalesced and seals them on the first tick after a create succeeds. The fault-matrix case of a release while the backlog is refused now runs with a 10 ms flush interval, repeats the release, and checks that the backlog does not grow and that recovery reads the record back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): keep rejected release records unsealed while failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): cite the drop procedure issue for the crash window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): let a stop seal rejected release records of a failing store Keeping the records of rejected releases out of sealed batches while the store is failing also kept them out of the seal a stop requests, so a stop whose last attempt of the backlog succeeded dropped them and the released region pinned collection after the restart. A stop now seals them with the backlog. A fault-matrix case rejects a release while the store is failing, stops, lets both creates succeed, and checks the record after a reopen. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the stop of a failing store with rejected releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep release records readable by older object store WAL readers Release records were written as a version 2 footer, which a build without it refuses to read, so rolling a node back after a drop failed to open the whole WAL prefix. They are now the single entry of a segment of the reserved region u64::MAX at the first position of a version 1 object: an older reader indexes that segment as a region it never opens, and recovery reads the segment back with one range read per object that holds one. The segment never holds the reclaim boundary and names no namespace. A durable stop whose last create of release records fails now reports the error instead of returning Ok and losing the records silently. The tests are trimmed: the release unit tests and fault-matrix cases that repeated each other are merged, and the Mito drop tests share one parameterized case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe release records as a segment of a reserved region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): report release records a poisoning stop drops A durable stop whose last create of release records met a conflict poisoned the store and returned Ok, losing the records silently. Stop now reports the error whenever release records are dropped after it began, by a rollback or by poisoning. The test of a failed release covers that conflict, and seals its record on a tick after a one-shot create failure, so the next tick cannot fail too. The documentation now says that recovery reads the segments of release records, and that one that does not decode fails recovery. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how recovery reads segments of release records Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): say that stop reports release records a conflict drops Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): restore the transiently failing release create in enqueued mode The fault matrix again covers a release whose create fails transiently in the enqueued mode: the release waits for the repeat under the same sequence and returns once it succeeds. The encode_batch doc now says the release segment is the last segment of the object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the release segment and count the restored case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL release Returning the release error from the drop left a metric physical region half dropped: the metric engine stops after the data region and never drops the metadata region or forgets the physical region, and a retried drop finds the data region gone and counts it as done. The release error gave the caller nothing to retry, since the region is already removed and the log store keeps the record for its next object. The drop now logs the error and succeeds. A metric engine test drops a physical region while the WAL releases of both mito regions fail, and checks that both regions and the physical region state are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed release no longer fails the drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL obsolete A store that the release of another region poisoned fails obsolete as well, which left the metric engine with its metadata region undropped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): drop a physical region whose WAL obsolete or release fails Cover a failed data release, two failed releases and a conflicting release that poisons the store, after the regions wrote to the WAL, and check that a store opened later keeps no segment of either region. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed obsolete does not fail the drop either Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fail the drop on an obsolete error of other WAL providers Only the object store WAL is shared by the regions of a store that the release of one can poison; Raft Engine and Kafka keep failing the drop when obsolete fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a stopped or poisoned store keeps no failed release Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 dennis zhuang
|
5c4d9eb4f8 |
fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE The bloom fulltext index stored tokens that did not follow matches_term's word boundaries (`_`, ASCII next to Han, Han substrings), so pruning dropped rows matches_term accepts. Tokens and query probes now share the character classes of matches_term, Han runs are indexed as bigrams, and LIKE patterns use the same probes. The bloom blob is written under a new key; blobs from older versions are no longer probed. Tantivy no longer serves matches_term. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: drop unused lazy_static from index and avoid typo-flagged test tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: link like_probes to tokenize instead of the downstream ScriptTokenizer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name probe extraction with verbs and document CharClass Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: rebuild legacy fulltext bloom indexes and probe lower(col) The manifest records the fulltext bloom version per column, so ADMIN BUILD_INDEX now rebuilds SSTs whose bloom index predates the current tokens, and ssts_index_meta shows the version. Case-insensitive bloom indexes also store the tokens of the whole lowercased text for rows where it can differ (U+0130, U+212A, sigma), so matches_term(lower(col), ..) and lower(col) LIKE .. use the index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover legacy bloom index rewrite by compaction in the compat case Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: log outdated fulltext bloom indexes in the consistency check Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: explain the unrecorded fulltext bloom version Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Weny Xu
|
f690bbb204 |
refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): update new WAL replay tests after index removal Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
863eebdecb |
feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve retry policy through error hints Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): restore sparse partition tags when rebuilding writes Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): stop detached write retries when callers cancel Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(frontend): configure retries across distributed write paths Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage. Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
55d753ca58 |
perf(log-store): fetch object store WAL replay segments concurrently (#9555)
* perf(log-store): fetch object store WAL replay segments concurrently A region read fetched each segment only after the previous object's entries were yielded, so a replay paid the full request latency once per object. Fetch up to READ_PREFETCH_DEPTH (8) segments ahead, each on its own task, and handle their outcomes in object sequence order: entries, collected-object skips, corrupted-segment holes and errors surface exactly as before, and dropping the stream aborts the fetches in flight. A read holds at most eight decoded segments besides the one it yields. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): raise object store WAL recovery concurrency to 64 On S3, recovery reads the header and tail of every object, about 57 to 91 ms per object per lane on 4, 8 and 16 vCPUs alike, and its time scales with objects divided by concurrency up to 128 lanes without throttling: about 9,500 objects recovered in 99 s at 8 and 10.5 s at 64. At 64 recovery issues about 1,800 reads per second, a third of S3's request rate per prefix, and holds one 64 KiB tail window per lane, 4 MiB in all. The recovery concurrency tests now scale with the constant. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): share one prefetch byte budget among object store WAL reads Each read fetched up to eight segments ahead on its own, so regions opened in parallel (16 by default) could together hold 128 segments of up to max_batch_bytes each. A read now always fetches the segment it yields next, as before, and fetches a later one only while its length fits in a 64 MiB budget shared by every read of the store. Regions replayed in parallel thus hold one segment each plus at most 64 MiB, however many open at once. Nothing waits on the budget: a read that cannot reserve stops fetching ahead until its next poll. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail an object store WAL read whose fetch task panics A fetch task that panicked was re-raised in the reader with resume_unwind, which unwinds through the caller, such as Mito's region opener task, past its cleanup. Return it as a WalSegmentFetch error instead, so the read ends at that object like any other failed fetch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): bound replay prefetch by decoded size held by fetch tasks The prefetch reservation sat in the read's queue, so dropping a read released it at once while an aborted fetch could still hold its buffers until it next yielded. The fetch task now holds the reservation and returns it with its outcome, so it is released with the buffers. A reservation also counted only the encoded segment length, while a fetched segment is held decoded. It now counts the length plus one Record per entry, and the comment states the whole bound. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): round the measured figures in the recovery concurrency comment Keep the magnitudes in the comment; the exact S3 figures belong to the measurement run rather than the code. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): abort replay fetch tasks with tokio-util's AbortOnDropHandle Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): rename the replay prefetch reservation to try_reserve_prefetch_bytes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): share the bounded polling helper across the object store WAL tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
d86741d0ad |
feat(operator): retry failed writes with refreshed partitions (#9542)
* feat(operator): retry failed writes with refreshed partitions Signed-off-by: WenyXu <wenymedia@gmail.com> * style(operator): simplify retry test type references Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): clarify write request rebuilding names Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): encapsulate write retry attempts Signed-off-by: WenyXu <wenymedia@gmail.com> * style(operator): group inserter configuration methods Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): share routed peers and separate retry decisions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): avoid temporary write routing allocations Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
684c8ff291 |
feat(client): support region request reuse and route refresh hints (#9540)
* feat(client): support shared region requests without deep cloning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate route refresh retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * style: preserve local error variant imports in retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): distinguish staging rejection from write backpressure Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve legacy retry hints in instruction replies Signed-off-by: WenyXu <wenymedia@gmail.com> * test(client): inline retry hint compatibility cases Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): align batch-open retry hint assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * style: use imported types in region request tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
18726c7602 |
fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(snapshot): bound writes and reduce logical export overhead Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: clarify metric snapshot acceptance criteria Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): split batches only for merged readers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(snapshot): cover supported types in both layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: consolidate metric snapshot experiment conclusions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): honor series statistics when splitting L0 scans Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito): explain memory risk of splitting short L0 series Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address snapshot export review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): accept source errors in lazy snapshot contexts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
e6f3ca10a5 |
fix(deps): bump OpenDAL to 0.59.4 for OSS RRSA credentials (#9520)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
fys
|
9a57e1195d |
feat(json2): json_get respect type hints (#9349)
Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
shuiyisong
|
a907cbfbe4 |
perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues for tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: bound asynchronous trace auxiliary writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf: bound trace auxiliary cache by estimated memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: size trace auxiliary cache from host or pod memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add metrics for cache hit and miss Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: unify frontend cache metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: merge and reduce Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add init for the metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 zhaiAohan
|
21536a092b |
build(deps): update opensrv-mysql to fix MariaDB 12.3 USE information_schema hang (#9357)
Update opensrv-mysql from f3135e5 to 1f3d88c, which replies a proper EOF packet to COM_FIELD_LIST instead of a malformed OK packet that MariaDB 12.3 misparses and hangs on USE information_schema. Fixes #9158 Signed-off-by: tian1220A <1573612521@qq.com> |
||
 dependabot[bot]
|
27c1aa76d3 |
chore(deps): bump xxhash-rust from 0.8.15 to 0.8.19 (#9465)
Bumps [xxhash-rust](https://github.com/DoumanAsh/xxhash-rust) from 0.8.15 to 0.8.19. - [Commits](https://github.com/DoumanAsh/xxhash-rust/commits) --- updated-dependencies: - dependency-name: xxhash-rust dependency-version: 0.8.19 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
fys
|
c46fbde9ac |
fix(json2): preserve metadata when adding columns via alter table (#8696)
* fix(json2): support adding columns via alter table Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): scope missing-column null handling to JSON2 --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
 yihong
|
beed56c305 |
chore: update jieba-rs (#9453)
Signed-off-by: yihong0618 <zouzou0208@gmail.com> |
||
|
Ning Sun
|
3e86422ef8 |
chore: bump version and start new 1.4 release cycle (#9403)
chore: start new 1.4 release cycle |
||
|
Ning Sun
|
3131cdbcb6 |
fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests The request-memory accounting only bounds and charges the encoded bytes on the wire, but a compressed body can expand far beyond that during decompression, so tiny requests could allocate disproportionate frontend memory before any protobuf validation or quota charge. - Handler-level decompression (Prometheus remote read/write v1+v2, Loki) now enforces a hard 512 MiB decoded-size cap, checked before any output buffer is allocated, and charges the decoded bytes to the shared ServerMemoryLimiter, holding the permits for the lifetime of the decompressed buffer. - gRPC requests with transport compression reserve the configured max_recv_message_size before tonic decompresses, so the decoding phase is admitted against max_in_flight_write_bytes; the later per-message charge is skipped to avoid double accounting. - The HTTP memory-limit middleware keeps its upfront Content-Length charge but now also accounts the bytes actually streamed beyond it, so chunked requests and understated Content-Length headers no longer bypass the aggregate quota. - Routes that decompress request bodies via RequestDecompressionLayer (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards) now charge the decompressed bytes as handlers consume them: the global middleware marks Content-Encoding requests, and a route-local accounting layer inside the decompression layer charges the decoded stream. Plain requests are skipped to avoid double-counting. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on memory admission guard lifetimes and 429 mapping - Retain the gRPC pre-decode reservation for the whole request: the extensions holding the guard were dropped when the request was consumed (into_inner), releasing the reservation while per-message charges stayed skipped. Both the unary and streaming handlers now clone and hold the reservation marker for the duration of request handling. - Retain the HTTP body permits across the handler: the AccountedBody wrapper and the request extensions are dropped once the extractors finish collecting the body, before the handler is done with the decoded data. Both accounting middlewares now keep their own accounting handle alive across next.run(req).await. - Return a ChargedBuffer from the Loki snappy decompressor so the reservation outlives the decompressed bytes through the caller's protobuf decoding, matching the Prometheus path. - Map mid-stream quota exhaustion to 429 instead of the generic body error (400): the accounting flags exhaustion and the middlewares rewrite the extractor rejection, so clients can distinguish backpressure from malformed input. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve the in-flight body acquisition before trying a new one A parked acquisition in AccountedBody::charge always belongs to the currently buffered frame, so it must be resolved before any new acquisition is tried. Letting the fast-path try_acquire succeed while a waiter is parked left the waiter alive, and its late completion would then be credited with a later frame's byte count, under-reserving memory relative to what was marked charged. Adds a regression test that reproduces the misattribution: with the buggy ordering the test delivers a body the quota cannot cover; with the fix the over-quota frame waits for its own acquisition and times out. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Weny Xu
|
133296f298 |
chore: bump version to 1.3.0-beta.1 (#9392)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
3c0e2a8d55 |
feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject empty packed export time ranges Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover cancellation during packed export I/O Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
  
|
75bd8e9ce6 |
feat: add HDFS object storage backend (#8701)
* feat: add HDFS object storage backend Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * fix: make HDFS storage operations durable Gate the native HDFS backend behind an explicit feature. Publish writes through same-directory temporary files and atomic HDFS Rename2 replacement, and provide streaming copy fallback for COPY_REGION. Add regression coverage for interrupted writes and the region-copy path. Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * ci: run HDFS object store tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: note HDFS temporary file cleanup follow-up Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: enable HDFS object storage by default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: remove redundant HDFS build feature notes Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: Minghan2005 <cambrianocean@gmail.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Minghan2005 <cambrianocean@gmail.com> Co-authored-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
ffdd6d09a6 |
perf(index): cut allocations when building bloom and inverted indexes (#9359)
* perf(index): build bloom filters from element hashes without per-token allocation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): speed up inverted index building with hashed buffers and sync pushes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): use BuildHasher::hash_one for element hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): require callers to act on spill requests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): insert bloom hashes one by one to keep segment set capacity bounded Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): add index build and bloom search benchmarks Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): keep applier setup out of the bloom search benchmark Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): skip empty spills and keep the old inverted sort memory estimate Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): stream fulltext token hashes and test spill dispatch Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): note non-ASCII tokens still allocate in analyze_text_hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): narrow the allocation note to case-insensitive non-ASCII tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
4ffd93d8dc |
fix(query): preserve global limits with DataFusion optimizer fixes (#9071)
* fix(query): preserve global fetch through physical-optimizer repartitioning Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch) below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then dropped or rewrote that fetched coalesce, losing the global limit. Root-caused and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo pins to that fix commit and adds regression coverage. - pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix) - global_limit.rs: adapt to new DF API (input_distribution_requirements, child_distribution, replace_children+Recompute); accept HashPartitioned and KeyPartitioned as partitioning to restore; add regression unit tests - sqlness: extend standalone common ssts and refresh distributed ssts_limit result for the preserved fetch plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: drop dead HashPartitioned compat arms and test wrapper - remove HashPartitioned compatibility matches (pinned DF only emits KeyPartitioned); drop their #[expect(deprecated)] attributes - remove single-use agg_with_limit helper whose seed limit is always overwritten by the soft-limit transform - restore Cargo.lock dependency edges unrelated to the DataFusion pin bump (cargo update --precise had downgraded 26 unrelated packages) Addresses oracle ablation review. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(deps): pin DataFusion to merged fetch preservation fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Yingwen
|
3c2aac0a55 |
feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(storage): route series index build requests Signed-off-by: evenyag <realevenyag@gmail.com> * feat(admin): add BUILD_SERIES_INDEX Signed-off-by: evenyag <realevenyag@gmail.com> * test: specify compaction type in series index fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * test: correct series index SQL fixtures and error assertions Signed-off-by: evenyag <realevenyag@gmail.com> * test(compat): preserve legacy index rebuild across upgrades Signed-off-by: evenyag <realevenyag@gmail.com> * test(sql): cover series index admin validation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): bound series index maintenance queue Signed-off-by: evenyag <realevenyag@gmail.com> * docs: remove series index how-to guide Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove index build upgrade compatibility case Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address series index reconciliation review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * fix: bound manual series index reconciliation admission Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto to merged index build options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
b9f991502c |
refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
d910183cc4 |
feat(flow): freeze recovery windows and retention bounds for incremental flows (#9312)
* feat(flow): discover recovery windows from exact sequence reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(flow): freeze explicit recovery windows and retention bounds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): validate retained source windows before recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): qualify recovery source tables before serialization Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
fys
|
045441e3cc |
feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Lei, HUANG
|
b5199bc59a |
feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: gate Jev filtering behind an opt-in Cargo feature Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: verify Jev registration with default features Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: clarify Jev concurrency scope and stabilization work Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: enable renamed ai-functions feature by default Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: remove extra AI feature-off checks Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: rename AI feature to ai_functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
19c127d9c3 |
feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: restore packed metric snapshots Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: stream large packed parquet entries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: allow default S3 endpoint in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: skip unconfigured S3 in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use portable file URLs in packed restore fixtures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate packed snapshot structure before restore Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve strict manifest decoding and verify fixture Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject packed layout on both database export paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: implement file size in coordinator test storage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use expect_err for rejected export layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
263e229103 |
feat: add experimental Metric export to V2 snapshots (#9233)
* feat: add experimental Metric export to V2 snapshots Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate the complete Metric export capability response Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: construct portable file URLs for Metric export fixtures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address Metric export review nits Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
6a084d9a4f |
fix: release completed SST write buffers during compaction (#9243)
* fix: upgrade OpenDAL and honor SST write buffer size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: track removal of the OpenDAL bridge fork Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): honor SST write buffers across upload paths Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 shuiyisongandLei, HUANG
|
66d38e8e1c |
feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add `InsertLimitInterceptor` hook to `Inserter` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: attribute insert limit checks to the target table's database Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: unify write admission through metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce write admission for pending row batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: distinguish internal requests for ingestion metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: admit split ingestion requests once per database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: OpenTSDB throws error reason Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use meter crate main rev Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude database ingest rate limit from table options Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: reserve channel 255 for internal requests Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
b332bb67d6 |
perf(telemetry): remove the shared lock from TraceLayer (#9269)
* perf(telemetry): remove the shared lock from TraceLayer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(telemetry): stop collecting span data while tracing is disabled Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
6647dbc437 |
feat!: upgrade DataFusion fork to 55.1.0 (#9177)
Rebase the GreptimeDB DataFusion fork from official 55.0.0 to 55.1.0. DataFusion 55.1.0 is a patch release on branch-55 containing eleven cherry-picked fixes (schema-adaptation struct filters, cast/projection metadata propagation, nested-nullability aggregation adaptation, UnnestExec batch_size, RightMark join ordering panic, empty-struct ScalarValue, and FFI codec fixes). All twenty GreptimeDB fork patches rebase onto it with no textual or semantic overlap; none of them is absorbed upstream, so all are retained. Fork pin moves to discord9/datafusion branch greptimedb-55.1.0, commit 2aa87d52cdce7006af492330064738f33ed294c1 (55.1.0 + 20 patches). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |