mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-29 07:06:20 +00:00
88cb2a986ef0776a1eb84b0df9581b3e7ffb189d
6458
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dragon Roar
|
88cb2a986e |
fix: reclaim HDFS temporary files left behind by a datanode crash (#9616)
* fix: reclaim HDFS temporary files left behind by a datanode crash Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): stop the temporary file sweep from starving GC jobs Signed-off-by: lczllx <2181719471@qq.com> * feat(mito2): add metrics for the temporary file sweep Signed-off-by: lczllx <2181719471@qq.com> * refactor(object-store): share one definition of the temporary file name Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): skip the temporary file sweep when no GC job is allowed Signed-off-by: lczllx <2181719471@qq.com> * refactor: gate the temporary file sweep behind the hdfs-object-store feature Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): sweep temporary files when a create opens an existing region Signed-off-by: lczllx <2181719471@qq.com> * docs(config): document the temporary file sweep in the GC settings Signed-off-by: lczllx <2181719471@qq.com> * refactor: align the temporary file sweep code with the surrounding style Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
  Dhruv VaishnavandWeny Xu
|
2e37e19af4 |
feat(datanode): expose ingest task panic and cancellation metrics (#9830)
* feat(datanode): record ingest task join failures Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(datanode): remove extra ingest task metric tests Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
 Ning Sun
|
80df1c397b |
feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support * feat: add postgres copy-in framework * feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher Follow-up to #9302: begin_copy_in now uses the batching-aware new_query_context, so copy-in batches are submitted to the pending-rows batcher like SQL INSERTs when PostgreSQL batching is enabled for the connection, and keep the direct write path when it is not. The batcher acknowledges synchronously, so the CommandComplete row count stays exact in both modes. Adds test_postgres_copy_in_with_batching asserting the batcher flush metric increments only when batching is enabled. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options The WITH (...) option list is now open instead of restricted to sqlparser's COPY grammar: - COPY ... FROM STDIN statements are parsed textually (new copy_in::parse module, tokenizer-based) so arbitrary KEY value entries reach the codecs; sqlparser would reject options it does not know before we ever see them. WITH is optional, matching PostgreSQL. - CopyOptionSet becomes an open ordered map of KEY -> String | Number | Ident | Boolean | Flag with typed take_* accessors (take_string, take_single_byte, take_flag, raw take_value) and case-insensitive keys; duplicate options are rejected. - Each format codec takes the options it understands and rejects the leftovers with one message naming them ('COPY option(s) X are not supported by format Y'), so a typo cannot be silently ignored. The text format now also rejects HEADER/QUOTE/ESCAPE instead of ignoring them, matching PostgreSQL. A future format (e.g. influxdb line protocol with WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec and a registry entry; no parser changes. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing Replace the hand-rolled CSV record parser and per-value conversion with arrow-csv's incremental Decoder — the same engine backing COPY ... FROM 'file.csv' in common-datasource — so both COPY routes accept identical data. The decoder handles record splitting (quoting, escaped quotes, custom delimiters, CRLF, headers, chunk boundaries) and typed conversion against the targeted columns' schema in one pass; decoded batches become insert rows via the existing api::helper::vectors_to_rows and datatypes Helper::try_into_vectors, and the decode schema comes from the greptime Schema's arrow projection. The codec framework now has parsers emit complete insertable rows (api Row) instead of raw byte fields: create_parser takes the targeted columns and is fallible; text converts with the shared text-representation helpers, csv with the arrow decoder. Documented consequences of sharing the engine: JSON and Binary columns and bare epoch-integer timestamps are rejected for csv, and an empty field — quoted or not — is NULL; a custom NULL string is rejected accordingly. Signed-off-by: Ning Sun <sunning@greptime.com> * docs(postgres): make the copy-in framework explicitly format-open Reframe the codec framework docs so nothing assumes csv is the only other text format: the shared pieces (Field, convert_text_field, check_text_representable_type) are described as building blocks for any format whose records carry values as text, while formats with their own typed decoding (csv today) build rows themselves. State explicitly that formats are not limited to PostgreSQL's own — GreptimeDB-specific formats (e.g. a line protocol) register like any other and define their own options through the open WITH clause. No code changes beyond a test rename and a doc clarifier. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): driver-level coverage for CSV copy-in options Add an end-to-end test through tokio-postgres exercising HEADER, custom DELIMITER, quoted fields with embedded newlines, escaped quotes and CRLF record separators, with read-back verification through the real insert path. WITH is omitted, matching PostgreSQL. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): drop the raw-socket copy-in tests The simple-protocol raw client duplicated what tokio-postgres and the integration tests already cover; keep the driver-based tests only. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode - text feed: take the longest complete line prefix in one drain and iterate lines over the slice, instead of draining per line (each drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of short lines) - csv check_column_type: delegate to the shared check_text_representable_type and additionally reject Binary/Json - decode_bytea: use the hex crate instead of a hand-rolled decoder - semantic_type_of: use ColumnSchema::is_time_index() like the flow adapter's column_schemas_to_proto, instead of index lookups - parse: drop the unused expect_ident parameter; document the identifier-case decision - PostgresServer:🆕 document that query_handler and copy_in_handler are the same frontend instance Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove unrelated files * feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format Remove the hand-rolled PostgreSQL text-format codec (backslash escapes, octal, \N nulls) that arrow-csv cannot express, keeping arrow-csv-based csv as the only implemented format. - COPY FROM STDIN without FORMAT now fails with guidance ('requires an explicit format in this version; use WITH (FORMAT csv)') rather than silently defaulting to the removed format - FORMAT text/txt and FORMAT binary report 'not supported yet' via registry stubs instead of unknown-format errors - codec.rs shrinks to the trait framework: the text-representation helpers (Field, convert_text_field, bytea hex, timestamp parsing) are removed with the codec; the module docs note where to find their shape should a future text-serializing format need them - csv implements check_column_type directly (no shared default) Breaking for clients relying on the PostgreSQL default text format; explicit WITH (FORMAT csv) is required. Signed-off-by: Ning Sun <sunning@greptime.com> * perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics - COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher each flush pays the full insert pipeline (catalog lookup, schema check, partition split, region write round trip), so 4x larger batches cut round trips 4x at linearly-bounded memory; the constant also sizes the arrow decoder batch. Comment documents the trade-off and points high-rate users at the pending-rows batcher, which turns flushes into enqueues. - rows buffer pre-allocates COPY_IN_FLUSH_ROWS. - flush_copy_in warns on unexpected output variants instead of silently counting 0 rows. - batching wire test feeds 70000 rows to stay above the new threshold. Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv) sends FORMAT csv and succeeds; default \copy (text) gets the 'requires an explicit format' guidance; COPY TO STDOUT round-trips. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): reset copy-in state on successful CopyDone on_copy_done left the CopyInState in place (inert but holding buffers) until the next COPY overwrote it; take it like on_copy_fail does, so buffers are freed eagerly and a stray post-done CopyData gets the protocol-violation error instead of a finished parser. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass typo check * chore: format change * fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries The csv feed loop flushed the arrow-csv decoder at every CopyData chunk end — including partial batches at record boundaries — but arrow-csv's contract permits flush() only after decode() returned 0 (a full batch of complete records). Flushing outside that point corrupts subsequent decoding: fields mis-parse shifted by the size of the record prefix buffered so far ('000001,host-001,...' parses as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse, silently writing wrong values. Which chunks trigger it depends on where record boundaries land, so it appears intermittent with random-length data (psql 8KB / pgx ~64KB chunking both hit it). Adopt the strict decoder protocol: feed decodes until input is exhausted and flushes only when decode returns 0 (batch full); trailing partial batches surface at the EOF signal in finish(). With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is unchanged; sub-threshold streams now emit their rows at CopyDone instead of per chunk, which is semantically identical. Adds test_csv_chunked_feeds_are_alignment_safe: the same payload fed at chunk sizes 1..65531 (plus shifted starts) with every row's content verified — reproduces the corruption on the old loop at chunk sizes 7, 8, 13, 64 and 1000. Verified with the reporter's Go reproducer (pgx, content-checked): 10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the previously failing benchmark sizes 500/1000/2000/200/800 all round- trip exactly. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass spell check * chore: bypass spell check * fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter Three review findings (shuiyisong, PR #9297): [P1] Enforce the catalog restriction on COPY targets. copy_in_table now applies validate_catalog_and_schema when QueryOptions:: disallow_cross_catalog_query is set, mirroring SQL INSERT's check_permission path, BEFORE begin_copy_in switches the query context to the target catalog (which previously made the restriction unenforceable downstream). Cross-catalog COPY is now rejected with 'Illegal access to catalog' like INSERT. [P2] Canonicalize unquoted identifiers like the SQL parser (ParserContext::canonicalize_identifier): unquoted table-name parts and columns fold to lowercase, quoted ones keep their case. COPY Metrics (Host, Ts) now addresses the table CREATE TABLE Metrics created, and no longer silently misroutes into a coexisting quoted "Metrics". [P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal tab while still rejecting newlines and other control characters, and the option parser now also accepts E'...' escape-string constants (DELIMITER E'\t'), processing PostgreSQL's backslash escapes. Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round- trip into the right tables, cross-catalog rejection asserted in a new frontend unit test (QueryOptions plugin + second catalog), and both DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure Two review findings (shuiyisong, PR #9297): [P2] Boolean columns round-trip between the COPY directions. arrow-csv only parses true/false while PostgreSQL COPY OUT emits t/f, so boolean tables failed to re-import their own export (22P02). Boolean columns now decode as Utf8 in the arrow schema and flush_batch converts them with PostgreSQL's boolean input grammar (true/false, t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact bytes are fed back via COPY FROM STDIN and verified. [P2] Drop the COPY state on every server-side failure. on_copy_data and on_copy_done now take ownership of the state up front, so parse, conversion or insert errors drop the buffered data instead of retaining it on long-lived or pooled connections until the next COPY or connection close (the reviewer measured ~17 MiB retained after a failed 8 MiB row); on_copy_data success puts the state back. An e2e test drives a failed COPY followed by a successful one on the same connection. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): interpret offset-free COPY timestamps in the session timezone Review finding (shuiyisong, PR #9297): the arrow decode schema carried Timestamp fields with no timezone, so the CSV decoder parsed offset-free timestamp strings as UTC while SQL INSERT parses them in the session timezone — the same input string silently stored a different instant depending on the ingestion method (8h apart in Asia/Shanghai). CopyInCodec::create_parser now receives the session timezone and the csv decode schema attaches it to Timestamp fields, so naive strings parse like INSERT; explicit offsets in the input (Z, +08:00) still take precedence, and UTC sessions are unchanged. Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z) and the end-to-end test_postgres_copy_in_session_timezone_parity, which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the same string, and asserts both CAST(ts AS BIGINT) readbacks agree on 1704038400000. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): don't unescape E-string COPY options twice Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already decodes PostgreSQL escape-string constants before returning Token::EscapedStringLiteral, so the second unescape_pg_string pass altered the decoded value — DELIMITER E'\\t' (a literal backslash followed by t, two characters) was turned into a tab and wrongly accepted as a single-byte delimiter. Use the EscapedStringLiteral payload as-is, like SingleQuotedString, and delete unescape_pg_string. Regression coverage distinguishes the three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t' -> rejected as a multi-character delimiter. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN Two review findings (shuiyisong, PR #9297): [P2] Preserve timestamp instants across COPY OUT/IN in non-UTC sessions. COPY OUT rendered naive UTC text while COPY IN (since the session-tz fix) interprets offset-free strings as local time, so re-importing an export shifted instants by the zone offset. COPY OUT now renders timestamps with an explicit session-timezone offset (2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the session tz (arrow-pg converts to a tz-aware DateTime) and the pg field type becomes TIMESTAMPTZ so the offset-bearing text format applies. Embedded offsets win on re-import regardless of session, keeping the OUT -> IN round trip exact; UTC sessions are unchanged. [P2] Preserve quoted empty strings. The shared arrow-csv decoder nulls every empty field, so a quoted empty string ("" — an empty STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner runs over the same bytes with the decoder's own splitting rules and records per field whether it was quoted; flush_batch restores quoted empties to empty strings for string columns, while unquoted empty fields stay NULL. Masks track records across chunk boundaries and align with decoder batches. Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip (Asia/Shanghai export asserted to carry +08, exact-epoch re-import comparison) and test_postgres_copy_in_quoted_empty_string (NULL, quoted empty and non-empty in one import), plus unit tests including a chunk-boundary sweep for the quoted-empty mask alignment. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns Two review findings (fengjiachun, PR #9297): [P2] Doubled quotes desynchronized the shadow quote-mask scanner. In QuoteInQuoted the doubled-quote branch skipped the second quote AND the following byte, so a closing quote or delimiter directly after an escaped quote was swallowed and later fields lost their masks -- import semantics depended on CopyData chunk boundaries. The branch now consumes only the escaped quote and returns to the in-quoted state, mirroring the decoder. Regression: "a""b","" ... verified across chunk sizes 1-64. [P2] A NOT NULL string column rejected quoted empty strings before restoration could run: the decode schema preserved the target nullability, and arrow-csv's batch validation rejects the temporary NULL a quoted empty decodes to. String fields now decode as nullable (safe: quoted-vs-unquoted is exactly the restoration's job), and the target NOT NULL constraint is enforced in flush_batch after restoration, so "" imports as an empty string while an unquoted empty field fails with a not-null-constraint error. Regressions at unit and e2e level: NOT NULL column accepting "" and rejecting unquoted NULL. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators Two review findings (fengjiachun, PR #9297): [P2] Blank lines emitted a spurious empty mask. csv-core skips blank lines, but the scanner queued one mask per blank record, so every subsequent decoded row consumed the wrong mask (a leading blank line turned a quoted empty into NULL and an unquoted empty into an empty string, and tripped NOT NULL on the wrong row). A blank line before the header also shifted the header expectation. end_record now returns no mask for a completely blank record without consuming the header expectation. [P2] Standalone CR was not a record terminator. The decoder accepts CR, CRLF and LF alike; the scanner only recognized CR before LF or at EOF, so CR-separated input desynchronized masks and flush_batch failed with 'quote masks out of sync'. is_record_end now treats CR alone as a terminator, consuming one byte (two for CRLF), while a CR at a chunk boundary still defers via carry so CRLF split across CopyData chunks stays one terminator. Regressions: blank lines before/inside records and before a header (NULL/empty preservation plus NOT NULL behavior), and mixed CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing CR-only record. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY The shadow quote-mask scanner — kept to preserve PostgreSQL's quoted-empty-string vs unquoted-NULL distinction — mirrored the decoder's record-splitting grammar by hand and required a fix for every rule it drifted on (doubled quotes, blank lines, CR terminators, NOT NULL interplay). Per maintainers' decision, that complexity is not worth the divergence: COPY FROM STDIN now behaves EXACTLY like the file-based COPY ... FROM 'file.csv' — an empty field, quoted or not, imports as NULL (arrow-csv's engine semantics, identical in both routes). Removed: QuoteMaskScanner and helpers, the always-nullable string decode schema, the restoration + NOT NULL enforcement in flush_batch, set_string_at, and the scanner-era unit tests. The decode schema again preserves target nullability, so the decoder enforces NOT NULL itself, same as the file route. Kept (small, schema-level, no second grammar): boolean spellings (t/f) and session-timezone timestamp decoding, both addressing separate review findings. E2E: test_postgres_copy_in_empty_field_is_null pins the parity — quoted empty, unquoted empty and non-empty in one import. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove empty file * chore: lint issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 jeremyhi
|
1c5d15fd36 |
perf(log-store): read object store WAL footers while the recovery listing goes on (#9889)
* perf(log-store): read object store WAL footers while the recovery listing goes on Recovery listed the whole prefix before it read any footer, so its LIST pages, about 80-120 ms each on S3 and one per 1000 objects, added to the footer reads. The listing now runs in a task of its own and feeds the footer reads as its pages arrive; a listing that fails part-way still fails recovery, and a failed footer read stops the listing. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): fail object store WAL recovery as soon as its listing fails A listing that failed part-way was reported only after every object it had already listed was read, which on S3 can be tens of thousands of reads. The stream now settles a finished listing before it yields another object, so its failure goes first. Tests cover a large listed backlog and a listing task that panics; the counting and parking test object accesses forward list_ahead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): hold the listing until the first footer reads wait in the backlog test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): check a failed listing settles before buffered objects with a finished task Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): bound the objects the recovery listing holds ahead of the reads The listing fed the footer reads through an unbounded channel, so a listing much faster than the reads could hold most of the prefix in memory. It now holds at most 4096 objects, four LIST pages of S3, and waits for the reads while that many are queued. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): poll the recovery listing directly to show it waits on a full backlog Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): check the bound with the channel the recovery listing uses Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
f2be11a69e |
fix: upload staging manifests in chunks (#9919)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 jeremyhiandYingwen
|
1cb01d0d44 |
fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898)
* fix(mito2): keep SSTs when a manifest save ends with RateLimited The object store's retry layer retries temporary errors and returns the last attempt's error. A manifest delta PUT can be stored by the server while the client sees a timeout, reset or 5xx, and the retries that follow can be throttled, so the save ends with RateLimited even though the delta exists. may_have_persisted_manifest_update treated RateLimited as proof that nothing was written, so flush and compaction deleted the SSTs the stored delta references. A later open or follower sync replays the delta and reads fail on the missing files. Treat RateLimited like other ambiguous errors: the SSTs are kept and become orphans when the save really failed. Fixes #9897 Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep SSTs when a manifest save ends with PermissionDenied The retry layer returns the last attempt's error, so a 403 from a revoked key, a changed bucket policy or clock skew can follow a delta PUT that the server already stored. Like RateLimited, PermissionDenied does not prove that nothing was written, so it no longer lets flush or compaction delete the SSTs; they become orphans when the save really failed. The compaction test that rejects a manifest write before persistence now injects Unsupported, which is raised before any request is sent. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
 shuiyisong
|
afbe77e31f |
perf: avoid duplicate OTLP histogram conversion (#9880)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
815da8a6ef |
feat(mito): expose scan concurrency metrics and dashboard (#9758)
* feat(mito): expose planned scan concurrency histogram Signed-off-by: evenyag <realevenyag@gmail.com> * feat(grafana): visualize planned scan concurrency Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): distinguish query and compaction scan metrics Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
5523cf3874 |
feat(meta): track per-node scheduling state and fence region placement (#9884)
feat: track per-node scheduling state and fence region placement
Add a cluster-wide, persistent "can this node receive new regions" flag and
make every metasrv placement path respect it. This is the fence half of
`ADMIN drain_node`; the drain procedure, gRPC entry and region evacuation land
separately.
State layout
- New namespace `__node_scheduling/{datanode,flownode}/{node_id}` holding
`{"schedulable": false}`. A missing key means schedulable, so undrain is a
delete and no stale `schedulable: true` records are left behind.
- `NodeSchedulingKey`/`NodeSchedulingValue` mirror the existing KV conventions
(`NodeInfoKey`'s `{role, node_id}` shape, `LeaseValue`'s `*Value` suffix).
The role is part of the prefix, so extending to flownodes needs no key change.
- `NodeSchedulingManager` writes with `put_if_not_exists`/`compare_and_put`, so
concurrent marks change the state exactly once, and reads a single prefix
range per placement decision (DDL/failover only, not per query).
Placement
- `NodeInfo` gains `#[serde(skip)] schedulable: Option<bool>` plus
`is_schedulable()`; `None` (unresolved) counts as schedulable, and the field
never reaches heartbeat bytes, so the wire/in-memory format is unchanged.
- `SchedulingAwarePeerDiscovery` stamps the flag on `active_datanodes`. It is
additive (never hides nodes) because failover still needs the full active set
to compute its exclusion list.
- All three built-in selectors now fetch candidates through
`selector::common::active_schedulable_datanodes`; `RegionSupervisor` adds
unschedulable datanodes to `exclude_peer_ids` for both selector variants.
Enforcing the single fence
- `MetaPeerClient` no longer implements `PeerDiscovery`; its discovery methods
are inherent, and `DatabaseOperator` (frontends only) takes the concrete
client. `SchedulingAwarePeerDiscovery`, built once in `MetasrvBuilder`, is
therefore the only way to obtain a `PeerDiscoveryRef`, so bypassing the fence
is a compile error rather than a silent misplacement.
Compatibility
- New key namespace only; `NodeInfo` serialization is byte-identical, so old
metasrv nodes and persisted heartbeats keep working. The flag itself is stored
in the durable KV, so a follower metasrv sees marks written by the leader.
Verification
- `cargo nextest run -p meta-srv node_scheduling` (13), `selector` (12),
`supervisor` (14), `-p common-meta cluster` (9), `-p meta-client` (36),
`-p standalone` (compiles, no tests).
- `cargo clippy -p meta-srv -p common-meta -p meta-client -p standalone
--all-targets -- -D warnings`, `cargo fmt --all -- --check`.
Signed-off-by: WenyXu <wenymedia@gmail.com>
|
||
|
Weny Xu
|
567849908e |
fix(ci): specify Kind storage class for datanodes (#9908)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
shuiyisong
|
c4043efc27 |
fix(mito2): require flat SST output for JSON2 (#9893)
* fix(mito2): require flat SST output for JSON2 Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito2): avoid contradictory JSON2 recovery warnings Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 LFC
|
6a34078706 |
fix(pipeline): accept auto-transformed values in existing JSON columns (#9895)
* fix(pipeline): accept auto-transformed values in existing JSON columns Signed-off-by: luofucong <luofc@foxmail.com> * test(pipeline): compare decoded JSON values in regression test Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 liyang
|
3483f4a284 |
fix(ci): move monitoring datanodeStorage out of standalone base (#9902)
The `datanodeStorage` field belongs directly under `monitoring.standalone` (GreptimeDBStandaloneSpec), not under `monitoring.standalone.base` (PodTemplateSpec). Nesting it under `base` renders an invalid GreptimeDBCluster resource and the CRD schema rejects it during server-side apply: .spec.monitoring.standalone.base.datanodeStorage: field not declared in schema |
||
 dennis zhuang
|
00ab09012d |
ci: fix CI setup steps that fail on Ubuntu 26.04 runners (#9901)
* ci: install protoc with taiki-e/install-action Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * ci: run multi-language SDK tests on Python 3.13 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * ci: drop the unused libfuzzer-14-dev package from fuzz builds Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * ci: build fuzz targets with the bundled libFuzzer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
  NightG1owZandNightG1owZ
|
b886ae525d |
feat(config): support overriding region engine options from env (#9700)
* feat(config): support overriding region engine options from env `region_engine` is a `Vec` of externally tagged enums, so plain dotted environment keys cannot address it: `config` walks the path, replaces the default sequence with a table and loading fails with `invalid type: map, expected a sequence`. Address the entries by tag name instead (`<PREFIX>__REGION_ENGINE__<TAG>__<FIELD>`) and merge the overrides onto the default entries in a dedicated source, so the engines and fields that are not mentioned keep their configured value. An entry the default sequence does not have is appended; a numeric index, an override of the whole sequence, a field below a non-table default and two variables that set a path and a field below it are reported with a dedicated error instead of a serde type error. A configuration file that configures the sequence as a whole still replaces the merged value, which the example configs document. Closes #8620 Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * docs(config): clarify that the metric engine is always enabled `region_engine` defaults to [mito, file], but the datanode builds and registers the metric engine unconditionally. Reword the example configs so `mito` and `file` read as the default configuration entries and naming `metric` only supplies its options, then regenerate config.md. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * docs(config): drop the env-var explanation from the config reference Environment variables are documented in the GreptimeTeam/docs repository, so the example configs and the generated config reference only describe the config file. Keep the note that the metric engine is always registered. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(config): cover tagged sequence environment collection Add unit tests for TaggedSeqEnvSource::collect: merging by tag, appending unknown tags, empty and null defaults, flat key removal, the error paths and the take_entries and is_index_like helpers. The variables are injected through Environment::source, so the tests do not touch the process environment. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> --------- Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> |
||
|
shuiyisong
|
2eff52c757 |
feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): guard point accounting and benchmark metrics ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove otlp_metric_compat logic Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): cache physical schema lookups during metric preflight Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: minor rename Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): batch metric rows and reuse schema validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: replace super in import path Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Lei, HUANG
|
bb3f07b482 |
fix(mito2): pick TWCS L1 merges by closed units so compaction terminates (#9741)
* refactor(mito2): pick TWCS L1 merges by closed units TWCS could keep rewriting L1 files without new writes. Overlap-only picks could split their inputs into one more file than they read, and L1 file reduction then rewrote up to 16 files to remove that single file. The outputs, split by key, created new overlaps and the cycle repeated. Replace the three global pick phases with one pick per window, scheduled by class across windows. Every pick now decreases a termination measure when the window receives no writes. - Active windows only take merges that remove files. L1 files are grouped by conflicts and merged in key order, so size-split tails are merged together. A reduction rewrites at most 4 output files' worth of bytes per removed file. - Inactive windows absorb L0 files, then take closed merges: no other file intrudes on the inputs' bounding box, so outputs cannot add overlaps. Each merge resolves conflicts or removes files, decreasing (conflicts, files). Files that do not take part in an overlap are not rewritten. - Overlaps that only share a boundary key no longer count as conflicts, so a size-split sorted run does not look overlapping. - Closed merges may take 4x the input limit to resolve a whole window once. - Cache primary key ranges per pick instead of resolving them per comparison. - Log the pick branch of each compaction output. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): state the compactor assumptions behind TWCS termination Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): resolve active window overlaps on manual compaction A window that stops receiving writes keeps being the active window, whose picks only remove files, so its overlaps were never resolved. Manual compactions now pick every window as inactive, so a closed merge can resolve them. Automatic compactions are unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): defer overlap resolution in the window before the active one Late data keeps landing in the window right before the active window. A closed merge that resolves its overlaps early rewrites the whole window, then the late data brings the overlaps back, and the balanced run check keeps them there. Overlap-only merges now skip that window until it is older. File-reducing merges and manual compactions are unchanged. On the 12h/1GB out-of-order bench this reaches overlap depth 1.0 instead of 2.08 at the same write amplification (3.01 vs 3.02). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): only check earlier-starting files when closing units Closing units only checks files that start before a unit ends, and the overlap checks compare key bounds without cloning them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep settling windows from rewriting L1 files The window before the active one still receives late data. Skipping only its overlap-only merges was not enough: file-reducing merges of conflicting files and mixed L0/L1 merges still rewrote its L1 files, and a mixed merge could rewrite a nearly full output for each small late file. A settling window now only merges L0 files among themselves and conflict-free files; the window resolves its overlaps once it is older. On the out-of-order bench, write amplification drops from 3.88 to 3.30 at 2GB/6h, 3.05 to 2.94 at 1GB/6h and 3.01 to 2.96 at 1GB/12h, with overlap depth 1.0 in all three. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve late overlaps once a window is quiet Late data that lands in a window older than the settling one is merged into a small file covering the whole key range. Resolving it rewrites the window's dominant sorted run, which the balanced run check refuses, so the overlap stayed. Removing the check instead rewrites the window for every late flush. A window is quiet once the region has written at least as many rows as the window holds since its newest file. Overlap merges in a quiet window may rewrite a dominant run, so each quiet period pays for at most one such rewrite. Sequences are per row and kept by compaction, so this needs no state. On the bench with late data up to 2 hours, mean overlap depth drops from 1.67 to 1.17 at 2GB/6h and from 2.33 to 1.83 at 1GB/6h, at write amplification 3.70 to 4.25 and 3.26 to 3.74; all inactive windows of 2GB/6h realtime reach depth 1 (write amplification 2.80 to 3.30). The remaining depth is in the final active window. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): check closed merges of quiet and settling windows terminate Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): split TWCS closures over the input limit A file spanning the keys of a large sorted run joins every run file into one closure. Once a closure held more files than one compaction accepts, closed merges skipped it, so the window never merged those files again and late files kept piling up. Split such closures into single files, like active-window picks do, and replace the closed check with a conflict bound: outputs only conflict with files intruding on the inputs' box, so a merge is taken if `outputs · I` does not exceed the conflicts its inputs had with other files. Intervals of whole closures have no such conflicts and still need to be closed, so other windows pick as before, and `(conflicts, files)` still strictly decreases. Small run files and late files over a large run can be merged again. A late file over a run that cannot shrink stays: resolving it would rewrite the whole run. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): address TWCS picker review findings - Manual compactions may rewrite a dominant sorted run to resolve overlaps. The active window is never quiet, so manual compactions could not resolve its late-data overlaps before. - Single-key intervals predict one output: the writer never splits a series, so large files of one series were never merged. - The active window falls back to single files when no group of conflicting files qualifies, so a large file no longer keeps its small peers unmerged. - Compute the settling window distance with checked_sub, which overflowed for windows at the timestamp limits. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): count inner keys as TWCS conflicts and check balance per group - A single key strictly inside another file's key range now conflicts, so a late file of one series inside a wide file is resolved like other overlaps. Files sharing only a boundary key still do not conflict. Intrusion covers the new pairs, which a randomized test checks. - Check the dominant-run guard per group of conflicting files. Runs of separate groups never overlap, so pairing one group's run with another group's peer made an interval of two dominant runs look balanced. - A window with unknown row counts, written before they were recorded, is not quiet: its threshold would drop to a single row. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): index TWCS files by key to avoid quadratic grouping Grouping a window's files compared every file with all files sharing its time range, and closing groups and checking intrusions scanned the whole window. A sorted run split by key shares one time range, so a window of N such files took O(N²) checks without any conflict. The 1024-file fallback only covers the sorted-run path, not these groups. Index the files by lower primary key in a segment tree that keeps each subtree's largest upper key and its time bounds. Grouping, closing, and the conflict bound only check the files whose key and time ranges meet, so a file of a key-sliced run only meets its neighbors. Conflicts among split files are precomputed per file and counted per interval with a binary search instead of pairwise. On a run of 20,000 key-sliced files with 5 late files spanning all keys, a closed merge pick takes ~62 ms instead of ~3.9 s and an active pick ~19 ms instead of ~3.5 s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(object-store): retry transient S3 transport failures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound TWCS closed-merge conflict graph work Budget the candidate files visited while building conflict units and split-unit adjacency. When the budget is exceeded, treat every file as a split unit and pick a bounded file reduction instead of materializing a quadratic graph. The fallback candidate is admitted only through adds_no_conflicts, so outputs do not add conflict pairs and the (P, N) termination invariant still holds. Dense picks are logged with their own branch. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): charge TWCS index nodes to the conflict-graph budget The conflict-graph work budget only counted files that reached a callback, so a traversal with poor pruning could still visit a quadratic number of internal nodes without triggering the dense fallback. Charge every ExtentIndex node, including pruned nodes, and stop the traversal as soon as the shared budget is exhausted. Add a regression test where the root straddles the query time but neither leaf matches, so a callback-only budget would never fire. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): retry smaller TWCS dense compaction candidates If a dense fallback candidate fails admission, try one single-output subinterval within it before giving up. Reuse the reduction guards and check both candidates against the full window's external conflicts, preserving bounded work and termination.\n\nCover the chain regression through both fallback unit constructions and reject one-output retries that introduce conflicts across time gaps. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): compact shared primary-key boundaries Count shared primary-key endpoints as conflicts and apply the same overlap rule to bounding-box intrusions. Existing compaction writers keep series intact, so fresh output ranges remain disjoint. Align simulated output splits with the writer and cover boundary-only cleanup in inactive and manual windows, plus stable real output files after a subsequent TWCS request. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): compact small L1 files in inactive windows Allow bounded small-L1 reductions in settling and older inactive windows when normal picks cannot proceed. Preserve full-window conflict checks, large-file protection, and LastNonNull closure semantics.\n\nCover selection limits, termination, and real SST compaction with tombstones and reopen checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify TWCS unit grouping guarantees Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fys
|
f35057d9c2 |
feat(json2): support inverted index (#9324)
feat: build inverted indexes for JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Weny Xu
|
4aa4aa63fe |
fix(test): avoid MinIO connection reuse after conditional writes (#9877)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
af52dfafd8 |
chore(deps): upgrade sqlx to 0.9 to pick up webpki-roots 1.0.10 (#9862)
sqlx 0.8 pins the unmaintained webpki-roots 0.26 line for its rustls trust store, which still contains the recently compromised e-Tugra CA root. Upgrading sqlx to 0.9 switches its dependency to webpki-roots 1.x, and the lockfile now resolves to 1.0.10 where the distrusted root has been removed from the compiled-in Mozilla trust store. sqlx 0.9 requires dynamic SQL strings to be explicitly audited via the new SqlSafeStr trait: only &'static str literals are accepted as-is. All affected call sites (MySQL KV backend, MySQL election, integration tests and fuzz targets) now wrap their internally generated SQL templates in AssertSqlSafe; the wrapped SQL is built from operator config or test-local identifiers only, and dynamic data keeps flowing through bind parameters. The mysql-rsa feature is enabled alongside mysql to preserve 0.8 authentication behavior: sqlx 0.9 gates non-TLS RSA password auth (caching_sha2_password / sha256_password) behind it. Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Yingwen
|
8b1d68369b |
fix(ci): align nightly Windows and macOS artifact test stack sizes (#9860)
* fix(ci): increase Windows nightly test stack size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(ci): align optional macOS artifact test stack size Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
b4aae2f73b |
fix(meta): release all PostgreSQL election advisory locks on step-down (#9864)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
f595e9bbf0 |
ci: allow independent Linux release architecture selection (#9865)
* ci: allow independent Linux release architecture selection Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: use Linux architecture checkboxes and preserve release defaults Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
34210f5022 |
feat(mito): add an environment switch to disable compaction (#9863)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
LFC
|
67bd279586 |
fix(query): preserve Arrow types when decoding dictionary output (#9858)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Weny Xu
|
9e9346d2ff |
fix: preserve datanode write batches when retrying (#9720)
* fix: preserve datanode write batches when retrying Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: isolate invalid entries in metric write batches Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: adapt batch writes to the unified metric API Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: explicitly accept partial results for batched retries Signed-off-by: WenyXu <wenymedia@gmail.com> * test: assert terminal retry hint instead of masked error text Signed-off-by: WenyXu <wenymedia@gmail.com> * test: preserve application status in region failure fixtures Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: refresh routes after region close Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
436d07bf2d |
fix(mito): return from manual index builds only after the index is published (#9643)
* fix(mito): return from manual index builds only after the index is published A manual build_index could report success while the index was missing: the request was rejected or coalesced behind another build, the manifest commit failed, the indexer dropped an error, or the region was staging. Manual builds now wait for active builds of the same SST, keep their waiters across coalescing and schema-change retries, are notified after the worker applies the edit, and verify the files against the manifest before returning. The consistency check also honors inverted_index.ignore_column_ids and a zero segment row count, so such SSTs are no longer rebuilt on every call. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito): cover manual index build waiting for a schema-change retry Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito): select manual index build files by the applied version Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito): log why index builds are skipped or aborted Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(mito): split IndexBuildTask::index_build into phases Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito): stop scheduling index builds when the region state changes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito): document region file id and silent aborts in index builds Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito): log why a manual index build is incomplete Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito): expose incomplete index builds and treat compacted files as done Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito): pin index build concurrency in the compaction test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>v1.4.0-nightly-20261022 |
||
|
Weny Xu
|
27c531336a |
fix(mito2): restore source splitting for cross-range series merges (#9818)
* fix: restore batched series merges across ranges Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): split range sources before last-row selection Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): avoid tiny splits for single-source cross-range merges Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
ec4ef45f40 |
feat: use pgwire auth mechanisms API to simplify startuphandler (#9634)
* feat: use pgwire auth mechanisms API to simplify startuphandler * chore: update pgwire * chore: update pgwire revision to merged one * refactor: model based on upstream api * refactor: simplify implementation by introducing authsource * refactor: update to latest pgwire apis * refactor: adopt pgwire api changes * chore: switch to released pgwire * test: harden pg scram mixed-auth integration test and reject channel-binding clients - Fix MixedAuthProvider to return a SCRAM verifier for the denied user, so the test exercises rejection by authorization after successful SCRAM verification; record the selected mechanism from the returned variant and assert authorize ran exactly once per verified user. - Assert the denied connection fails with 28P01 before any terminal message, protecting the completion-ordering introduced by pgwire's authentication gate. - Add a raw-protocol test proving a client-first with a p= channel binding flag is rejected (RFC 5802 section 6), while n-flag clients (tokio_postgres) continue normally. Bump pgwire to 0.42.1, which enforces RFC 5802 channel binding negotiation and rejects short/overlong SCRAM proofs on the verifier path (sunng87/pgwire#476). Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Weny Xu
|
0dc649e028 |
fix(datanode): release region map guard before awaiting role changes (#9824)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
LFC
|
5af1e3301e |
fix(mysql): cap timestamp output at microsecond precision (#9831)
* fix(mysql): cap timestamp output at microsecond precision Truncate MySQL text timestamp results to microseconds while preserving the existing second and millisecond formatting. Keep stored nanoseconds and explicit integer/string representations unchanged. Fixes #9828 Signed-off-by: luofucong <luofc@foxmail.com> * chore: ignore local output directory Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 Chenglong Zheng
|
9f9b35c270 |
feat: support wait and timeout options for TRUNCATE TABLE (#9753)
* feat: support wait and timeout options for TRUNCATE TABLE * fix: address review feedback for truncate DDL options * fix: address review feedback for truncate DDL options |
||
|
Yingwen
|
e6fa1a0b21 |
fix(ci): increase release integration test stack size (#9817)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
f866ef2bd9 |
feat: return per-entry region write failures (#9728)
* feat: return per-entry region write failures Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve retry hints for single-entry region writes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve region write error compatibility Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify region partial result handling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align partial failure summaries with input order Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate region write response handling Signed-off-by: WenyXu <wenymedia@gmail.com> * test: fix partial result expectations and region map guard Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: limit lockfile changes to greptime proto Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: make partial region results an explicit client opt-in Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin greptime-proto to merged partial-result changes Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
a1b2362e58 |
fix: avoid duplicate keys in update flow metadata txn (#9816)
CREATE OR REPLACE FLOW deleted and then put the same flow route, flownode flow and table flow keys in a single txn when the replacement kept the partitions, flownodes or source tables, which etcd rejects with "duplicate key given in txn request". Only delete the old keys that are not put again. Closes #9810. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
LFC
|
9a2ac7f9de |
feat: support dot access for JSON columns (#9775)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
 Aum
|
6f53d57d15 |
fix: avoid exact scan row counts for SSTs with unknown counts (#9759)
fix: avoid exact scan row counts for legacy SSTs Treat selected SSTs with unknown row counts as non-exact before checking region partition visibility. Preserve exact statistics for known counts and cover legacy metadata mixed with known-count files. Fixes GreptimeTeam/greptimedb#9718 Signed-off-by: Aumnertic <hznuzc@gmail.com> |
||
|
Ning Sun
|
a7230f7a92 |
fix(partition): coerce partition exprs against the batch schema before routing (#9752)
* fix(partition): coerce partition exprs against the batch schema before routing Batch partition routing (MultiDimPartitionRule::split_record_batch) builds physical expressions via PartitionExpr::try_as_physical_expr, which did no type coercion. After widening the time index unit (e.g. ms -> ns via ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition bounds keep their original unit, so evaluating the bounds against the batch failed with: Invalid comparison operation: Timestamp(ns) < Timestamp(ms) Row inserts were unaffected because find_region compares Values and Timestamp::cmp converts units; scans were unaffected because PredicateGroup::new coerces the region partition expr against the region schema first. Coerce the logical partition expression against the evaluation schema in try_as_physical_expr before creating the physical expression, the same way PredicateGroup does. This also covers the region partition filter built for SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this conversion. Fixes #9747 Signed-off-by: Ning Sun <sunning@greptime.com> * test: regenerate change_col_type result after comment update The ts_widen_part comment in change_col_type.sql was reworded after the result file was last generated; sqlness echoes SQL comments into the result, so the stale comment made the case fail. Regenerate it. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
dennis zhuang
|
b9aa627180 |
fix(query): forward scalar subquery filters to region scans (#9738)
* fix(query): forward scalar subquery filters to region scans Since DataFusion 55, uncorrelated scalar subqueries run in ScalarSubqueryExec instead of being rewritten into joins, so filters such as ts = (SELECT ts ... ORDER BY ts DESC LIMIT 1) no longer reach region scans through a join dynamic filter. MergeScanExec now captures such filters as remote dynamic filters, keeps them in the parent FilterExec, and replaces the subquery with its result before querying regions. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): check the remote dyn filter budget across pushdown phases Filters captured in the Pre and Post phases are sent in one request, but each phase validated the initial registration budget alone. When their sum exceeded the budget, serialization dropped every registration, including join and TopK filters. Validate the combined set and keep filters accepted by the other phase. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): keep IN-bearing scalar subquery filters at the parent Oversized remote dyn filter payloads replace IN lists with true. That only widens a conjunction of bounds, but inside an arbitrary predicate such as (host IN (...)) = (SELECT flag ...) it can drop rows the parent filter would keep. Do not forward scalar subquery filters that contain IN lists. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(query): note the IN fallback restriction for scalar subquery filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(query): document the scalar subquery dyn filter bridge removal condition Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(query): keep only dyn_filters of region scans in the scalar subquery plan case #9565 added scan diagnostics to EXPLAIN ANALYZE VERBOSE and partition filter rendering changed; the case only checks the forwarded dyn filter. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
977e76d0a4 |
fix(mito2): apply SST internal column encodings to bulk-encoded parts (#9735)
* fix(mito2): apply SST internal column encodings to bulk-encoded parts Bulk memtable parts are encoded to parquet and flushed as SSTs as-is, but the bulk encoder did not apply the time index / __sequence DELTA_BINARY_PACKED encoding (with dictionary disabled) and the uncompressed op_type setting that the regular SST writer uses. Share these settings through one helper. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): drop the answered bulk encoder compression TODO Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
41ec5e0763 |
fix: decouple slow-query logging from the global log level (#9765)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
c51293836e |
fix(query): push LIKE filters down into MergeScan remote plans (#9648)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
3e2751d71d |
feat: improve SQL analyze scan diagnostics (#9565)
* feat: improve SQL analyze scan diagnostics Signed-off-by: evenyag <realevenyag@gmail.com> * test: redact dynamic filters in sqlness results Signed-off-by: evenyag <realevenyag@gmail.com> * test: stabilize analyze snapshots for merge queue Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove obsolete scan_details assertion Signed-off-by: evenyag <realevenyag@gmail.com> * test: refresh explain snapshots after partition filter changes Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
4645ab9ed7 |
perf(mito2): gather last rows with one take per input batch (#9737)
* perf(mito2): gather last rows with one take per input batch FlatLastTimestampSelector buffered a single-row slice per series and concatenated up to 8192 of them. Concatenating slices with dictionary columns merges the dictionaries once per slice, which made high-cardinality instant PromQL queries using the LastRow scan path (#9034) slower. Select the rows of series that end inside a batch with one take instead. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito2): cover duplicate last timestamps in the take path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): group last rows by primary key bytes Flat batches may assign different dictionary keys to the same primary key, for example when binary keys are wrapped with identity keys. Merge adjacent ranges with equal key bytes before selecting last rows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
9c7e1e1605 |
fix: record slow INSERT SELECT queries (#9733)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
jeremyhi
|
a436213bf3 |
ci: compare several images in one object store WAL S3 benchmark run (#9691)
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
|
||
|
jeremyhi
|
93c3b116da |
fix(mito2): re-check a filled write as sent after a schema change (#9701)
* fix(mito2): re-check a filled write as sent after a schema change The engine fills the columns a write leaves out with their defaults when it accepts the write, appending them to the rows without adding them to `name_to_index` and `has_null`. If the region schema changes before the worker handles the write, the worker checks the request again and `check_schema` looks up a filled column in `name_to_index`, which panics and stops the region worker. If the filled column was dropped, the write fails with an unknown column that the client never sent. Remove the filled columns before checking, so the request is checked as sent and filled again for the current schema. Nothing is filled yet when the engine checks the request, so that check does no extra work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): rename put_request helper to build_put_request Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
da88ff8653 |
fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697)
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep equality two-valued for NULL rows and accept unsigned -0 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): treat NULL masks as false in batch routing and keep comparisons bare Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(compat): update partition filter in analyze verbose expected output Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
7397ab6b23 |
perf(datanode): send the mito puts of one request to each worker at once (#9692)
* perf(datanode): send the mito puts of one request to each worker at once The region server handled the puts of a request to several regions as separate requests. A Mito worker woke on the first of them and wrote it alone; in the durable object store WAL mode its append then blocked the worker until the next seal, so the rest of the request went into the next object, and clients writing a partitioned table locked into one phase that wastes a tick. When every put of a request goes to a ready mito region, the region server now hands them to `MitoEngine::put_regions_batch`, which submits the puts of each worker in one message, so the worker writes its regions of the request in one batch and one WAL append. Requests to one region and to other engines are handled as before. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): await each region of a batched mito put on its own task The batched path waited for the regions of a request in order and sent the worker messages one after another, so an error or a full worker queue of one region held back the others, and its metrics counted the request as a whole. `MitoEngine::put_regions_batch` now sends the worker messages concurrently on their own task and returns each region's result. The region server awaits and accounts every region on its own ingest task, as the per-region path does: the first error returns at once, and each region records its latency, changed rows or failure and is named in its error. A request with several puts to one region keeps the per-region path. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): spawn the accounting of batched mito puts from the batch task The request spawned the per-region accounting tasks only after the batch task returned, so a request dropped in between left accepted writes unaccounted. The batch task now spawns them and the request only joins their handles. Also state in the architecture invariants that the region server's batched puts to the metric and mito engines are the one request dispatch that does not go through the RegionEngine trait. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch mito puts through the RegionEngine trait Add `RegionEngine::handle_batch_put_requests`, which returns each region's response of the puts of one request. The default handles each put with `handle_request`, so other engines keep their behaviour; Mito overrides it with the one message per worker submission. The region server calls the trait method instead of downcasting to `MitoEngine`, and the architecture invariants no longer need an exception. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): trace each region of a batched mito put Attach the per-region `RegionServer::handle_region_request` span of the per-region path to each region's task of a batched mito put, so the region ids stay in distributed traces. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): let a batch put response answer for several regions `RegionEngine::handle_batch_put_requests` now returns each response with the regions it answers for. The default and Mito answer each region on its own; an engine that writes several puts together, such as the metric engine's batched puts, can answer them with one response. The region server accounts a response for each of its regions. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch any request with several mito puts Take the batch path for a request with at least two puts to ready mito regions, also when they all target one region: a request of many puts to one region otherwise still sends one worker message per put. Each put keeps its own response, affected rows and error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): account batched mito puts in one task Await the responses of a batched mito put in the batch task with a FuturesUnordered instead of one spawned task per put. The task accounts each response as it completes, hands the first error back to the caller at once and keeps accounting the rest, so a request of many puts no longer spawns a task per put. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): name the batched worker message BatchWrite Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): submit batched writes on the ingest runtime Grouping and submitting the writes of a request is part of ingestion, so the task runs on the ingest runtime and takes the write task class when the workload scheduler is enabled. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): share ingest result accounting across put paths The per-region path, the metric batch path and the mito batch path each recorded the changed rows or the failures of an ingest response. They now call one helper, so a metric or label change happens in one place. A batch put response lists the region of each put it answers for, so the failure count stays per put when two puts go to one region; the trait documentation now says so. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * fix(mito2): give batched puts of a stopped worker the worker's error A batched write message that could not reach a stopped worker was dropped, so its puts failed with "Failed to recv result" while a put submitted on its own fails with "Worker <id> is stopped". The worker now returns the message it could not take, and each of its puts receives the WorkerStopped error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): look up the batch latency histogram once The label of the request latency histogram is the same for every put of a batch, so the batch task looks it up once instead of per observation. Signed-off-by: jeremyhi <jiachun_feng@proton.me> --------- Signed-off-by: jeremyhi <jiachun_feng@proton.me> |
||
 fengyue-xve
|
1e29e8d4b6 |
fix(meta): validate persisted partition bounds before altering regions (#9487)
* fix(meta): validate persisted partition bounds before altering regions
A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.
Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.
Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.
Fixes #9454
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix: access ColumnSchema fields directly and use plain format args
ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: access ColumnSchema data_type field in alter table tests
Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop borrows now that data_type is read as a field
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop useless format! in nested expression test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: cover the partition tail in change_col_type sqlness case
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop spurious blank line in change_col_type result
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): compare partition bounds with the scan comparison coercion
The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.
Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: update the sqlness fixture for the new incompatibility message
The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency
common-meta gained a datafusion-expr dependency in
|