mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-29 23:16:57 +00:00
6f5c064f543eedfa65d312bebe64f33870c3ca60
145
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ning Sun
|
80df1c397b |
feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support * feat: add postgres copy-in framework * feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher Follow-up to #9302: begin_copy_in now uses the batching-aware new_query_context, so copy-in batches are submitted to the pending-rows batcher like SQL INSERTs when PostgreSQL batching is enabled for the connection, and keep the direct write path when it is not. The batcher acknowledges synchronously, so the CommandComplete row count stays exact in both modes. Adds test_postgres_copy_in_with_batching asserting the batcher flush metric increments only when batching is enabled. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options The WITH (...) option list is now open instead of restricted to sqlparser's COPY grammar: - COPY ... FROM STDIN statements are parsed textually (new copy_in::parse module, tokenizer-based) so arbitrary KEY value entries reach the codecs; sqlparser would reject options it does not know before we ever see them. WITH is optional, matching PostgreSQL. - CopyOptionSet becomes an open ordered map of KEY -> String | Number | Ident | Boolean | Flag with typed take_* accessors (take_string, take_single_byte, take_flag, raw take_value) and case-insensitive keys; duplicate options are rejected. - Each format codec takes the options it understands and rejects the leftovers with one message naming them ('COPY option(s) X are not supported by format Y'), so a typo cannot be silently ignored. The text format now also rejects HEADER/QUOTE/ESCAPE instead of ignoring them, matching PostgreSQL. A future format (e.g. influxdb line protocol with WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec and a registry entry; no parser changes. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing Replace the hand-rolled CSV record parser and per-value conversion with arrow-csv's incremental Decoder — the same engine backing COPY ... FROM 'file.csv' in common-datasource — so both COPY routes accept identical data. The decoder handles record splitting (quoting, escaped quotes, custom delimiters, CRLF, headers, chunk boundaries) and typed conversion against the targeted columns' schema in one pass; decoded batches become insert rows via the existing api::helper::vectors_to_rows and datatypes Helper::try_into_vectors, and the decode schema comes from the greptime Schema's arrow projection. The codec framework now has parsers emit complete insertable rows (api Row) instead of raw byte fields: create_parser takes the targeted columns and is fallible; text converts with the shared text-representation helpers, csv with the arrow decoder. Documented consequences of sharing the engine: JSON and Binary columns and bare epoch-integer timestamps are rejected for csv, and an empty field — quoted or not — is NULL; a custom NULL string is rejected accordingly. Signed-off-by: Ning Sun <sunning@greptime.com> * docs(postgres): make the copy-in framework explicitly format-open Reframe the codec framework docs so nothing assumes csv is the only other text format: the shared pieces (Field, convert_text_field, check_text_representable_type) are described as building blocks for any format whose records carry values as text, while formats with their own typed decoding (csv today) build rows themselves. State explicitly that formats are not limited to PostgreSQL's own — GreptimeDB-specific formats (e.g. a line protocol) register like any other and define their own options through the open WITH clause. No code changes beyond a test rename and a doc clarifier. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): driver-level coverage for CSV copy-in options Add an end-to-end test through tokio-postgres exercising HEADER, custom DELIMITER, quoted fields with embedded newlines, escaped quotes and CRLF record separators, with read-back verification through the real insert path. WITH is omitted, matching PostgreSQL. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): drop the raw-socket copy-in tests The simple-protocol raw client duplicated what tokio-postgres and the integration tests already cover; keep the driver-based tests only. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode - text feed: take the longest complete line prefix in one drain and iterate lines over the slice, instead of draining per line (each drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of short lines) - csv check_column_type: delegate to the shared check_text_representable_type and additionally reject Binary/Json - decode_bytea: use the hex crate instead of a hand-rolled decoder - semantic_type_of: use ColumnSchema::is_time_index() like the flow adapter's column_schemas_to_proto, instead of index lookups - parse: drop the unused expect_ident parameter; document the identifier-case decision - PostgresServer:🆕 document that query_handler and copy_in_handler are the same frontend instance Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove unrelated files * feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format Remove the hand-rolled PostgreSQL text-format codec (backslash escapes, octal, \N nulls) that arrow-csv cannot express, keeping arrow-csv-based csv as the only implemented format. - COPY FROM STDIN without FORMAT now fails with guidance ('requires an explicit format in this version; use WITH (FORMAT csv)') rather than silently defaulting to the removed format - FORMAT text/txt and FORMAT binary report 'not supported yet' via registry stubs instead of unknown-format errors - codec.rs shrinks to the trait framework: the text-representation helpers (Field, convert_text_field, bytea hex, timestamp parsing) are removed with the codec; the module docs note where to find their shape should a future text-serializing format need them - csv implements check_column_type directly (no shared default) Breaking for clients relying on the PostgreSQL default text format; explicit WITH (FORMAT csv) is required. Signed-off-by: Ning Sun <sunning@greptime.com> * perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics - COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher each flush pays the full insert pipeline (catalog lookup, schema check, partition split, region write round trip), so 4x larger batches cut round trips 4x at linearly-bounded memory; the constant also sizes the arrow decoder batch. Comment documents the trade-off and points high-rate users at the pending-rows batcher, which turns flushes into enqueues. - rows buffer pre-allocates COPY_IN_FLUSH_ROWS. - flush_copy_in warns on unexpected output variants instead of silently counting 0 rows. - batching wire test feeds 70000 rows to stay above the new threshold. Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv) sends FORMAT csv and succeeds; default \copy (text) gets the 'requires an explicit format' guidance; COPY TO STDOUT round-trips. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): reset copy-in state on successful CopyDone on_copy_done left the CopyInState in place (inert but holding buffers) until the next COPY overwrote it; take it like on_copy_fail does, so buffers are freed eagerly and a stray post-done CopyData gets the protocol-violation error instead of a finished parser. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass typo check * chore: format change * fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries The csv feed loop flushed the arrow-csv decoder at every CopyData chunk end — including partial batches at record boundaries — but arrow-csv's contract permits flush() only after decode() returned 0 (a full batch of complete records). Flushing outside that point corrupts subsequent decoding: fields mis-parse shifted by the size of the record prefix buffered so far ('000001,host-001,...' parses as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse, silently writing wrong values. Which chunks trigger it depends on where record boundaries land, so it appears intermittent with random-length data (psql 8KB / pgx ~64KB chunking both hit it). Adopt the strict decoder protocol: feed decodes until input is exhausted and flushes only when decode returns 0 (batch full); trailing partial batches surface at the EOF signal in finish(). With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is unchanged; sub-threshold streams now emit their rows at CopyDone instead of per chunk, which is semantically identical. Adds test_csv_chunked_feeds_are_alignment_safe: the same payload fed at chunk sizes 1..65531 (plus shifted starts) with every row's content verified — reproduces the corruption on the old loop at chunk sizes 7, 8, 13, 64 and 1000. Verified with the reporter's Go reproducer (pgx, content-checked): 10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the previously failing benchmark sizes 500/1000/2000/200/800 all round- trip exactly. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass spell check * chore: bypass spell check * fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter Three review findings (shuiyisong, PR #9297): [P1] Enforce the catalog restriction on COPY targets. copy_in_table now applies validate_catalog_and_schema when QueryOptions:: disallow_cross_catalog_query is set, mirroring SQL INSERT's check_permission path, BEFORE begin_copy_in switches the query context to the target catalog (which previously made the restriction unenforceable downstream). Cross-catalog COPY is now rejected with 'Illegal access to catalog' like INSERT. [P2] Canonicalize unquoted identifiers like the SQL parser (ParserContext::canonicalize_identifier): unquoted table-name parts and columns fold to lowercase, quoted ones keep their case. COPY Metrics (Host, Ts) now addresses the table CREATE TABLE Metrics created, and no longer silently misroutes into a coexisting quoted "Metrics". [P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal tab while still rejecting newlines and other control characters, and the option parser now also accepts E'...' escape-string constants (DELIMITER E'\t'), processing PostgreSQL's backslash escapes. Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round- trip into the right tables, cross-catalog rejection asserted in a new frontend unit test (QueryOptions plugin + second catalog), and both DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure Two review findings (shuiyisong, PR #9297): [P2] Boolean columns round-trip between the COPY directions. arrow-csv only parses true/false while PostgreSQL COPY OUT emits t/f, so boolean tables failed to re-import their own export (22P02). Boolean columns now decode as Utf8 in the arrow schema and flush_batch converts them with PostgreSQL's boolean input grammar (true/false, t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact bytes are fed back via COPY FROM STDIN and verified. [P2] Drop the COPY state on every server-side failure. on_copy_data and on_copy_done now take ownership of the state up front, so parse, conversion or insert errors drop the buffered data instead of retaining it on long-lived or pooled connections until the next COPY or connection close (the reviewer measured ~17 MiB retained after a failed 8 MiB row); on_copy_data success puts the state back. An e2e test drives a failed COPY followed by a successful one on the same connection. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): interpret offset-free COPY timestamps in the session timezone Review finding (shuiyisong, PR #9297): the arrow decode schema carried Timestamp fields with no timezone, so the CSV decoder parsed offset-free timestamp strings as UTC while SQL INSERT parses them in the session timezone — the same input string silently stored a different instant depending on the ingestion method (8h apart in Asia/Shanghai). CopyInCodec::create_parser now receives the session timezone and the csv decode schema attaches it to Timestamp fields, so naive strings parse like INSERT; explicit offsets in the input (Z, +08:00) still take precedence, and UTC sessions are unchanged. Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z) and the end-to-end test_postgres_copy_in_session_timezone_parity, which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the same string, and asserts both CAST(ts AS BIGINT) readbacks agree on 1704038400000. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): don't unescape E-string COPY options twice Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already decodes PostgreSQL escape-string constants before returning Token::EscapedStringLiteral, so the second unescape_pg_string pass altered the decoded value — DELIMITER E'\\t' (a literal backslash followed by t, two characters) was turned into a tab and wrongly accepted as a single-byte delimiter. Use the EscapedStringLiteral payload as-is, like SingleQuotedString, and delete unescape_pg_string. Regression coverage distinguishes the three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t' -> rejected as a multi-character delimiter. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN Two review findings (shuiyisong, PR #9297): [P2] Preserve timestamp instants across COPY OUT/IN in non-UTC sessions. COPY OUT rendered naive UTC text while COPY IN (since the session-tz fix) interprets offset-free strings as local time, so re-importing an export shifted instants by the zone offset. COPY OUT now renders timestamps with an explicit session-timezone offset (2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the session tz (arrow-pg converts to a tz-aware DateTime) and the pg field type becomes TIMESTAMPTZ so the offset-bearing text format applies. Embedded offsets win on re-import regardless of session, keeping the OUT -> IN round trip exact; UTC sessions are unchanged. [P2] Preserve quoted empty strings. The shared arrow-csv decoder nulls every empty field, so a quoted empty string ("" — an empty STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner runs over the same bytes with the decoder's own splitting rules and records per field whether it was quoted; flush_batch restores quoted empties to empty strings for string columns, while unquoted empty fields stay NULL. Masks track records across chunk boundaries and align with decoder batches. Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip (Asia/Shanghai export asserted to carry +08, exact-epoch re-import comparison) and test_postgres_copy_in_quoted_empty_string (NULL, quoted empty and non-empty in one import), plus unit tests including a chunk-boundary sweep for the quoted-empty mask alignment. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns Two review findings (fengjiachun, PR #9297): [P2] Doubled quotes desynchronized the shadow quote-mask scanner. In QuoteInQuoted the doubled-quote branch skipped the second quote AND the following byte, so a closing quote or delimiter directly after an escaped quote was swallowed and later fields lost their masks -- import semantics depended on CopyData chunk boundaries. The branch now consumes only the escaped quote and returns to the in-quoted state, mirroring the decoder. Regression: "a""b","" ... verified across chunk sizes 1-64. [P2] A NOT NULL string column rejected quoted empty strings before restoration could run: the decode schema preserved the target nullability, and arrow-csv's batch validation rejects the temporary NULL a quoted empty decodes to. String fields now decode as nullable (safe: quoted-vs-unquoted is exactly the restoration's job), and the target NOT NULL constraint is enforced in flush_batch after restoration, so "" imports as an empty string while an unquoted empty field fails with a not-null-constraint error. Regressions at unit and e2e level: NOT NULL column accepting "" and rejecting unquoted NULL. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators Two review findings (fengjiachun, PR #9297): [P2] Blank lines emitted a spurious empty mask. csv-core skips blank lines, but the scanner queued one mask per blank record, so every subsequent decoded row consumed the wrong mask (a leading blank line turned a quoted empty into NULL and an unquoted empty into an empty string, and tripped NOT NULL on the wrong row). A blank line before the header also shifted the header expectation. end_record now returns no mask for a completely blank record without consuming the header expectation. [P2] Standalone CR was not a record terminator. The decoder accepts CR, CRLF and LF alike; the scanner only recognized CR before LF or at EOF, so CR-separated input desynchronized masks and flush_batch failed with 'quote masks out of sync'. is_record_end now treats CR alone as a terminator, consuming one byte (two for CRLF), while a CR at a chunk boundary still defers via carry so CRLF split across CopyData chunks stays one terminator. Regressions: blank lines before/inside records and before a header (NULL/empty preservation plus NOT NULL behavior), and mixed CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing CR-only record. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY The shadow quote-mask scanner — kept to preserve PostgreSQL's quoted-empty-string vs unquoted-NULL distinction — mirrored the decoder's record-splitting grammar by hand and required a fix for every rule it drifted on (doubled quotes, blank lines, CR terminators, NOT NULL interplay). Per maintainers' decision, that complexity is not worth the divergence: COPY FROM STDIN now behaves EXACTLY like the file-based COPY ... FROM 'file.csv' — an empty field, quoted or not, imports as NULL (arrow-csv's engine semantics, identical in both routes). Removed: QuoteMaskScanner and helpers, the always-nullable string decode schema, the restoration + NOT NULL enforcement in flush_batch, set_string_at, and the scanner-era unit tests. The decode schema again preserves target nullability, so the decoder enforces NOT NULL itself, same as the file route. Kept (small, schema-level, no second grammar): boolean spellings (t/f) and session-timezone timestamp decoding, both addressing separate review findings. E2E: test_postgres_copy_in_empty_field_is_null pins the parity — quoted empty, unquoted empty and non-empty in one import. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove empty file * chore: lint issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Ning Sun
|
40770ddb72 |
test: only assert genuinely invalid read-preference values at request ingresses (#9576)
The flow and nested-MergeScan ingress tests hardcoded "follower" among the read-preference values that must be rejected, coupling them to the current leader-only ReadPreference enum. Assert only genuinely invalid values there, matching the wire-values test which already limits itself to ["", "unknown"]. This keeps the ingress tests valid for builds that extend ReadPreference with more variants. Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 discord9
|
0e40281262 |
fix(promql): stop claiming scalar calculate preserves order (#9523)
* test: cover scalar calculate order contract Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): stop claiming scalar calculate preserves order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(test): describe scalar ordering regression as pre-fix behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
75bc4ce46d |
fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): only claim projected primary key prefix ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): keep physical key ordering for metric projections Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): compare unknown partition counts explicitly Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): model scoped internal TSID scan properties Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
2039f508fc |
fix(promql): stop fabricating histogram fold hash partitioning (#9509)
* fix(promql): stop fabricating histogram fold hash partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover histogram fraction binary join missing rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): normalize dictionary labels in histogram regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover histogram joins on logical metric tables Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: specify insert columns for logical metric fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use supported histogram child replacement API Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): keep histogram fold fixtures truthful Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
483d1e49a3 |
fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep filter fetch when rewriting matches term Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
32e8467cda |
fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): use current DataFusion plan downcasts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): match null rendering in join regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh union plan for unknown scan partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh shared plans after distribution rule removal Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover semi joins with remote dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen partitioned merge scan regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove unused merge scan test helpers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): measure merge scan hash repartition cost Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt merge scan deserialization to two-arg constructor Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): adapt nested merge scan integration constructor Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
 jeremyhi
|
f9ab3c296b |
feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region A segment that does not decode, because its checksum does not match or its content disagrees with its footer entry, is fetched once more. If it still does not decode, the read follows `on_corrupted_segment`: `skip` records a WAL hole for the region with the object key, sequence and the footer's entry id range, increments `greptime_logstore_object_store_wal_skipped_segments_total`, logs a warning and continues with the next object; `fail` fails the read. A failed fetch fails the read in both modes. Holes are kept in memory per region and can be listed from the store. The standalone test builder allocates WAL options through `build_standalone_wal_provider`, and integration tests run a standalone instance on the object store WAL across restarts, on S3 when the `GT_S3_*` variables are set. The standalone shutdown runs every step and returns the first error. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cmd): cover the best-effort standalone stop Build a standalone instance whose leader services fail to stop and whose datanode servers fail afterwards, and check that `Instance::stop` still stops the object store WAL and returns the leader services error. Rename `skip_segment` to `record_skipped_segment` and the file store integration test to `test_standalone_object_store_wal_survives_restarts_on_file`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Ning Sun
|
3131cdbcb6 |
fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests The request-memory accounting only bounds and charges the encoded bytes on the wire, but a compressed body can expand far beyond that during decompression, so tiny requests could allocate disproportionate frontend memory before any protobuf validation or quota charge. - Handler-level decompression (Prometheus remote read/write v1+v2, Loki) now enforces a hard 512 MiB decoded-size cap, checked before any output buffer is allocated, and charges the decoded bytes to the shared ServerMemoryLimiter, holding the permits for the lifetime of the decompressed buffer. - gRPC requests with transport compression reserve the configured max_recv_message_size before tonic decompresses, so the decoding phase is admitted against max_in_flight_write_bytes; the later per-message charge is skipped to avoid double accounting. - The HTTP memory-limit middleware keeps its upfront Content-Length charge but now also accounts the bytes actually streamed beyond it, so chunked requests and understated Content-Length headers no longer bypass the aggregate quota. - Routes that decompress request bodies via RequestDecompressionLayer (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards) now charge the decompressed bytes as handlers consume them: the global middleware marks Content-Encoding requests, and a route-local accounting layer inside the decompression layer charges the decoded stream. Plain requests are skipped to avoid double-counting. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on memory admission guard lifetimes and 429 mapping - Retain the gRPC pre-decode reservation for the whole request: the extensions holding the guard were dropped when the request was consumed (into_inner), releasing the reservation while per-message charges stayed skipped. Both the unary and streaming handlers now clone and hold the reservation marker for the duration of request handling. - Retain the HTTP body permits across the handler: the AccountedBody wrapper and the request extensions are dropped once the extractors finish collecting the body, before the handler is done with the decoded data. Both accounting middlewares now keep their own accounting handle alive across next.run(req).await. - Return a ChargedBuffer from the Loki snappy decompressor so the reservation outlives the decompressed bytes through the caller's protobuf decoding, matching the Prometheus path. - Map mid-stream quota exhaustion to 429 instead of the generic body error (400): the accounting flags exhaustion and the middlewares rewrite the extractor rejection, so clients can distinguish backpressure from malformed input. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve the in-flight body acquisition before trying a new one A parked acquisition in AccountedBody::charge always belongs to the currently buffered frame, so it must be resolved before any new acquisition is tried. Letting the fast-path try_acquire succeed while a waiter is parked left the waiter alive, and its late completion would then be credited with a later frame's byte count, under-reserving memory relative to what was marked charged. Adds a regression test that reproduces the misattribution: with the buggy ordering the test delivers a body the quota cannot cover; with the fix the over-quota frame waits for its own acquisition and times out. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 shuiyisong
|
20dde2601f |
feat: enable native histogram ingestion by default (#9301)
* feat: enable native histogram ingestion by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 dennis zhuang
|
f3eb8e6c72 |
test: cut integration test time and make the storage matrix meaningful (#9308)
* test: cut integration test time and make the storage matrix meaningful
tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.
- Point the PR CI object-store matrix at the MinIO already started by
tests-integration/fixtures. Three GT_S3_* consumers did not read
GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
release integration-test container. The release previously ran every
remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
/tmp/greptimedb_cache, which the datanode wipes on startup, so a
starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
non-trivial SST representation: JSON/JSON2 columns, native histograms,
metric-engine logical tables, and tables carrying fulltext or skipping
indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
shorten it in the event tests, which otherwise wait a 5s window per DDL
they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
test_grpc_message_size_ok and is now rewritten to assert the negotiated
grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
prefixes of their copy_from siblings; two standalone/distributed event
test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
in a parallelism-dependent order, so its last digits are unstable; the
test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
ttl=7d with 2025 timestamps, so its rows were only readable as long as
they stayed in the memtable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: address review — wire nightly real-S3 job into check-status, keep the short event interval
The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.
In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
 shuiyisongandLei, HUANG
|
66d38e8e1c |
feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add `InsertLimitInterceptor` hook to `Inserter` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: attribute insert limit checks to the target table's database Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: unify write admission through metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce write admission for pending row batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: distinguish internal requests for ingestion metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: admit split ingestion requests once per database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: OpenTSDB throws error reason Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use meter crate main rev Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude database ingest rate limit from table options Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: reserve channel 255 for internal requests Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
discord9
|
13c69cdaee |
perf(gc): pack file reference exchange (#9009)
* perf(gc): pack file reference exchange Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): address packed reference review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): stop without retry when maintenance is enabled Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(meta): avoid logging malformed mailbox payloads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Weny Xu
|
a673e084b2 |
test: cover request-level insert WAL skipping end to end (#9093)
* test: cover request-level WAL skipping end to end Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover session WAL policy and COPY recovery in sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(test): isolate Mito test feature in dev dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> * test: parameterize WAL protocol cases and make setup explicit Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover skip-WAL hints across streaming messages Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
da5cb1a190 |
perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: guard instant last row correctness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update instant query explain results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply last row after source deduplication Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: scope post-merge last row selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover instant PromQL last row Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): sort generated SST rows before writing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover instant last row selection in sqlness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): avoid last row hints for lossy timestamp casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): preserve stale marker semantics across flushes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): decode dictionary labels in stale regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): avoid reserved column name in stale fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): exercise LastRow hints and filtered results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: keep after-merge mode in LastRow selector Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: reject instant LastRow across residual filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: expect after-merge selector in instant vector guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain instant LastRow filter eligibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restrict instant LastRow to safe selector nodes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: show LastRow merge mode directly in diagnostics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh LastRow display in explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 LFC
|
529f046110 |
refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * rethinking when "needs_remainder" Signed-off-by: luofucong <luofc@foxmail.com> * restore "ReadColumns" Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 Lei, HUANG
|
7539e60139 |
refactor: remove trivial tests (#8877)
* refactor: remove trivial tests Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: remove unused trace test import Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
72f6cf09bf |
refactor(procedure): centralize event context handling (#8834)
* refactor(procedure): centralize event context handling Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): simplify migration trigger reason handling Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): avoid cloning event context Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Yingwen
|
78084a9d44 |
feat: add admin function to discard unflushed data (#8768)
* feat: add admin function to discard unflushed data Signed-off-by: evenyag <realevenyag@gmail.com> * test: cover discarding unflushed data by table Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix license header Signed-off-by: evenyag <realevenyag@gmail.com> * fix: reject discarding logical metric table data Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: defer table name formatting in error paths Signed-off-by: evenyag <realevenyag@gmail.com> * chore(deps): update greptime-proto revision Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename discard unflushed admin function Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
1f1c9270a8 |
feat(event): add event context to procedure events (#8734)
* feat(event): add trigger context to procedure events Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix trigger context event contracts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve trigger context origins Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve lifecycle trigger contexts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): gate enterprise trigger imports Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): assert trigger contexts exactly Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(test): restore migration test literals Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event): add trigger context to non-table ddl Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): simplify trigger context encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve auto alter trigger reason Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): require explicit trigger context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): derive trigger context at ddl boundary Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): propagate DDL trigger reasons Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): resolve alter table rebase conflict Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): fix alter table trigger context setup Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): pass trigger context to region migration Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix unknown trigger context assertions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): centralize trigger context protocol mapping Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix migration trigger context assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(procedure): rename event runtime context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): rename trigger context Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix migration event context assertion Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
448f973593 |
fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
c52db42b61 |
test: stabilize remote dynamic filter left join e2e (#8711)
test: stabilize remote dynamic filter left join Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
7344d47756 |
feat(event-recorder): configure lifecycle event recording (#8648)
* refactor(event-recorder): centralize event table helpers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(procedure): wire lifecycle event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event-recorder): filter events by type Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): derive event type filter default Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): decouple frontend filtering Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove docs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): complete configuration support Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event-recorder): centralize filter ownership Signed-off-by: WenyXu <wenymedia@gmail.com> * test(config): update event recorder snapshot Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(frontend): decouple slow query event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
b30765e5e9 |
fix: preserve distributed topk merge ordering (#8432)
* fix: preserve distributed topk for scalar latest Signed-off-by: discord9 <discord9@163.com> * fix: preserve scalar latest ordering across merge scan Signed-off-by: discord9 <discord9@163.com> * fix: carry merge scan ordering from planner rewrite Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan ordering metadata Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan partition ordering gate Signed-off-by: discord9 <discord9@163.com> * fix: require adjacent merge sort for ordering metadata Signed-off-by: discord9 <discord9@163.com> * test: update ordering sqlness plans Signed-off-by: discord9 <discord9@163.com> * fix: preserve distributed topk merge ordering Signed-off-by: discord9 <discord9@163.com> * test: cover latest per series queries Signed-off-by: discord9 <discord9@163.com> * test: update distributed merge sort sqlness Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan ordering partition gate Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan over-partition ordering Signed-off-by: discord9 <discord9@163.com> * test: cover distributed latest with low parallelism Signed-off-by: discord9 <discord9@163.com> * fix: keep distributed merge sort opaque Signed-off-by: discord9 <discord9@163.com> * test: assert distributed scalar latest merge sort Signed-off-by: discord9 <discord9@163.com> * test: clarify merge scan ordering helper Signed-off-by: discord9 <discord9@163.com> * fix: complete merge sort exec delegation Signed-off-by: discord9 <discord9@163.com> * docs: explain merge sort limit pushdown Signed-off-by: discord9 <discord9@163.com> * test: cover merge sort optimizer opacity Signed-off-by: discord9 <discord9@163.com> * docs: explain merge sort optimizer hooks Signed-off-by: discord9 <discord9@163.com> * test: cover merge sort optimizer hooks Signed-off-by: discord9 <discord9@163.com> * fix: preserve merge sort child topk Signed-off-by: discord9 <discord9@163.com> * test: update order by topk plan Signed-off-by: discord9 <discord9@163.com> * fix: recognize merge sort global fetch Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Yingwen
|
758664eefa |
fix: Use prepared file locations for CSV strict headers integration test (#8493)
fix: normalize Windows COPY test paths Signed-off-by: Codex <codex@openai.com> |
||
 QuakeWang
|
4e14ac8676 |
feat: add strict CSV header validation (#8426)
* feat: add strict CSV header validation Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * style: simplify strict header parse error Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
 fys
|
3bf8b9722d |
feat(json2): validate append mode for tables with JSON2 columns (#8434)
* feat: validate append mode for JSON2 tables * fix: cr by codex * fix: unit test * fix: validate JSON2 append_mode for all ALTER TABLE operations |
||
|
fys
|
aa563628fb |
feat(json2): type hint (#8247)
* feat(json2): type hint * test(datatypes): add JsonSettings serde round-trip test * minor refactor This reverts commit 7ff5a5249a09be5396536284fe822b5761ef4e6a. * fix: code review |
||
|
discord9
|
8fc5a3b1c7 |
feat: apply remote dynamic filters on datanode scans (#8262)
* feat: apply rdf Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * fix: drop remote dyn filter fallback exec Signed-off-by: discord9 <discord9@163.com> * Revert "fix: drop remote dyn filter fallback exec" This reverts commit bb757a596c3c2c078a836aa623226b7934103634. Signed-off-by: discord9 <discord9@163.com> * refactor: use rdf receiver logical plan instead Signed-off-by: discord9 <discord9@163.com> * test: update sqlness Signed-off-by: discord9 <discord9@163.com> * feat: rdf disable option Signed-off-by: discord9 <discord9@163.com> * tests: large int tests Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * test: update prec fix Signed-off-by: discord9 <discord9@163.com> * fix: make receiver node works Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * fix: tql disable rdf Signed-off-by: discord9 <discord9@163.com> * chore: rm useless joins Signed-off-by: discord9 <discord9@163.com> * fix: also disable in flow tql Signed-off-by: discord9 <discord9@163.com> * refactor: per review rm to promql Signed-off-by: discord9 <discord9@163.com> * chore: promql ut Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * test: rm misleading&add some nested/cleanup Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
QuakeWang
|
6ed67167bb |
feat: support headerless CSV copy from (#8233)
* feat: support headerless CSV copy from Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: update csv copy sqlness result Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * test: cover headerless CSV copy from Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * test: cover headerless CSV column count mismatch Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
|
Weny Xu
|
bb1de2a744 |
test: move CSV skip bad records coverage to integration (#8237)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
LFC
|
209880b991 |
feat: json2 flush (#8011)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Ning Sun
|
e14404c677 |
chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21 * chore: new format * fix: lint * chore: resolve lint issues * chore: remove as_millis_f64 * chore: deps up |
||
 discord9
|
30e895abbe |
fix: prom cast to f64 (#7840)
* fix: cast to f64 Signed-off-by: discord9 <discord9@163.com> * test: div case Signed-off-by: discord9 <discord9@163.com> * test: int test Signed-off-by: discord9 <discord9@163.com> * chore: sqlness update Signed-off-by: discord9 <discord9@163.com> * chore: test Signed-off-by: discord9 <discord9@163.com> * chore: update test Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
56ee8baa3f |
feat: admin gc table/regions (#7619)
* feat: gc table Signed-off-by: discord9 <discord9@163.com> * test: admin gc Signed-off-by: discord9 <discord9@163.com> * chore: after rebase fix Signed-off-by: discord9 <discord9@163.com> * refactor: GcStats Signed-off-by: discord9 <discord9@163.com> * refactor: use gc ticker for admin gc Signed-off-by: discord9 <discord9@163.com> * fix: region routes override Signed-off-by: discord9 <discord9@163.com> * test: non happy path Signed-off-by: discord9 <discord9@163.com> * refactor: gc job report enum Signed-off-by: discord9 <discord9@163.com> * test: process 0 regions Signed-off-by: discord9 <discord9@163.com> * after rebase Signed-off-by: discord9 <discord9@163.com> * feat: allow manual gc to return error Signed-off-by: discord9 <discord9@163.com> * chore: update proto Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * chore: timeout and update proto Signed-off-by: discord9 <discord9@163.com> * chore: udpate proto Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
b17947b5d8 |
feat: add COPY support for GCS/Azblob with OSS/GCS/Azblob integration coverage (#7743)
* feat: add gcs and azblob backends to copy datasource Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add copy to/from oss integration coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add copy to/from gcs/azure integration coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use existing layers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: hide `credential_path` in cli Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove logs Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Ruihang Xia
|
4ac73a7e9e |
feat: adapt prometheus 3.x selector and lookback behavior (#7688)
* feat: adapt prometheus 3.x selector and lookback behavior Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove commit hash from comment Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Ruihang Xia
|
38b5df574c |
refactor: SQL and gRPC server handlers (#7637)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
discord9
|
00f568ed28 |
fix: gc update repart map properly (#7606)
* feat: update repart map Signed-off-by: discord9 <discord9@163.com> * fix: table id write lock Signed-off-by: discord9 <discord9@163.com> * chore: default value Signed-off-by: discord9 <discord9@163.com> * chore: config Signed-off-by: discord9 <discord9@163.com> * test: update repartition map Signed-off-by: discord9 <discord9@163.com> * fix: empty file ref set Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * chore: properly log error Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xuanddiscord9
|
4fb61047cb |
test: add integration tests for repartition (#7560)
* test: add integration tests for mito repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update test result Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add integration tests for metric repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: correct results Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: enable tests for object store Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add compaction and gc Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit test Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * more cases Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: file ref also in repart mapping Signed-off-by: discord9 <discord9@163.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: set a longer timeout for mock metasrv channel manager Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: discord9 <discord9@163.com> Co-authored-by: discord9 <discord9@163.com> |
||
|
discord9
|
67e51b4573 |
feat: gc worker on dropped region (#7537)
* feat: allow clean up for dropped region Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * fix: get access layer correct Signed-off-by: discord9 <discord9@163.com> * chore: invalid gc args Signed-off-by: discord9 <discord9@163.com> * chore: fix test Signed-off-by: discord9 <discord9@163.com> * feat: more defend check Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * feat: messy impl of drop region Signed-off-by: discord9 <discord9@163.com> * feat: add dropped region GC handling module and integrate with GcScheduler Signed-off-by: discord9 <discord9@163.com> * refactor: simplify access layer creation Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> * fix: path type Signed-off-by: discord9 <discord9@163.com> * feat: gc handle drop Signed-off-by: discord9 <discord9@163.com> * chore: use proper const Signed-off-by: discord9 <discord9@163.com> * fix: recursive list when check empty dir Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * refactor: with gc only delete if metadata region Signed-off-by: discord9 <discord9@163.com> * feat: add batch_get_table_route method to SchedulerCtx and MockSchedulerCtx Signed-off-by: discord9 <discord9@163.com> * chore: comment Signed-off-by: discord9 <discord9@163.com> * refactor: retry delete method Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
6487f14f70 |
feat: gc schd update repart mapping (#7517)
* feat(gc): batch gc now alos handle routing Signed-off-by: discord9 <discord9@163.com> typo Signed-off-by: discord9 <discord9@163.com> s Signed-off-by: discord9 <discord9@163.com> feat: use batch gc procedure Signed-off-by: discord9 <discord9@163.com> feat: cross region refs Signed-off-by: discord9 <discord9@163.com> feat: clean up repartition Signed-off-by: discord9 <discord9@163.com> chore: cleanup Signed-off-by: discord9 <discord9@163.com> per review Signed-off-by: discord9 <discord9@163.com> test: update mock test Signed-off-by: discord9 <discord9@163.com> refactor: rm unused Signed-off-by: discord9 <discord9@163.com> refactor: invert related_regions Signed-off-by: discord9 <discord9@163.com> clippy Signed-off-by: discord9 <discord9@163.com> pcr Signed-off-by: discord9 <discord9@163.com> chore: remove unused Signed-off-by: discord9 <discord9@163.com> fix: after invert fix Signed-off-by: discord9 <discord9@163.com> chore: rm unused Signed-off-by: discord9 <discord9@163.com> refactor: eff Signed-off-by: discord9 <discord9@163.com> docs: chore Signed-off-by: discord9 <discord9@163.com> * after rebase fix Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * fix: mssing region Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
f3e2d333e4 |
feat(repartition): implement region allocation for repartition procedure (#7534)
* refactor: rename WalOptionsAllocator to WalProvider The name "WalOptionsAllocator" was misleading because: - For RaftEngine variant, it doesn't actually allocate anything - The actual allocation logic lives in KafkaTopicPool "WalProvider" better describes its role as providing WAL options based on the configured WAL backend (RaftEngine or Kafka). Changes: - Rename `WalOptionsAllocator` to `WalProvider` - Rename `WalOptionsAllocatorRef` to `WalProviderRef` - Rename `build_wal_options_allocator` to `build_wal_provider` - Rename module `wal_options_allocator` to `wal_provider` - Rename error types: `BuildWalOptionsAllocator` -> `BuildWalProvider`, `StartWalOptionsAllocator` -> `StartWalProvider` Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): extract allocator traits from TableMetadataAllocator Refactor TableMetadataAllocator to use trait-based dependency injection for better testability and separation of concerns. Changes: - Add `ResourceIdAllocator` trait to abstract ID allocation - Add `WalOptionsAllocator` trait to abstract WAL options allocation - Implement traits for `Sequence` and `WalProvider` - Remove duplicate `allocate_region_wal_options` function - Rename `table_id_sequence` to `table_id_allocator` for consistency - Rename `TableIdSequenceHandler` to `TableIdAllocatorHandler` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): add max_region_number tracking to PhysicalTableRouteValue Add `max_region_number` field to track the highest region number ever allocated for a table. This value only increases when regions are added and never decreases when regions are dropped, ensuring unique region numbers across the table's lifetime. Changes: - Add `max_region_number` field to `PhysicalTableRouteValue` - Implement custom `Deserialize` for backward compatibility - Update `update_region_routes` to maintain max_region_number - Calculate max_region_number from region_routes in `new()` Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: extract TableRouteAllocator trait from TableMetadataAllocator - Add TableRouteAllocator trait for abstracting region route allocation - Implement blanket impl for all PeerAllocator types - Add PeerAllocator impl for Arc<T> to support trait object delegation - Update TableMetadataAllocator to use TableRouteAllocatorRef Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: rename TableRouteAllocator to RegionRoutesAllocator - Rename table_route.rs to region_routes.rs - Rename TableRouteAllocator trait to RegionRoutesAllocator - Rename wal_option.rs to wal_options.rs for consistency - Update TableMetadataAllocator to use new naming Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta-srv): implement region allocation for repartition procedure This commit implements the region allocation phase of the repartition procedure, which handles allocating new regions when a table needs to be split into more partitions. Key changes: - Refactor `RegionRoutesAllocator::allocate` to accept `(region_number, partition_expr)` tuples for more flexible region number assignment - Simplify `AllocationPlanEntry` by removing `regions_to_allocate` and `regions_to_deallocate` fields (now derived from source/target counts) - Add `convert_allocation_plan_to_repartition_plan` function to handle allocation, equal, and deallocation cases - Fix `RepartitionPlanEntry::allocate_regions()` to return target regions (was incorrectly returning source regions) - Implement complete `AllocateRegion` state with: - Region route allocation via `RegionRoutesAllocator` - WAL options allocation via `WalOptionsAllocator` - Operating region registration for concurrency control - Region creation on datanodes via `CreateTableExecutor` - Table route metadata update - Add `TableRouteValue::max_region_number()` helper method - Add comprehensive unit tests for plan conversion and allocation logic Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
LFC
|
dc9f3a702e |
refactor: explicitly define json struct to ingest jsonbench data (#7462)
ingest jsonbench data Signed-off-by: luofucong <luofc@foxmail.com> |
||
 dennis zhuangandCopilot
|
1f91422bae |
feat!: improve mysql/pg compatibility (#7315)
* feat(mysql): add SHOW WARNINGS support and return warnings for unsupported SET variables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(function): add MySQL IF() function and PostgreSQL description functions for connector compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: show tables for mysql Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: partitions table in information_schema and add starrocks external catalog compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: async udf Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: set warnings Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: impl pg_my_temp_schema and make description functions simple Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add test for issue 7313 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: apply suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: partition_expression and partition_description Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: test Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: unit tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: saerch_path only works for pg Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: improve warnings processing Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: warnings while writing affected rows and refactor Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: improve ShobjDescriptionFunction signature Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: array_to_boolean Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
discord9
|
ee76d50569 |
test: gc integration test (#7306)
* test: basic infra for set gc Signed-off-by: discord9 <discord9@163.com> * more stuff Signed-off-by: discord9 <discord9@163.com> * test: basic gc integration test Signed-off-by: discord9 <discord9@163.com> * rm unused Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * refactor: remove loader Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * fix: allow default endpoint Signed-off-by: discord9 <discord9@163.com> * filter out files Signed-off-by: discord9 <discord9@163.com> * chore: rm minio support Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
LFC
|
4a7c16586b |
refactor: remove Vectors from RecordBatch completely (#7184)
* refactor: remove `Vector`s from `RecordBatch` completely Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |