Commit Graph
159 Commits
Author SHA1 Message Date
Ning Sun 7e76d28908 fix(postgres): account for buffered InfluxDB and OTLP COPY bodies (#10450)
Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-16 15:59:31 +00:00
shuiyisong 0a93810c23 fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): reserve memory for table batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): prefix the labels column

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): hide exemplar table from metadata

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): reconcile source tags before writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): prune tags and retain source IDs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): record slow queries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: account for held memory in batch admission

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: config comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: simplify exemplar and memory tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: deduplicate exemplar setup

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-16 11:55:20 +00:00
Ning SunandLei, HUANG b30e002cef feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>])
ingests one Prometheus Remote Write request body over the postgres wire
protocol — a GreptimeDB-specific, binary COPY format (wire format code
1). The payload is byte-for-byte the body a remote-write client POSTs
to /v1/prometheus/write, compression included: buffered until CopyDone
and completed through the exact ingestion core that endpoint runs.

To share that core, the HTTP handlers' ingestion bodies are extracted
into axum-free functions (ingest_remote_write_v1/v2 plus the context
extension stamping), and the HTTP routes become thin wrappers — their
behavior is unchanged, and the copy layer runs the identical decode,
validation, conversion, batched/prepared write paths, per-batch
write-progress accounting and context extensions, under the
connection's regular postgres context.

The WITH options carry what the HTTP endpoint takes as its
content-type/-encoding and query parameters; session-provided values
(database, user) are not repeated:
- version 1|2 (default 1), matching the content-type's proto
- zstd, matching Content-Encoding: zstd (snappy by default)
- physical_table, matching the ?physical_table= parameter
- pipeline_name/pipeline_version, matching the pipeline headers
  (v1 only; for v2 they are accepted and ignored, exactly like the
  HTTP route ignores them)

Table-less like the other ingestion formats; one COPY statement is one
bounded request (shared 64 MiB limit); the row count reports written
rows (v1) or written samples plus histograms (v2). The flush enforces
the frontend's shared request-memory quota (max_in_flight_write_bytes)
— the same limiter the HTTP ingestion protocols use, so COPY cannot
bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write
carries the handler, metric-engine flag, validation mode and shared
memory limiter.

e2e: v1 body chunked across CopyData boundaries; v2 samples with
readback; physical_table option routing; garbage-body decode failure
with connection reuse; a tiny-quota regression where a compressed body
whose decompressed size exceeds the budget fails like the HTTP route.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(servers): propagate COPY body reservations and test write accounting

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): share Prometheus logical batching with COPY ingestion

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): narrow Prometheus COPY scope and share write outcomes

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-11-16 02:45:34 +00:00
shuiyisong ee1fc7782f feat(prometheus): query persisted metric exemplars (#9924)
* feat(prometheus): add exemplar queries and lifecycle coverage

Expose /v1/prometheus/api/v1/query_exemplars with exact time bounds and series selectors. Resolve and authorize source selectors before scanning exemplar storage, enforce query limits and cancellation, preserve exact observation identity, and return warnings for malformed stored rows.

Includes HTTP and authorization tests, pre-created-table and concurrent-DDL coverage, OTLP Arrow ingestion checks, and cross-protocol lifecycle tests across memory, WAL, SST, compaction and reopen. The query-dependent ingestion tests live here so every lower layer builds independently. Add metric-exemplar usage and retention guidance.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): preserve query labels with source identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): floor bounds and add audit hooks

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* • fix(exemplar): include aggregation parameter selectors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): avoid v1 exemplar quota stalls

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align exemplar query timestamps with Prometheus

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove unnecessory doc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-10 08:11:49 +00:00
Weny Xu 65dc999180 feat: support cancelling compaction tasks by task ID (#10229)
* feat: add ADMIN compaction task cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject metadata region compaction cancellation before routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin greptime-proto to merged cancellation commit

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 07:38:08 +00:00
Weny Xu 8bdb7d5509 feat: add information_schema.compaction_tasks (#10173)
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.

Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 03:17:48 +00:00
shuiyisong fd74b9239e feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars

OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.

Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(otlp): bind exemplars to source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): bound exemplar memory and preserve series

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars without metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars for ordinary tables

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(exemplar)!: prefix the shared table name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): avoid waiting with body permits held

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:09:57 +00:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
Ning Sun a09ebc70e6 feat(postgres): OTLP binary payload ingest via COPY FROM STDIN (metrics, traces, logs) (#10050)
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:

- signal 'metrics' (default): ExportMetricsServiceRequest, completed
  through OpenTelemetryProtocolHandler::metrics with the same context
  preparation as the HTTP route (OtlpMetricCtx from the options); rows
  are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
  trace table name and pipeline name/version/params from the options
  (an explicit pipeline name is required, exactly like the HTTP traces
  endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
  table name, pipeline options and extract keys from the options, and
  the same OtlpLogDirect default as the HTTP route; rows are affected
  rows.

The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.

Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.

e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-02 08:57:05 +00:00
shuiyisong f2d74650d9 fix: optimize Prometheus remote write v2 ingestion (#9928)
* fix: optimize Prometheus remote write v2 ingestion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: correct remote write v2 memory accounting and response counts

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-30 08:56:34 +00:00
shuiyisong ec240ad7e6 feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables

Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions.

Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): preserve reserved sparse key semantics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): clarify sparse errors and avoid panics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito): reduce sparse key validation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito-codec): reduce sparse validation cost

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(mito): remove ordinary sparse support

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-29 05:45:07 +00:00
Ning Sun c818b6ac2a feat(postgres): influxdb line protocol ingest via COPY FROM STDIN (#9982)
* feat(postgres): influxdb line protocol ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT influxdb [, precision 'ns']) ingests
InfluxDB line protocol over the postgres wire protocol, reusing the
HTTP ingestion pipeline wholesale for correctness and performance
parity: the whole payload is buffered and completed once at CopyDone
through the frontend's InfluxdbLineProtocolHandler::exec — conversion,
permissions, line-protocol interceptors, timestamp alignment to table
time-index units, merge mode and inserts are the same path an HTTP
write of the same bytes takes, under the connection's regular postgres
query context.

Formats are an explicit, closed enum (CopyFormat::Csv |
CopyFormat::Influxdb(options)); each variant states its own rules in
the copy lifecycle's match arms — csv requires a target table and
parses incrementally, influxdb forbids one (each line names its own
measurement) and buffers one bounded request body. The COPY layer owns
only the buffering: record boundaries, including newlines inside quoted
field values and after escapes, belong to the line-protocol parser
alone — one COPY statement is one bounded ingestion request (64 MiB;
larger imports are multiple COPY statements).

- Wiring: PostgresServer::with_influxdb_handler attaches the shared
  line-protocol handler; unwired servers reject the format when a copy
  starts.
- Grammar: COPY FROM STDIN without a table is scanned (an unquoted
  FROM cannot be a table name); table-bound formats reject it with
  guidance, influxdb rejects a named table.
- Options: precision accepts the HTTP endpoint's values via the shared
  parse_time_precision (moved to servers::influxdb), defaulting to
  nanoseconds.

Review findings (fengjiachun) are fixed at the root by removing the
pre-splitter rather than repairing it: the retired naive line splitter
broke quoted multiline records at chunk/flush boundaries (found by
removing record splitting entirely), drained the buffer quadratically
(found by append-only buffering), and capped nothing (found by an
explicit payload limit).

e2e: multi-measurement ingest with chunked feeds; a 32,768-physical-line
flush-threshold regression with a multiline quoted record split across
CopyData chunks; precision 'ms' epoch verification; invalid UTF-8 and
trailing-record-without-newline handling; payload limit; table-named
rejection; malformed-line abort with connection recovery.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: typo

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-28 08:00:41 +00:00
discord9 984f80e277 fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit- or timezone-changing time casts

A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.

The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit-changing timestamp wrappers

to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.

The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regen windowed sort sqlness after wrapper guard

The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): simplify windowed sort cast regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 11:51:43 +00:00
dennis zhuang 2e84833ac8 feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): reject partition column type changes and quote rule identifiers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(table): keep time index widening for partition columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(partition): replace PartitionFunction with IcebergHash

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(partition): describe iceberg_hash in the function module

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: quote partition rule identifiers in SHOW CREATE TABLE expectations

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(partition): check hash rules in information_schema and repartition target columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): coerce hash partition rules against the evaluation schema

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): keep quotes in partition column names when building filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 09:23:00 +00:00
Ning Sun 80df1c397b feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support

* feat: add postgres copy-in framework

* feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher

Follow-up to #9302: begin_copy_in now uses the batching-aware
new_query_context, so copy-in batches are submitted to the
pending-rows batcher like SQL INSERTs when PostgreSQL batching is
enabled for the connection, and keep the direct write path when it
is not. The batcher acknowledges synchronously, so the
CommandComplete row count stays exact in both modes.

Adds test_postgres_copy_in_with_batching asserting the batcher
flush metric increments only when batching is enabled.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options

The WITH (...) option list is now open instead of restricted to
sqlparser's COPY grammar:

- COPY ... FROM STDIN statements are parsed textually (new
  copy_in::parse module, tokenizer-based) so arbitrary KEY value
  entries reach the codecs; sqlparser would reject options it does
  not know before we ever see them. WITH is optional, matching
  PostgreSQL.
- CopyOptionSet becomes an open ordered map of KEY -> String |
  Number | Ident | Boolean | Flag with typed take_* accessors
  (take_string, take_single_byte, take_flag, raw take_value) and
  case-insensitive keys; duplicate options are rejected.
- Each format codec takes the options it understands and rejects the
  leftovers with one message naming them ('COPY option(s) X are not
  supported by format Y'), so a typo cannot be silently ignored.
  The text format now also rejects HEADER/QUOTE/ESCAPE instead of
  ignoring them, matching PostgreSQL.

A future format (e.g. influxdb line protocol with
WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec
and a registry entry; no parser changes.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing

Replace the hand-rolled CSV record parser and per-value conversion
with arrow-csv's incremental Decoder — the same engine backing
COPY ... FROM 'file.csv' in common-datasource — so both COPY routes
accept identical data. The decoder handles record splitting (quoting,
escaped quotes, custom delimiters, CRLF, headers, chunk boundaries)
and typed conversion against the targeted columns' schema in one
pass; decoded batches become insert rows via the existing
api::helper::vectors_to_rows and datatypes Helper::try_into_vectors,
and the decode schema comes from the greptime Schema's arrow
projection.

The codec framework now has parsers emit complete insertable rows
(api Row) instead of raw byte fields: create_parser takes the
targeted columns and is fallible; text converts with the shared
text-representation helpers, csv with the arrow decoder.

Documented consequences of sharing the engine: JSON and Binary
columns and bare epoch-integer timestamps are rejected for csv, and
an empty field — quoted or not — is NULL; a custom NULL string is
rejected accordingly.

Signed-off-by: Ning Sun <sunning@greptime.com>

* docs(postgres): make the copy-in framework explicitly format-open

Reframe the codec framework docs so nothing assumes csv is the only
other text format: the shared pieces (Field, convert_text_field,
check_text_representable_type) are described as building blocks for
any format whose records carry values as text, while formats with
their own typed decoding (csv today) build rows themselves. State
explicitly that formats are not limited to PostgreSQL's own —
GreptimeDB-specific formats (e.g. a line protocol) register like any
other and define their own options through the open WITH clause.

No code changes beyond a test rename and a doc clarifier.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test(postgres): driver-level coverage for CSV copy-in options

Add an end-to-end test through tokio-postgres exercising HEADER,
custom DELIMITER, quoted fields with embedded newlines, escaped
quotes and CRLF record separators, with read-back verification
through the real insert path. WITH is omitted, matching PostgreSQL.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test(postgres): drop the raw-socket copy-in tests

The simple-protocol raw client duplicated what tokio-postgres and the
integration tests already cover; keep the driver-based tests only.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode

- text feed: take the longest complete line prefix in one drain and
  iterate lines over the slice, instead of draining per line (each
  drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of
  short lines)
- csv check_column_type: delegate to the shared
  check_text_representable_type and additionally reject Binary/Json
- decode_bytea: use the hex crate instead of a hand-rolled decoder
- semantic_type_of: use ColumnSchema::is_time_index() like the flow
  adapter's column_schemas_to_proto, instead of index lookups
- parse: drop the unused expect_ident parameter; document the
  identifier-case decision
- PostgresServer:🆕 document that query_handler and copy_in_handler
  are the same frontend instance

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: remove unrelated files

* feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format

Remove the hand-rolled PostgreSQL text-format codec (backslash
escapes, octal, \N nulls) that arrow-csv cannot express, keeping
arrow-csv-based csv as the only implemented format.

- COPY FROM STDIN without FORMAT now fails with guidance ('requires
  an explicit format in this version; use WITH (FORMAT csv)') rather
  than silently defaulting to the removed format
- FORMAT text/txt and FORMAT binary report 'not supported yet' via
  registry stubs instead of unknown-format errors
- codec.rs shrinks to the trait framework: the text-representation
  helpers (Field, convert_text_field, bytea hex, timestamp parsing)
  are removed with the codec; the module docs note where to find
  their shape should a future text-serializing format need them
- csv implements check_column_type directly (no shared default)

Breaking for clients relying on the PostgreSQL default text format;
explicit WITH (FORMAT csv) is required.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics

- COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher
  each flush pays the full insert pipeline (catalog lookup, schema
  check, partition split, region write round trip), so 4x larger
  batches cut round trips 4x at linearly-bounded memory; the constant
  also sizes the arrow decoder batch. Comment documents the trade-off
  and points high-rate users at the pending-rows batcher, which turns
  flushes into enqueues.
- rows buffer pre-allocates COPY_IN_FLUSH_ROWS.
- flush_copy_in warns on unexpected output variants instead of
  silently counting 0 rows.
- batching wire test feeds 70000 rows to stay above the new threshold.

Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv)
sends FORMAT csv and succeeds; default \copy (text) gets the
'requires an explicit format' guidance; COPY TO STDOUT round-trips.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): reset copy-in state on successful CopyDone

on_copy_done left the CopyInState in place (inert but holding buffers)
until the next COPY overwrote it; take it like on_copy_fail does, so
buffers are freed eagerly and a stray post-done CopyData gets the
protocol-violation error instead of a finished parser.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: bypass typo check

* chore: format change

* fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries

The csv feed loop flushed the arrow-csv decoder at every CopyData
chunk end — including partial batches at record boundaries — but
arrow-csv's contract permits flush() only after decode() returned 0
(a full batch of complete records). Flushing outside that point
corrupts subsequent decoding: fields mis-parse shifted by the size
of the record prefix buffered so far ('000001,host-001,...' parses
as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse,
silently writing wrong values. Which chunks trigger it depends on
where record boundaries land, so it appears intermittent with
random-length data (psql 8KB / pgx ~64KB chunking both hit it).

Adopt the strict decoder protocol: feed decodes until input is
exhausted and flushes only when decode returns 0 (batch full);
trailing partial batches surface at the EOF signal in finish().
With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is
unchanged; sub-threshold streams now emit their rows at CopyDone
instead of per chunk, which is semantically identical.

Adds test_csv_chunked_feeds_are_alignment_safe: the same payload
fed at chunk sizes 1..65531 (plus shifted starts) with every row's
content verified — reproduces the corruption on the old loop at
chunk sizes 7, 8, 13, 64 and 1000.

Verified with the reporter's Go reproducer (pgx, content-checked):
10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the
previously failing benchmark sizes 500/1000/2000/200/800 all round-
trip exactly.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: bypass spell check

* chore: bypass spell check

* fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter

Three review findings (shuiyisong, PR #9297):

[P1] Enforce the catalog restriction on COPY targets. copy_in_table now
applies validate_catalog_and_schema when QueryOptions::
disallow_cross_catalog_query is set, mirroring SQL INSERT's
check_permission path, BEFORE begin_copy_in switches the query context
to the target catalog (which previously made the restriction
unenforceable downstream). Cross-catalog COPY is now rejected with
'Illegal access to catalog' like INSERT.

[P2] Canonicalize unquoted identifiers like the SQL parser
(ParserContext::canonicalize_identifier): unquoted table-name parts and
columns fold to lowercase, quoted ones keep their case. COPY Metrics
(Host, Ts) now addresses the table CREATE TABLE Metrics created, and
no longer silently misroutes into a coexisting quoted "Metrics".

[P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal
tab while still rejecting newlines and other control characters, and
the option parser now also accepts E'...' escape-string constants
(DELIMITER E'\t'), processing PostgreSQL's backslash escapes.

Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round-
trip into the right tables, cross-catalog rejection asserted in a new
frontend unit test (QueryOptions plugin + second catalog), and both
DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure

Two review findings (shuiyisong, PR #9297):

[P2] Boolean columns round-trip between the COPY directions. arrow-csv
only parses true/false while PostgreSQL COPY OUT emits t/f, so
boolean tables failed to re-import their own export (22P02). Boolean
columns now decode as Utf8 in the arrow schema and flush_batch
converts them with PostgreSQL's boolean input grammar (true/false,
t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace
trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact
bytes are fed back via COPY FROM STDIN and verified.

[P2] Drop the COPY state on every server-side failure. on_copy_data
and on_copy_done now take ownership of the state up front, so parse,
conversion or insert errors drop the buffered data instead of
retaining it on long-lived or pooled connections until the next COPY
or connection close (the reviewer measured ~17 MiB retained after a
failed 8 MiB row); on_copy_data success puts the state back. An
e2e test drives a failed COPY followed by a successful one on the
same connection.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): interpret offset-free COPY timestamps in the session timezone

Review finding (shuiyisong, PR #9297): the arrow decode schema carried
Timestamp fields with no timezone, so the CSV decoder parsed
offset-free timestamp strings as UTC while SQL INSERT parses them in
the session timezone — the same input string silently stored a
different instant depending on the ingestion method (8h apart in
Asia/Shanghai).

CopyInCodec::create_parser now receives the session timezone and the
csv decode schema attaches it to Timestamp fields, so naive strings
parse like INSERT; explicit offsets in the input (Z, +08:00) still
take precedence, and UTC sessions are unchanged.

Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z)
and the end-to-end test_postgres_copy_in_session_timezone_parity,
which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the
same string, and asserts both CAST(ts AS BIGINT) readbacks agree on
1704038400000.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): don't unescape E-string COPY options twice

Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already
decodes PostgreSQL escape-string constants before returning
Token::EscapedStringLiteral, so the second unescape_pg_string pass
altered the decoded value — DELIMITER E'\\t' (a literal backslash
followed by t, two characters) was turned into a tab and wrongly
accepted as a single-byte delimiter.

Use the EscapedStringLiteral payload as-is, like SingleQuotedString,
and delete unescape_pg_string. Regression coverage distinguishes the
three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t'
-> rejected as a multi-character delimiter.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN

Two review findings (shuiyisong, PR #9297):

[P2] Preserve timestamp instants across COPY OUT/IN in non-UTC
sessions. COPY OUT rendered naive UTC text while COPY IN (since the
session-tz fix) interprets offset-free strings as local time, so
re-importing an export shifted instants by the zone offset. COPY OUT
now renders timestamps with an explicit session-timezone offset
(2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the
session tz (arrow-pg converts to a tz-aware DateTime) and the pg field
type becomes TIMESTAMPTZ so the offset-bearing text format applies.
Embedded offsets win on re-import regardless of session, keeping the
OUT -> IN round trip exact; UTC sessions are unchanged.

[P2] Preserve quoted empty strings. The shared arrow-csv decoder
nulls every empty field, so a quoted empty string ("" — an empty
STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner
runs over the same bytes with the decoder's own splitting rules and
records per field whether it was quoted; flush_batch restores quoted
empties to empty strings for string columns, while unquoted empty
fields stay NULL. Masks track records across chunk boundaries and
align with decoder batches.

Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip
(Asia/Shanghai export asserted to carry +08, exact-epoch re-import
comparison) and test_postgres_copy_in_quoted_empty_string (NULL,
quoted empty and non-empty in one import), plus unit tests including
a chunk-boundary sweep for the quoted-empty mask alignment.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns

Two review findings (fengjiachun, PR #9297):

[P2] Doubled quotes desynchronized the shadow quote-mask scanner. In
QuoteInQuoted the doubled-quote branch skipped the second quote AND the
following byte, so a closing quote or delimiter directly after an
escaped quote was swallowed and later fields lost their masks --
import semantics depended on CopyData chunk boundaries. The branch now
consumes only the escaped quote and returns to the in-quoted state,
mirroring the decoder. Regression: "a""b","" ... verified across
chunk sizes 1-64.

[P2] A NOT NULL string column rejected quoted empty strings before
restoration could run: the decode schema preserved the target
nullability, and arrow-csv's batch validation rejects the temporary
NULL a quoted empty decodes to. String fields now decode as nullable
(safe: quoted-vs-unquoted is exactly the restoration's job), and the
target NOT NULL constraint is enforced in flush_batch after
restoration, so "" imports as an empty string while an unquoted empty
field fails with a not-null-constraint error. Regressions at unit and
e2e level: NOT NULL column accepting "" and rejecting unquoted NULL.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators

Two review findings (fengjiachun, PR #9297):

[P2] Blank lines emitted a spurious empty mask. csv-core skips blank
lines, but the scanner queued one mask per blank record, so every
subsequent decoded row consumed the wrong mask (a leading blank line
turned a quoted empty into NULL and an unquoted empty into an empty
string, and tripped NOT NULL on the wrong row). A blank line before the
header also shifted the header expectation. end_record now returns no
mask for a completely blank record without consuming the header
expectation.

[P2] Standalone CR was not a record terminator. The decoder accepts CR,
CRLF and LF alike; the scanner only recognized CR before LF or at EOF,
so CR-separated input desynchronized masks and flush_batch failed with
'quote masks out of sync'. is_record_end now treats CR alone as a
terminator, consuming one byte (two for CRLF), while a CR at a chunk
boundary still defers via carry so CRLF split across CopyData chunks
stays one terminator.

Regressions: blank lines before/inside records and before a header
(NULL/empty preservation plus NOT NULL behavior), and mixed
CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing
CR-only record.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY

The shadow quote-mask scanner — kept to preserve PostgreSQL's
quoted-empty-string vs unquoted-NULL distinction — mirrored the
decoder's record-splitting grammar by hand and required a fix for every
rule it drifted on (doubled quotes, blank lines, CR terminators, NOT
NULL interplay). Per maintainers' decision, that complexity is not
worth the divergence: COPY FROM STDIN now behaves EXACTLY like the
file-based COPY ... FROM 'file.csv' — an empty field, quoted or not,
imports as NULL (arrow-csv's engine semantics, identical in both
routes).

Removed: QuoteMaskScanner and helpers, the always-nullable string
decode schema, the restoration + NOT NULL enforcement in flush_batch,
set_string_at, and the scanner-era unit tests. The decode schema again
preserves target nullability, so the decoder enforces NOT NULL itself,
same as the file route.

Kept (small, schema-level, no second grammar): boolean spellings
(t/f) and session-timezone timestamp decoding, both addressing
separate review findings.

E2E: test_postgres_copy_in_empty_field_is_null pins the parity —
quoted empty, unquoted empty and non-empty in one import.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: remove empty file

* chore: lint issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-26 07:08:12 +00:00
Ning Sun 40770ddb72 test: only assert genuinely invalid read-preference values at request ingresses (#9576)
The flow and nested-MergeScan ingress tests hardcoded "follower" among
the read-preference values that must be rejected, coupling them to the
current leader-only ReadPreference enum. Assert only genuinely invalid
values there, matching the wire-values test which already limits itself
to ["", "unknown"]. This keeps the ingress tests valid for builds that
extend ReadPreference with more variants.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-14 12:49:56 +00:00
discord9 0e40281262 fix(promql): stop claiming scalar calculate preserves order (#9523)
* test: cover scalar calculate order contract

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): stop claiming scalar calculate preserves order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(test): describe scalar ordering regression as pre-fix behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-14 04:20:39 +00:00
discord9 75bc4ce46d fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(table): only claim projected primary key prefix ordering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(table): keep physical key ordering for metric projections

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(table): compare unknown partition counts explicitly

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(table): model scoped internal TSID scan properties

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-14 03:35:58 +00:00
discord9 2039f508fc fix(promql): stop fabricating histogram fold hash partitioning (#9509)
* fix(promql): stop fabricating histogram fold hash partitioning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): cover histogram fraction binary join missing rows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): normalize dictionary labels in histogram regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover histogram joins on logical metric tables

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: specify insert columns for logical metric fixture

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use supported histogram child replacement API

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): keep histogram fold fixtures truthful

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-13 07:58:31 +00:00
discord9 483d1e49a3 fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep filter fetch when rewriting matches term

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 06:12:56 +00:00
discord9 32e8467cda fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): use current DataFusion plan downcasts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): match null rendering in join regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh union plan for unknown scan partitioning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh shared plans after distribution rule removal

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): cover semi joins with remote dynamic filtering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): strengthen partitioned merge scan regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove unused merge scan test helpers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): measure merge scan hash repartition cost

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt merge scan deserialization to two-arg constructor

Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): adapt nested merge scan integration constructor

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 11:18:46 +00:00
discord9 28697c9e55 feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode

Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.

Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
  engine catalog manager, and decodes a MergeScan payload's tables by
  (catalog, schema, table) name through the engine catalog rather than
  the region-aware request catalog list (which binds every table to the
  request's region and breaks on mismatched columns).

Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
  for the MergeScan nodes of a received plan, resolving region leaders
  via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
  the datanode's catalog/cache registry (with_dist_planner stays false:
  a datanode never rewrites its own plan).

Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
  partition_info caches the datanode's catalog manager and partition
  rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
  on a current-thread runtime from within another executor (spawn a
  dedicated thread + runtime when not on a multi-thread runtime).

Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep GreptimeDB function bindings after rebuilding decode state

SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.

Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): address review on nested MergeScan capability

Three follow-ups from review of the nested MergeScan execution capability:

1. Layer the datanode caches (P2b). table_cache derives from
   table_info_cache + table_name_cache, and partition_info_cache derives
   from table_route_cache, but they were all registered in a single flat
   CacheRegistry whose invalidate() runs concurrently with no ordering.
   A derived cache could be invalidated and refilled from a base cache
   that had not been invalidated yet, leaving stale metadata. Build the
   base caches and the derived caches as separate layers of a
   LayeredCacheRegistry (reusing the existing layering mechanism) so base
   caches finish invalidating before derived ones. Both the production
   path (cmd) and the test scaffolding now build the same layered
   registry; the flat build_datanode_cache_registry is removed.

2. Don't let a nested MergeScan's internal region query compete for the
   concurrency permit (P2a). An outer query holds its permit until its
   stream is fully consumed, and a nested MergeScan makes the datanode
   issue region queries to peer datanodes as execution stages of that
   same query. If such an inner stage also took a permit, the outer query
   waiting on it would deadlock against its own held permit (or time out
   when max_concurrent_queries > 0). Mark inner DN->DN region queries as
   internal stages: QueryRequest gains an  flag carried over
   the wire in the request header's query_context extensions (reserved
   key, not injectable via hints), and the region server skips acquiring
   a concurrency permit for internal-stage requests on both acquire
   sites. Outer queries still acquire normally.

3. Add an end-to-end capability test
   (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
   Join(<local probe region>, MergeScan(build regions on both datanodes))
   with mismatched column names and duplicate build keys, ship it to a
   datanode over a real TCP Flight endpoint, consume the stream fully and
   compare the row multiset against the frontend's reference join result,
   assert a real cross-datanode DoGet via an RPC counter, and assert an
   inner region failure fails the whole query. Also migrate the
   with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt date_format regression test to the updated DataFusion fork

ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover the internal stage under the concurrency limit

Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.

Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover cold route loads under unrelated invalidations

The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): trim nested plan decoder commentary

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: remove redundant nested scan support plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): decode nested MergeScan payloads asynchronously

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): avoid unnecessary extension name allocation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain extension decoding for manual session states

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datanode): keep nested scan query admission unchanged

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): honor query handler factories and trim nested scan fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): retain required cache error import

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): reuse registered state for nested scan decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): verify payload information schema table functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): retain nested scan runtime and cold-route guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): address nested scan review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(session): preserve read preference across region query hops

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(session): verify strict context rejection at Flow ingress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove redundant serializer error case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:49:38 +00:00
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
jeremyhi f9ab3c296b feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region

A segment that does not decode, because its checksum does not match or
its content disagrees with its footer entry, is fetched once more. If it
still does not decode, the read follows `on_corrupted_segment`: `skip`
records a WAL hole for the region with the object key, sequence and the
footer's entry id range, increments
`greptime_logstore_object_store_wal_skipped_segments_total`, logs a
warning and continues with the next object; `fail` fails the read. A
failed fetch fails the read in both modes. Holes are kept in memory per
region and can be listed from the store.

The standalone test builder allocates WAL options through
`build_standalone_wal_provider`, and integration tests run a standalone
instance on the object store WAL across restarts, on S3 when the
`GT_S3_*` variables are set. The standalone shutdown runs every step and
returns the first error.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cmd): cover the best-effort standalone stop

Build a standalone instance whose leader services fail to stop and whose
datanode servers fail afterwards, and check that `Instance::stop` still
stops the object store WAL and returns the leader services error.

Rename `skip_segment` to `record_skipped_segment` and the file store
integration test to `test_standalone_object_store_wal_survives_restarts_on_file`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-08 06:33:40 +00:00
Ning Sun 3131cdbcb6 fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests

The request-memory accounting only bounds and charges the encoded bytes on
the wire, but a compressed body can expand far beyond that during
decompression, so tiny requests could allocate disproportionate frontend
memory before any protobuf validation or quota charge.

- Handler-level decompression (Prometheus remote read/write v1+v2, Loki)
  now enforces a hard 512 MiB decoded-size cap, checked before any output
  buffer is allocated, and charges the decoded bytes to the shared
  ServerMemoryLimiter, holding the permits for the lifetime of the
  decompressed buffer.
- gRPC requests with transport compression reserve the configured
  max_recv_message_size before tonic decompresses, so the decoding phase
  is admitted against max_in_flight_write_bytes; the later per-message
  charge is skipped to avoid double accounting.
- The HTTP memory-limit middleware keeps its upfront Content-Length charge
  but now also accounts the bytes actually streamed beyond it, so chunked
  requests and understated Content-Length headers no longer bypass the
  aggregate quota.
- Routes that decompress request bodies via RequestDecompressionLayer
  (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards)
  now charge the decompressed bytes as handlers consume them: the global
  middleware marks Content-Encoding requests, and a route-local
  accounting layer inside the decompression layer charges the decoded
  stream. Plain requests are skipped to avoid double-counting.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on memory admission guard lifetimes and 429 mapping

- Retain the gRPC pre-decode reservation for the whole request: the
  extensions holding the guard were dropped when the request was consumed
  (into_inner), releasing the reservation while per-message charges
  stayed skipped. Both the unary and streaming handlers now clone and
  hold the reservation marker for the duration of request handling.
- Retain the HTTP body permits across the handler: the AccountedBody
  wrapper and the request extensions are dropped once the extractors
  finish collecting the body, before the handler is done with the decoded
  data. Both accounting middlewares now keep their own accounting handle
  alive across next.run(req).await.
- Return a ChargedBuffer from the Loki snappy decompressor so the
  reservation outlives the decompressed bytes through the caller's
  protobuf decoding, matching the Prometheus path.
- Map mid-stream quota exhaustion to 429 instead of the generic body
  error (400): the accounting flags exhaustion and the middlewares
  rewrite the extractor rejection, so clients can distinguish
  backpressure from malformed input.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: resolve the in-flight body acquisition before trying a new one

A parked acquisition in AccountedBody::charge always belongs to the
currently buffered frame, so it must be resolved before any new
acquisition is tried. Letting the fast-path try_acquire succeed while a
waiter is parked left the waiter alive, and its late completion would
then be credited with a later frame's byte count, under-reserving memory
relative to what was marked charged.

Adds a regression test that reproduces the misattribution: with the
buggy ordering the test delivers a body the quota cannot cover; with the
fix the over-quota frame waits for its own acquisition and times out.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-29 10:07:18 +00:00
shuiyisong 20dde2601f feat: enable native histogram ingestion by default (#9301)
* feat: enable native histogram ingestion by default

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-09-23 13:03:51 +00:00
dennis zhuang f3eb8e6c72 test: cut integration test time and make the storage matrix meaningful (#9308)
* test: cut integration test time and make the storage matrix meaningful

tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.

- Point the PR CI object-store matrix at the MinIO already started by
  tests-integration/fixtures. Three GT_S3_* consumers did not read
  GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
  they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
  release integration-test container. The release previously ran every
  remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
  /tmp/greptimedb_cache, which the datanode wipes on startup, so a
  starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
  non-trivial SST representation: JSON/JSON2 columns, native histograms,
  metric-engine logical tables, and tables carrying fulltext or skipping
  indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
  shorten it in the event tests, which otherwise wait a 5s window per DDL
  they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
  test_grpc_message_size_ok and is now rewritten to assert the negotiated
  grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
  prefixes of their copy_from siblings; two standalone/distributed event
  test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
  in a parallelism-dependent order, so its last digits are unstable; the
  test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
  ttl=7d with 2025 timestamps, so its rows were only readable as long as
  they stayed in the memtable.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: address review — wire nightly real-S3 job into check-status, keep the short event interval

The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.

In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-23 01:52:52 +00:00
shuiyisongandLei, HUANG 66d38e8e1c feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add `InsertLimitInterceptor` hook to `Inserter`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: attribute insert limit checks to the target table's database

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: unify write admission through metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce write admission for pending row batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: distinguish internal requests for ingestion metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: admit split ingestion requests once per database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: OpenTSDB throws error reason

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use meter crate main rev

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude database ingest rate limit from table options

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: reserve channel 255 for internal requests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 07:27:22 +00:00
discord9andNing Sun 94d7e2c7fc feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: migrate DataFusion 55 APIs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve table function planning behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve PostgreSQL query compatibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve distributed execution plan behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover DataFusion 55 behavior regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update DataFusion 55 SQLness expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DataFusion 55 test API migration

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address DataFusion 55 CI regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address remaining DataFusion 55 regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: adapt latest base code to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: normalize environment-specific DataFusion 55 plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align final DataFusion 55 expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: isolate DataFusion 55 regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve empty result schema in timestamp widening

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve JSON source column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: use released DataFusion 55 integrations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: adapt latest execution plan mock to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: pin DataFusion recursive schema and date repairs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): align dictionary temporality match keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: retain Greptime DataFusion fork behaviors on version 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore ordinary function error expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh distributed count compatibility plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt last-row cast hint to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh instant last-row empty results for Arrow 59

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: simplify DataFusion expression visitor imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore sorting and PostgreSQL column-order assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): restore primitive numeric coercion signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(function): share geo integer signature types

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover timestamp widening overflow boundaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix decimal coercion regression imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): preserve scalar count_hash NULL state semantics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: simplify decimal clamp case type inference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: retain historical count_hash wrapper result

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restore timestamp widening equality and IN pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream aggregate dynamic filter correctness fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream null and predicate simplification fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore baseline JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore histogram JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh empty PromQL range result schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align native timestamp plan with DF55 decimal display

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh native timestamp SQLness results for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regenerate NULL sample empty result headers for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use DF55 child replacement API in timestamp regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: expose pushed scan dynamic filters to DF55 producers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: encode string-backed PostgreSQL OID aliases in binary results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify REGPROC binary and text over PostgreSQL protocol

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: register real PostgreSQL catalogs in server fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DF55 expression inventories for custom query plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: correct RangeSelect expression fixture and column identities

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* ci: wait for Kafka WAL helper deployment rollout

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update custom storage empty result headers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: require exact row counts in scan statistics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: suppress deprecated partition_statistics warning in test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-09-15 11:42:38 +00:00
discord9 13c69cdaee perf(gc): pack file reference exchange (#9009)
* perf(gc): pack file reference exchange

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): address packed reference review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): stop without retry when maintenance is enabled

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(meta): avoid logging malformed mailbox payloads

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 07:30:39 +00:00
Weny Xu a673e084b2 test: cover request-level insert WAL skipping end to end (#9093)
* test: cover request-level WAL skipping end to end

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover session WAL policy and COPY recovery in sqlness

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): isolate Mito test feature in dev dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: parameterize WAL protocol cases and make setup explicit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover skip-WAL hints across streaming messages

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-11 07:48:57 +00:00
discord9 da5cb1a190 perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: guard instant last row correctness

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update instant query explain results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: apply last row after source deduplication

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: scope post-merge last row selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): cover instant PromQL last row

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): sort generated SST rows before writing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): cover instant last row selection in sqlness

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): avoid last row hints for lossy timestamp casts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): preserve stale marker semantics across flushes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): decode dictionary labels in stale regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): avoid reserved column name in stale fixture

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): exercise LastRow hints and filtered results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: keep after-merge mode in LastRow selector

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: reject instant LastRow across residual filters

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: expect after-merge selector in instant vector guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs: explain instant LastRow filter eligibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restrict instant LastRow to safe selector nodes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: show LastRow merge mode directly in diagnostics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh LastRow display in explain expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-09 12:38:52 +00:00
LFC 529f046110 refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* rethinking when "needs_remainder"

Signed-off-by: luofucong <luofc@foxmail.com>

* restore "ReadColumns"

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-01 13:39:56 +00:00
Lei, HUANG 7539e60139 refactor: remove trivial tests (#8877)
* refactor: remove trivial tests

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: remove unused trace test import

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-08-13 12:10:49 +00:00
Weny Xu 72f6cf09bf refactor(procedure): centralize event context handling (#8834)
* refactor(procedure): centralize event context handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): simplify migration trigger reason handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): avoid cloning event context

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-08-11 09:05:26 +00:00
Yingwen 78084a9d44 feat: add admin function to discard unflushed data (#8768)
* feat: add admin function to discard unflushed data

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: cover discarding unflushed data by table

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix license header

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: reject discarding logical metric table data

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: defer table name formatting in error paths

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore(deps): update greptime-proto revision

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: rename discard unflushed admin function

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-08-10 12:19:27 +00:00
Weny Xu 1f1c9270a8 feat(event): add event context to procedure events (#8734)
* feat(event): add trigger context to procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix trigger context event contracts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve trigger context origins

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve lifecycle trigger contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): gate enterprise trigger imports

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): assert trigger contexts exactly

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): restore migration test literals

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event): add trigger context to non-table ddl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): simplify trigger context encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve auto alter trigger reason

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): require explicit trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): derive trigger context at ddl boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): propagate DDL trigger reasons

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve alter table rebase conflict

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): fix alter table trigger context setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): pass trigger context to region migration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix unknown trigger context assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): centralize trigger context protocol mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration trigger context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(procedure): rename event runtime context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): rename trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration event context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-08-07 12:37:58 +00:00
jeremyhi 448f973593 fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file sandbox review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support Windows local copy paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve sandbox path errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: simplify local path error context

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: stream secure filesystem listings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* style: derive local file access default

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve local file access errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file access review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify local file access coverage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: harden sandboxed local file backends

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject directory copy targets before creation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: avoid implicit string clone in file table listing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-07-31 13:23:15 +00:00
discord9 c52db42b61 test: stabilize remote dynamic filter left join e2e (#8711)
test: stabilize remote dynamic filter left join

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-07-31 10:56:52 +00:00
Weny Xu 7344d47756 feat(event-recorder): configure lifecycle event recording (#8648)
* refactor(event-recorder): centralize event table helpers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(procedure): wire lifecycle event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event-recorder): filter events by type

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): derive event type filter default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): decouple frontend filtering

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove docs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): complete configuration support

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event-recorder): centralize filter ownership

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(config): update event recorder snapshot

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(frontend): decouple slow query event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-27 13:05:56 +00:00
discord9 b30765e5e9 fix: preserve distributed topk merge ordering (#8432)
* fix: preserve distributed topk for scalar latest

Signed-off-by: discord9 <discord9@163.com>

* fix: preserve scalar latest ordering across merge scan

Signed-off-by: discord9 <discord9@163.com>

* fix: carry merge scan ordering from planner rewrite

Signed-off-by: discord9 <discord9@163.com>

* test: cover merge scan ordering metadata

Signed-off-by: discord9 <discord9@163.com>

* test: cover merge scan partition ordering gate

Signed-off-by: discord9 <discord9@163.com>

* fix: require adjacent merge sort for ordering metadata

Signed-off-by: discord9 <discord9@163.com>

* test: update ordering sqlness plans

Signed-off-by: discord9 <discord9@163.com>

* fix: preserve distributed topk merge ordering

Signed-off-by: discord9 <discord9@163.com>

* test: cover latest per series queries

Signed-off-by: discord9 <discord9@163.com>

* test: update distributed merge sort sqlness

Signed-off-by: discord9 <discord9@163.com>

* test: cover merge scan ordering partition gate

Signed-off-by: discord9 <discord9@163.com>

* test: cover merge scan over-partition ordering

Signed-off-by: discord9 <discord9@163.com>

* test: cover distributed latest with low parallelism

Signed-off-by: discord9 <discord9@163.com>

* fix: keep distributed merge sort opaque

Signed-off-by: discord9 <discord9@163.com>

* test: assert distributed scalar latest merge sort

Signed-off-by: discord9 <discord9@163.com>

* test: clarify merge scan ordering helper

Signed-off-by: discord9 <discord9@163.com>

* fix: complete merge sort exec delegation

Signed-off-by: discord9 <discord9@163.com>

* docs: explain merge sort limit pushdown

Signed-off-by: discord9 <discord9@163.com>

* test: cover merge sort optimizer opacity

Signed-off-by: discord9 <discord9@163.com>

* docs: explain merge sort optimizer hooks

Signed-off-by: discord9 <discord9@163.com>

* test: cover merge sort optimizer hooks

Signed-off-by: discord9 <discord9@163.com>

* fix: preserve merge sort child topk

Signed-off-by: discord9 <discord9@163.com>

* test: update order by topk plan

Signed-off-by: discord9 <discord9@163.com>

* fix: recognize merge sort global fetch

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-14 12:53:07 +00:00
Yingwen 758664eefa fix: Use prepared file locations for CSV strict headers integration test (#8493)
fix: normalize Windows COPY test paths

Signed-off-by: Codex <codex@openai.com>
2026-07-13 10:19:57 +00:00
QuakeWang 4e14ac8676 feat: add strict CSV header validation (#8426)
* feat: add strict CSV header validation

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* style: simplify strict header parse error

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2026-07-10 02:15:08 +00:00
fys 3bf8b9722d feat(json2): validate append mode for tables with JSON2 columns (#8434)
* feat: validate append mode for JSON2 tables

* fix: cr by codex

* fix: unit test

* fix: validate JSON2 append_mode for all ALTER TABLE operations
2026-07-08 03:33:36 +00:00
fys aa563628fb feat(json2): type hint (#8247)
* feat(json2): type hint

* test(datatypes): add JsonSettings serde round-trip test

* minor refactor

This reverts commit 7ff5a5249a09be5396536284fe822b5761ef4e6a.

* fix: code review
2026-06-30 07:37:31 +00:00
discord9 8fc5a3b1c7 feat: apply remote dynamic filters on datanode scans (#8262)
* feat: apply rdf

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* fix: drop remote dyn filter fallback exec

Signed-off-by: discord9 <discord9@163.com>

* Revert "fix: drop remote dyn filter fallback exec"

This reverts commit bb757a596c3c2c078a836aa623226b7934103634.

Signed-off-by: discord9 <discord9@163.com>

* refactor: use rdf receiver logical plan instead

Signed-off-by: discord9 <discord9@163.com>

* test: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* feat: rdf disable option

Signed-off-by: discord9 <discord9@163.com>

* tests: large int tests

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: update prec fix

Signed-off-by: discord9 <discord9@163.com>

* fix: make receiver node works

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* fix: tql disable rdf

Signed-off-by: discord9 <discord9@163.com>

* chore: rm useless joins

Signed-off-by: discord9 <discord9@163.com>

* fix: also disable in flow tql

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review rm to promql

Signed-off-by: discord9 <discord9@163.com>

* chore: promql ut

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* test: rm misleading&add some nested/cleanup

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-10 12:57:23 +00:00
QuakeWang 6ed67167bb feat: support headerless CSV copy from (#8233)
* feat: support headerless CSV copy from

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix: update csv copy sqlness result

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* test: cover headerless CSV copy from

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* test: cover headerless CSV column count mismatch

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2026-06-04 12:04:15 +00:00
Weny Xu bb1de2a744 test: move CSV skip bad records coverage to integration (#8237)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-04 07:58:58 +00:00
LFC 209880b991 feat: json2 flush (#8011)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-04-23 03:03:37 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
2026-03-30 12:13:14 +00:00