A prepared statement keeps its logical plan, and that plan holds the
table as it was at prepare time. The analyzer reads the partition
columns from that table while the regions to scan come from the current
route. After a table's partition columns change, the analyzer can
consider a query safe to push down, so each region runs the whole query
and the frontend concatenates their results. The same happens to the
source of a prepared INSERT ... SELECT, which then stores wrong values.
Before executing a kept plan, compare each scanned table's partition
columns with the current table found by id. Scans of a table whose
columns changed get the current table when the schema is unchanged.
Otherwise they keep their schema, because a scan's projection holds
indices into it, and report the current columns as absent from it, so
the query keeps merging on the frontend.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.
Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(otlp): add v1 schema for auto-created OTLP log tables
New tables created by direct OTLP logs ingestion get a v1 built-in schema:
service_name tag from resource service.name ahead of scope_name in the
primary key, a skipping index on trace_id, and event_name and
observed_timestamp fields. The table is marked with
table_data_model=greptime_log_v1. Existing tables without the marker keep
the legacy schema.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(otlp): share resource service.name parsing between logs and traces
Log tables now derive service_name with the same rules as trace spans, so
both signals agree on the value used to correlate them. The integration test
writes twice to cover the existing v1 table path.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(catalog): use advertised peer address as process identity
The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.
Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
in list_all_processes, and for local/remote routing in kill_process.
Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.
The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: keep local peer address reserved in kill-routing test
The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.
Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(function): require ADMIN statement privileges to execute admin functions
Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.
Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.
Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(function): expose admin privilege check to downstream macros
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(otlp): persist metric exemplars
OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.
Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat(otlp): bind exemplars to source metric identities
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): bound exemplar memory and preserve series
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): skip exemplars without metric engine
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): skip exemplars for ordinary tables
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* refactor(exemplar)!: prefix the shared table name
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): avoid waiting with body permits held
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN
COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:
- signal 'metrics' (default): ExportMetricsServiceRequest, completed
through OpenTelemetryProtocolHandler::metrics with the same context
preparation as the HTTP route (OtlpMetricCtx from the options); rows
are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
trace table name and pipeline name/version/params from the options
(an explicit pipeline name is required, exactly like the HTTP traces
endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
table name, pipeline options and extract keys from the options, and
the same OtlpLogDirect default as the HTTP route; rows are affected
rows.
The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.
Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.
e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(mito2): merge compaction inputs on a separate task
Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.
The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.
One output of 4.8M rows took 1.28s instead of 1.65s.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): parallelize decoding and encoding of compaction outputs
Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.
The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.
The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): cancel compaction producers before consumer polling
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): support per-table compaction output parallelism
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoding tasks before returning failures
Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.
Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoders on SST write errors
Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.
Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain compaction outputs on cancellation
Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.
Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(postgres): influxdb line protocol ingest via COPY FROM STDIN
COPY FROM STDIN WITH (FORMAT influxdb [, precision 'ns']) ingests
InfluxDB line protocol over the postgres wire protocol, reusing the
HTTP ingestion pipeline wholesale for correctness and performance
parity: the whole payload is buffered and completed once at CopyDone
through the frontend's InfluxdbLineProtocolHandler::exec — conversion,
permissions, line-protocol interceptors, timestamp alignment to table
time-index units, merge mode and inserts are the same path an HTTP
write of the same bytes takes, under the connection's regular postgres
query context.
Formats are an explicit, closed enum (CopyFormat::Csv |
CopyFormat::Influxdb(options)); each variant states its own rules in
the copy lifecycle's match arms — csv requires a target table and
parses incrementally, influxdb forbids one (each line names its own
measurement) and buffers one bounded request body. The COPY layer owns
only the buffering: record boundaries, including newlines inside quoted
field values and after escapes, belong to the line-protocol parser
alone — one COPY statement is one bounded ingestion request (64 MiB;
larger imports are multiple COPY statements).
- Wiring: PostgresServer::with_influxdb_handler attaches the shared
line-protocol handler; unwired servers reject the format when a copy
starts.
- Grammar: COPY FROM STDIN without a table is scanned (an unquoted
FROM cannot be a table name); table-bound formats reject it with
guidance, influxdb rejects a named table.
- Options: precision accepts the HTTP endpoint's values via the shared
parse_time_precision (moved to servers::influxdb), defaulting to
nanoseconds.
Review findings (fengjiachun) are fixed at the root by removing the
pre-splitter rather than repairing it: the retired naive line splitter
broke quoted multiline records at chunk/flush boundaries (found by
removing record splitting entirely), drained the buffer quadratically
(found by append-only buffering), and capped nothing (found by an
explicit payload limit).
e2e: multi-measurement ingest with chunked feeds; a 32,768-physical-line
flush-threshold regression with a multiline quoted record split across
CopyData chunks; precision 'ms' epoch verification; invalid UTF-8 and
trailing-record-without-newline handling; payload limit; table-named
rejection; malformed-line abort with connection recovery.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: typo
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(catalog): propagate frontend process-list failures
Return the existing invocation error when a selected frontend cannot list processes, so a successful response cannot omit that peer.
Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
* test(frontend): cover remote process-list errors and aggregation
Exercise real metadata discovery and gRPC listeners for a missing management service and successful catalog-scoped aggregation.
Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
---------
Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
* test: cover windowed sort cast unit mismatch
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit- or timezone-changing time casts
A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.
The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit-changing timestamp wrappers
to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.
The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: regen windowed sort sqlness after wrapper guard
The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): simplify windowed sort cast regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(partition): support iceberg_hash in partition rules
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): reject partition column type changes and quote rule identifiers
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(table): keep time index widening for partition columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(partition): replace PartitionFunction with IcebergHash
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(partition): describe iceberg_hash in the function module
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: quote partition rule identifiers in SHOW CREATE TABLE expectations
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(partition): check hash rules in information_schema and repartition target columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): coerce hash partition rules against the evaluation schema
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): keep quotes in partition column names when building filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: Support procedure_state() using the last procedure ID
Signed-off-by: hahaha <a310608318@gmail.com>
* feat: Fix and add tests according to review comments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: the zero-argument signature in ADMIN swallows explicit arguments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: fix the CI failures caused by SQL version issues
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: apply the changes based on the review
Signed-off-by: hahaha <a310608318@gmail.com>
---------
Signed-off-by: hahaha <a310608318@gmail.com>
* feat: postgres copy-in support
* feat: add postgres copy-in framework
* feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher
Follow-up to #9302: begin_copy_in now uses the batching-aware
new_query_context, so copy-in batches are submitted to the
pending-rows batcher like SQL INSERTs when PostgreSQL batching is
enabled for the connection, and keep the direct write path when it
is not. The batcher acknowledges synchronously, so the
CommandComplete row count stays exact in both modes.
Adds test_postgres_copy_in_with_batching asserting the batcher
flush metric increments only when batching is enabled.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options
The WITH (...) option list is now open instead of restricted to
sqlparser's COPY grammar:
- COPY ... FROM STDIN statements are parsed textually (new
copy_in::parse module, tokenizer-based) so arbitrary KEY value
entries reach the codecs; sqlparser would reject options it does
not know before we ever see them. WITH is optional, matching
PostgreSQL.
- CopyOptionSet becomes an open ordered map of KEY -> String |
Number | Ident | Boolean | Flag with typed take_* accessors
(take_string, take_single_byte, take_flag, raw take_value) and
case-insensitive keys; duplicate options are rejected.
- Each format codec takes the options it understands and rejects the
leftovers with one message naming them ('COPY option(s) X are not
supported by format Y'), so a typo cannot be silently ignored.
The text format now also rejects HEADER/QUOTE/ESCAPE instead of
ignoring them, matching PostgreSQL.
A future format (e.g. influxdb line protocol with
WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec
and a registry entry; no parser changes.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing
Replace the hand-rolled CSV record parser and per-value conversion
with arrow-csv's incremental Decoder — the same engine backing
COPY ... FROM 'file.csv' in common-datasource — so both COPY routes
accept identical data. The decoder handles record splitting (quoting,
escaped quotes, custom delimiters, CRLF, headers, chunk boundaries)
and typed conversion against the targeted columns' schema in one
pass; decoded batches become insert rows via the existing
api::helper::vectors_to_rows and datatypes Helper::try_into_vectors,
and the decode schema comes from the greptime Schema's arrow
projection.
The codec framework now has parsers emit complete insertable rows
(api Row) instead of raw byte fields: create_parser takes the
targeted columns and is fallible; text converts with the shared
text-representation helpers, csv with the arrow decoder.
Documented consequences of sharing the engine: JSON and Binary
columns and bare epoch-integer timestamps are rejected for csv, and
an empty field — quoted or not — is NULL; a custom NULL string is
rejected accordingly.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(postgres): make the copy-in framework explicitly format-open
Reframe the codec framework docs so nothing assumes csv is the only
other text format: the shared pieces (Field, convert_text_field,
check_text_representable_type) are described as building blocks for
any format whose records carry values as text, while formats with
their own typed decoding (csv today) build rows themselves. State
explicitly that formats are not limited to PostgreSQL's own —
GreptimeDB-specific formats (e.g. a line protocol) register like any
other and define their own options through the open WITH clause.
No code changes beyond a test rename and a doc clarifier.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(postgres): driver-level coverage for CSV copy-in options
Add an end-to-end test through tokio-postgres exercising HEADER,
custom DELIMITER, quoted fields with embedded newlines, escaped
quotes and CRLF record separators, with read-back verification
through the real insert path. WITH is omitted, matching PostgreSQL.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(postgres): drop the raw-socket copy-in tests
The simple-protocol raw client duplicated what tokio-postgres and the
integration tests already cover; keep the driver-based tests only.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode
- text feed: take the longest complete line prefix in one drain and
iterate lines over the slice, instead of draining per line (each
drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of
short lines)
- csv check_column_type: delegate to the shared
check_text_representable_type and additionally reject Binary/Json
- decode_bytea: use the hex crate instead of a hand-rolled decoder
- semantic_type_of: use ColumnSchema::is_time_index() like the flow
adapter's column_schemas_to_proto, instead of index lookups
- parse: drop the unused expect_ident parameter; document the
identifier-case decision
- PostgresServer:🆕 document that query_handler and copy_in_handler
are the same frontend instance
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: remove unrelated files
* feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format
Remove the hand-rolled PostgreSQL text-format codec (backslash
escapes, octal, \N nulls) that arrow-csv cannot express, keeping
arrow-csv-based csv as the only implemented format.
- COPY FROM STDIN without FORMAT now fails with guidance ('requires
an explicit format in this version; use WITH (FORMAT csv)') rather
than silently defaulting to the removed format
- FORMAT text/txt and FORMAT binary report 'not supported yet' via
registry stubs instead of unknown-format errors
- codec.rs shrinks to the trait framework: the text-representation
helpers (Field, convert_text_field, bytea hex, timestamp parsing)
are removed with the codec; the module docs note where to find
their shape should a future text-serializing format need them
- csv implements check_column_type directly (no shared default)
Breaking for clients relying on the PostgreSQL default text format;
explicit WITH (FORMAT csv) is required.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics
- COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher
each flush pays the full insert pipeline (catalog lookup, schema
check, partition split, region write round trip), so 4x larger
batches cut round trips 4x at linearly-bounded memory; the constant
also sizes the arrow decoder batch. Comment documents the trade-off
and points high-rate users at the pending-rows batcher, which turns
flushes into enqueues.
- rows buffer pre-allocates COPY_IN_FLUSH_ROWS.
- flush_copy_in warns on unexpected output variants instead of
silently counting 0 rows.
- batching wire test feeds 70000 rows to stay above the new threshold.
Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv)
sends FORMAT csv and succeeds; default \copy (text) gets the
'requires an explicit format' guidance; COPY TO STDOUT round-trips.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): reset copy-in state on successful CopyDone
on_copy_done left the CopyInState in place (inert but holding buffers)
until the next COPY overwrote it; take it like on_copy_fail does, so
buffers are freed eagerly and a stray post-done CopyData gets the
protocol-violation error instead of a finished parser.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: bypass typo check
* chore: format change
* fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries
The csv feed loop flushed the arrow-csv decoder at every CopyData
chunk end — including partial batches at record boundaries — but
arrow-csv's contract permits flush() only after decode() returned 0
(a full batch of complete records). Flushing outside that point
corrupts subsequent decoding: fields mis-parse shifted by the size
of the record prefix buffered so far ('000001,host-001,...' parses
as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse,
silently writing wrong values. Which chunks trigger it depends on
where record boundaries land, so it appears intermittent with
random-length data (psql 8KB / pgx ~64KB chunking both hit it).
Adopt the strict decoder protocol: feed decodes until input is
exhausted and flushes only when decode returns 0 (batch full);
trailing partial batches surface at the EOF signal in finish().
With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is
unchanged; sub-threshold streams now emit their rows at CopyDone
instead of per chunk, which is semantically identical.
Adds test_csv_chunked_feeds_are_alignment_safe: the same payload
fed at chunk sizes 1..65531 (plus shifted starts) with every row's
content verified — reproduces the corruption on the old loop at
chunk sizes 7, 8, 13, 64 and 1000.
Verified with the reporter's Go reproducer (pgx, content-checked):
10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the
previously failing benchmark sizes 500/1000/2000/200/800 all round-
trip exactly.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: bypass spell check
* chore: bypass spell check
* fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter
Three review findings (shuiyisong, PR #9297):
[P1] Enforce the catalog restriction on COPY targets. copy_in_table now
applies validate_catalog_and_schema when QueryOptions::
disallow_cross_catalog_query is set, mirroring SQL INSERT's
check_permission path, BEFORE begin_copy_in switches the query context
to the target catalog (which previously made the restriction
unenforceable downstream). Cross-catalog COPY is now rejected with
'Illegal access to catalog' like INSERT.
[P2] Canonicalize unquoted identifiers like the SQL parser
(ParserContext::canonicalize_identifier): unquoted table-name parts and
columns fold to lowercase, quoted ones keep their case. COPY Metrics
(Host, Ts) now addresses the table CREATE TABLE Metrics created, and
no longer silently misroutes into a coexisting quoted "Metrics".
[P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal
tab while still rejecting newlines and other control characters, and
the option parser now also accepts E'...' escape-string constants
(DELIMITER E'\t'), processing PostgreSQL's backslash escapes.
Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round-
trip into the right tables, cross-catalog rejection asserted in a new
frontend unit test (QueryOptions plugin + second catalog), and both
DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure
Two review findings (shuiyisong, PR #9297):
[P2] Boolean columns round-trip between the COPY directions. arrow-csv
only parses true/false while PostgreSQL COPY OUT emits t/f, so
boolean tables failed to re-import their own export (22P02). Boolean
columns now decode as Utf8 in the arrow schema and flush_batch
converts them with PostgreSQL's boolean input grammar (true/false,
t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace
trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact
bytes are fed back via COPY FROM STDIN and verified.
[P2] Drop the COPY state on every server-side failure. on_copy_data
and on_copy_done now take ownership of the state up front, so parse,
conversion or insert errors drop the buffered data instead of
retaining it on long-lived or pooled connections until the next COPY
or connection close (the reviewer measured ~17 MiB retained after a
failed 8 MiB row); on_copy_data success puts the state back. An
e2e test drives a failed COPY followed by a successful one on the
same connection.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): interpret offset-free COPY timestamps in the session timezone
Review finding (shuiyisong, PR #9297): the arrow decode schema carried
Timestamp fields with no timezone, so the CSV decoder parsed
offset-free timestamp strings as UTC while SQL INSERT parses them in
the session timezone — the same input string silently stored a
different instant depending on the ingestion method (8h apart in
Asia/Shanghai).
CopyInCodec::create_parser now receives the session timezone and the
csv decode schema attaches it to Timestamp fields, so naive strings
parse like INSERT; explicit offsets in the input (Z, +08:00) still
take precedence, and UTC sessions are unchanged.
Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z)
and the end-to-end test_postgres_copy_in_session_timezone_parity,
which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the
same string, and asserts both CAST(ts AS BIGINT) readbacks agree on
1704038400000.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): don't unescape E-string COPY options twice
Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already
decodes PostgreSQL escape-string constants before returning
Token::EscapedStringLiteral, so the second unescape_pg_string pass
altered the decoded value — DELIMITER E'\\t' (a literal backslash
followed by t, two characters) was turned into a tab and wrongly
accepted as a single-byte delimiter.
Use the EscapedStringLiteral payload as-is, like SingleQuotedString,
and delete unescape_pg_string. Regression coverage distinguishes the
three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t'
-> rejected as a multi-character delimiter.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN
Two review findings (shuiyisong, PR #9297):
[P2] Preserve timestamp instants across COPY OUT/IN in non-UTC
sessions. COPY OUT rendered naive UTC text while COPY IN (since the
session-tz fix) interprets offset-free strings as local time, so
re-importing an export shifted instants by the zone offset. COPY OUT
now renders timestamps with an explicit session-timezone offset
(2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the
session tz (arrow-pg converts to a tz-aware DateTime) and the pg field
type becomes TIMESTAMPTZ so the offset-bearing text format applies.
Embedded offsets win on re-import regardless of session, keeping the
OUT -> IN round trip exact; UTC sessions are unchanged.
[P2] Preserve quoted empty strings. The shared arrow-csv decoder
nulls every empty field, so a quoted empty string ("" — an empty
STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner
runs over the same bytes with the decoder's own splitting rules and
records per field whether it was quoted; flush_batch restores quoted
empties to empty strings for string columns, while unquoted empty
fields stay NULL. Masks track records across chunk boundaries and
align with decoder batches.
Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip
(Asia/Shanghai export asserted to carry +08, exact-epoch re-import
comparison) and test_postgres_copy_in_quoted_empty_string (NULL,
quoted empty and non-empty in one import), plus unit tests including
a chunk-boundary sweep for the quoted-empty mask alignment.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns
Two review findings (fengjiachun, PR #9297):
[P2] Doubled quotes desynchronized the shadow quote-mask scanner. In
QuoteInQuoted the doubled-quote branch skipped the second quote AND the
following byte, so a closing quote or delimiter directly after an
escaped quote was swallowed and later fields lost their masks --
import semantics depended on CopyData chunk boundaries. The branch now
consumes only the escaped quote and returns to the in-quoted state,
mirroring the decoder. Regression: "a""b","" ... verified across
chunk sizes 1-64.
[P2] A NOT NULL string column rejected quoted empty strings before
restoration could run: the decode schema preserved the target
nullability, and arrow-csv's batch validation rejects the temporary
NULL a quoted empty decodes to. String fields now decode as nullable
(safe: quoted-vs-unquoted is exactly the restoration's job), and the
target NOT NULL constraint is enforced in flush_batch after
restoration, so "" imports as an empty string while an unquoted empty
field fails with a not-null-constraint error. Regressions at unit and
e2e level: NOT NULL column accepting "" and rejecting unquoted NULL.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators
Two review findings (fengjiachun, PR #9297):
[P2] Blank lines emitted a spurious empty mask. csv-core skips blank
lines, but the scanner queued one mask per blank record, so every
subsequent decoded row consumed the wrong mask (a leading blank line
turned a quoted empty into NULL and an unquoted empty into an empty
string, and tripped NOT NULL on the wrong row). A blank line before the
header also shifted the header expectation. end_record now returns no
mask for a completely blank record without consuming the header
expectation.
[P2] Standalone CR was not a record terminator. The decoder accepts CR,
CRLF and LF alike; the scanner only recognized CR before LF or at EOF,
so CR-separated input desynchronized masks and flush_batch failed with
'quote masks out of sync'. is_record_end now treats CR alone as a
terminator, consuming one byte (two for CRLF), while a CR at a chunk
boundary still defers via carry so CRLF split across CopyData chunks
stays one terminator.
Regressions: blank lines before/inside records and before a header
(NULL/empty preservation plus NOT NULL behavior), and mixed
CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing
CR-only record.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY
The shadow quote-mask scanner — kept to preserve PostgreSQL's
quoted-empty-string vs unquoted-NULL distinction — mirrored the
decoder's record-splitting grammar by hand and required a fix for every
rule it drifted on (doubled quotes, blank lines, CR terminators, NOT
NULL interplay). Per maintainers' decision, that complexity is not
worth the divergence: COPY FROM STDIN now behaves EXACTLY like the
file-based COPY ... FROM 'file.csv' — an empty field, quoted or not,
imports as NULL (arrow-csv's engine semantics, identical in both
routes).
Removed: QuoteMaskScanner and helpers, the always-nullable string
decode schema, the restoration + NOT NULL enforcement in flush_batch,
set_string_at, and the scanner-era unit tests. The decode schema again
preserves target nullability, so the decoder enforces NOT NULL itself,
same as the file route.
Kept (small, schema-level, no second grammar): boolean spellings
(t/f) and session-timezone timestamp decoding, both addressing
separate review findings.
E2E: test_postgres_copy_in_empty_field_is_null pins the parity —
quoted empty, unquoted empty and non-empty in one import.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: remove empty file
* chore: lint issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
sqlx 0.8 pins the unmaintained webpki-roots 0.26 line for its rustls
trust store, which still contains the recently compromised e-Tugra CA
root. Upgrading sqlx to 0.9 switches its dependency to webpki-roots
1.x, and the lockfile now resolves to 1.0.10 where the distrusted
root has been removed from the compiled-in Mozilla trust store.
sqlx 0.9 requires dynamic SQL strings to be explicitly audited via
the new SqlSafeStr trait: only &'static str literals are accepted
as-is. All affected call sites (MySQL KV backend, MySQL election,
integration tests and fuzz targets) now wrap their internally
generated SQL templates in AssertSqlSafe; the wrapped SQL is built
from operator config or test-local identifiers only, and dynamic
data keeps flowing through bind parameters.
The mysql-rsa feature is enabled alongside mysql to preserve 0.8
authentication behavior: sqlx 0.9 gates non-TLS RSA password auth
(caching_sha2_password / sha256_password) behind it.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(partition): coerce partition exprs against the batch schema before routing
Batch partition routing (MultiDimPartitionRule::split_record_batch) builds
physical expressions via PartitionExpr::try_as_physical_expr, which did no
type coercion. After widening the time index unit (e.g. ms -> ns via
ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition
bounds keep their original unit, so evaluating the bounds against the batch
failed with:
Invalid comparison operation: Timestamp(ns) < Timestamp(ms)
Row inserts were unaffected because find_region compares Values and
Timestamp::cmp converts units; scans were unaffected because
PredicateGroup::new coerces the region partition expr against the region
schema first.
Coerce the logical partition expression against the evaluation schema in
try_as_physical_expr before creating the physical expression, the same way
PredicateGroup does. This also covers the region partition filter built for
SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this
conversion.
Fixes#9747
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: regenerate change_col_type result after comment update
The ts_widen_part comment in change_col_type.sql was reworded after the
result file was last generated; sqlness echoes SQL comments into the
result, so the stale comment made the case fail. Regenerate it.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(otlp): chunk trace spans across resource groups
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): plan spans of a mixed binary/JSON chunk individually
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(otlp): describe cross-group trace chunking
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add column encoding option and set it for trace span ids
COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'.
Either disables the dictionary for the column; delta_binary_packed is
limited to integer and timestamp columns.
OTLP trace v1/v2 tables now create span_id and parent_span_id with plain
encoding and 64KiB pages, and timestamp_end with delta_binary_packed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add compatibility cases for column encoding options
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: move column encoding downgrade checks into downgrade_compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(servers): accept application/x-json-stream for log ingestion
Support fluentbit's json_stream output format on the HTTP log ingestion
endpoints by accepting `application/x-json-stream` as a payload content
type.
The payload is parsed as concatenated JSON -- multiple JSON values placed
back-to-back without a separator, e.g. `{"a":1}{"b":2}` -- reusing the
streaming value iterator already used for `application/json`, so the
behavior for concatenated or mixed payloads is identical.
Aimed at #5404 (msgpack support is left as a possible follow-up).
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
* chore(servers): remove redundant comments per review feedback
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
---------
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
* fix(jaeger): return the most recent traces when a search hits its limit
Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search
matching more traces than the limit returned an arbitrary subset that
changed between identical requests. Order traces by their latest matching
span before applying the limit.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(jaeger): name the trace dataframe builder with a verb and document execution
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The flow and nested-MergeScan ingress tests hardcoded "follower" among
the read-preference values that must be rejected, coupling them to the
current leader-only ReadPreference enum. Assert only genuinely invalid
values there, matching the wire-values test which already limits itself
to ["", "unknown"]. This keeps the ingress tests valid for builds that
extend ReadPreference with more variants.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(operator): route shared requests through NodeManager
Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): preserve retry policy through error hints
Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): restore sparse partition tags when rebuilding writes
Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): stop detached write retries when callers cancel
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(frontend): configure retries across distributed write paths
Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(jaeger): support v3 HTTP query APIs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(jaeger): preserve full traces with v3 time hints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(jaeger): clarify validation of v3 lookup hints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(jaeger): bound v3 searches and preserve v1 binary attributes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(jaeger): match resource attributes in v0 tag filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>