Commit Graph
509 Commits
Author SHA1 Message Date
Ning Sun c679ca5fdc feat(ingestion): configurable time index unit for auto-created physical metric tables (#10189)
* feat(ingestion): configurable time index unit for auto-created physical metric tables

Auto-created physical metric tables hardcoded a millisecond time index.
Add an `auto_create_physical_table_timestamp_unit` option (frontend and
standalone configs, default "Millisecond") that selects the time index
unit for physical metric tables created by auto table creation. New
logical tables created under such a physical table inherit its unit, so
Prometheus remote write, OTLP metrics, and other metric-engine ingestion
paths write timestamps in the configured unit.

The option threads through all auto-create sites: the operator inserter's
`create_physical_table_on_demand`, the frontend instance's batched
logical-table flush path (`create_metric_physical_table_if_missing`),
the logical-table batcher's unit alignment fallback, and the OTLP metric
preflight validation unit. Millisecond remains the default, so existing
behavior is unchanged unless configured.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(otlp): match default physical schema to configured timestamp unit

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-16 17:00:51 +00:00
Ning Sun 7e76d28908 fix(postgres): account for buffered InfluxDB and OTLP COPY bodies (#10450)
Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-16 15:59:31 +00:00
shuiyisong 0a93810c23 fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): reserve memory for table batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): prefix the labels column

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): hide exemplar table from metadata

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): reconcile source tags before writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): prune tags and retain source IDs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): record slow queries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: account for held memory in batch admission

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: config comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: simplify exemplar and memory tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: deduplicate exemplar setup

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-16 11:55:20 +00:00
Ning SunandLei, HUANG b30e002cef feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>])
ingests one Prometheus Remote Write request body over the postgres wire
protocol — a GreptimeDB-specific, binary COPY format (wire format code
1). The payload is byte-for-byte the body a remote-write client POSTs
to /v1/prometheus/write, compression included: buffered until CopyDone
and completed through the exact ingestion core that endpoint runs.

To share that core, the HTTP handlers' ingestion bodies are extracted
into axum-free functions (ingest_remote_write_v1/v2 plus the context
extension stamping), and the HTTP routes become thin wrappers — their
behavior is unchanged, and the copy layer runs the identical decode,
validation, conversion, batched/prepared write paths, per-batch
write-progress accounting and context extensions, under the
connection's regular postgres context.

The WITH options carry what the HTTP endpoint takes as its
content-type/-encoding and query parameters; session-provided values
(database, user) are not repeated:
- version 1|2 (default 1), matching the content-type's proto
- zstd, matching Content-Encoding: zstd (snappy by default)
- physical_table, matching the ?physical_table= parameter
- pipeline_name/pipeline_version, matching the pipeline headers
  (v1 only; for v2 they are accepted and ignored, exactly like the
  HTTP route ignores them)

Table-less like the other ingestion formats; one COPY statement is one
bounded request (shared 64 MiB limit); the row count reports written
rows (v1) or written samples plus histograms (v2). The flush enforces
the frontend's shared request-memory quota (max_in_flight_write_bytes)
— the same limiter the HTTP ingestion protocols use, so COPY cannot
bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write
carries the handler, metric-engine flag, validation mode and shared
memory limiter.

e2e: v1 body chunked across CopyData boundaries; v2 samples with
readback; physical_table option routing; garbage-body decode failure
with connection reuse; a tiny-quota regression where a compressed body
whose decompressed size exceeds the budget fails like the HTTP route.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(servers): propagate COPY body reservations and test write accounting

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): share Prometheus logical batching with COPY ingestion

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): narrow Prometheus COPY scope and share write outcomes

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-11-16 02:45:34 +00:00
jeremyhi 9eb027f738 test: disable the event recorder in tests that count WAL writes (#10316)
The object store WAL restart test and the skip-WAL tests built on
MockInstanceImpl assume only the tables they create write to the WAL.
The event recorder also writes greptime_private.events into the same
WAL, so when it flushes during the test (the first tick of its interval
lands after the first DDL event on Windows), the extra region keeps WAL
objects from being collected or shows up in the WAL snapshot.

Record no events in these tests.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-10 09:29:35 +00:00
shuiyisong ee1fc7782f feat(prometheus): query persisted metric exemplars (#9924)
* feat(prometheus): add exemplar queries and lifecycle coverage

Expose /v1/prometheus/api/v1/query_exemplars with exact time bounds and series selectors. Resolve and authorize source selectors before scanning exemplar storage, enforce query limits and cancellation, preserve exact observation identity, and return warnings for malformed stored rows.

Includes HTTP and authorization tests, pre-created-table and concurrent-DDL coverage, OTLP Arrow ingestion checks, and cross-protocol lifecycle tests across memory, WAL, SST, compaction and reopen. The query-dependent ingestion tests live here so every lower layer builds independently. Add metric-exemplar usage and retention guidance.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): preserve query labels with source identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): floor bounds and add audit hooks

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* • fix(exemplar): include aggregation parameter selectors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): avoid v1 exemplar quota stalls

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align exemplar query timestamps with Prometheus

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove unnecessory doc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-10 08:11:49 +00:00
Weny Xu 65dc999180 feat: support cancelling compaction tasks by task ID (#10229)
* feat: add ADMIN compaction task cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject metadata region compaction cancellation before routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin greptime-proto to merged cancellation commit

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 07:38:08 +00:00
jeremyhi 297a343fd4 fix(query): use current partition columns in prepared plans (#10244)
A prepared statement keeps its logical plan, and that plan holds the
table as it was at prepare time. The analyzer reads the partition
columns from that table while the regions to scan come from the current
route. After a table's partition columns change, the analyzer can
consider a query safe to push down, so each region runs the whole query
and the frontend concatenates their results. The same happens to the
source of a prepared INSERT ... SELECT, which then stores wrong values.

Before executing a kept plan, compare each scanned table's partition
columns with the current table found by id. Scans of a table whose
columns changed get the current table when the schema is unchanged.
Otherwise they keep their schema, because a scan's projection holds
indices into it, and report the current columns as absent from it, so
the query keeps merging on the frontend.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-09 11:08:47 +00:00
Weny Xu 8bdb7d5509 feat: add information_schema.compaction_tasks (#10173)
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.

Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 03:17:48 +00:00
fys c4cd76c607 perf: narrow remote read scans using time hints (#10180)
Signed-off-by: fys <fengys1996@gmail.com>
2026-11-09 03:12:22 +00:00
Lei, HUANG 557843dd68 fix(catalog): use advertised peer address as process identity (#10153)
* fix(catalog): use advertised peer address as process identity

The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.

Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
  manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
  in list_all_processes, and for local/remote routing in kill_process.

Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.

The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: keep local peer address reserved in kill-routing test

The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.

Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 18:18:38 +00:00
Ning Sun b21b1cdb69 fix(function): require ADMIN statement privileges to execute admin functions (#10120)
* fix(function): require ADMIN statement privileges to execute admin functions

Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.

Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.

Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(function): expose admin privilege check to downstream macros

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-04 18:18:33 +00:00
Weny Xu f7e0913ff0 test(meta): cover datanode draining with Kafka and local WAL (#10060)
* test(meta): cover datanode draining with Kafka and local WAL

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): fence preferred automatic drain target

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-04 18:18:26 +00:00
shuiyisong fd74b9239e feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars

OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.

Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(otlp): bind exemplars to source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): bound exemplar memory and preserve series

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars without metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars for ordinary tables

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(exemplar)!: prefix the shared table name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): avoid waiting with body permits held

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:09:57 +00:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
Ning Sun a09ebc70e6 feat(postgres): OTLP binary payload ingest via COPY FROM STDIN (metrics, traces, logs) (#10050)
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:

- signal 'metrics' (default): ExportMetricsServiceRequest, completed
  through OpenTelemetryProtocolHandler::metrics with the same context
  preparation as the HTTP route (OtlpMetricCtx from the options); rows
  are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
  trace table name and pipeline name/version/params from the options
  (an explicit pipeline name is required, exactly like the HTTP traces
  endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
  table name, pipeline options and extract keys from the options, and
  the same OtlpLogDirect default as the HTTP route; rows are affected
  rows.

The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.

Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.

e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-02 08:57:05 +00:00
shuiyisong f2d74650d9 fix: optimize Prometheus remote write v2 ingestion (#9928)
* fix: optimize Prometheus remote write v2 ingestion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: correct remote write v2 memory accounting and response counts

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-30 08:56:34 +00:00
shuiyisong ec240ad7e6 feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables

Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions.

Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): preserve reserved sparse key semantics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): clarify sparse errors and avoid panics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito): reduce sparse key validation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito-codec): reduce sparse validation cost

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(mito): remove ordinary sparse support

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-29 05:45:07 +00:00
Dragon Roar b21cd16f1d feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup

Signed-off-by: lczllx <2181719471@qq.com>

* fix(datanode): harden the startup object storage self-test

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): cover the File store skip in the object store self-test

Signed-off-by: lczllx <2181719471@qq.com>

* refactor(datanode): rename the object storage self-test to probe

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): disable object store self-test in sqlness and integration tests

Signed-off-by: lczllx <2181719471@qq.com>

* docs(datanode): correct what the object storage probe claims to cover

Signed-off-by: lczllx <2181719471@qq.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
2026-10-28 17:09:29 +00:00
Ning Sun c818b6ac2a feat(postgres): influxdb line protocol ingest via COPY FROM STDIN (#9982)
* feat(postgres): influxdb line protocol ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT influxdb [, precision 'ns']) ingests
InfluxDB line protocol over the postgres wire protocol, reusing the
HTTP ingestion pipeline wholesale for correctness and performance
parity: the whole payload is buffered and completed once at CopyDone
through the frontend's InfluxdbLineProtocolHandler::exec — conversion,
permissions, line-protocol interceptors, timestamp alignment to table
time-index units, merge mode and inserts are the same path an HTTP
write of the same bytes takes, under the connection's regular postgres
query context.

Formats are an explicit, closed enum (CopyFormat::Csv |
CopyFormat::Influxdb(options)); each variant states its own rules in
the copy lifecycle's match arms — csv requires a target table and
parses incrementally, influxdb forbids one (each line names its own
measurement) and buffers one bounded request body. The COPY layer owns
only the buffering: record boundaries, including newlines inside quoted
field values and after escapes, belong to the line-protocol parser
alone — one COPY statement is one bounded ingestion request (64 MiB;
larger imports are multiple COPY statements).

- Wiring: PostgresServer::with_influxdb_handler attaches the shared
  line-protocol handler; unwired servers reject the format when a copy
  starts.
- Grammar: COPY FROM STDIN without a table is scanned (an unquoted
  FROM cannot be a table name); table-bound formats reject it with
  guidance, influxdb rejects a named table.
- Options: precision accepts the HTTP endpoint's values via the shared
  parse_time_precision (moved to servers::influxdb), defaulting to
  nanoseconds.

Review findings (fengjiachun) are fixed at the root by removing the
pre-splitter rather than repairing it: the retired naive line splitter
broke quoted multiline records at chunk/flush boundaries (found by
removing record splitting entirely), drained the buffer quadratically
(found by append-only buffering), and capped nothing (found by an
explicit payload limit).

e2e: multi-measurement ingest with chunked feeds; a 32,768-physical-line
flush-threshold regression with a multiline quoted record split across
CopyData chunks; precision 'ms' epoch verification; invalid UTF-8 and
trailing-record-without-newline handling; payload limit; table-named
rejection; malformed-line abort with connection recovery.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: typo

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-28 08:00:41 +00:00
discord9 984f80e277 fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit- or timezone-changing time casts

A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.

The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit-changing timestamp wrappers

to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.

The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regen windowed sort sqlness after wrapper guard

The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): simplify windowed sort cast regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 11:51:43 +00:00
dennis zhuang 2e84833ac8 feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): reject partition column type changes and quote rule identifiers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(table): keep time index widening for partition columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(partition): replace PartitionFunction with IcebergHash

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(partition): describe iceberg_hash in the function module

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: quote partition rule identifiers in SHOW CREATE TABLE expectations

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(partition): check hash rules in information_schema and repartition target columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): coerce hash partition rules against the evaluation schema

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): keep quotes in partition column names when building filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 09:23:00 +00:00
shuiyisong 5e9bb3c689 feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align native histogram ingestion with remote read validation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-27 03:09:13 +00:00
Ning Sun 80df1c397b feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support

* feat: add postgres copy-in framework

* feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher

Follow-up to #9302: begin_copy_in now uses the batching-aware
new_query_context, so copy-in batches are submitted to the
pending-rows batcher like SQL INSERTs when PostgreSQL batching is
enabled for the connection, and keep the direct write path when it
is not. The batcher acknowledges synchronously, so the
CommandComplete row count stays exact in both modes.

Adds test_postgres_copy_in_with_batching asserting the batcher
flush metric increments only when batching is enabled.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options

The WITH (...) option list is now open instead of restricted to
sqlparser's COPY grammar:

- COPY ... FROM STDIN statements are parsed textually (new
  copy_in::parse module, tokenizer-based) so arbitrary KEY value
  entries reach the codecs; sqlparser would reject options it does
  not know before we ever see them. WITH is optional, matching
  PostgreSQL.
- CopyOptionSet becomes an open ordered map of KEY -> String |
  Number | Ident | Boolean | Flag with typed take_* accessors
  (take_string, take_single_byte, take_flag, raw take_value) and
  case-insensitive keys; duplicate options are rejected.
- Each format codec takes the options it understands and rejects the
  leftovers with one message naming them ('COPY option(s) X are not
  supported by format Y'), so a typo cannot be silently ignored.
  The text format now also rejects HEADER/QUOTE/ESCAPE instead of
  ignoring them, matching PostgreSQL.

A future format (e.g. influxdb line protocol with
WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec
and a registry entry; no parser changes.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing

Replace the hand-rolled CSV record parser and per-value conversion
with arrow-csv's incremental Decoder — the same engine backing
COPY ... FROM 'file.csv' in common-datasource — so both COPY routes
accept identical data. The decoder handles record splitting (quoting,
escaped quotes, custom delimiters, CRLF, headers, chunk boundaries)
and typed conversion against the targeted columns' schema in one
pass; decoded batches become insert rows via the existing
api::helper::vectors_to_rows and datatypes Helper::try_into_vectors,
and the decode schema comes from the greptime Schema's arrow
projection.

The codec framework now has parsers emit complete insertable rows
(api Row) instead of raw byte fields: create_parser takes the
targeted columns and is fallible; text converts with the shared
text-representation helpers, csv with the arrow decoder.

Documented consequences of sharing the engine: JSON and Binary
columns and bare epoch-integer timestamps are rejected for csv, and
an empty field — quoted or not — is NULL; a custom NULL string is
rejected accordingly.

Signed-off-by: Ning Sun <sunning@greptime.com>

* docs(postgres): make the copy-in framework explicitly format-open

Reframe the codec framework docs so nothing assumes csv is the only
other text format: the shared pieces (Field, convert_text_field,
check_text_representable_type) are described as building blocks for
any format whose records carry values as text, while formats with
their own typed decoding (csv today) build rows themselves. State
explicitly that formats are not limited to PostgreSQL's own —
GreptimeDB-specific formats (e.g. a line protocol) register like any
other and define their own options through the open WITH clause.

No code changes beyond a test rename and a doc clarifier.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test(postgres): driver-level coverage for CSV copy-in options

Add an end-to-end test through tokio-postgres exercising HEADER,
custom DELIMITER, quoted fields with embedded newlines, escaped
quotes and CRLF record separators, with read-back verification
through the real insert path. WITH is omitted, matching PostgreSQL.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test(postgres): drop the raw-socket copy-in tests

The simple-protocol raw client duplicated what tokio-postgres and the
integration tests already cover; keep the driver-based tests only.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode

- text feed: take the longest complete line prefix in one drain and
  iterate lines over the slice, instead of draining per line (each
  drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of
  short lines)
- csv check_column_type: delegate to the shared
  check_text_representable_type and additionally reject Binary/Json
- decode_bytea: use the hex crate instead of a hand-rolled decoder
- semantic_type_of: use ColumnSchema::is_time_index() like the flow
  adapter's column_schemas_to_proto, instead of index lookups
- parse: drop the unused expect_ident parameter; document the
  identifier-case decision
- PostgresServer:🆕 document that query_handler and copy_in_handler
  are the same frontend instance

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: remove unrelated files

* feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format

Remove the hand-rolled PostgreSQL text-format codec (backslash
escapes, octal, \N nulls) that arrow-csv cannot express, keeping
arrow-csv-based csv as the only implemented format.

- COPY FROM STDIN without FORMAT now fails with guidance ('requires
  an explicit format in this version; use WITH (FORMAT csv)') rather
  than silently defaulting to the removed format
- FORMAT text/txt and FORMAT binary report 'not supported yet' via
  registry stubs instead of unknown-format errors
- codec.rs shrinks to the trait framework: the text-representation
  helpers (Field, convert_text_field, bytea hex, timestamp parsing)
  are removed with the codec; the module docs note where to find
  their shape should a future text-serializing format need them
- csv implements check_column_type directly (no shared default)

Breaking for clients relying on the PostgreSQL default text format;
explicit WITH (FORMAT csv) is required.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics

- COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher
  each flush pays the full insert pipeline (catalog lookup, schema
  check, partition split, region write round trip), so 4x larger
  batches cut round trips 4x at linearly-bounded memory; the constant
  also sizes the arrow decoder batch. Comment documents the trade-off
  and points high-rate users at the pending-rows batcher, which turns
  flushes into enqueues.
- rows buffer pre-allocates COPY_IN_FLUSH_ROWS.
- flush_copy_in warns on unexpected output variants instead of
  silently counting 0 rows.
- batching wire test feeds 70000 rows to stay above the new threshold.

Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv)
sends FORMAT csv and succeeds; default \copy (text) gets the
'requires an explicit format' guidance; COPY TO STDOUT round-trips.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): reset copy-in state on successful CopyDone

on_copy_done left the CopyInState in place (inert but holding buffers)
until the next COPY overwrote it; take it like on_copy_fail does, so
buffers are freed eagerly and a stray post-done CopyData gets the
protocol-violation error instead of a finished parser.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: bypass typo check

* chore: format change

* fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries

The csv feed loop flushed the arrow-csv decoder at every CopyData
chunk end — including partial batches at record boundaries — but
arrow-csv's contract permits flush() only after decode() returned 0
(a full batch of complete records). Flushing outside that point
corrupts subsequent decoding: fields mis-parse shifted by the size
of the record prefix buffered so far ('000001,host-001,...' parses
as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse,
silently writing wrong values. Which chunks trigger it depends on
where record boundaries land, so it appears intermittent with
random-length data (psql 8KB / pgx ~64KB chunking both hit it).

Adopt the strict decoder protocol: feed decodes until input is
exhausted and flushes only when decode returns 0 (batch full);
trailing partial batches surface at the EOF signal in finish().
With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is
unchanged; sub-threshold streams now emit their rows at CopyDone
instead of per chunk, which is semantically identical.

Adds test_csv_chunked_feeds_are_alignment_safe: the same payload
fed at chunk sizes 1..65531 (plus shifted starts) with every row's
content verified — reproduces the corruption on the old loop at
chunk sizes 7, 8, 13, 64 and 1000.

Verified with the reporter's Go reproducer (pgx, content-checked):
10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the
previously failing benchmark sizes 500/1000/2000/200/800 all round-
trip exactly.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: bypass spell check

* chore: bypass spell check

* fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter

Three review findings (shuiyisong, PR #9297):

[P1] Enforce the catalog restriction on COPY targets. copy_in_table now
applies validate_catalog_and_schema when QueryOptions::
disallow_cross_catalog_query is set, mirroring SQL INSERT's
check_permission path, BEFORE begin_copy_in switches the query context
to the target catalog (which previously made the restriction
unenforceable downstream). Cross-catalog COPY is now rejected with
'Illegal access to catalog' like INSERT.

[P2] Canonicalize unquoted identifiers like the SQL parser
(ParserContext::canonicalize_identifier): unquoted table-name parts and
columns fold to lowercase, quoted ones keep their case. COPY Metrics
(Host, Ts) now addresses the table CREATE TABLE Metrics created, and
no longer silently misroutes into a coexisting quoted "Metrics".

[P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal
tab while still rejecting newlines and other control characters, and
the option parser now also accepts E'...' escape-string constants
(DELIMITER E'\t'), processing PostgreSQL's backslash escapes.

Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round-
trip into the right tables, cross-catalog rejection asserted in a new
frontend unit test (QueryOptions plugin + second catalog), and both
DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure

Two review findings (shuiyisong, PR #9297):

[P2] Boolean columns round-trip between the COPY directions. arrow-csv
only parses true/false while PostgreSQL COPY OUT emits t/f, so
boolean tables failed to re-import their own export (22P02). Boolean
columns now decode as Utf8 in the arrow schema and flush_batch
converts them with PostgreSQL's boolean input grammar (true/false,
t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace
trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact
bytes are fed back via COPY FROM STDIN and verified.

[P2] Drop the COPY state on every server-side failure. on_copy_data
and on_copy_done now take ownership of the state up front, so parse,
conversion or insert errors drop the buffered data instead of
retaining it on long-lived or pooled connections until the next COPY
or connection close (the reviewer measured ~17 MiB retained after a
failed 8 MiB row); on_copy_data success puts the state back. An
e2e test drives a failed COPY followed by a successful one on the
same connection.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): interpret offset-free COPY timestamps in the session timezone

Review finding (shuiyisong, PR #9297): the arrow decode schema carried
Timestamp fields with no timezone, so the CSV decoder parsed
offset-free timestamp strings as UTC while SQL INSERT parses them in
the session timezone — the same input string silently stored a
different instant depending on the ingestion method (8h apart in
Asia/Shanghai).

CopyInCodec::create_parser now receives the session timezone and the
csv decode schema attaches it to Timestamp fields, so naive strings
parse like INSERT; explicit offsets in the input (Z, +08:00) still
take precedence, and UTC sessions are unchanged.

Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z)
and the end-to-end test_postgres_copy_in_session_timezone_parity,
which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the
same string, and asserts both CAST(ts AS BIGINT) readbacks agree on
1704038400000.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): don't unescape E-string COPY options twice

Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already
decodes PostgreSQL escape-string constants before returning
Token::EscapedStringLiteral, so the second unescape_pg_string pass
altered the decoded value — DELIMITER E'\\t' (a literal backslash
followed by t, two characters) was turned into a tab and wrongly
accepted as a single-byte delimiter.

Use the EscapedStringLiteral payload as-is, like SingleQuotedString,
and delete unescape_pg_string. Regression coverage distinguishes the
three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t'
-> rejected as a multi-character delimiter.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN

Two review findings (shuiyisong, PR #9297):

[P2] Preserve timestamp instants across COPY OUT/IN in non-UTC
sessions. COPY OUT rendered naive UTC text while COPY IN (since the
session-tz fix) interprets offset-free strings as local time, so
re-importing an export shifted instants by the zone offset. COPY OUT
now renders timestamps with an explicit session-timezone offset
(2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the
session tz (arrow-pg converts to a tz-aware DateTime) and the pg field
type becomes TIMESTAMPTZ so the offset-bearing text format applies.
Embedded offsets win on re-import regardless of session, keeping the
OUT -> IN round trip exact; UTC sessions are unchanged.

[P2] Preserve quoted empty strings. The shared arrow-csv decoder
nulls every empty field, so a quoted empty string ("" — an empty
STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner
runs over the same bytes with the decoder's own splitting rules and
records per field whether it was quoted; flush_batch restores quoted
empties to empty strings for string columns, while unquoted empty
fields stay NULL. Masks track records across chunk boundaries and
align with decoder batches.

Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip
(Asia/Shanghai export asserted to carry +08, exact-epoch re-import
comparison) and test_postgres_copy_in_quoted_empty_string (NULL,
quoted empty and non-empty in one import), plus unit tests including
a chunk-boundary sweep for the quoted-empty mask alignment.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns

Two review findings (fengjiachun, PR #9297):

[P2] Doubled quotes desynchronized the shadow quote-mask scanner. In
QuoteInQuoted the doubled-quote branch skipped the second quote AND the
following byte, so a closing quote or delimiter directly after an
escaped quote was swallowed and later fields lost their masks --
import semantics depended on CopyData chunk boundaries. The branch now
consumes only the escaped quote and returns to the in-quoted state,
mirroring the decoder. Regression: "a""b","" ... verified across
chunk sizes 1-64.

[P2] A NOT NULL string column rejected quoted empty strings before
restoration could run: the decode schema preserved the target
nullability, and arrow-csv's batch validation rejects the temporary
NULL a quoted empty decodes to. String fields now decode as nullable
(safe: quoted-vs-unquoted is exactly the restoration's job), and the
target NOT NULL constraint is enforced in flush_batch after
restoration, so "" imports as an empty string while an unquoted empty
field fails with a not-null-constraint error. Regressions at unit and
e2e level: NOT NULL column accepting "" and rejecting unquoted NULL.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators

Two review findings (fengjiachun, PR #9297):

[P2] Blank lines emitted a spurious empty mask. csv-core skips blank
lines, but the scanner queued one mask per blank record, so every
subsequent decoded row consumed the wrong mask (a leading blank line
turned a quoted empty into NULL and an unquoted empty into an empty
string, and tripped NOT NULL on the wrong row). A blank line before the
header also shifted the header expectation. end_record now returns no
mask for a completely blank record without consuming the header
expectation.

[P2] Standalone CR was not a record terminator. The decoder accepts CR,
CRLF and LF alike; the scanner only recognized CR before LF or at EOF,
so CR-separated input desynchronized masks and flush_batch failed with
'quote masks out of sync'. is_record_end now treats CR alone as a
terminator, consuming one byte (two for CRLF), while a CR at a chunk
boundary still defers via carry so CRLF split across CopyData chunks
stays one terminator.

Regressions: blank lines before/inside records and before a header
(NULL/empty preservation plus NOT NULL behavior), and mixed
CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing
CR-only record.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY

The shadow quote-mask scanner — kept to preserve PostgreSQL's
quoted-empty-string vs unquoted-NULL distinction — mirrored the
decoder's record-splitting grammar by hand and required a fix for every
rule it drifted on (doubled quotes, blank lines, CR terminators, NOT
NULL interplay). Per maintainers' decision, that complexity is not
worth the divergence: COPY FROM STDIN now behaves EXACTLY like the
file-based COPY ... FROM 'file.csv' — an empty field, quoted or not,
imports as NULL (arrow-csv's engine semantics, identical in both
routes).

Removed: QuoteMaskScanner and helpers, the always-nullable string
decode schema, the restoration + NOT NULL enforcement in flush_batch,
set_string_at, and the scanner-era unit tests. The decode schema again
preserves target nullability, so the decoder enforces NOT NULL itself,
same as the file route.

Kept (small, schema-level, no second grammar): boolean spellings
(t/f) and session-timezone timestamp decoding, both addressing
separate review findings.

E2E: test_postgres_copy_in_empty_field_is_null pins the parity —
quoted empty, unquoted empty and non-empty in one import.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: remove empty file

* chore: lint issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-26 07:08:12 +00:00
shuiyisong 2eff52c757 feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): guard point accounting and benchmark metrics ingestion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove otlp_metric_compat logic

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(otlp): cache physical schema lookups during metric preflight

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: minor rename

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(otlp): batch metric rows and reuse schema validation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: replace super in import path

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-22 13:36:30 +00:00
Ning Sun a7230f7a92 fix(partition): coerce partition exprs against the batch schema before routing (#9752)
* fix(partition): coerce partition exprs against the batch schema before routing

Batch partition routing (MultiDimPartitionRule::split_record_batch) builds
physical expressions via PartitionExpr::try_as_physical_expr, which did no
type coercion. After widening the time index unit (e.g. ms -> ns via
ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition
bounds keep their original unit, so evaluating the bounds against the batch
failed with:

  Invalid comparison operation: Timestamp(ns) < Timestamp(ms)

Row inserts were unaffected because find_region compares Values and
Timestamp::cmp converts units; scans were unaffected because
PredicateGroup::new coerces the region partition expr against the region
schema first.

Coerce the logical partition expression against the evaluation schema in
try_as_physical_expr before creating the physical expression, the same way
PredicateGroup does. This also covers the region partition filter built for
SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this
conversion.

Fixes #9747

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: regenerate change_col_type result after comment update

The ts_widen_part comment in change_col_type.sql was reworded after the
result file was last generated; sqlness echoes SQL comments into the
result, so the stale comment made the case fail. Regenerate it.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-20 08:17:52 +00:00
fys 4e8bc6df86 fix(prom): align remote read label matcher semantics (#9633)
* fix(prom): align remote read label matcher semantics

Signed-off-by: fys <fengys1996@gmail.com>

* chore: optimize comment of validate_matchers

* fix(prom): preserve existing binary label regex handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: condition

* test(prom): simplify remote read matcher tests and clarify comments

* fix(prom): return binary remote read labels as UTF-8

* fix(prom): reject binary labels in remote read

* refactor(prom): align remote read null label handling with PromQL

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(prom): simplify null handling in label matcher branches

Signed-off-by: fys <fengys1996@gmail.com>

* fix(prom): restrict remote read labels to string types

Signed-off-by: fys <fengys1996@gmail.com>

* test(prom): use string label in default value column test

Signed-off-by: fys <fengys1996@gmail.com>

* fix: code review

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-19 06:58:40 +00:00
dennis zhuang 672e94f61d feat: add column encoding option and use it for trace span ids (#9629)
* feat: add column encoding option and set it for trace span ids

COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'.
Either disables the dictionary for the column; delta_binary_packed is
limited to integer and timestamp columns.

OTLP trace v1/v2 tables now create span_id and parent_span_id with plain
encoding and 64KiB pages, and timestamp_end with delta_binary_packed.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: add compatibility cases for column encoding options

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: move column encoding downgrade checks into downgrade_compatibility

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-16 09:40:54 +00:00
Weny Xu 4859e57e0b feat(meta): apply affinity to failover and repartition (#9604)
* feat(meta): apply affinity to failover and repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): share repartition affinity until group allocation

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-15 14:29:10 +00:00
fys cf0f6c308a fix(prom): return empty remote read results for missing metrics (#9613)
* fix: return empty remote read results for missing metrics

Signed-off-by: fys <fengys1996@gmail.com>

* test(prom): cover empty reads for missing databases

Signed-off-by: fys <fengys1996@gmail.com>

* test(prom): simplify remote read regression cases

Signed-off-by: fys <fengys1996@gmail.com>

* fix(prom): log missing tables during remote read

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-15 12:16:14 +00:00
shuiyisong ad823db020 fix(pipeline)!: disable VRL host access by default (#9610)
fix(pipeline): disable VRL host access by default

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-15 10:10:50 +00:00
Weny Xu cbfbad469f feat(meta): add affinity hooks (#9596)
* feat(meta): add affinity hooks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): cover load-based affinity fallback queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): borrow affinity requirements during selection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): remove unused affinity error variant

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-15 09:41:31 +00:00
Ning Sun 40770ddb72 test: only assert genuinely invalid read-preference values at request ingresses (#9576)
The flow and nested-MergeScan ingress tests hardcoded "follower" among
the read-preference values that must be rejected, coupling them to the
current leader-only ReadPreference enum. Assert only genuinely invalid
values there, matching the wire-values test which already limits itself
to ["", "unknown"]. This keeps the ingress tests valid for builds that
extend ReadPreference with more variants.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-14 12:49:56 +00:00
discord9 0e40281262 fix(promql): stop claiming scalar calculate preserves order (#9523)
* test: cover scalar calculate order contract

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): stop claiming scalar calculate preserves order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(test): describe scalar ordering regression as pre-fix behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-14 04:20:39 +00:00
discord9 75bc4ce46d fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(table): only claim projected primary key prefix ordering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(table): keep physical key ordering for metric projections

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(table): compare unknown partition counts explicitly

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(table): model scoped internal TSID scan properties

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-14 03:35:58 +00:00
Weny Xu 863eebdecb feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager

Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): preserve retry policy through error hints

Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): restore sparse partition tags when rebuilding writes

Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): stop detached write retries when callers cancel

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(frontend): configure retries across distributed write paths

Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 14:51:35 +00:00
discord9 2039f508fc fix(promql): stop fabricating histogram fold hash partitioning (#9509)
* fix(promql): stop fabricating histogram fold hash partitioning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): cover histogram fraction binary join missing rows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): normalize dictionary labels in histogram regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover histogram joins on logical metric tables

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: specify insert columns for logical metric fixture

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use supported histogram child replacement API

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): keep histogram fold fixtures truthful

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-13 07:58:31 +00:00
jeremyhi 83e62a7725 feat: remove the server-side Metric export gate (#9547)
* feat: remove the server-side Metric export gate

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: remove unused export capability import

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: remove unused CLI capability import

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* style: import the packed export error selector

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-13 02:32:30 +00:00
discord9 483d1e49a3 fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep filter fetch when rewriting matches term

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 06:12:56 +00:00
discord9 32e8467cda fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): use current DataFusion plan downcasts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): match null rendering in join regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh union plan for unknown scan partitioning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh shared plans after distribution rule removal

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): cover semi joins with remote dynamic filtering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): strengthen partitioned merge scan regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove unused merge scan test helpers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): measure merge scan hash repartition cost

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt merge scan deserialization to two-arg constructor

Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): adapt nested merge scan integration constructor

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 11:18:46 +00:00
discord9 28697c9e55 feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode

Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.

Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
  engine catalog manager, and decodes a MergeScan payload's tables by
  (catalog, schema, table) name through the engine catalog rather than
  the region-aware request catalog list (which binds every table to the
  request's region and breaks on mismatched columns).

Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
  for the MergeScan nodes of a received plan, resolving region leaders
  via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
  the datanode's catalog/cache registry (with_dist_planner stays false:
  a datanode never rewrites its own plan).

Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
  partition_info caches the datanode's catalog manager and partition
  rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
  on a current-thread runtime from within another executor (spawn a
  dedicated thread + runtime when not on a multi-thread runtime).

Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep GreptimeDB function bindings after rebuilding decode state

SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.

Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): address review on nested MergeScan capability

Three follow-ups from review of the nested MergeScan execution capability:

1. Layer the datanode caches (P2b). table_cache derives from
   table_info_cache + table_name_cache, and partition_info_cache derives
   from table_route_cache, but they were all registered in a single flat
   CacheRegistry whose invalidate() runs concurrently with no ordering.
   A derived cache could be invalidated and refilled from a base cache
   that had not been invalidated yet, leaving stale metadata. Build the
   base caches and the derived caches as separate layers of a
   LayeredCacheRegistry (reusing the existing layering mechanism) so base
   caches finish invalidating before derived ones. Both the production
   path (cmd) and the test scaffolding now build the same layered
   registry; the flat build_datanode_cache_registry is removed.

2. Don't let a nested MergeScan's internal region query compete for the
   concurrency permit (P2a). An outer query holds its permit until its
   stream is fully consumed, and a nested MergeScan makes the datanode
   issue region queries to peer datanodes as execution stages of that
   same query. If such an inner stage also took a permit, the outer query
   waiting on it would deadlock against its own held permit (or time out
   when max_concurrent_queries > 0). Mark inner DN->DN region queries as
   internal stages: QueryRequest gains an  flag carried over
   the wire in the request header's query_context extensions (reserved
   key, not injectable via hints), and the region server skips acquiring
   a concurrency permit for internal-stage requests on both acquire
   sites. Outer queries still acquire normally.

3. Add an end-to-end capability test
   (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
   Join(<local probe region>, MergeScan(build regions on both datanodes))
   with mismatched column names and duplicate build keys, ship it to a
   datanode over a real TCP Flight endpoint, consume the stream fully and
   compare the row multiset against the frontend's reference join result,
   assert a real cross-datanode DoGet via an RPC counter, and assert an
   inner region failure fails the whole query. Also migrate the
   with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt date_format regression test to the updated DataFusion fork

ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover the internal stage under the concurrency limit

Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.

Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover cold route loads under unrelated invalidations

The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): trim nested plan decoder commentary

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: remove redundant nested scan support plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): decode nested MergeScan payloads asynchronously

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): avoid unnecessary extension name allocation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain extension decoding for manual session states

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datanode): keep nested scan query admission unchanged

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): honor query handler factories and trim nested scan fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): retain required cache error import

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): reuse registered state for nested scan decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): verify payload information schema table functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): retain nested scan runtime and cold-route guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): address nested scan review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(session): preserve read preference across region query hops

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(session): verify strict context rejection at Flow ingress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove redundant serializer error case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:49:38 +00:00
shuiyisong a907cbfbe4 perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues for tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: bound asynchronous trace auxiliary writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: bound trace auxiliary cache by estimated memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: size trace auxiliary cache from host or pod memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add metrics for cache hit and miss

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: unify frontend cache metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: merge and reduce

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add init for the metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-10 02:45:43 +00:00
discord9 be91fb5ad2 test: support metasrv plugins in integration cluster builder (#9423)
* test: support plugins and flownodes in integration cluster builder

Allow the meta-srv mock and the tests-integration distributed cluster
builder to install metasrv Plugins and to start flownodes that register
with the mock metasrv, so integration tests can exercise plugin-provided
DDL handlers and flows against a real multi-datanode cluster.

The default path is unchanged: mock()/mock_with_client_channel_config()
keep their signatures, and GreptimeDbClusterBuilder builds the metasrv
plugin-free and starts no flownode unless with_plugins/with_flownodes is
called.

Signed-off-by: discord9 <discord9@163.com>

* test: narrow cluster plugins to metasrv injection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: remove unnecessary plugin container mutability

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-09 07:01:51 +00:00
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
jeremyhi f9ab3c296b feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region

A segment that does not decode, because its checksum does not match or
its content disagrees with its footer entry, is fetched once more. If it
still does not decode, the read follows `on_corrupted_segment`: `skip`
records a WAL hole for the region with the object key, sequence and the
footer's entry id range, increments
`greptime_logstore_object_store_wal_skipped_segments_total`, logs a
warning and continues with the next object; `fail` fails the read. A
failed fetch fails the read in both modes. Holes are kept in memory per
region and can be listed from the store.

The standalone test builder allocates WAL options through
`build_standalone_wal_provider`, and integration tests run a standalone
instance on the object store WAL across restarts, on S3 when the
`GT_S3_*` variables are set. The standalone shutdown runs every step and
returns the first error.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cmd): cover the best-effort standalone stop

Build a standalone instance whose leader services fail to stop and whose
datanode servers fail afterwards, and check that `Instance::stop` still
stops the object store WAL and returns the leader services error.

Rename `skip_segment` to `record_skipped_segment` and the file store
integration test to `test_standalone_object_store_wal_survives_restarts_on_file`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-08 06:33:40 +00:00
Ning Sun 3131cdbcb6 fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests

The request-memory accounting only bounds and charges the encoded bytes on
the wire, but a compressed body can expand far beyond that during
decompression, so tiny requests could allocate disproportionate frontend
memory before any protobuf validation or quota charge.

- Handler-level decompression (Prometheus remote read/write v1+v2, Loki)
  now enforces a hard 512 MiB decoded-size cap, checked before any output
  buffer is allocated, and charges the decoded bytes to the shared
  ServerMemoryLimiter, holding the permits for the lifetime of the
  decompressed buffer.
- gRPC requests with transport compression reserve the configured
  max_recv_message_size before tonic decompresses, so the decoding phase
  is admitted against max_in_flight_write_bytes; the later per-message
  charge is skipped to avoid double accounting.
- The HTTP memory-limit middleware keeps its upfront Content-Length charge
  but now also accounts the bytes actually streamed beyond it, so chunked
  requests and understated Content-Length headers no longer bypass the
  aggregate quota.
- Routes that decompress request bodies via RequestDecompressionLayer
  (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards)
  now charge the decompressed bytes as handlers consume them: the global
  middleware marks Content-Encoding requests, and a route-local
  accounting layer inside the decompression layer charges the decoded
  stream. Plain requests are skipped to avoid double-counting.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on memory admission guard lifetimes and 429 mapping

- Retain the gRPC pre-decode reservation for the whole request: the
  extensions holding the guard were dropped when the request was consumed
  (into_inner), releasing the reservation while per-message charges
  stayed skipped. Both the unary and streaming handlers now clone and
  hold the reservation marker for the duration of request handling.
- Retain the HTTP body permits across the handler: the AccountedBody
  wrapper and the request extensions are dropped once the extractors
  finish collecting the body, before the handler is done with the decoded
  data. Both accounting middlewares now keep their own accounting handle
  alive across next.run(req).await.
- Return a ChargedBuffer from the Loki snappy decompressor so the
  reservation outlives the decompressed bytes through the caller's
  protobuf decoding, matching the Prometheus path.
- Map mid-stream quota exhaustion to 429 instead of the generic body
  error (400): the accounting flags exhaustion and the middlewares
  rewrite the extractor rejection, so clients can distinguish
  backpressure from malformed input.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: resolve the in-flight body acquisition before trying a new one

A parked acquisition in AccountedBody::charge always belongs to the
currently buffered frame, so it must be resolved before any new
acquisition is tried. Letting the fast-path try_acquire succeed while a
waiter is parked left the waiter alive, and its late completion would
then be credited with a later frame's byte count, under-reserving memory
relative to what was marked charged.

Adds a regression test that reproduces the misattribution: with the
buggy ordering the test delivers a body the quota cannot cover; with the
fix the over-quota frame waits for its own acquisition and times out.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-29 10:07:18 +00:00
discord9 abf1396c28 fix(client): yield Flight batches and affected rows without waiting for next message (#8918)
* fix(client): avoid Flight metrics lookahead stalls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(client): extract trailing Flight metrics task

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(client): synchronize trailing metrics and bound cancellation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-29 08:05:50 +00:00
Lei, HUANG 5ccfcd4644 feat: support automatic column addition for Flight bulk inserts (#9285)
* fix: auto-add columns when initializing bulk insert streams

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject nested columns in bulk schema auto-add

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject unknown columns in non-empty bulk inserts

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-29 06:47:06 +00:00
Ning Sun fedce5c5ec feat: support non-millisecond time index units in the logical batcher (#9346)
* feat: allow customized time index unit for metric engine table

* test: provide query tests

* refactor: revert unnecessary change

* refactor: share timestamp unit conversions in api helper

Address review feedback on the time index unit changeset:

- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
  (the only crate that sees both proto ColumnDataType and TimeUnit due
  to layering; common-time and datatypes have no greptime-proto dep).
  This removes the ColumnDataType -> TimeUnit match duplicated between
  operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
  convert_timestamp_value_data by reusing api::helper::to_grpc_value
  for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
  values, so an overflow mid-batch leaves the request half-converted;
  harmless because the error aborts the whole insert request.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: align time units per destination table and floor remote-read timestamps

Address review feedback on PR #9236:

- Align each metric insert request to the unit of the table it actually
  targets: an existing logical table keeps its own unit (it may be bound
  to a different physical table than the one selected by the request),
  and only new tables use the selected physical table's unit. The
  previous blanket conversion rewrote valid millisecond samples to the
  selected physical table's unit and the engine rejected them.
  Regression test: writing an existing millisecond logical table and a
  new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
  negative infinity (div_euclid), consistent with
  Timestamp::convert_to on the ingestion path; arrow's cast truncates
  towards zero and returned -1ms for a stored -1001us. Widening
  (second -> millisecond) keeps the exact arrow cast. Regression test:
  a negative, non-aligned timestamp round-trips as -2ms.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: fold time unit alignment into existing table lookups

Address review feedback on PR #9236:

- The per-destination unit alignment no longer runs its own pass of
  table lookups: create_or_alter_tables_on_demand gains an
  align_time_index_unit parameter (metric engine path only) and
  converts each request inside the lookups it already performs —
  existing tables to their own unit, new tables to the selected
  physical table's. Default ingest paths now issue zero additional
  catalog lookups compared to main; the separate alignment pass remains
  only in the opt-in logical batcher pre-gate, next to the eligibility
  check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
  (validate_column_count_match guarantees row widths) instead of
  Optional get_mut; the gate-side alignment validates widths itself.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: resolve the batcher time index guard once per write target

All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.

Signed-off-by: Ning Sun <sunning@greptime.com>

* feat: support non-millisecond time index units in the logical batcher

Make the logical table batcher's bulk encode path unit-aware so physical
metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can
use logical batching instead of falling back to the ordinary insert path.

- rows_to_aligned_record_batch builds the time index column in the
  TARGET schema's unit, converting any timestamp encoding via
  Timestamp::convert_to (flooring on narrowing, consistent with the
  ordinary insert path).
- New tables created by the batcher use the selected physical table's
  time index unit (resolved once per submit; a missing physical table
  keeps the millisecond auto-create default).
- columns_taxonomy and the can_batch_metric_rows schema whitelist accept
  any timestamp unit; the prometheus remote write v1/v2 batcher gates
  and the OTLP pre-gate alignment are removed together with
  Inserter::align_metric_row_inserts_time_unit, as the batcher now
  converts internally.

Closes #9342

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

* fix: address review issue

* refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path

The main merge of #9236 (squash) resurrected the OTLP pre-gate
alignment and Inserter::align_metric_row_inserts_time_unit, which this
branch had removed. Drop them again:

- The pre-gate existed because the #9236-era bulk eligibility gate only
  accepted millisecond schemas, so nanosecond-encoded OTLP requests had
  to be converted before the check. This branch makes the bulk path
  unit-aware (the gate accepts all time index units and batch alignment
  converts each request to its destination's unit), so the pre-gate is
  redundant and only added N+1 catalog lookups per batched request —
  the very lookup-count overhead raised in the #9236 review.
- The per-destination unit semantics it implemented remain enforced in
  the two paths that need them: the ordinary insert path
  (create_or_alter_tables_on_demand converts inside its existing table
  lookups) and the batched path (batch alignment resolves each
  destination schema and converts to it).

test_otlp_logical_batcher_alignment (the test the pre-gate originally
fixed) and the mixed-physical-table regression both pass without it.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: cover OTLP batcher cross-physical fallback and nanosecond physical

Extend the logical batcher integration coverage for the cases
previously guarded by the removed OTLP pre-gate alignment:

- test_otlp_logical_batcher_fallback_for_cross_physical_destination:
  with the batcher enabled, an OTLP request targeting an existing
  logical table bound to another physical table must NOT enter the
  batcher (the bulk eligibility check rejects the destination binding)
  and the ordinary insert path must convert it to the destination's
  unit (60s -> 60_000_000us).
- test_otlp_logical_batcher_non_millisecond_physical_table now covers
  both microsecond and nanosecond physical tables (parameterized),
  asserting batcher submissions and unit-precise stored values.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: validate per-batch physical bindings and avoid intermediate timestamp buffers

Address review feedback on PR #9346:

- accepts_bulk_destinations dedupes on (schema, table, selected physical)
  instead of (schema, table): one request can select different physical
  tables per batch (per-series x_greptime_physical_table labels), and the
  old key let a second selection skip validation and flush rows through
  the wrong physical's regions. Missing tables additionally reject
  conflicting physical selections within the same request. Regression
  test covers an existing destination, a missing destination, and a
  consistent selection (which must still batch).
- The timestamp column builder appends each value directly into the
  target-unit Arrow builder; values already in the target unit (the
  unchanged millisecond fast path) are appended without conversion, so
  the default millisecond physical pays no Timestamp construction or
  intermediate Vec allocation.
- The non-millisecond batching tests assert the submit_build_and_align
  counter, which increments on every batcher submission in both
  acknowledgement modes, so a silent fallback to ordinary insertion
  fails the tests instead of passing on stored values alone.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-28 12:27:36 +00:00
Ning Sun 55b7e08a1a feat: allow customized time index unit for metric engine table (#9236)
* feat: allow customized time index unit for metric engine table

* test: provide query tests

* refactor: revert unnecessary change

* refactor: share timestamp unit conversions in api helper

Address review feedback on the time index unit changeset:

- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
  (the only crate that sees both proto ColumnDataType and TimeUnit due
  to layering; common-time and datatypes have no greptime-proto dep).
  This removes the ColumnDataType -> TimeUnit match duplicated between
  operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
  convert_timestamp_value_data by reusing api::helper::to_grpc_value
  for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
  values, so an overflow mid-batch leaves the request half-converted;
  harmless because the error aborts the whole insert request.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: align time units per destination table and floor remote-read timestamps

Address review feedback on PR #9236:

- Align each metric insert request to the unit of the table it actually
  targets: an existing logical table keeps its own unit (it may be bound
  to a different physical table than the one selected by the request),
  and only new tables use the selected physical table's unit. The
  previous blanket conversion rewrote valid millisecond samples to the
  selected physical table's unit and the engine rejected them.
  Regression test: writing an existing millisecond logical table and a
  new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
  negative infinity (div_euclid), consistent with
  Timestamp::convert_to on the ingestion path; arrow's cast truncates
  towards zero and returned -1ms for a stored -1001us. Widening
  (second -> millisecond) keeps the exact arrow cast. Regression test:
  a negative, non-aligned timestamp round-trips as -2ms.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: fold time unit alignment into existing table lookups

Address review feedback on PR #9236:

- The per-destination unit alignment no longer runs its own pass of
  table lookups: create_or_alter_tables_on_demand gains an
  align_time_index_unit parameter (metric engine path only) and
  converts each request inside the lookups it already performs —
  existing tables to their own unit, new tables to the selected
  physical table's. Default ingest paths now issue zero additional
  catalog lookups compared to main; the separate alignment pass remains
  only in the opt-in logical batcher pre-gate, next to the eligibility
  check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
  (validate_column_count_match guarantees row widths) instead of
  Optional get_mut; the gate-side alignment validates widths itself.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: resolve the batcher time index guard once per write target

All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

* fix: address review issue

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-28 07:15:36 +00:00