mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-18 00:34:11 +00:00
c679ca5fdcc67f0195a65ee33ad9c7df79bad527
509
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c679ca5fdc |
feat(ingestion): configurable time index unit for auto-created physical metric tables (#10189)
* feat(ingestion): configurable time index unit for auto-created physical metric tables Auto-created physical metric tables hardcoded a millisecond time index. Add an `auto_create_physical_table_timestamp_unit` option (frontend and standalone configs, default "Millisecond") that selects the time index unit for physical metric tables created by auto table creation. New logical tables created under such a physical table inherit its unit, so Prometheus remote write, OTLP metrics, and other metric-engine ingestion paths write timestamps in the configured unit. The option threads through all auto-create sites: the operator inserter's `create_physical_table_on_demand`, the frontend instance's batched logical-table flush path (`create_metric_physical_table_if_missing`), the logical-table batcher's unit alignment fallback, and the OTLP metric preflight validation unit. Millisecond remains the default, so existing behavior is unchanged unless configured. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(otlp): match default physical schema to configured timestamp unit Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
7e76d28908 |
fix(postgres): account for buffered InfluxDB and OTLP COPY bodies (#10450)
Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
0a93810c23 |
fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(ingest): reserve memory for table batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): prefix the labels column Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): hide exemplar table from metadata Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): reconcile source tags before writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(exemplar): prune tags and retain source IDs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): record slow queries Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: account for held memory in batch admission Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: config comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: simplify exemplar and memory tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: deduplicate exemplar setup Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
b30e002cef |
feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>]) ingests one Prometheus Remote Write request body over the postgres wire protocol — a GreptimeDB-specific, binary COPY format (wire format code 1). The payload is byte-for-byte the body a remote-write client POSTs to /v1/prometheus/write, compression included: buffered until CopyDone and completed through the exact ingestion core that endpoint runs. To share that core, the HTTP handlers' ingestion bodies are extracted into axum-free functions (ingest_remote_write_v1/v2 plus the context extension stamping), and the HTTP routes become thin wrappers — their behavior is unchanged, and the copy layer runs the identical decode, validation, conversion, batched/prepared write paths, per-batch write-progress accounting and context extensions, under the connection's regular postgres context. The WITH options carry what the HTTP endpoint takes as its content-type/-encoding and query parameters; session-provided values (database, user) are not repeated: - version 1|2 (default 1), matching the content-type's proto - zstd, matching Content-Encoding: zstd (snappy by default) - physical_table, matching the ?physical_table= parameter - pipeline_name/pipeline_version, matching the pipeline headers (v1 only; for v2 they are accepted and ignored, exactly like the HTTP route ignores them) Table-less like the other ingestion formats; one COPY statement is one bounded request (shared 64 MiB limit); the row count reports written rows (v1) or written samples plus histograms (v2). The flush enforces the frontend's shared request-memory quota (max_in_flight_write_bytes) — the same limiter the HTTP ingestion protocols use, so COPY cannot bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write carries the handler, metric-engine flag, validation mode and shared memory limiter. e2e: v1 body chunked across CopyData boundaries; v2 samples with readback; physical_table option routing; garbage-body decode failure with connection reuse; a tiny-quota regression where a compressed body whose decompressed size exceeds the budget fails like the HTTP route. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(servers): propagate COPY body reservations and test write accounting Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): share Prometheus logical batching with COPY ingestion Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): narrow Prometheus COPY scope and share write outcomes Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> |
||
|
|
9eb027f738 |
test: disable the event recorder in tests that count WAL writes (#10316)
The object store WAL restart test and the skip-WAL tests built on MockInstanceImpl assume only the tables they create write to the WAL. The event recorder also writes greptime_private.events into the same WAL, so when it flushes during the test (the first tick of its interval lands after the first DDL event on Windows), the extra region keeps WAL objects from being collected or shows up in the WAL snapshot. Record no events in these tests. Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
ee1fc7782f |
feat(prometheus): query persisted metric exemplars (#9924)
* feat(prometheus): add exemplar queries and lifecycle coverage Expose /v1/prometheus/api/v1/query_exemplars with exact time bounds and series selectors. Resolve and authorize source selectors before scanning exemplar storage, enforce query limits and cancellation, preserve exact observation identity, and return warnings for malformed stored rows. Includes HTTP and authorization tests, pre-created-table and concurrent-DDL coverage, OTLP Arrow ingestion checks, and cross-protocol lifecycle tests across memory, WAL, SST, compaction and reopen. The query-dependent ingestion tests live here so every lower layer builds independently. Add metric-exemplar usage and retention guidance. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): preserve query labels with source identities Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): floor bounds and add audit hooks Signed-off-by: shuiyisong <xixing.sys@gmail.com> * • fix(exemplar): include aggregation parameter selectors Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): avoid v1 exemplar quota stalls Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: docs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: align exemplar query timestamps with Prometheus Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove unnecessory doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
65dc999180 |
feat: support cancelling compaction tasks by task ID (#10229)
* feat: add ADMIN compaction task cancellation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: reject metadata region compaction cancellation before routing Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin greptime-proto to merged cancellation commit Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
297a343fd4 |
fix(query): use current partition columns in prepared plans (#10244)
A prepared statement keeps its logical plan, and that plan holds the table as it was at prepare time. The analyzer reads the partition columns from that table while the regions to scan come from the current route. After a table's partition columns change, the analyzer can consider a query safe to push down, so each region runs the whole query and the frontend concatenates their results. The same happens to the source of a prepared INSERT ... SELECT, which then stores wrong values. Before executing a kept plan, compare each scanned table's partition columns with the current table found by id. Scans of a table whose columns changed get the current table when the schema is unchanged. Otherwise they keep their schema, because a scan's projection holds indices into it, and report the current columns as absent from it, so the query keeps merging on the frontend. Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
8bdb7d5509 |
feat: add information_schema.compaction_tasks (#10173)
Expose live compaction tasks through the datanode inspection path with node pruning, lazy projected batches, and residual SQL filtering. Reuse inspection request and plan helpers across compaction and SST tables. Cover routing, projection, predicates, and lifecycle behavior in datanode, standalone and distributed integration, and SQL tests. Use the production information extension in the standalone test fixture. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
c4cd76c607 |
perf: narrow remote read scans using time hints (#10180)
Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
557843dd68 |
fix(catalog): use advertised peer address as process identity (#10153)
* fix(catalog): use advertised peer address as process identity The process manager identified the local frontend by its public gRPC address, while metasrv discovers peers by the address advertised in the heartbeat (the internal gRPC address when configured). With separate public and internal listeners, list_all_processes failed to exclude the local node, queried its own management service as a remote frontend, and appended the local registry again, so every local query was listed twice. The same mismatch broke routing of KILL to other frontends. Identify a frontend by the address it advertises to metasrv: - add FrontendOptions::peer_addr, shared by the heartbeat and the process manager, and FrontendOptions::create_process_manager to wire it once; - use the advertised address as the process identity, for self-exclusion in list_all_processes, and for local/remote routing in kill_process. Process ids now carry the address other frontends use to reach the node. When no internal gRPC listener is configured this is the public gRPC address, so the previous behavior is unchanged. The internal listener is the only one serving the frontend management service, so process ids stay routable. The new distributed regression tests exercise separate public and internal addresses: the local query must be listed exactly once while remote frontends are still listed, and KILL must route to a remote frontend. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: keep local peer address reserved in kill-routing test The test reserved the local frontend address with a temporary listener that was dropped immediately, then reserved the remote address separately. The kernel could assign the same port to the remote frontend, so the remote kill request would take the local branch and fail to find the remote process. Bind the local address once and keep the listener alive for the whole test so the local and remote frontend identities cannot collide. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
b21b1cdb69 |
fix(function): require ADMIN statement privileges to execute admin functions (#10120)
* fix(function): require ADMIN statement privileges to execute admin functions Read-only accounts could execute the admin UDFs (flush_table, compact_table, flush_flow, gc_regions, migrate_region, reconcile_*, build_index, ...) through ordinary SELECT statements: SELECT is a read-only statement so it passes the statement-level privilege check, and the functions were only guarded by the greptime-catalog scope check inside the UDF. That let read-only users trigger WAL flushes, manifest version bumps, compactions, flow-sink writes and, in distributed mode, region GC/migration and SST deletion. Wrap the privilege check into the admin_fn macro itself, next to the existing greptime-catalog check: before executing, every #[admin_fn] function now requires the same privileges as its ADMIN statement. The check synthesizes the PermissionReq::SqlStatement request an ADMIN <fn>(...) statement produces and evaluates it with the same PermissionCheckerRef plugin the frontend installs, so query invocation (SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path are gated identically — in particular read-only accounts are denied with PermissionDenied (7006), while write-privileged users keep both invocation styles. With auth disabled (no permission checker), the functions stay invocable by everyone, exactly like ADMIN statements. The denial is wrapped as External(BoxedError) so the 7006 status code survives both the query execution path and the admin statement executor. Regression tests: unit tests for the check (no-checker allows, read-only denied 7006, read-write allowed), an end-to-end test over the macro-generated flush_table UDF, and an integration test where a read-only account is denied 7006 on all 13 functions from SELECT while a read-write account keeps SELECT and ADMIN FLUSH_TABLE working. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(function): expose admin privilege check to downstream macros Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
f7e0913ff0 |
test(meta): cover datanode draining with Kafka and local WAL (#10060)
* test(meta): cover datanode draining with Kafka and local WAL Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): fence preferred automatic drain target Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
fd74b9239e |
feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible. Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(otlp): bind exemplars to source metric identities Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): bound exemplar memory and preserve series Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): skip exemplars without metric engine Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): skip exemplars for ordinary tables Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(exemplar)!: prefix the shared table name Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): avoid waiting with body permits held Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
5fe9bdc814 |
feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision. Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(exemplar): persist source metric identities Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): bound exemplar write memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): correct batching with exemplars Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): accept mixed-case sparse encoding Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(exemplar): reduce write preparation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(prom): reuse exemplar source targets Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: clippy Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): defer exemplar series expansion Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
a09ebc70e6 |
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN (metrics, traces, logs) (#10050)
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN
COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:
- signal 'metrics' (default): ExportMetricsServiceRequest, completed
through OpenTelemetryProtocolHandler::metrics with the same context
preparation as the HTTP route (OtlpMetricCtx from the options); rows
are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
trace table name and pipeline name/version/params from the options
(an explicit pipeline name is required, exactly like the HTTP traces
endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
table name, pipeline options and extract keys from the options, and
the same OtlpLogDirect default as the HTTP route; rows are affected
rows.
The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.
Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.
e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.
Signed-off-by: Ning Sun <sunning@greptime.com>
|
||
|
|
f2d74650d9 |
fix: optimize Prometheus remote write v2 ingestion (#9928)
* fix: optimize Prometheus remote write v2 ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: correct remote write v2 memory accounting and response counts Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
ec240ad7e6 |
feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions. Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): preserve reserved sparse key semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): clarify sparse errors and avoid panics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito): reduce sparse key validation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito-codec): reduce sparse validation cost Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(mito): remove ordinary sparse support Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
b21cd16f1d |
feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup Signed-off-by: lczllx <2181719471@qq.com> * fix(datanode): harden the startup object storage self-test Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): cover the File store skip in the object store self-test Signed-off-by: lczllx <2181719471@qq.com> * refactor(datanode): rename the object storage self-test to probe Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): disable object store self-test in sqlness and integration tests Signed-off-by: lczllx <2181719471@qq.com> * docs(datanode): correct what the object storage probe claims to cover Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
|
|
c818b6ac2a |
feat(postgres): influxdb line protocol ingest via COPY FROM STDIN (#9982)
* feat(postgres): influxdb line protocol ingest via COPY FROM STDIN COPY FROM STDIN WITH (FORMAT influxdb [, precision 'ns']) ingests InfluxDB line protocol over the postgres wire protocol, reusing the HTTP ingestion pipeline wholesale for correctness and performance parity: the whole payload is buffered and completed once at CopyDone through the frontend's InfluxdbLineProtocolHandler::exec — conversion, permissions, line-protocol interceptors, timestamp alignment to table time-index units, merge mode and inserts are the same path an HTTP write of the same bytes takes, under the connection's regular postgres query context. Formats are an explicit, closed enum (CopyFormat::Csv | CopyFormat::Influxdb(options)); each variant states its own rules in the copy lifecycle's match arms — csv requires a target table and parses incrementally, influxdb forbids one (each line names its own measurement) and buffers one bounded request body. The COPY layer owns only the buffering: record boundaries, including newlines inside quoted field values and after escapes, belong to the line-protocol parser alone — one COPY statement is one bounded ingestion request (64 MiB; larger imports are multiple COPY statements). - Wiring: PostgresServer::with_influxdb_handler attaches the shared line-protocol handler; unwired servers reject the format when a copy starts. - Grammar: COPY FROM STDIN without a table is scanned (an unquoted FROM cannot be a table name); table-bound formats reject it with guidance, influxdb rejects a named table. - Options: precision accepts the HTTP endpoint's values via the shared parse_time_precision (moved to servers::influxdb), defaulting to nanoseconds. Review findings (fengjiachun) are fixed at the root by removing the pre-splitter rather than repairing it: the retired naive line splitter broke quoted multiline records at chunk/flush boundaries (found by removing record splitting entirely), drained the buffer quadratically (found by append-only buffering), and capped nothing (found by an explicit payload limit). e2e: multi-measurement ingest with chunked feeds; a 32,768-physical-line flush-threshold regression with a multiline quoted record split across CopyData chunks; precision 'ms' epoch verification; invalid UTF-8 and trailing-record-without-newline handling; payload limit; table-named rejection; malformed-line abort with connection recovery. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: typo --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
984f80e277 |
fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit- or timezone-changing time casts A cast on the time index column keeps the windowed-sort optimization eligible only when it preserves the underlying timestamp values. Casts that change the time unit (e.g. ms -> s) or that add a timezone to a naive timestamp re-interpret the values, so the working ranges derived from the time index no longer describe the sort column and the scan fails with a working-range error or silently wrong order. Reject both cast shapes so such queries fall back to the normal SortExec path. The integration fixture uses two rows 1000 seconds apart so the cast output stays distinct and the ordering assertions are not vacuous; the descending arm gets its own expectation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit-changing timestamp wrappers to_timestamp_seconds/millis/... cast their argument to their own unit inside the function, so the hidden cast bypassed the explicit-cast guard and kept the windowed-sort optimization eligible on a sort column whose values no longer match the working ranges. The wrapper branch now applies the same unit- and value-preserving predicate as the cast branch, with the predicate extracted into preserves_timestamp_values and shared by both. The integration regression gains to_timestamp_seconds ASC/DESC arms and a to_timestamp_millis value-preserving control; unit tests cover unit-changing, tz-adding, and non-timestamp wrapper inputs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regen windowed sort sqlness after wrapper guard The windowed_sort_nanos and windowed_sort_advance cases sort by to_timestamp_nanos over a millisecond index and to_timestamp_millis over a nanosecond index. Both wrappers change the time unit inside the function, so the windowed-sort rule no longer applies and the plans render a plain SortExec. Query results are unchanged; only the EXPLAIN plan line differs. Regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): simplify windowed sort cast regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
2e84833ac8 |
feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): reject partition column type changes and quote rule identifiers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(table): keep time index widening for partition columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(partition): replace PartitionFunction with IcebergHash Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(partition): describe iceberg_hash in the function module Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: quote partition rule identifiers in SHOW CREATE TABLE expectations Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(partition): check hash rules in information_schema and repartition target columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): coerce hash partition rules against the evaluation schema Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep quotes in partition column names when building filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
5e9bb3c689 |
feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: align native histogram ingestion with remote read validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
80df1c397b |
feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support * feat: add postgres copy-in framework * feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher Follow-up to #9302: begin_copy_in now uses the batching-aware new_query_context, so copy-in batches are submitted to the pending-rows batcher like SQL INSERTs when PostgreSQL batching is enabled for the connection, and keep the direct write path when it is not. The batcher acknowledges synchronously, so the CommandComplete row count stays exact in both modes. Adds test_postgres_copy_in_with_batching asserting the batcher flush metric increments only when batching is enabled. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options The WITH (...) option list is now open instead of restricted to sqlparser's COPY grammar: - COPY ... FROM STDIN statements are parsed textually (new copy_in::parse module, tokenizer-based) so arbitrary KEY value entries reach the codecs; sqlparser would reject options it does not know before we ever see them. WITH is optional, matching PostgreSQL. - CopyOptionSet becomes an open ordered map of KEY -> String | Number | Ident | Boolean | Flag with typed take_* accessors (take_string, take_single_byte, take_flag, raw take_value) and case-insensitive keys; duplicate options are rejected. - Each format codec takes the options it understands and rejects the leftovers with one message naming them ('COPY option(s) X are not supported by format Y'), so a typo cannot be silently ignored. The text format now also rejects HEADER/QUOTE/ESCAPE instead of ignoring them, matching PostgreSQL. A future format (e.g. influxdb line protocol with WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec and a registry entry; no parser changes. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing Replace the hand-rolled CSV record parser and per-value conversion with arrow-csv's incremental Decoder — the same engine backing COPY ... FROM 'file.csv' in common-datasource — so both COPY routes accept identical data. The decoder handles record splitting (quoting, escaped quotes, custom delimiters, CRLF, headers, chunk boundaries) and typed conversion against the targeted columns' schema in one pass; decoded batches become insert rows via the existing api::helper::vectors_to_rows and datatypes Helper::try_into_vectors, and the decode schema comes from the greptime Schema's arrow projection. The codec framework now has parsers emit complete insertable rows (api Row) instead of raw byte fields: create_parser takes the targeted columns and is fallible; text converts with the shared text-representation helpers, csv with the arrow decoder. Documented consequences of sharing the engine: JSON and Binary columns and bare epoch-integer timestamps are rejected for csv, and an empty field — quoted or not — is NULL; a custom NULL string is rejected accordingly. Signed-off-by: Ning Sun <sunning@greptime.com> * docs(postgres): make the copy-in framework explicitly format-open Reframe the codec framework docs so nothing assumes csv is the only other text format: the shared pieces (Field, convert_text_field, check_text_representable_type) are described as building blocks for any format whose records carry values as text, while formats with their own typed decoding (csv today) build rows themselves. State explicitly that formats are not limited to PostgreSQL's own — GreptimeDB-specific formats (e.g. a line protocol) register like any other and define their own options through the open WITH clause. No code changes beyond a test rename and a doc clarifier. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): driver-level coverage for CSV copy-in options Add an end-to-end test through tokio-postgres exercising HEADER, custom DELIMITER, quoted fields with embedded newlines, escaped quotes and CRLF record separators, with read-back verification through the real insert path. WITH is omitted, matching PostgreSQL. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): drop the raw-socket copy-in tests The simple-protocol raw client duplicated what tokio-postgres and the integration tests already cover; keep the driver-based tests only. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode - text feed: take the longest complete line prefix in one drain and iterate lines over the slice, instead of draining per line (each drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of short lines) - csv check_column_type: delegate to the shared check_text_representable_type and additionally reject Binary/Json - decode_bytea: use the hex crate instead of a hand-rolled decoder - semantic_type_of: use ColumnSchema::is_time_index() like the flow adapter's column_schemas_to_proto, instead of index lookups - parse: drop the unused expect_ident parameter; document the identifier-case decision - PostgresServer:🆕 document that query_handler and copy_in_handler are the same frontend instance Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove unrelated files * feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format Remove the hand-rolled PostgreSQL text-format codec (backslash escapes, octal, \N nulls) that arrow-csv cannot express, keeping arrow-csv-based csv as the only implemented format. - COPY FROM STDIN without FORMAT now fails with guidance ('requires an explicit format in this version; use WITH (FORMAT csv)') rather than silently defaulting to the removed format - FORMAT text/txt and FORMAT binary report 'not supported yet' via registry stubs instead of unknown-format errors - codec.rs shrinks to the trait framework: the text-representation helpers (Field, convert_text_field, bytea hex, timestamp parsing) are removed with the codec; the module docs note where to find their shape should a future text-serializing format need them - csv implements check_column_type directly (no shared default) Breaking for clients relying on the PostgreSQL default text format; explicit WITH (FORMAT csv) is required. Signed-off-by: Ning Sun <sunning@greptime.com> * perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics - COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher each flush pays the full insert pipeline (catalog lookup, schema check, partition split, region write round trip), so 4x larger batches cut round trips 4x at linearly-bounded memory; the constant also sizes the arrow decoder batch. Comment documents the trade-off and points high-rate users at the pending-rows batcher, which turns flushes into enqueues. - rows buffer pre-allocates COPY_IN_FLUSH_ROWS. - flush_copy_in warns on unexpected output variants instead of silently counting 0 rows. - batching wire test feeds 70000 rows to stay above the new threshold. Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv) sends FORMAT csv and succeeds; default \copy (text) gets the 'requires an explicit format' guidance; COPY TO STDOUT round-trips. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): reset copy-in state on successful CopyDone on_copy_done left the CopyInState in place (inert but holding buffers) until the next COPY overwrote it; take it like on_copy_fail does, so buffers are freed eagerly and a stray post-done CopyData gets the protocol-violation error instead of a finished parser. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass typo check * chore: format change * fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries The csv feed loop flushed the arrow-csv decoder at every CopyData chunk end — including partial batches at record boundaries — but arrow-csv's contract permits flush() only after decode() returned 0 (a full batch of complete records). Flushing outside that point corrupts subsequent decoding: fields mis-parse shifted by the size of the record prefix buffered so far ('000001,host-001,...' parses as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse, silently writing wrong values. Which chunks trigger it depends on where record boundaries land, so it appears intermittent with random-length data (psql 8KB / pgx ~64KB chunking both hit it). Adopt the strict decoder protocol: feed decodes until input is exhausted and flushes only when decode returns 0 (batch full); trailing partial batches surface at the EOF signal in finish(). With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is unchanged; sub-threshold streams now emit their rows at CopyDone instead of per chunk, which is semantically identical. Adds test_csv_chunked_feeds_are_alignment_safe: the same payload fed at chunk sizes 1..65531 (plus shifted starts) with every row's content verified — reproduces the corruption on the old loop at chunk sizes 7, 8, 13, 64 and 1000. Verified with the reporter's Go reproducer (pgx, content-checked): 10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the previously failing benchmark sizes 500/1000/2000/200/800 all round- trip exactly. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass spell check * chore: bypass spell check * fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter Three review findings (shuiyisong, PR #9297): [P1] Enforce the catalog restriction on COPY targets. copy_in_table now applies validate_catalog_and_schema when QueryOptions:: disallow_cross_catalog_query is set, mirroring SQL INSERT's check_permission path, BEFORE begin_copy_in switches the query context to the target catalog (which previously made the restriction unenforceable downstream). Cross-catalog COPY is now rejected with 'Illegal access to catalog' like INSERT. [P2] Canonicalize unquoted identifiers like the SQL parser (ParserContext::canonicalize_identifier): unquoted table-name parts and columns fold to lowercase, quoted ones keep their case. COPY Metrics (Host, Ts) now addresses the table CREATE TABLE Metrics created, and no longer silently misroutes into a coexisting quoted "Metrics". [P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal tab while still rejecting newlines and other control characters, and the option parser now also accepts E'...' escape-string constants (DELIMITER E'\t'), processing PostgreSQL's backslash escapes. Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round- trip into the right tables, cross-catalog rejection asserted in a new frontend unit test (QueryOptions plugin + second catalog), and both DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure Two review findings (shuiyisong, PR #9297): [P2] Boolean columns round-trip between the COPY directions. arrow-csv only parses true/false while PostgreSQL COPY OUT emits t/f, so boolean tables failed to re-import their own export (22P02). Boolean columns now decode as Utf8 in the arrow schema and flush_batch converts them with PostgreSQL's boolean input grammar (true/false, t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact bytes are fed back via COPY FROM STDIN and verified. [P2] Drop the COPY state on every server-side failure. on_copy_data and on_copy_done now take ownership of the state up front, so parse, conversion or insert errors drop the buffered data instead of retaining it on long-lived or pooled connections until the next COPY or connection close (the reviewer measured ~17 MiB retained after a failed 8 MiB row); on_copy_data success puts the state back. An e2e test drives a failed COPY followed by a successful one on the same connection. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): interpret offset-free COPY timestamps in the session timezone Review finding (shuiyisong, PR #9297): the arrow decode schema carried Timestamp fields with no timezone, so the CSV decoder parsed offset-free timestamp strings as UTC while SQL INSERT parses them in the session timezone — the same input string silently stored a different instant depending on the ingestion method (8h apart in Asia/Shanghai). CopyInCodec::create_parser now receives the session timezone and the csv decode schema attaches it to Timestamp fields, so naive strings parse like INSERT; explicit offsets in the input (Z, +08:00) still take precedence, and UTC sessions are unchanged. Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z) and the end-to-end test_postgres_copy_in_session_timezone_parity, which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the same string, and asserts both CAST(ts AS BIGINT) readbacks agree on 1704038400000. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): don't unescape E-string COPY options twice Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already decodes PostgreSQL escape-string constants before returning Token::EscapedStringLiteral, so the second unescape_pg_string pass altered the decoded value — DELIMITER E'\\t' (a literal backslash followed by t, two characters) was turned into a tab and wrongly accepted as a single-byte delimiter. Use the EscapedStringLiteral payload as-is, like SingleQuotedString, and delete unescape_pg_string. Regression coverage distinguishes the three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t' -> rejected as a multi-character delimiter. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN Two review findings (shuiyisong, PR #9297): [P2] Preserve timestamp instants across COPY OUT/IN in non-UTC sessions. COPY OUT rendered naive UTC text while COPY IN (since the session-tz fix) interprets offset-free strings as local time, so re-importing an export shifted instants by the zone offset. COPY OUT now renders timestamps with an explicit session-timezone offset (2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the session tz (arrow-pg converts to a tz-aware DateTime) and the pg field type becomes TIMESTAMPTZ so the offset-bearing text format applies. Embedded offsets win on re-import regardless of session, keeping the OUT -> IN round trip exact; UTC sessions are unchanged. [P2] Preserve quoted empty strings. The shared arrow-csv decoder nulls every empty field, so a quoted empty string ("" — an empty STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner runs over the same bytes with the decoder's own splitting rules and records per field whether it was quoted; flush_batch restores quoted empties to empty strings for string columns, while unquoted empty fields stay NULL. Masks track records across chunk boundaries and align with decoder batches. Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip (Asia/Shanghai export asserted to carry +08, exact-epoch re-import comparison) and test_postgres_copy_in_quoted_empty_string (NULL, quoted empty and non-empty in one import), plus unit tests including a chunk-boundary sweep for the quoted-empty mask alignment. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns Two review findings (fengjiachun, PR #9297): [P2] Doubled quotes desynchronized the shadow quote-mask scanner. In QuoteInQuoted the doubled-quote branch skipped the second quote AND the following byte, so a closing quote or delimiter directly after an escaped quote was swallowed and later fields lost their masks -- import semantics depended on CopyData chunk boundaries. The branch now consumes only the escaped quote and returns to the in-quoted state, mirroring the decoder. Regression: "a""b","" ... verified across chunk sizes 1-64. [P2] A NOT NULL string column rejected quoted empty strings before restoration could run: the decode schema preserved the target nullability, and arrow-csv's batch validation rejects the temporary NULL a quoted empty decodes to. String fields now decode as nullable (safe: quoted-vs-unquoted is exactly the restoration's job), and the target NOT NULL constraint is enforced in flush_batch after restoration, so "" imports as an empty string while an unquoted empty field fails with a not-null-constraint error. Regressions at unit and e2e level: NOT NULL column accepting "" and rejecting unquoted NULL. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators Two review findings (fengjiachun, PR #9297): [P2] Blank lines emitted a spurious empty mask. csv-core skips blank lines, but the scanner queued one mask per blank record, so every subsequent decoded row consumed the wrong mask (a leading blank line turned a quoted empty into NULL and an unquoted empty into an empty string, and tripped NOT NULL on the wrong row). A blank line before the header also shifted the header expectation. end_record now returns no mask for a completely blank record without consuming the header expectation. [P2] Standalone CR was not a record terminator. The decoder accepts CR, CRLF and LF alike; the scanner only recognized CR before LF or at EOF, so CR-separated input desynchronized masks and flush_batch failed with 'quote masks out of sync'. is_record_end now treats CR alone as a terminator, consuming one byte (two for CRLF), while a CR at a chunk boundary still defers via carry so CRLF split across CopyData chunks stays one terminator. Regressions: blank lines before/inside records and before a header (NULL/empty preservation plus NOT NULL behavior), and mixed CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing CR-only record. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY The shadow quote-mask scanner — kept to preserve PostgreSQL's quoted-empty-string vs unquoted-NULL distinction — mirrored the decoder's record-splitting grammar by hand and required a fix for every rule it drifted on (doubled quotes, blank lines, CR terminators, NOT NULL interplay). Per maintainers' decision, that complexity is not worth the divergence: COPY FROM STDIN now behaves EXACTLY like the file-based COPY ... FROM 'file.csv' — an empty field, quoted or not, imports as NULL (arrow-csv's engine semantics, identical in both routes). Removed: QuoteMaskScanner and helpers, the always-nullable string decode schema, the restoration + NOT NULL enforcement in flush_batch, set_string_at, and the scanner-era unit tests. The decode schema again preserves target nullability, so the decoder enforces NOT NULL itself, same as the file route. Kept (small, schema-level, no second grammar): boolean spellings (t/f) and session-timezone timestamp decoding, both addressing separate review findings. E2E: test_postgres_copy_in_empty_field_is_null pins the parity — quoted empty, unquoted empty and non-empty in one import. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove empty file * chore: lint issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
2eff52c757 |
feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): guard point accounting and benchmark metrics ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove otlp_metric_compat logic Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): cache physical schema lookups during metric preflight Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: minor rename Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): batch metric rows and reuse schema validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: replace super in import path Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
a7230f7a92 |
fix(partition): coerce partition exprs against the batch schema before routing (#9752)
* fix(partition): coerce partition exprs against the batch schema before routing Batch partition routing (MultiDimPartitionRule::split_record_batch) builds physical expressions via PartitionExpr::try_as_physical_expr, which did no type coercion. After widening the time index unit (e.g. ms -> ns via ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition bounds keep their original unit, so evaluating the bounds against the batch failed with: Invalid comparison operation: Timestamp(ns) < Timestamp(ms) Row inserts were unaffected because find_region compares Values and Timestamp::cmp converts units; scans were unaffected because PredicateGroup::new coerces the region partition expr against the region schema first. Coerce the logical partition expression against the evaluation schema in try_as_physical_expr before creating the physical expression, the same way PredicateGroup does. This also covers the region partition filter built for SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this conversion. Fixes #9747 Signed-off-by: Ning Sun <sunning@greptime.com> * test: regenerate change_col_type result after comment update The ts_widen_part comment in change_col_type.sql was reworded after the result file was last generated; sqlness echoes SQL comments into the result, so the stale comment made the case fail. Regenerate it. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
4e8bc6df86 |
fix(prom): align remote read label matcher semantics (#9633)
* fix(prom): align remote read label matcher semantics Signed-off-by: fys <fengys1996@gmail.com> * chore: optimize comment of validate_matchers * fix(prom): preserve existing binary label regex handling Signed-off-by: fys <fengys1996@gmail.com> * refactor: condition * test(prom): simplify remote read matcher tests and clarify comments * fix(prom): return binary remote read labels as UTF-8 * fix(prom): reject binary labels in remote read * refactor(prom): align remote read null label handling with PromQL Signed-off-by: fys <fengys1996@gmail.com> * refactor(prom): simplify null handling in label matcher branches Signed-off-by: fys <fengys1996@gmail.com> * fix(prom): restrict remote read labels to string types Signed-off-by: fys <fengys1996@gmail.com> * test(prom): use string label in default value column test Signed-off-by: fys <fengys1996@gmail.com> * fix: code review --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
672e94f61d |
feat: add column encoding option and use it for trace span ids (#9629)
* feat: add column encoding option and set it for trace span ids COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'. Either disables the dictionary for the column; delta_binary_packed is limited to integer and timestamp columns. OTLP trace v1/v2 tables now create span_id and parent_span_id with plain encoding and 64KiB pages, and timestamp_end with delta_binary_packed. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add compatibility cases for column encoding options Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: move column encoding downgrade checks into downgrade_compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
4859e57e0b |
feat(meta): apply affinity to failover and repartition (#9604)
* feat(meta): apply affinity to failover and repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): share repartition affinity until group allocation Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
cf0f6c308a |
fix(prom): return empty remote read results for missing metrics (#9613)
* fix: return empty remote read results for missing metrics Signed-off-by: fys <fengys1996@gmail.com> * test(prom): cover empty reads for missing databases Signed-off-by: fys <fengys1996@gmail.com> * test(prom): simplify remote read regression cases Signed-off-by: fys <fengys1996@gmail.com> * fix(prom): log missing tables during remote read Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
ad823db020 |
fix(pipeline)!: disable VRL host access by default (#9610)
fix(pipeline): disable VRL host access by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
cbfbad469f |
feat(meta): add affinity hooks (#9596)
* feat(meta): add affinity hooks Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): cover load-based affinity fallback queries Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): borrow affinity requirements during selection Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): remove unused affinity error variant Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
40770ddb72 |
test: only assert genuinely invalid read-preference values at request ingresses (#9576)
The flow and nested-MergeScan ingress tests hardcoded "follower" among the read-preference values that must be rejected, coupling them to the current leader-only ReadPreference enum. Assert only genuinely invalid values there, matching the wire-values test which already limits itself to ["", "unknown"]. This keeps the ingress tests valid for builds that extend ReadPreference with more variants. Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
0e40281262 |
fix(promql): stop claiming scalar calculate preserves order (#9523)
* test: cover scalar calculate order contract Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): stop claiming scalar calculate preserves order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(test): describe scalar ordering regression as pre-fix behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
75bc4ce46d |
fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): only claim projected primary key prefix ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): keep physical key ordering for metric projections Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): compare unknown partition counts explicitly Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): model scoped internal TSID scan properties Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
863eebdecb |
feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve retry policy through error hints Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): restore sparse partition tags when rebuilding writes Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): stop detached write retries when callers cancel Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(frontend): configure retries across distributed write paths Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage. Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
2039f508fc |
fix(promql): stop fabricating histogram fold hash partitioning (#9509)
* fix(promql): stop fabricating histogram fold hash partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover histogram fraction binary join missing rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): normalize dictionary labels in histogram regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover histogram joins on logical metric tables Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: specify insert columns for logical metric fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use supported histogram child replacement API Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): keep histogram fold fixtures truthful Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
83e62a7725 |
feat: remove the server-side Metric export gate (#9547)
* feat: remove the server-side Metric export gate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused export capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused CLI capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: import the packed export error selector Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
483d1e49a3 |
fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep filter fetch when rewriting matches term Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
32e8467cda |
fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): use current DataFusion plan downcasts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): match null rendering in join regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh union plan for unknown scan partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh shared plans after distribution rule removal Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover semi joins with remote dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen partitioned merge scan regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove unused merge scan test helpers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): measure merge scan hash repartition cost Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt merge scan deserialization to two-arg constructor Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): adapt nested merge scan integration constructor Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
a907cbfbe4 |
perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues for tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: bound asynchronous trace auxiliary writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf: bound trace auxiliary cache by estimated memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: size trace auxiliary cache from host or pod memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add metrics for cache hit and miss Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: unify frontend cache metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: merge and reduce Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add init for the metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
be91fb5ad2 |
test: support metasrv plugins in integration cluster builder (#9423)
* test: support plugins and flownodes in integration cluster builder Allow the meta-srv mock and the tests-integration distributed cluster builder to install metasrv Plugins and to start flownodes that register with the mock metasrv, so integration tests can exercise plugin-provided DDL handlers and flows against a real multi-datanode cluster. The default path is unchanged: mock()/mock_with_client_channel_config() keep their signatures, and GreptimeDbClusterBuilder builds the metasrv plugin-free and starts no flownode unless with_plugins/with_flownodes is called. Signed-off-by: discord9 <discord9@163.com> * test: narrow cluster plugins to metasrv injection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: remove unnecessary plugin container mutability Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
|
|
f9ab3c296b |
feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region A segment that does not decode, because its checksum does not match or its content disagrees with its footer entry, is fetched once more. If it still does not decode, the read follows `on_corrupted_segment`: `skip` records a WAL hole for the region with the object key, sequence and the footer's entry id range, increments `greptime_logstore_object_store_wal_skipped_segments_total`, logs a warning and continues with the next object; `fail` fails the read. A failed fetch fails the read in both modes. Holes are kept in memory per region and can be listed from the store. The standalone test builder allocates WAL options through `build_standalone_wal_provider`, and integration tests run a standalone instance on the object store WAL across restarts, on S3 when the `GT_S3_*` variables are set. The standalone shutdown runs every step and returns the first error. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cmd): cover the best-effort standalone stop Build a standalone instance whose leader services fail to stop and whose datanode servers fail afterwards, and check that `Instance::stop` still stops the object store WAL and returns the leader services error. Rename `skip_segment` to `record_skipped_segment` and the file store integration test to `test_standalone_object_store_wal_survives_restarts_on_file`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
3131cdbcb6 |
fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests The request-memory accounting only bounds and charges the encoded bytes on the wire, but a compressed body can expand far beyond that during decompression, so tiny requests could allocate disproportionate frontend memory before any protobuf validation or quota charge. - Handler-level decompression (Prometheus remote read/write v1+v2, Loki) now enforces a hard 512 MiB decoded-size cap, checked before any output buffer is allocated, and charges the decoded bytes to the shared ServerMemoryLimiter, holding the permits for the lifetime of the decompressed buffer. - gRPC requests with transport compression reserve the configured max_recv_message_size before tonic decompresses, so the decoding phase is admitted against max_in_flight_write_bytes; the later per-message charge is skipped to avoid double accounting. - The HTTP memory-limit middleware keeps its upfront Content-Length charge but now also accounts the bytes actually streamed beyond it, so chunked requests and understated Content-Length headers no longer bypass the aggregate quota. - Routes that decompress request bodies via RequestDecompressionLayer (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards) now charge the decompressed bytes as handlers consume them: the global middleware marks Content-Encoding requests, and a route-local accounting layer inside the decompression layer charges the decoded stream. Plain requests are skipped to avoid double-counting. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on memory admission guard lifetimes and 429 mapping - Retain the gRPC pre-decode reservation for the whole request: the extensions holding the guard were dropped when the request was consumed (into_inner), releasing the reservation while per-message charges stayed skipped. Both the unary and streaming handlers now clone and hold the reservation marker for the duration of request handling. - Retain the HTTP body permits across the handler: the AccountedBody wrapper and the request extensions are dropped once the extractors finish collecting the body, before the handler is done with the decoded data. Both accounting middlewares now keep their own accounting handle alive across next.run(req).await. - Return a ChargedBuffer from the Loki snappy decompressor so the reservation outlives the decompressed bytes through the caller's protobuf decoding, matching the Prometheus path. - Map mid-stream quota exhaustion to 429 instead of the generic body error (400): the accounting flags exhaustion and the middlewares rewrite the extractor rejection, so clients can distinguish backpressure from malformed input. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve the in-flight body acquisition before trying a new one A parked acquisition in AccountedBody::charge always belongs to the currently buffered frame, so it must be resolved before any new acquisition is tried. Letting the fast-path try_acquire succeed while a waiter is parked left the waiter alive, and its late completion would then be credited with a later frame's byte count, under-reserving memory relative to what was marked charged. Adds a regression test that reproduces the misattribution: with the buggy ordering the test delivers a body the quota cannot cover; with the fix the over-quota frame waits for its own acquisition and times out. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
abf1396c28 |
fix(client): yield Flight batches and affected rows without waiting for next message (#8918)
* fix(client): avoid Flight metrics lookahead stalls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(client): extract trailing Flight metrics task Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(client): synchronize trailing metrics and bound cancellation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
5ccfcd4644 |
feat: support automatic column addition for Flight bulk inserts (#9285)
* fix: auto-add columns when initializing bulk insert streams Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: reject nested columns in bulk schema auto-add Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: reject unknown columns in non-empty bulk inserts Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
fedce5c5ec |
feat: support non-millisecond time index units in the logical batcher (#9346)
* feat: allow customized time index unit for metric engine table * test: provide query tests * refactor: revert unnecessary change * refactor: share timestamp unit conversions in api helper Address review feedback on the time index unit changeset: - Add shared timestamp_unit/timestamp_datatype helpers to api::helper (the only crate that sees both proto ColumnDataType and TimeUnit due to layering; common-time and datatypes have no greptime-proto dep). This removes the ColumnDataType -> TimeUnit match duplicated between operator's insert path and the OTLP logs path. - Collapse the two TimeUnit <-> ValueData matches in convert_timestamp_value_data by reusing api::helper::to_grpc_value for the construction side. - Note that convert_rows_time_unit rewrites the schema before the values, so an overflow mid-batch leaves the request half-converted; harmless because the error aborts the whole insert request. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: align time units per destination table and floor remote-read timestamps Address review feedback on PR #9236: - Align each metric insert request to the unit of the table it actually targets: an existing logical table keeps its own unit (it may be bound to a different physical table than the one selected by the request), and only new tables use the selected physical table's unit. The previous blanket conversion rewrote valid millisecond samples to the selected physical table's unit and the engine rejected them. Regression test: writing an existing millisecond logical table and a new table in one request that selects a microsecond physical table. - Remote read now floors narrowing timestamp conversions towards negative infinity (div_euclid), consistent with Timestamp::convert_to on the ingestion path; arrow's cast truncates towards zero and returned -1ms for a stored -1001us. Widening (second -> millisecond) keeps the exact arrow cast. Regression test: a negative, non-aligned timestamp round-trips as -2ms. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: fold time unit alignment into existing table lookups Address review feedback on PR #9236: - The per-destination unit alignment no longer runs its own pass of table lookups: create_or_alter_tables_on_demand gains an align_time_index_unit parameter (metric engine path only) and converts each request inside the lookups it already performs — existing tables to their own unit, new tables to the selected physical table's. Default ingest paths now issue zero additional catalog lookups compared to main; the separate alignment pass remains only in the opt-in logical batcher pre-gate, next to the eligibility check that already looks up the same tables. - convert_rows_time_unit indexes the time index position directly (validate_column_count_match guarantees row widths) instead of Optional get_mut; the gate-side alignment validates widths itself. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: resolve the batcher time index guard once per write target All batches of one remote write request share the same write target (catalog, schema, physical table), so the batcher time index guard now resolves each distinct target once instead of once per batch. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: support non-millisecond time index units in the logical batcher Make the logical table batcher's bulk encode path unit-aware so physical metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can use logical batching instead of falling back to the ordinary insert path. - rows_to_aligned_record_batch builds the time index column in the TARGET schema's unit, converting any timestamp encoding via Timestamp::convert_to (flooring on narrowing, consistent with the ordinary insert path). - New tables created by the batcher use the selected physical table's time index unit (resolved once per submit; a missing physical table keeps the millisecond auto-create default). - columns_taxonomy and the can_batch_metric_rows schema whitelist accept any timestamp unit; the prometheus remote write v1/v2 batcher gates and the OTLP pre-gate alignment are removed together with Inserter::align_metric_row_inserts_time_unit, as the batcher now converts internally. Closes #9342 Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address review issue * refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path The main merge of #9236 (squash) resurrected the OTLP pre-gate alignment and Inserter::align_metric_row_inserts_time_unit, which this branch had removed. Drop them again: - The pre-gate existed because the #9236-era bulk eligibility gate only accepted millisecond schemas, so nanosecond-encoded OTLP requests had to be converted before the check. This branch makes the bulk path unit-aware (the gate accepts all time index units and batch alignment converts each request to its destination's unit), so the pre-gate is redundant and only added N+1 catalog lookups per batched request — the very lookup-count overhead raised in the #9236 review. - The per-destination unit semantics it implemented remain enforced in the two paths that need them: the ordinary insert path (create_or_alter_tables_on_demand converts inside its existing table lookups) and the batched path (batch alignment resolves each destination schema and converts to it). test_otlp_logical_batcher_alignment (the test the pre-gate originally fixed) and the mixed-physical-table regression both pass without it. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover OTLP batcher cross-physical fallback and nanosecond physical Extend the logical batcher integration coverage for the cases previously guarded by the removed OTLP pre-gate alignment: - test_otlp_logical_batcher_fallback_for_cross_physical_destination: with the batcher enabled, an OTLP request targeting an existing logical table bound to another physical table must NOT enter the batcher (the bulk eligibility check rejects the destination binding) and the ordinary insert path must convert it to the destination's unit (60s -> 60_000_000us). - test_otlp_logical_batcher_non_millisecond_physical_table now covers both microsecond and nanosecond physical tables (parameterized), asserting batcher submissions and unit-precise stored values. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: validate per-batch physical bindings and avoid intermediate timestamp buffers Address review feedback on PR #9346: - accepts_bulk_destinations dedupes on (schema, table, selected physical) instead of (schema, table): one request can select different physical tables per batch (per-series x_greptime_physical_table labels), and the old key let a second selection skip validation and flush rows through the wrong physical's regions. Missing tables additionally reject conflicting physical selections within the same request. Regression test covers an existing destination, a missing destination, and a consistent selection (which must still batch). - The timestamp column builder appends each value directly into the target-unit Arrow builder; values already in the target unit (the unchanged millisecond fast path) are appended without conversion, so the default millisecond physical pays no Timestamp construction or intermediate Vec allocation. - The non-millisecond batching tests assert the submit_build_and_align counter, which increments on every batcher submission in both acknowledgement modes, so a silent fallback to ordinary insertion fails the tests instead of passing on stored values alone. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
55b7e08a1a |
feat: allow customized time index unit for metric engine table (#9236)
* feat: allow customized time index unit for metric engine table * test: provide query tests * refactor: revert unnecessary change * refactor: share timestamp unit conversions in api helper Address review feedback on the time index unit changeset: - Add shared timestamp_unit/timestamp_datatype helpers to api::helper (the only crate that sees both proto ColumnDataType and TimeUnit due to layering; common-time and datatypes have no greptime-proto dep). This removes the ColumnDataType -> TimeUnit match duplicated between operator's insert path and the OTLP logs path. - Collapse the two TimeUnit <-> ValueData matches in convert_timestamp_value_data by reusing api::helper::to_grpc_value for the construction side. - Note that convert_rows_time_unit rewrites the schema before the values, so an overflow mid-batch leaves the request half-converted; harmless because the error aborts the whole insert request. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: align time units per destination table and floor remote-read timestamps Address review feedback on PR #9236: - Align each metric insert request to the unit of the table it actually targets: an existing logical table keeps its own unit (it may be bound to a different physical table than the one selected by the request), and only new tables use the selected physical table's unit. The previous blanket conversion rewrote valid millisecond samples to the selected physical table's unit and the engine rejected them. Regression test: writing an existing millisecond logical table and a new table in one request that selects a microsecond physical table. - Remote read now floors narrowing timestamp conversions towards negative infinity (div_euclid), consistent with Timestamp::convert_to on the ingestion path; arrow's cast truncates towards zero and returned -1ms for a stored -1001us. Widening (second -> millisecond) keeps the exact arrow cast. Regression test: a negative, non-aligned timestamp round-trips as -2ms. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: fold time unit alignment into existing table lookups Address review feedback on PR #9236: - The per-destination unit alignment no longer runs its own pass of table lookups: create_or_alter_tables_on_demand gains an align_time_index_unit parameter (metric engine path only) and converts each request inside the lookups it already performs — existing tables to their own unit, new tables to the selected physical table's. Default ingest paths now issue zero additional catalog lookups compared to main; the separate alignment pass remains only in the opt-in logical batcher pre-gate, next to the eligibility check that already looks up the same tables. - convert_rows_time_unit indexes the time index position directly (validate_column_count_match guarantees row widths) instead of Optional get_mut; the gate-side alignment validates widths itself. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: resolve the batcher time index guard once per write target All batches of one remote write request share the same write target (catalog, schema, physical table), so the batcher time index guard now resolves each distinct target once instead of once per batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address review issue --------- Signed-off-by: Ning Sun <sunning@greptime.com> |