Commit Graph
5406 Commits
Author SHA1 Message Date
WenyXu e29dadfe7b fix(common-recordbatch): preserve pruning for SQL numeric casts
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 09:03:05 +00:00
Weny Xu d750d25965 perf: push down COUNT and SUM for SST manifest queries (#9949)
* feat: execute logical plans through distributed inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: push down SST manifest scalar aggregates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: remove redundant SST manifest inspection guide

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: generalize SST manifest aggregate pushdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: retain frontend aggregation for unsupported inspection states

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: expose remote SST aggregate plans in EXPLAIN

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: normalize SST analyzer in TQL explain snapshots

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: guard SST predicate types before remote inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 02:37:00 +00:00
Lei, HUANG 557843dd68 fix(catalog): use advertised peer address as process identity (#10153)
* fix(catalog): use advertised peer address as process identity

The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.

Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
  manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
  in list_all_processes, and for local/remote routing in kill_process.

Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.

The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: keep local peer address reserved in kill-routing test

The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.

Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 18:18:38 +00:00
Ning Sun b21b1cdb69 fix(function): require ADMIN statement privileges to execute admin functions (#10120)
* fix(function): require ADMIN statement privileges to execute admin functions

Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.

Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.

Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(function): expose admin privilege check to downstream macros

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-04 18:18:33 +00:00
discord9 65a89945dd feat(flow): support caller-owned fenced incremental deltas (#9996)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
2026-11-04 18:18:23 +00:00
Weny Xu c0cd86335d feat(meta): stream and batch datanode drain migrations (#10131)
feat(meta): stream and batch datanode drain migrations (#10068)

* feat(meta): stream and batch datanode drain migrations



* test(meta): align drain cursor tests with streaming state



* fix(meta): enforce drain deadlines and clarify scan boundaries



* fix(meta): reconcile drain batch construction failures



* refactor(meta): clarify drain scan buffer and reconciliation naming



---------


(cherry picked from commit 710f8ae683)

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-04 14:13:12 +00:00
Lei, HUANG f4ebe0b34b perf(mito2): read compaction input footers concurrently (#10000)
* perf(mito2): read compaction input footers concurrently

Compaction read the parquet metadata of its inputs one file at a time
before merging. It now reads up to 8 footers at once, keeping the input
order. Remote compaction also caches SST metadata now, so the merge
reuses those footers instead of reading each one a second time.

With 20ms object store reads, a remote compaction of 8 inputs issued 57
read requests instead of 65, and one output took 0.51s instead of 0.75s
with `experimental_compaction_output_parallelism` set to 8.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): stop accounting metadata bytes when caches drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover compaction with a missing input SST

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): skip output metadata warmup for remote compaction

Per-job compaction caches are not reused by readers of newly written SSTs. Keep input footer reuse, but preserve output prewarming only for long-lived engine caches. Cover both cache policies and the unchanged flush path with real SST reads.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 11:09:24 +00:00
Floris SmitandWeny Xu 351f4207b4 fix(meta): invalidate all datanode tables on address change (#9986)
When a datanode restarts with a new address, metasrv invalidated cached
table routes only for the tables in that heartbeat's region stats. The
first heartbeat after a restart often reports few or no regions (e.g.
with init_regions_in_background), so frontends kept dialing the old
address until metadata_cache_ttl expired.

Read the previous address via prev_kv when saving it, and when it
changed (or a failed save of the same epoch is retried) invalidate
every table registered for the datanode in __dn_table.

Signed-off-by: Floris Smit <florissmit10@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-11-04 10:29:42 +00:00
shuiyisong 4633c37dbb fix: test (#10149)
fix: fixture

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:21:42 +00:00
shuiyisong fd74b9239e feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars

OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.

Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(otlp): bind exemplars to source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): bound exemplar memory and preserve series

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars without metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars for ordinary tables

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(exemplar)!: prefix the shared table name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): avoid waiting with body permits held

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:09:57 +00:00
Lei, HUANG e38b84a9ba fix(mito2): prevent automatic compaction candidate starvation (#9946)
* fix(mito2): prevent automatic compaction candidate starvation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): share fair compaction candidate planning

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): include L0 peers in fair closed merges

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): observe TWCS planning and promoted picks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): move compaction pick history into scheduler

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): simplify compaction pick history helpers

Separate history observation from selection and gate test-only access.\nSimplify candidate collection and consolidate redundant tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): acknowledge fair picks at selection

A selected key consumes its fairness turn when the plan is accepted instead of after a successful execution. Drop the pending-selection state, input-id matching and execution hooks; rejected or failed attempts retry with normal priority, so a promoted key can no longer hold the front slot indefinitely.

Add a regression test that selects the older overdue candidate without any completion acknowledgement and verifies the next pick serves the younger one.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): preserve fairness for budget-excluded compaction seeds

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): simplify compaction fairness coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): format compaction fairness regressions

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): skip unused manual pick history snapshots

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): exercise real LastRow fairness planning

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): verify accepted plans install pick history

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): finalize complete TWCS plans

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): name compaction priority fields

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore(mito2): align TWCS metric help text style

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): restore stateless compaction discard logging

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): normalize fairness test imports

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): isolate compaction planning metrics

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 09:29:18 +00:00
Weny Xu 4d8537a622 refactor(meta): separate migration construction and drain child tracking (#10130)
refactor(meta): separate migration construction and drain child tracking (#10098)

refactor(meta): separate migration construction and own drain child tracking


(cherry picked from commit 6a731a444f)

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-04 06:52:10 +00:00
Minh Vu 25003cea16 fix(time): parse zero-hour timezone offsets correctly (#10091)
fix(time): preserve zero-hour timezone offset signs

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
2026-11-04 06:49:45 +00:00
fys b6cf3384dc fix: register trigger procedure loaders through DdlManager (#10123)
Add a loader registration hook to TriggerDdlManager and invoke it
  before procedure recovery. Test that registration reaches the
  configured trigger DDL manager.

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-04 04:31:23 +00:00
shuiyisong adf29740b2 fix(auth): redact inline user-provider credentials (#10115)
* fix(auth): redact inline user-provider credentials

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(auth): close user-provider redaction gaps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 03:39:51 +00:00
Weny Xu 8deb3872b3 fix: include peer context in distributed query errors (#10118)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-03 13:25:14 +00:00
shuiyisong c17a5e09f7 fix(auth): preserve physical metric table targets (#10084)
* fix(auth): preserve physical metric table targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: nit

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 12:52:05 +00:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
Weny Xu 7339a56abd feat: share streaming range results across scans (#9794)
* feat: share streaming range results across scans

Share post-merge range batches across concurrent readers while preserving stream boundaries, cancellation and retry behavior. Encapsulate range ownership, compaction and publication in RangeResultCache, backed by the common SyncCache primitive.

Retain SST retirement fencing and memory accounting. Track live shared-buffer bytes and worker events alongside existing cache metrics.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): clean up shared range registrations on release

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-03 11:33:34 +00:00
AumandClaude Opus 5.5 5cecc99cbe fix(mito-codec): return the first field from dense decode_leftmost (#10090)
DensePrimaryKeyCodec::decode_leftmost decoded the whole key and popped
the last value, so codecs with more than one field returned the
rightmost primary key column. Decode only the first field instead,
which also removes the full-key decoding noted in the TODO.

Current callers build single-field codecs and are unaffected.

Fixes #10089

Signed-off-by: Aumnertic <hznuzc@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 03:16:10 +00:00
Weny Xu 6491700fe6 fix(meta): forward flushed Parquet bytes in region statistics (#10100)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 11:42:56 +00:00
Weny Xu 295bed98ec feat: share Parquet byte page loads (#9793)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 10:52:31 +00:00
Ning Sun a09ebc70e6 feat(postgres): OTLP binary payload ingest via COPY FROM STDIN (metrics, traces, logs) (#10050)
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:

- signal 'metrics' (default): ExportMetricsServiceRequest, completed
  through OpenTelemetryProtocolHandler::metrics with the same context
  preparation as the HTTP route (OtlpMetricCtx from the options); rows
  are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
  trace table name and pipeline name/version/params from the options
  (an explicit pipeline name is required, exactly like the HTTP traces
  endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
  table name, pipeline options and extract keys from the options, and
  the same OtlpLogDirect default as the HTTP route; rows are affected
  rows.

The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.

Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.

e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-02 08:57:05 +00:00
Weny Xu 94d71196cf feat(mito2): track per-region flushed Parquet bytes (#10070)
* feat(mito2): track per-region flushed Parquet bytes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(store-api): inline region statistic compatibility payload

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 04:34:48 +00:00
discord9 01b0076f3e fix(promql): collapse duplicate samples at one timestamp before range functions (#10069)
* fix(promql): collapse duplicate samples at one timestamp before range functions

A read path that does not deduplicate, such as an append-mode region, can
hand PromQL two rows for one series at one timestamp. Every range function
reads its samples from RangeManipulate, so the duplicate entries were
treated as independent samples: count_over_time counted both, idelta and
irate returned NULL because the last two entries shared a timestamp, and
delta/increase/rate deflated because the average sampling interval divides
the observed span by sample_count - 1.

Prometheus sample identity is one sample per series per timestamp, so its
range functions can never see this input. Restore that identity once, at
the operator every range function shares, instead of special-casing each
function's arithmetic.

The collapse compares native ticks, never the millisecond payload, so
distinct sub-millisecond samples that share a millisecond stay distinct as
calculate_range documents and tests. Batches without duplicate timestamps
take a single adjacency scan and are returned untouched; only when
duplicates exist is a take built, keeping tag and value columns aligned.
Keeping the last row of a group matches MergeMode::LastRow.

The normalize test that fed three rows at one timestamp now uses distinct
timestamps: it exercises histogram start-timestamp offsetting, and its
repeated timestamp was incidental, not asserted behavior.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover duplicate samples end to end and keep the newer value

The operator now keeps the first row of a tied timestamp group, which is the row a
deduplicating read keeps: storage orders equal primary key and timestamp by sequence
descending, and FlatDedupReader takes the first row of each group. The end-to-end case
pins that choice by writing 100 and then 101 at the same timestamp: the window reports
increase 6, while keeping the stale 100 would report 8.

The sqlness case writes duplicate rows into an append_mode region, which is the read
path that does not deduplicate, and asserts every range function plus the unaffected
ordinary table in both the standalone and distributed environments.

Signed-off-by: discord9 <discord9@163.com>

* docs(promql): qualify duplicate-timestamp conflict selection

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): avoid copying tags when collapsing duplicate samples

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-02 04:15:19 +00:00
shuiyisong a57ab18eaa feat(ingest): report confirmed progress for batched writes (#9921)
* feat(ingest): track confirmed writes through batching and retries

Batched writes can partially reach storage before another partition fails. Callers need the successful row count and the failure together to report accurate ingestion results. Add WriteProgress and a confirmed-write context, carry row ownership through partitioning and retry rebuilds, and retain completions through ordinary and logical-table batchers.

Existing callers retain the completion/result wrappers. Tests cover partial progress, request ownership, batching and retries. This supplies the accounting required by Remote Write exemplar written-count headers.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(servers): coalesce flow notifications

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(ingest): reduce write progress overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): drain confirmed writes on cancellation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(batcher): reduce row owner allocations

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(batcher): restore per-flush metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* revert: scope retry-hint fix to storage writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): restore flush metrics and benchmark

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): propagate sync batch write errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve peer batching for confirmed writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: avoid unnecessary row owner copies

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: reuse IPC buffers in tracked retries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* style: order write dispatcher imports

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: rebase main

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-02 03:09:13 +00:00
jeremyhi c40e86c653 fix(datanode): pick a node epoch above the recorded one (#9984)
* fix(datanode): pick a node epoch above the recorded one

The metasrv fences a datanode whose node epoch is lower than the
highest one it has recorded for the node id. A datanode used its start
time as its epoch, so after a restart on a host whose clock is behind
the previous process's, or after the clock stepped back, the new
process picked a lower epoch and was fenced for good: no region lease,
no keep-alive.

Before it creates the heartbeat task, the datanode now reads the
recorded epoch through its metadata backend and uses one more than it
if its start time is not higher. The node info still reports the local
start time. A failed read fails the startup, like the other metadata
reads.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(datanode): tighten the node epoch tests

Use a lower record that leaves room below the start time, so the test
fails if the start time is ignored, and a nonzero node id, so it fails
if the node id is ignored. Clarify the doc comment of the epoch choice.

This assumes the previous process of a node id has stopped before a new
one starts.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-31 16:25:49 +00:00
shuiyisong f2d74650d9 fix: optimize Prometheus remote write v2 ingestion (#9928)
* fix: optimize Prometheus remote write v2 ingestion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: correct remote write v2 memory accounting and response counts

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-30 08:56:34 +00:00
Weny Xu c035670ec0 feat: share Puffin metadata and index page loads (#9792)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-30 07:25:55 +00:00
zhaiAohan 8a38b90135 feat: add relational region engine skeleton and registration (#9953)
* feat: add relational region engine skeleton and registration

Add a read-only relational RegionEngine (create/open/drop lifecycle) and register it as the 4th engine in build_store_engines alongside mito/metric/file. Add a datanode-level test asserting all 4 engines are registered.

Signed-off-by: tian1220A <1573612521@qq.com>

* feat: add relational connection and idempotent region lifecycle

Move ConnectionInfo into relational-engine (DSN parsing + redacted rendering), place the RelationalOptions/RELATIONAL_META_KEY serialization contract in table::requests, make CREATE idempotent by comparing the persisted manifest definition against the incoming request (RegionExists on mismatch), and drop the empty EngineConfig placeholder.

Signed-off-by: tian1220A <1573612521@qq.com>

* refactor: address review comments on relational region engine

- remove connection.rs (only used by its own tests); it will be
  reintroduced when connection handling is wired up
- drop RelationalOptions::version (not validated or used to select a
  deserialization format)
- validate an idempotent create against the existing region definition
  (region id / schema / primary key / options) and return Unexpected on
  mismatch, with a reason field on RegionExists
- use serde.workspace and remove the empty [features] section
- cargo fmt

Signed-off-by: tian1220A <1573612521@qq.com>

* fix: adapt relational engine to latest main

- implement RegionEngine::set_region_role_with_lease (no-op lease, the
  engine takes no writes so no lease is needed)
- pass node_id to build_object_store_manager in the registration test
- drop stale url dependency from the relational-engine lock entry

Signed-off-by: tian1220A <1573612521@qq.com>

* ci: register relational-engine in scope rules

Signed-off-by: tian1220A <1573612521@qq.com>

* fix: remove table self-dependency from Cargo.lock

Signed-off-by: tian1220A <1573612521@qq.com>

---------

Signed-off-by: tian1220A <1573612521@qq.com>
2026-10-30 04:06:01 +00:00
Weny Xu db03b2341e feat(meta): orchestrate datanode draining with detached migrations (#9961)
* feat(meta): orchestrate datanode draining with detached migrations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): recheck drain source after migration route races

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): skip dropped tables during datanode drain

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): skip missing table info during datanode drain

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-30 00:12:42 +00:00
fys d33c664183 feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* chore: remove unused code files

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): pass build type to bloom filter indexer

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): share bloom filter field and JSON update loop

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: better name

* refactor(mito2): simplify JSON path bloom index handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): expose index targets from creators

Signed-off-by: fys <fengys1996@gmail.com>

* docs(mito2): note JSON path index rebuild risk

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto

* chore: update greptime-proto

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-29 14:29:00 +00:00
Weny XuandLei, HUANG ee55c0c2fa feat: share concurrent SST metadata loads (#9729)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-10-29 10:03:35 +00:00
Lei, HUANG c7ea80558f fix(mito2): preserve live index files across remote compactions (#10046)
fix(mito2): reuse executor-owned compaction index resources

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-29 17:58:37 +08:00
Weny Xu 694ca48517 feat(meta): add drain migration batch operations (#10035)
* feat(meta): add drain migration batch operations

Build migrations through the shared manager and collect batch results with route reconciliation and failure strategies. Keep the existing fence-and-plan flow while preparing tested operations for drain orchestration.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): gate procedure receiver helpers to unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-29 16:34:41 +08:00
shuiyisong ec240ad7e6 feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables

Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions.

Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): preserve reserved sparse key semantics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): clarify sparse errors and avoid panics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito): reduce sparse key validation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito-codec): reduce sparse validation cost

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(mito): remove ordinary sparse support

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-29 05:45:07 +00:00
shuiyisong efed847ca6 fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce sparse primary key schema

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: fix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-28 17:09:33 +00:00
Dragon Roar b21cd16f1d feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup

Signed-off-by: lczllx <2181719471@qq.com>

* fix(datanode): harden the startup object storage self-test

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): cover the File store skip in the object store self-test

Signed-off-by: lczllx <2181719471@qq.com>

* refactor(datanode): rename the object storage self-test to probe

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): disable object store self-test in sqlness and integration tests

Signed-off-by: lczllx <2181719471@qq.com>

* docs(datanode): correct what the object storage probe claims to cover

Signed-off-by: lczllx <2181719471@qq.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
2026-10-28 17:09:29 +00:00
Lei, HUANG 045cf74c3f perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task

Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.

The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.

One output of 4.8M rows took 1.28s instead of 1.65s.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): parallelize decoding and encoding of compaction outputs

Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.

The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.

The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): cancel compaction producers before consumer polling

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): support per-table compaction output parallelism

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoding tasks before returning failures

Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.

Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoders on SST write errors

Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.

Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain compaction outputs on cancellation

Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.

Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-28 17:09:26 +00:00
Weny Xu b83ceb5e7b feat: add reusable paged byte cache (#9791)
* feat: add reusable paged byte cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): include common cache in scope rules

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 17:09:23 +00:00
Yvan Wang 4c37ed4a0e feat(pipeline): add logfmt processor (#9586)
Add a built-in pipeline processor that parses logfmt lines
(https://brandur.org/logfmt) into structured objects, with support for
quoted values, escapes, bare-key flags, and ignore_missing.

Closes #5049

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
2026-10-28 17:09:21 +00:00
jeremyhi 8e16710425 fix(log-store): accept a retried first create of the conditional create probe (#10034)
* fix(log-store): accept a retried first create of the conditional create probe

The startup probe of the object store WAL wrote its first create by hand
and failed on any error. When that create is stored but its response is
lost, the retry layer repeats it, the repeat meets the object, and the
probe reported a failed write, so the WAL refused to open.

Move the create of put_if_absent, which reads the object back and accepts
identical content, into a path-based core and use it for the probe's
first create too. The second create, which the store must refuse, is
unchanged.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): reuse LosingWriter for the retried probe create test

Move LosingWriter from the fault matrix into the test module and use it,
armed with the lose_next pattern, in place of RetriedWriter. Say why the
probe's second create bypasses put_path_if_absent: a read-back would find
the first create's object and let an overwriting store pass.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-28 13:33:39 +00:00
jeremyhi 89a5df3141 fix: ignore stale leader grants after a region is downgraded (#9987)
* fix: ignore stale leader grants after a region is downgraded

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: compare stale leader grants with the longest lease seen

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: treat a lease too long to store as unknown

A lease longer than a u64 of nanoseconds saturated to a shorter one, so the
derived send time could be later than the real one. Now such a lease makes
the send time unknown, which counts as earlier than any gate.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address review comments on stale leader grants

Scope the early send time claim to after a lease has been seen, drop the
unused role parameter of demote, clarify a test comment and check that a
fresh grant still makes a reopened region writable.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: compare stale leader grants with the heartbeat send time

Pass the send time of the heartbeat behind a lease to set_region_role_with_lease,
and return whether the role was applied. Mito compares the send time with the
grant gate directly, so it no longer derives it from the longest lease seen.
The datanode finalizes leadership only when the grant was applied.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover stale grants through the datanode and repeated downgrades

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: keep leader grant rules in one region method

Apply a role grant in MitoRegion::apply_role_grant, which renews the lease
and decides whether the role changes; the engine only looks up the region.
Rename the gate to downgrade_started_at and demote to begin_downgrade, keep
the field private, and copy it on reopen through
inherit_downgrade_state_from.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: drop a needless borrow

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: keep lease concepts in the engine and role ordering in the region

ManifestContext, which owns the role, records when the role was last
demoted and promotes a region only for a promotion issued after that. The
time starts when the context is created, so a reopened region ignores
grants for heartbeats sent before the reopen, without copying state. The
engine translates leases: it renews the deadline and promotes with the
heartbeat send time. MitoRegion keeps no grant state.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep the data region of a metric region off a grant its metadata region ignores

The metric engine applies a grant to the metadata region first and stops
if it is ignored, so the data region, whose role the engine reports, never
becomes a leader that skips finalization. Also describe what promoting
returns.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: describe when a lease grant is ignored and flatten the finalize check

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: move lease types out of the region and the Leader deadline into ManifestContext

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: name the promotion fence after the downgrade it records

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-28 11:18:25 +00:00
jeremyhi 05536cd7f5 fix(meta): register failure detectors for regions switched before a rollback (#9989)
* fix(meta): register failure detectors for regions switched before a rollback

A region migration switches the routes of its tables one table at a
time. If it aborts after some tables were switched, those regions are
already served by the target, but the rollback deregistered the
target's failure detectors for every region and registered only the
source's. The target's next heartbeat registers them again, so if the
target stopped before sending one, nothing detected its failure and
the switched regions stayed unavailable until the metasrv leader
changed or the target came back.

The rollback now collects, from the routes it already reads, the
regions whose leader is the target and resets the target's failure
detectors for them at the end. It also skips a table whose route no
longer exists, so a table dropped meanwhile no longer fails the
rollback before the detectors are handled.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(meta): roll back a downgrading leader in the rollback detector test

The table that was not switched yet keeps a downgrading leader in a real
abort, so the test now builds it that way and checks that the rollback
clears it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): monitor regions switched before a rollback only on the target

The rollback registered the source's failure detectors for every region
of the migration, including the regions already switched to the target.
Those regions are served by the target after the rollback, so their
source detectors watched the wrong peer: when one fired, the failover
found the leader had changed and only removed it. The rollback now
registers the source's detectors only for the regions that were not
switched, and keeps resetting the target's detectors for the others.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(meta): rename reset_failure_detectors_for_candidate_regions to reset_failure_detectors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-28 09:39:45 +00:00
Weny Xu df33737e9d feat: add shared asynchronous cache primitives (#9790)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 08:45:10 +00:00
Ning Sun c818b6ac2a feat(postgres): influxdb line protocol ingest via COPY FROM STDIN (#9982)
* feat(postgres): influxdb line protocol ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT influxdb [, precision 'ns']) ingests
InfluxDB line protocol over the postgres wire protocol, reusing the
HTTP ingestion pipeline wholesale for correctness and performance
parity: the whole payload is buffered and completed once at CopyDone
through the frontend's InfluxdbLineProtocolHandler::exec — conversion,
permissions, line-protocol interceptors, timestamp alignment to table
time-index units, merge mode and inserts are the same path an HTTP
write of the same bytes takes, under the connection's regular postgres
query context.

Formats are an explicit, closed enum (CopyFormat::Csv |
CopyFormat::Influxdb(options)); each variant states its own rules in
the copy lifecycle's match arms — csv requires a target table and
parses incrementally, influxdb forbids one (each line names its own
measurement) and buffers one bounded request body. The COPY layer owns
only the buffering: record boundaries, including newlines inside quoted
field values and after escapes, belong to the line-protocol parser
alone — one COPY statement is one bounded ingestion request (64 MiB;
larger imports are multiple COPY statements).

- Wiring: PostgresServer::with_influxdb_handler attaches the shared
  line-protocol handler; unwired servers reject the format when a copy
  starts.
- Grammar: COPY FROM STDIN without a table is scanned (an unquoted
  FROM cannot be a table name); table-bound formats reject it with
  guidance, influxdb rejects a named table.
- Options: precision accepts the HTTP endpoint's values via the shared
  parse_time_precision (moved to servers::influxdb), defaulting to
  nanoseconds.

Review findings (fengjiachun) are fixed at the root by removing the
pre-splitter rather than repairing it: the retired naive line splitter
broke quoted multiline records at chunk/flush boundaries (found by
removing record splitting entirely), drained the buffer quadratically
(found by append-only buffering), and capped nothing (found by an
explicit payload limit).

e2e: multi-measurement ingest with chunked feeds; a 32,768-physical-line
flush-threshold regression with a multiline quoted record split across
CopyData chunks; precision 'ms' epoch verification; invalid UTF-8 and
trailing-record-without-newline handling; payload limit; table-named
rejection; malformed-line abort with connection recovery.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: typo

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-28 08:00:41 +00:00
discord9 bd77b599cc refactor(flow): remove tracked-write protocol (#9992)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-28 07:27:04 +00:00
Weny Xu cdc0d3a055 feat(procedure): support detached subprocedure recovery (#10009)
Select Discard or Detach when submitting children via Status::Suspended.
Persist the policy from the child's first normal or rollback checkpoint.
Recover detached children as independent roots with their original id and
context, using existing execution and cleanup without rebuilding the graph.

Keep fresh root admission serialized while allowing accepted procedures to
recover above the submission limit. Cover first persistence, repeated
recovery, rollback, cleanup, defaults, and concurrent admission.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 07:26:22 +00:00
Chenglong Zheng 724072bb33 feat: Add last_sequence for max allocated sequence and add tests for … (#9951)
* feat: Add last_sequence for max allocated sequence and add tests for the reproduced issue cases

Signed-off-by: hahaha <a310608318@gmail.com>

* test: Add a regression test for Kafka remote catchup

Signed-off-by: hahaha <a310608318@gmail.com>

---------

Signed-off-by: hahaha <a310608318@gmail.com>
2026-10-28 06:56:11 +00:00
Dragon Roar 5f0e22171e fix(log-store): skip records with unparsable keys instead of aborting the read (#9749)
* fix(log-store): skip records with unparsable keys instead of aborting the read

Signed-off-by: lczllx <2181719471@qq.com>

* fix(log-store): keep undecodable RecordMeta keys fatal

Signed-off-by: lczllx <2181719471@qq.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
2026-10-28 06:55:55 +00:00