* fix(catalog): use advertised peer address as process identity
The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.
Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
in list_all_processes, and for local/remote routing in kill_process.
Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.
The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: keep local peer address reserved in kill-routing test
The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.
Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(function): require ADMIN statement privileges to execute admin functions
Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.
Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.
Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(function): expose admin privilege check to downstream macros
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(mito2): read compaction input footers concurrently
Compaction read the parquet metadata of its inputs one file at a time
before merging. It now reads up to 8 footers at once, keeping the input
order. Remote compaction also caches SST metadata now, so the merge
reuses those footers instead of reading each one a second time.
With 20ms object store reads, a remote compaction of 8 inputs issued 57
read requests instead of 65, and one output took 0.51s instead of 0.75s
with `experimental_compaction_output_parallelism` set to 8.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): stop accounting metadata bytes when caches drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover compaction with a missing input SST
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): skip output metadata warmup for remote compaction
Per-job compaction caches are not reused by readers of newly written SSTs. Keep input footer reuse, but preserve output prewarming only for long-lived engine caches. Cover both cache policies and the unchanged flush path with real SST reads.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
When a datanode restarts with a new address, metasrv invalidated cached
table routes only for the tables in that heartbeat's region stats. The
first heartbeat after a restart often reports few or no regions (e.g.
with init_regions_in_background), so frontends kept dialing the old
address until metadata_cache_ttl expired.
Read the previous address via prev_kv when saving it, and when it
changed (or a failed save of the same epoch is retried) invalidate
every table registered for the datanode in __dn_table.
Signed-off-by: Floris Smit <florissmit10@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* feat(otlp): persist metric exemplars
OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.
Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat(otlp): bind exemplars to source metric identities
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): bound exemplar memory and preserve series
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): skip exemplars without metric engine
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): skip exemplars for ordinary tables
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* refactor(exemplar)!: prefix the shared table name
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): avoid waiting with body permits held
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
refactor(meta): separate migration construction and drain child tracking (#10098)
refactor(meta): separate migration construction and own drain child tracking
(cherry picked from commit 6a731a444f)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Add a loader registration hook to TriggerDdlManager and invoke it
before procedure recovery. Test that registration reaches the
configured trigger DDL manager.
Signed-off-by: fys <fengys1996@gmail.com>
* feat: share streaming range results across scans
Share post-merge range batches across concurrent readers while preserving stream boundaries, cancellation and retry behavior. Encapsulate range ownership, compaction and publication in RangeResultCache, backed by the common SyncCache primitive.
Retain SST retirement fencing and memory accounting. Track live shared-buffer bytes and worker events alongside existing cache metrics.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): clean up shared range registrations on release
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
DensePrimaryKeyCodec::decode_leftmost decoded the whole key and popped
the last value, so codecs with more than one field returned the
rightmost primary key column. Decode only the first field instead,
which also removes the full-key decoding noted in the TODO.
Current callers build single-field codecs and are unaffected.
Fixes#10089
Signed-off-by: Aumnertic <hznuzc@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN
COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:
- signal 'metrics' (default): ExportMetricsServiceRequest, completed
through OpenTelemetryProtocolHandler::metrics with the same context
preparation as the HTTP route (OtlpMetricCtx from the options); rows
are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
trace table name and pipeline name/version/params from the options
(an explicit pipeline name is required, exactly like the HTTP traces
endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
table name, pipeline options and extract keys from the options, and
the same OtlpLogDirect default as the HTTP route; rows are affected
rows.
The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.
Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.
e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(promql): collapse duplicate samples at one timestamp before range functions
A read path that does not deduplicate, such as an append-mode region, can
hand PromQL two rows for one series at one timestamp. Every range function
reads its samples from RangeManipulate, so the duplicate entries were
treated as independent samples: count_over_time counted both, idelta and
irate returned NULL because the last two entries shared a timestamp, and
delta/increase/rate deflated because the average sampling interval divides
the observed span by sample_count - 1.
Prometheus sample identity is one sample per series per timestamp, so its
range functions can never see this input. Restore that identity once, at
the operator every range function shares, instead of special-casing each
function's arithmetic.
The collapse compares native ticks, never the millisecond payload, so
distinct sub-millisecond samples that share a millisecond stay distinct as
calculate_range documents and tests. Batches without duplicate timestamps
take a single adjacency scan and are returned untouched; only when
duplicates exist is a take built, keeping tag and value columns aligned.
Keeping the last row of a group matches MergeMode::LastRow.
The normalize test that fed three rows at one timestamp now uses distinct
timestamps: it exercises histogram start-timestamp offsetting, and its
repeated timestamp was incidental, not asserted behavior.
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover duplicate samples end to end and keep the newer value
The operator now keeps the first row of a tied timestamp group, which is the row a
deduplicating read keeps: storage orders equal primary key and timestamp by sequence
descending, and FlatDedupReader takes the first row of each group. The end-to-end case
pins that choice by writing 100 and then 101 at the same timestamp: the window reports
increase 6, while keeping the stale 100 would report 8.
The sqlness case writes duplicate rows into an append_mode region, which is the read
path that does not deduplicate, and asserts every range function plus the unaffected
ordinary table in both the standalone and distributed environments.
Signed-off-by: discord9 <discord9@163.com>
* docs(promql): qualify duplicate-timestamp conflict selection
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): avoid copying tags when collapsing duplicate samples
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* fix(datanode): pick a node epoch above the recorded one
The metasrv fences a datanode whose node epoch is lower than the
highest one it has recorded for the node id. A datanode used its start
time as its epoch, so after a restart on a host whose clock is behind
the previous process's, or after the clock stepped back, the new
process picked a lower epoch and was fenced for good: no region lease,
no keep-alive.
Before it creates the heartbeat task, the datanode now reads the
recorded epoch through its metadata backend and uses one more than it
if its start time is not higher. The node info still reports the local
start time. A failed read fails the startup, like the other metadata
reads.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(datanode): tighten the node epoch tests
Use a lower record that leaves room below the start time, so the test
fails if the start time is ignored, and a nonzero node id, so it fails
if the node id is ignored. Clarify the doc comment of the epoch choice.
This assumes the previous process of a node id has stopped before a new
one starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs: link the observability database definition from README
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: add the new required checks to the Docs CI placeholders
The main ruleset now requires five more checks. Their workflows skip
docs-only PRs via paths-ignore, so without placeholders those PRs can
never satisfy the required checks.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add relational region engine skeleton and registration
Add a read-only relational RegionEngine (create/open/drop lifecycle) and register it as the 4th engine in build_store_engines alongside mito/metric/file. Add a datanode-level test asserting all 4 engines are registered.
Signed-off-by: tian1220A <1573612521@qq.com>
* feat: add relational connection and idempotent region lifecycle
Move ConnectionInfo into relational-engine (DSN parsing + redacted rendering), place the RelationalOptions/RELATIONAL_META_KEY serialization contract in table::requests, make CREATE idempotent by comparing the persisted manifest definition against the incoming request (RegionExists on mismatch), and drop the empty EngineConfig placeholder.
Signed-off-by: tian1220A <1573612521@qq.com>
* refactor: address review comments on relational region engine
- remove connection.rs (only used by its own tests); it will be
reintroduced when connection handling is wired up
- drop RelationalOptions::version (not validated or used to select a
deserialization format)
- validate an idempotent create against the existing region definition
(region id / schema / primary key / options) and return Unexpected on
mismatch, with a reason field on RegionExists
- use serde.workspace and remove the empty [features] section
- cargo fmt
Signed-off-by: tian1220A <1573612521@qq.com>
* fix: adapt relational engine to latest main
- implement RegionEngine::set_region_role_with_lease (no-op lease, the
engine takes no writes so no lease is needed)
- pass node_id to build_object_store_manager in the registration test
- drop stale url dependency from the relational-engine lock entry
Signed-off-by: tian1220A <1573612521@qq.com>
* ci: register relational-engine in scope rules
Signed-off-by: tian1220A <1573612521@qq.com>
* fix: remove table self-dependency from Cargo.lock
Signed-off-by: tian1220A <1573612521@qq.com>
---------
Signed-off-by: tian1220A <1573612521@qq.com>
* perf(mito2): merge compaction inputs on a separate task
Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.
The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.
One output of 4.8M rows took 1.28s instead of 1.65s.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): parallelize decoding and encoding of compaction outputs
Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.
The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.
The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): cancel compaction producers before consumer polling
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): support per-table compaction output parallelism
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoding tasks before returning failures
Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.
Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoders on SST write errors
Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.
Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain compaction outputs on cancellation
Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.
Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
WenyXu
Lei, HUANG
Ning Sun
discord9
Floris SmitandWeny Xu
shuiyisong
LFC
Minh Vu
fys
liyang
AumandClaude Opus 5.5
jeremyhi
dennis zhuang
zhaiAohan
Dragon Roar