Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.
Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(mito2): expose live compaction task state
Expose active and pending compaction tasks through shared region state,
including live phases, request and effective options, progress, and time
ranges. Prune metadata predicates before reading task state without
queueing worker requests or retaining completed-task history.
Share projected inspection batch and plan construction with SST entries,
and cover task lifecycle, admission, pruning, and worker-independent reads.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): share compaction unit state across publications
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: prioritize region ALTER over queued edits
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: use valid edits in ALTER priority regression
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: make region edit dispatch outcomes explicit
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: reject queued alters and edits when dropping regions
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: cover queued ALTER handoff before region edits
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat(otlp): add v1 schema for auto-created OTLP log tables
New tables created by direct OTLP logs ingestion get a v1 built-in schema:
service_name tag from resource service.name ahead of scope_name in the
primary key, a skipping index on trace_id, and event_name and
observed_timestamp fields. The table is marked with
table_data_model=greptime_log_v1. Existing tables without the marker keep
the legacy schema.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(otlp): share resource service.name parsing between logs and traces
Log tables now derive service_name with the same rules as trace spans, so
both signals agree on the value used to correlate them. The integration test
writes twice to cover the existing v1 table path.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
UpgradeCandidateRegion acquired RemoteWalLock::Read for each topic on every
retry attempt while keeping the guards from earlier attempts. If a
WalPruneProcedure queued RemoteWalLock::Write on the same topic during the first
attempt, the retry's second read acquire waited behind the writer (the lock is a
fair tokio RwLock), and the writer waited on the first read guard. The region
migration and the WAL prune then hung until metasrv restarted.
Acquire the read guards once before the retry loop and hold them until the
function returns.
Fixes#10204
Each region worker built its own MemtableBuilderProvider, and every
provider created its own CompactDispatcher. The limit on background
bulk memtable compactions was therefore num_workers times
max_background_compactions, while SST compaction uses one
engine-wide pool of max_background_compactions.
Create one dispatcher in WorkerGroup::start (and start_for_test),
pass it to every worker, and make MemtableBuilderProvider::new take
it instead of creating its own. SST and memtable compaction keep
separate budgets of the same size.
Signed-off-by: luckys <190197589+luckysxx@users.noreply.github.com>
Co-authored-by: luckys <190197589+luckysxx@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* fix(datatypes): escape object keys when rendering jsonb as text
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): skip unicode normalization for escaped jsonb keys
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* refactor(datatypes): serialize jsonb to text directly with serde_json
Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that
both object keys and string values are escaped by serde_json. This drops
fix_unicode_point, its regex and the hand-written key escaping. Errors are
propagated instead of falling back to lossy raw bytes.
Characters such as U+2028 are now emitted as-is, which is still valid
JSON, so the affected sqlness result lines are regenerated.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): accept JSON text bytes in jsonb_to_string
Some write paths (COPY FROM csv/json/parquet, JSON bound as text via
MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode
such bytes as jsonb before serializing with serde_json, so they still
render (properly escaped) instead of failing with InvalidJsonb.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
---------
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* feat(mito2): add lossless multi-region compaction executor
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): use bounded workers for multi-region compaction
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): materialize compaction routing fragments on demand
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(index): clean up Tantivy resources on cancellation
Keep temporary directories alive through initialization and in-flight writer finalization. Release writers and remove directories on the shared blocking runtime, and retain the finishing task so abort can join it safely.
Cover creator lifecycle cleanup and multi-region cancellation and sibling failure after Tantivy initialization.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): stream single-window multi-region compaction
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): read published multi-region outputs with real metadata
Multi-region output metadata was only inspected field by field; every
successful read cleared the sequence bound and forced the preserve marker,
so partition filtering and sequence override were never exercised with the
metadata a caller publishes.
Read each output again with the returned FileMeta and the target metadata,
and require the same row count and the same reader-visible sequences.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover multi-region routing mask rejection
The exclusive-routing guards were unreachable: every fixture used a
partition rule whose masks happened to be disjoint and correctly sized.
An unchecked caller-supplied rule can still violate both invariants, and a
violation would route one physical row to several target regions.
Cover both guards: overlapping predicates from a real rule with
check_exprs=false, and an over-short mask from a minimal rule
implementation. Both must fail before any output is protected.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover JSON2 rewrite through multi-region compaction
The multi-region executor wires the shared JSON2 rewrite plan by column
id and target layout, but no fixture carried a JSON2 column, so neither the
per-target mapping nor the materialized values were exercised.
Write a source SST with a narrower JSON2 layout (one key stays in the
remainder), compact it into a wider target layout, and read the output back:
require the target physical layout and the original key values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover every validate_plan rejection
Plan validation is the boundary that keeps malformed inputs away from the
routing and writer stages, but only the happy path was exercised. Mutate a
valid plan for each rejection (budgets, control state, empty targets, window
alignment/overflow, source ownership, duplicates, missing inputs, mismatched
schemas) and require the specific reason under a timeout, with no protected
outputs.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): assert multi-region output bytes and retained groups
output_bytes had no positive assertion, and the peak check was an upper
bound that also holds when the peak is never recorded. Require the reported
bytes to equal the sum of the produced files and the peak equal-key group
storage to be non-zero.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(index): cover retained Tantivy finalization on abort and drop
Cancellation was only exercised after finish_writer returned, so the
retained finalization handle and the directory lease it holds were never
tested. Simulate a commit that is still running, then require abort to join
it before removing the directory and creator drop to clean up after the
lease is released.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): share sequence provenance helpers on FileMeta
The multi-region writer duplicated the trust predicate and maximum-bound
fold that read paths and DefaultSstMerger already rely on, so the two could
disagree about whether a file's row sequences are trustworthy.
Move both rules onto FileMeta, delegate from FileHandle, and reuse them from
the multi-region writer and the compactor.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document the guarded SST write path
write_sst_with_guard is the enterprise seam whose behaviour is not visible
from its signature, and the comment above the non-cache arm still claimed
the cache was unconditionally disabled. Describe the bypass, the single
upload stream and the pre-create guard, and correct the stale comment.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document multi-region plan budget fields
The public plan left max_output_file_size and window_seconds without units or
semantics, and memory_limit without the multipart-minimum floor implied by
admission. State all three so callers cannot budget against the wrong unit.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): explain the multi-region row-group cap
Capping output row groups at 1024 rows diverged from ordinary compaction
without saying why. Record that the cap bounds the encoder working set that
each resident target admits, and keep the behaviour unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(mito2): separate new error variants consistently
Every other variant in the enum is separated by a blank line; the new
enterprise variants were glued to their neighbours.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): quote the new module identifier in the module map
Every other row of the module map starts with a backticked identifier; the
multi-region row did not.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): prefix multi-region tests with test_
Every other test under compaction/ is named test_*, so name-filtered runs
and IDE grouping silently skipped this module. Rename only the test
functions; assertions are unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(mito2): wrap the multi-region scan stream bodies
rustfmt does not reflow async_stream macro bodies, so the shared scan kept
statements far past the column limit. Split them into one statement per
line; behaviour is unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): make the table-stage folding cases independent
Both parametrized cases used identical options and differed only by
merge_mode, which the table stage never reads, so the second case re-ran
the first. Keep the ignored-merge-mode case and spend the second on
preserve_row_sequence, asserting the per-file marker as well as the row
count.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): fix partition mask test name
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(mito2): wrap multi-region worker macro bodies
Wrap receive and accounting expressions and expand the scan completion arm without changing worker behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document multi-region output levels
Document that returned files default to L1 and callers must adjust metadata before publication when another level is intended. Distinguish this convention from changing existing persisted levels.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): identify invalid multi-region plan fields
Report each invalid source or execution limit separately while preserving validation order and thresholds. Update rejection cases and cover negative window widths.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): consolidate multi-region compaction regressions
Share SST fixtures and complete-row oracles while preserving non-append, JSON2, sequence, and cancellation coverage. Drive Tantivy finalization tests through the real finish path.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): use advertised peer address as process identity
The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.
Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
in list_all_processes, and for local/remote routing in kill_process.
Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.
The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: keep local peer address reserved in kill-routing test
The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.
Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(function): require ADMIN statement privileges to execute admin functions
Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.
Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.
Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(function): expose admin privilege check to downstream macros
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(mito2): read compaction input footers concurrently
Compaction read the parquet metadata of its inputs one file at a time
before merging. It now reads up to 8 footers at once, keeping the input
order. Remote compaction also caches SST metadata now, so the merge
reuses those footers instead of reading each one a second time.
With 20ms object store reads, a remote compaction of 8 inputs issued 57
read requests instead of 65, and one output took 0.51s instead of 0.75s
with `experimental_compaction_output_parallelism` set to 8.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): stop accounting metadata bytes when caches drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover compaction with a missing input SST
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): skip output metadata warmup for remote compaction
Per-job compaction caches are not reused by readers of newly written SSTs. Keep input footer reuse, but preserve output prewarming only for long-lived engine caches. Cover both cache policies and the unchanged flush path with real SST reads.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
When a datanode restarts with a new address, metasrv invalidated cached
table routes only for the tables in that heartbeat's region stats. The
first heartbeat after a restart often reports few or no regions (e.g.
with init_regions_in_background), so frontends kept dialing the old
address until metadata_cache_ttl expired.
Read the previous address via prev_kv when saving it, and when it
changed (or a failed save of the same epoch is retried) invalidate
every table registered for the datanode in __dn_table.
Signed-off-by: Floris Smit <florissmit10@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* feat(otlp): persist metric exemplars
OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.
Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat(otlp): bind exemplars to source metric identities
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): bound exemplar memory and preserve series
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): skip exemplars without metric engine
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): skip exemplars for ordinary tables
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* refactor(exemplar)!: prefix the shared table name
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(otlp): avoid waiting with body permits held
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
refactor(meta): separate migration construction and drain child tracking (#10098)
refactor(meta): separate migration construction and own drain child tracking
(cherry picked from commit 6a731a444f)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Add a loader registration hook to TriggerDdlManager and invoke it
before procedure recovery. Test that registration reaches the
configured trigger DDL manager.
Signed-off-by: fys <fengys1996@gmail.com>
* feat: share streaming range results across scans
Share post-merge range batches across concurrent readers while preserving stream boundaries, cancellation and retry behavior. Encapsulate range ownership, compaction and publication in RangeResultCache, backed by the common SyncCache primitive.
Retain SST retirement fencing and memory accounting. Track live shared-buffer bytes and worker events alongside existing cache metrics.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): clean up shared range registrations on release
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
DensePrimaryKeyCodec::decode_leftmost decoded the whole key and popped
the last value, so codecs with more than one field returned the
rightmost primary key column. Decode only the first field instead,
which also removes the full-key decoding noted in the TODO.
Current callers build single-field codecs and are unaffected.
Fixes#10089
Signed-off-by: Aumnertic <hznuzc@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN
COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:
- signal 'metrics' (default): ExportMetricsServiceRequest, completed
through OpenTelemetryProtocolHandler::metrics with the same context
preparation as the HTTP route (OtlpMetricCtx from the options); rows
are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
trace table name and pipeline name/version/params from the options
(an explicit pipeline name is required, exactly like the HTTP traces
endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
table name, pipeline options and extract keys from the options, and
the same OtlpLogDirect default as the HTTP route; rows are affected
rows.
The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.
Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.
e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(promql): collapse duplicate samples at one timestamp before range functions
A read path that does not deduplicate, such as an append-mode region, can
hand PromQL two rows for one series at one timestamp. Every range function
reads its samples from RangeManipulate, so the duplicate entries were
treated as independent samples: count_over_time counted both, idelta and
irate returned NULL because the last two entries shared a timestamp, and
delta/increase/rate deflated because the average sampling interval divides
the observed span by sample_count - 1.
Prometheus sample identity is one sample per series per timestamp, so its
range functions can never see this input. Restore that identity once, at
the operator every range function shares, instead of special-casing each
function's arithmetic.
The collapse compares native ticks, never the millisecond payload, so
distinct sub-millisecond samples that share a millisecond stay distinct as
calculate_range documents and tests. Batches without duplicate timestamps
take a single adjacency scan and are returned untouched; only when
duplicates exist is a take built, keeping tag and value columns aligned.
Keeping the last row of a group matches MergeMode::LastRow.
The normalize test that fed three rows at one timestamp now uses distinct
timestamps: it exercises histogram start-timestamp offsetting, and its
repeated timestamp was incidental, not asserted behavior.
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover duplicate samples end to end and keep the newer value
The operator now keeps the first row of a tied timestamp group, which is the row a
deduplicating read keeps: storage orders equal primary key and timestamp by sequence
descending, and FlatDedupReader takes the first row of each group. The end-to-end case
pins that choice by writing 100 and then 101 at the same timestamp: the window reports
increase 6, while keeping the stale 100 would report 8.
The sqlness case writes duplicate rows into an append_mode region, which is the read
path that does not deduplicate, and asserts every range function plus the unaffected
ordinary table in both the standalone and distributed environments.
Signed-off-by: discord9 <discord9@163.com>
* docs(promql): qualify duplicate-timestamp conflict selection
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): avoid copying tags when collapsing duplicate samples
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
WenyXu
Yingwen
dennis zhuang
Andrew Mains
sun
Yvan Wang
Lei, HUANG
LFC
Ning Sun
discord9
Floris SmitandWeny Xu
shuiyisong
Minh Vu
fys
liyang
AumandClaude Opus 5.5