Commit Graph
6566 Commits
Author SHA1 Message Date
WenyXu ec07e55a6d feat: add information_schema.compaction_tasks
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.

Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-06 14:52:45 +08:00
Weny Xu 1692c9f812 feat(mito2): expose live compaction task state (#10172)
* feat(mito2): expose live compaction task state

Expose active and pending compaction tasks through shared region state,
including live phases, request and effective options, progress, and time
ranges. Prune metadata predicates before reading task state without
queueing worker requests or retaining completed-task history.

Share projected inspection batch and plan construction with SST entries,
and cover task lifecycle, admission, pruning, and worker-independent reads.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): share compaction unit state across publications

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-06 06:20:39 +00:00
Yingwen ae422ef465 fix: prioritize region ALTER over queued edits (#10193)
* fix: prioritize region ALTER over queued edits

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: use valid edits in ALTER priority regression

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: make region edit dispatch outcomes explicit

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: reject queued alters and edits when dropping regions

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: cover queued ALTER handoff before region edits

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-11-06 06:01:31 +00:00
Weny Xu bba6818c38 fix(ci): honor configured region in metric snapshot benchmark (#10157)
* fix(ci): honor configured region in metric snapshot benchmark

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(ci): clarify AWS runner region configuration

Include the documentation changes from #10158 alongside the region fix.

Signed-off-by: Weny Xu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: Weny Xu <wenymedia@gmail.com>
2026-11-06 04:06:14 +00:00
dennis zhuang 2714687ab7 feat(otlp): add v1 schema for auto-created OTLP log tables (#10207)
* feat(otlp): add v1 schema for auto-created OTLP log tables

New tables created by direct OTLP logs ingestion get a v1 built-in schema:
service_name tag from resource service.name ahead of scope_name in the
primary key, a skipping index on trace_id, and event_name and
observed_timestamp fields. The table is marked with
table_data_model=greptime_log_v1. Existing tables without the marker keep
the legacy schema.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(otlp): share resource service.name parsing between logs and traces

Log tables now derive service_name with the same rules as trace spans, so
both signals agree on the value used to correlate them. The integration test
writes twice to cover the existing v1 table path.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-06 02:40:23 +00:00
Andrew Mains c9acb55a56 fix(meta-srv): acquire remote WAL read lock once before upgrade retries (#10205)
UpgradeCandidateRegion acquired RemoteWalLock::Read for each topic on every
retry attempt while keeping the guards from earlier attempts. If a
WalPruneProcedure queued RemoteWalLock::Write on the same topic during the first
attempt, the retry's second read acquire waited behind the writer (the lock is a
fair tokio RwLock), and the writer waited on the first read guard. The region
migration and the WAL prune then hung until metasrv restarted.

Acquire the read guards once before the retry loop and hold them until the
function returns.

Fixes #10204
2026-11-06 02:30:08 +00:00
Weny Xu 5bcb576b49 ci: compare GreptimeDB File and S3 in observability benchmarks (#10187)
* ci: compare GreptimeDB File and S3 in observability benchmarks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: separate File and S3 benchmark steps and verify report artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 18:19:56 +00:00
Weny Xu 4263f5fc01 fix(mito): decode cached manifest payloads (#10179)
* test(mito2): cover compressed manifest cache recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): decode cached manifest object payloads

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): address manifest cache review feedback

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 18:19:50 +00:00
Weny Xu 8e4aa4c86f fix(grafana): correct autopilot counters and show baseline reasons (#10160)
* fix(grafana): correct autopilot counters and show baseline reasons

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): retain baseline totals when details are unavailable

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 18:19:45 +00:00
9f7cedf58f fix(mito2): share bulk memtable compaction dispatcher across workers (#10022)
Each region worker built its own MemtableBuilderProvider, and every
provider created its own CompactDispatcher. The limit on background
bulk memtable compactions was therefore num_workers times
max_background_compactions, while SST compaction uses one
engine-wide pool of max_background_compactions.

Create one dispatcher in WorkerGroup::start (and start_for_test),
pass it to every worker, and make MemtableBuilderProvider::new take
it instead of creating its own. SST and memtable compaction keep
separate budgets of the same size.

Signed-off-by: luckys <190197589+luckysxx@users.noreply.github.com>
Co-authored-by: luckys <190197589+luckysxx@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-11-05 16:29:07 +00:00
dennis zhuang f4e612fc26 docs: tighten README facts and move quick start up (#10194)
* docs: tighten README facts and move quick start up

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: align README wording with docs style

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-05 13:01:16 +00:00
Weny Xu 7f5b02f4f5 feat(mito): expose manifest size metrics above a threshold (#10143)
* feat(mito): expose per-region manifest size metric

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): propagate standalone manifest metrics options

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito): publish manifest size metrics from tracked updates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(mito): tidy manifest metric imports and comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): filter manifest size metrics by configurable threshold

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito): configure manifest metrics at engine construction

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: keep per-region metrics configuration notes concise

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: distinguish per-region metrics by component

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito): enable manifest metrics independently of logging

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 10:28:17 +00:00
sun 2447ea24c2 feat: update dashboard to v0.14.3 (#10186) 2026-11-05 10:07:27 +00:00
Yvan Wang ec1ad1f15b fix(datatypes): escape object keys when rendering jsonb as text (#10112)
* fix(datatypes): escape object keys when rendering jsonb as text

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* fix(datatypes): skip unicode normalization for escaped jsonb keys

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* refactor(datatypes): serialize jsonb to text directly with serde_json

Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that
both object keys and string values are escaped by serde_json. This drops
fix_unicode_point, its regex and the hand-written key escaping. Errors are
propagated instead of falling back to lossy raw bytes.

Characters such as U+2028 are now emitted as-is, which is still valid
JSON, so the affected sqlness result lines are regenerated.

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* fix(datatypes): accept JSON text bytes in jsonb_to_string

Some write paths (COPY FROM csv/json/parquet, JSON bound as text via
MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode
such bytes as jsonb before serializing with serde_json, so they still
render (properly escaped) instead of failing with InvalidJsonb.

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

---------

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
2026-11-05 09:42:33 +00:00
sunandCursor 3dce3be163 docs: add demo link (#10184)
Signed-off-by: scl <sunchang_long@163.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-05 08:54:31 +00:00
Weny Xu 4a85e6fcf3 feat(meta): expose datanode drain and scheduling HTTP APIs (#10083)
* feat(meta): expose idempotent datanode drain HTTP APIs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): simplify drain admission and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): reject concurrent datanode drains

Align the drain tracker with region migration and return DrainDatanodeRunning for tracked nodes. Restore the original recover hook and rebuild reservations in the loader without procedure IDs.

Retain admission coordination and recovery readiness to protect scheduling restoration. Update concurrent submission and persisted recovery tests for rejection semantics.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): allow procedure recovery retry after startup failure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): classify active drain conflicts as illegal state

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 08:34:13 +00:00
Lei, HUANG 47bf7de138 feat(mito2): add lossless multi-region compaction workers (#10104)
* feat(mito2): add lossless multi-region compaction executor

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): use bounded workers for multi-region compaction

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): materialize compaction routing fragments on demand

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(index): clean up Tantivy resources on cancellation

Keep temporary directories alive through initialization and in-flight writer finalization. Release writers and remove directories on the shared blocking runtime, and retain the finishing task so abort can join it safely.

Cover creator lifecycle cleanup and multi-region cancellation and sibling failure after Tantivy initialization.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): stream single-window multi-region compaction

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): read published multi-region outputs with real metadata

Multi-region output metadata was only inspected field by field; every
successful read cleared the sequence bound and forced the preserve marker,
so partition filtering and sequence override were never exercised with the
metadata a caller publishes.

Read each output again with the returned FileMeta and the target metadata,
and require the same row count and the same reader-visible sequences.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover multi-region routing mask rejection

The exclusive-routing guards were unreachable: every fixture used a
partition rule whose masks happened to be disjoint and correctly sized.
An unchecked caller-supplied rule can still violate both invariants, and a
violation would route one physical row to several target regions.

Cover both guards: overlapping predicates from a real rule with
check_exprs=false, and an over-short mask from a minimal rule
implementation. Both must fail before any output is protected.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover JSON2 rewrite through multi-region compaction

The multi-region executor wires the shared JSON2 rewrite plan by column
id and target layout, but no fixture carried a JSON2 column, so neither the
per-target mapping nor the materialized values were exercised.

Write a source SST with a narrower JSON2 layout (one key stays in the
remainder), compact it into a wider target layout, and read the output back:
require the target physical layout and the original key values.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover every validate_plan rejection

Plan validation is the boundary that keeps malformed inputs away from the
routing and writer stages, but only the happy path was exercised. Mutate a
valid plan for each rejection (budgets, control state, empty targets, window
alignment/overflow, source ownership, duplicates, missing inputs, mismatched
schemas) and require the specific reason under a timeout, with no protected
outputs.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): assert multi-region output bytes and retained groups

output_bytes had no positive assertion, and the peak check was an upper
bound that also holds when the peak is never recorded. Require the reported
bytes to equal the sum of the produced files and the peak equal-key group
storage to be non-zero.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(index): cover retained Tantivy finalization on abort and drop

Cancellation was only exercised after finish_writer returned, so the
retained finalization handle and the directory lease it holds were never
tested. Simulate a commit that is still running, then require abort to join
it before removing the directory and creator drop to clean up after the
lease is released.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): share sequence provenance helpers on FileMeta

The multi-region writer duplicated the trust predicate and maximum-bound
fold that read paths and DefaultSstMerger already rely on, so the two could
disagree about whether a file's row sequences are trustworthy.

Move both rules onto FileMeta, delegate from FileHandle, and reuse them from
the multi-region writer and the compactor.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): document the guarded SST write path

write_sst_with_guard is the enterprise seam whose behaviour is not visible
from its signature, and the comment above the non-cache arm still claimed
the cache was unconditionally disabled. Describe the bypass, the single
upload stream and the pre-create guard, and correct the stale comment.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): document multi-region plan budget fields

The public plan left max_output_file_size and window_seconds without units or
semantics, and memory_limit without the multipart-minimum floor implied by
admission. State all three so callers cannot budget against the wrong unit.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): explain the multi-region row-group cap

Capping output row groups at 1024 rows diverged from ordinary compaction
without saying why. Record that the cap bounds the encoder working set that
each resident target admits, and keep the behaviour unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): separate new error variants consistently

Every other variant in the enum is separated by a blank line; the new
enterprise variants were glued to their neighbours.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): quote the new module identifier in the module map

Every other row of the module map starts with a backticked identifier; the
multi-region row did not.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): prefix multi-region tests with test_

Every other test under compaction/ is named test_*, so name-filtered runs
and IDE grouping silently skipped this module. Rename only the test
functions; assertions are unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): wrap the multi-region scan stream bodies

rustfmt does not reflow async_stream macro bodies, so the shared scan kept
statements far past the column limit. Split them into one statement per
line; behaviour is unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): make the table-stage folding cases independent

Both parametrized cases used identical options and differed only by
merge_mode, which the table stage never reads, so the second case re-ran
the first. Keep the ignored-merge-mode case and spend the second on
preserve_row_sequence, asserting the per-file marker as well as the row
count.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): fix partition mask test name

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): wrap multi-region worker macro bodies

Wrap receive and accounting expressions and expand the scan completion arm without changing worker behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): document multi-region output levels

Document that returned files default to L1 and callers must adjust metadata before publication when another level is intended. Distinguish this convention from changing existing persisted levels.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): identify invalid multi-region plan fields

Report each invalid source or execution limit separately while preserving validation order and thresholds. Update rejection cases and cover negative window widths.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): consolidate multi-region compaction regressions

Share SST fixtures and complete-row oracles while preserving non-append, JSON2, sequence, and cancellation coverage. Drive Tantivy finalization tests through the real finish path.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-05 07:45:07 +00:00
LFC 2cb473607d fix(grpc): propagate cancellation for read query execution (#10166)
* x

* fix(grpc): restrict cascading cancellation to read query variants

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-11-05 05:54:55 +00:00
Weny Xu d7036df73c fix(mito): use stored recovery-set bytes for manifest size (#10142)
* fix(mito): track stored bytes for manifest recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): retain manifest sizes on checkpoint and sync

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover logical manifest size accounting

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(mito): clarify manifest size retention invariants

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 04:55:53 +00:00
Weny Xu c02e1f58af fix(ci): prepare Ubuntu build runners and default EC2 disks to 500 GiB (#10156)
* ci: share Linux runner dependency setup for builds and benchmarks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: configure EC2 build runner volumes with a 500 GiB default

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 02:40:14 +00:00
Weny Xu d750d25965 perf: push down COUNT and SUM for SST manifest queries (#9949)
* feat: execute logical plans through distributed inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: push down SST manifest scalar aggregates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: remove redundant SST manifest inspection guide

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: generalize SST manifest aggregate pushdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: retain frontend aggregation for unsupported inspection states

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: expose remote SST aggregate plans in EXPLAIN

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: normalize SST analyzer in TQL explain snapshots

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: guard SST predicate types before remote inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 02:37:00 +00:00
Lei, HUANG 557843dd68 fix(catalog): use advertised peer address as process identity (#10153)
* fix(catalog): use advertised peer address as process identity

The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.

Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
  manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
  in list_all_processes, and for local/remote routing in kill_process.

Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.

The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: keep local peer address reserved in kill-routing test

The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.

Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 18:18:38 +00:00
Ning Sun b21b1cdb69 fix(function): require ADMIN statement privileges to execute admin functions (#10120)
* fix(function): require ADMIN statement privileges to execute admin functions

Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.

Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.

Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(function): expose admin privilege check to downstream macros

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-04 18:18:33 +00:00
Weny Xu f7e0913ff0 test(meta): cover datanode draining with Kafka and local WAL (#10060)
* test(meta): cover datanode draining with Kafka and local WAL

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): fence preferred automatic drain target

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-04 18:18:26 +00:00
discord9 65a89945dd feat(flow): support caller-owned fenced incremental deltas (#9996)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
2026-11-04 18:18:23 +00:00
Weny Xu c0cd86335d feat(meta): stream and batch datanode drain migrations (#10131)
feat(meta): stream and batch datanode drain migrations (#10068)

* feat(meta): stream and batch datanode drain migrations



* test(meta): align drain cursor tests with streaming state



* fix(meta): enforce drain deadlines and clarify scan boundaries



* fix(meta): reconcile drain batch construction failures



* refactor(meta): clarify drain scan buffer and reconciliation naming



---------


(cherry picked from commit 710f8ae683)

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-04 14:13:12 +00:00
Lei, HUANG f4ebe0b34b perf(mito2): read compaction input footers concurrently (#10000)
* perf(mito2): read compaction input footers concurrently

Compaction read the parquet metadata of its inputs one file at a time
before merging. It now reads up to 8 footers at once, keeping the input
order. Remote compaction also caches SST metadata now, so the merge
reuses those footers instead of reading each one a second time.

With 20ms object store reads, a remote compaction of 8 inputs issued 57
read requests instead of 65, and one output took 0.51s instead of 0.75s
with `experimental_compaction_output_parallelism` set to 8.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): stop accounting metadata bytes when caches drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover compaction with a missing input SST

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): skip output metadata warmup for remote compaction

Per-job compaction caches are not reused by readers of newly written SSTs. Keep input footer reuse, but preserve output prewarming only for long-lived engine caches. Cover both cache policies and the unchanged flush path with real SST reads.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 11:09:24 +00:00
Floris SmitandWeny Xu 351f4207b4 fix(meta): invalidate all datanode tables on address change (#9986)
When a datanode restarts with a new address, metasrv invalidated cached
table routes only for the tables in that heartbeat's region stats. The
first heartbeat after a restart often reports few or no regions (e.g.
with init_regions_in_background), so frontends kept dialing the old
address until metadata_cache_ttl expired.

Read the previous address via prev_kv when saving it, and when it
changed (or a failed save of the same epoch is retried) invalidate
every table registered for the datanode in __dn_table.

Signed-off-by: Floris Smit <florissmit10@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-11-04 10:29:42 +00:00
shuiyisong 4633c37dbb fix: test (#10149)
fix: fixture

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:21:42 +00:00
shuiyisong fd74b9239e feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars

OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.

Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(otlp): bind exemplars to source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): bound exemplar memory and preserve series

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars without metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars for ordinary tables

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(exemplar)!: prefix the shared table name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): avoid waiting with body permits held

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:09:57 +00:00
Lei, HUANG e38b84a9ba fix(mito2): prevent automatic compaction candidate starvation (#9946)
* fix(mito2): prevent automatic compaction candidate starvation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): share fair compaction candidate planning

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): include L0 peers in fair closed merges

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): observe TWCS planning and promoted picks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): move compaction pick history into scheduler

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): simplify compaction pick history helpers

Separate history observation from selection and gate test-only access.\nSimplify candidate collection and consolidate redundant tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): acknowledge fair picks at selection

A selected key consumes its fairness turn when the plan is accepted instead of after a successful execution. Drop the pending-selection state, input-id matching and execution hooks; rejected or failed attempts retry with normal priority, so a promoted key can no longer hold the front slot indefinitely.

Add a regression test that selects the older overdue candidate without any completion acknowledgement and verifies the next pick serves the younger one.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): preserve fairness for budget-excluded compaction seeds

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): simplify compaction fairness coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): format compaction fairness regressions

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): skip unused manual pick history snapshots

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): exercise real LastRow fairness planning

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): verify accepted plans install pick history

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): finalize complete TWCS plans

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): name compaction priority fields

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore(mito2): align TWCS metric help text style

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): restore stateless compaction discard logging

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): normalize fairness test imports

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): isolate compaction planning metrics

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 09:29:18 +00:00
LFC e80cf9a9ae fix(ci): increase nightly JSONBench runner root volume (#10146)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-11-04 08:23:09 +00:00
Weny Xu 4d8537a622 refactor(meta): separate migration construction and drain child tracking (#10130)
refactor(meta): separate migration construction and drain child tracking (#10098)

refactor(meta): separate migration construction and own drain child tracking


(cherry picked from commit 6a731a444f)

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-04 06:52:10 +00:00
Minh Vu 25003cea16 fix(time): parse zero-hour timezone offsets correctly (#10091)
fix(time): preserve zero-hour timezone offset signs

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
2026-11-04 06:49:45 +00:00
fys b6cf3384dc fix: register trigger procedure loaders through DdlManager (#10123)
Add a loader registration hook to TriggerDdlManager and invoke it
  before procedure recovery. Test that registration reaches the
  configured trigger DDL manager.

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-04 04:31:23 +00:00
shuiyisong adf29740b2 fix(auth): redact inline user-provider credentials (#10115)
* fix(auth): redact inline user-provider credentials

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(auth): close user-provider redaction gaps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 03:39:51 +00:00
Weny Xu 8deb3872b3 fix: include peer context in distributed query errors (#10118)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-03 13:25:14 +00:00
shuiyisong c17a5e09f7 fix(auth): preserve physical metric table targets (#10084)
* fix(auth): preserve physical metric table targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: nit

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 12:52:05 +00:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
Weny Xu 7339a56abd feat: share streaming range results across scans (#9794)
* feat: share streaming range results across scans

Share post-merge range batches across concurrent readers while preserving stream boundaries, cancellation and retry behavior. Encapsulate range ownership, compaction and publication in RangeResultCache, backed by the common SyncCache primitive.

Retain SST retirement fencing and memory accounting. Track live shared-buffer bytes and worker events alongside existing cache metrics.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): clean up shared range registrations on release

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-03 11:33:34 +00:00
liyang f58d598ca7 chore(ci): use standard AWS secrets in CI workflows (#10116) 2026-11-03 10:42:50 +00:00
AumandClaude Opus 5.5 5cecc99cbe fix(mito-codec): return the first field from dense decode_leftmost (#10090)
DensePrimaryKeyCodec::decode_leftmost decoded the whole key and popped
the last value, so codecs with more than one field returned the
rightmost primary key column. Decode only the first field instead,
which also removes the full-key decoding noted in the TODO.

Current callers build single-field codecs and are unaffected.

Fixes #10089

Signed-off-by: Aumnertic <hznuzc@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 03:16:10 +00:00
Weny Xu eef2d58a48 fix(ci): skip unsupported stacked PRs in nightly merge (#10063)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 18:19:28 +00:00
Weny Xu 6491700fe6 fix(meta): forward flushed Parquet bytes in region statistics (#10100)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 11:42:56 +00:00
Weny Xu 295bed98ec feat: share Parquet byte page loads (#9793)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 10:52:31 +00:00
Ning Sun a09ebc70e6 feat(postgres): OTLP binary payload ingest via COPY FROM STDIN (metrics, traces, logs) (#10050)
feat(postgres): OTLP binary payload ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT otlp [, signal '...'] [, <options>]) ingests
one binary OTLP protobuf message over the postgres wire protocol — the
first binary COPY format (wire format code 1) — for all three signals:

- signal 'metrics' (default): ExportMetricsServiceRequest, completed
  through OpenTelemetryProtocolHandler::metrics with the same context
  preparation as the HTTP route (OtlpMetricCtx from the options); rows
  are accepted data points.
- signal 'traces': ExportTraceServiceRequest through ::traces, with the
  trace table name and pipeline name/version/params from the options
  (an explicit pipeline name is required, exactly like the HTTP traces
  endpoint); rows are accepted spans.
- signal 'logs': ExportLogsServiceRequest through ::logs, with the log
  table name, pipeline options and extract keys from the options, and
  the same OtlpLogDirect default as the HTTP route; rows are affected
  rows.

The WITH options carry what the HTTP endpoints take as request headers,
reusing the HTTP path's own parsing helpers (truthy, SelectInfo,
GreptimePipelineParams, PipelineWay) so the two routes cannot drift;
values already provided by the postgres session (database, user,
permissions) are not repeated. Option semantics match the headers:
promote_all_resource_attrs / resource_attrs (';'-separated include or
exclude list) / promote_scope_attrs / translation_strategy for metrics;
table_name / pipeline_name / pipeline_version / pipeline_params /
extract_keys for traces and logs. Signal-inapplicable options are
rejected.

Like the influxdb format: table-less (a named table is rejected), the
whole payload is buffered until CopyDone exactly like one OTLP/HTTP
request body and bounded by the shared 64 MiB request limit, decoding
uses the same protobuf types, and the server must be wired —
PostgresServer::with_otlp_handler(handler, with_metric_engine) carries
the same metric-engine flag the HTTP wiring passes from its options.
The pipeline handler is derived from the protocol handler via trait
upcasting, as the HTTP routes do.

e2e: chunked binary metrics ingest; traces with table_name and
pipeline_name options; logs on the default table; metrics
promote_all_resource_attrs making a resource attribute queryable;
invalid-payload failure with connection reuse.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-02 08:57:05 +00:00
Weny Xu 94d71196cf feat(mito2): track per-region flushed Parquet bytes (#10070)
* feat(mito2): track per-region flushed Parquet bytes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(store-api): inline region statistic compatibility payload

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 04:34:48 +00:00
discord9 01b0076f3e fix(promql): collapse duplicate samples at one timestamp before range functions (#10069)
* fix(promql): collapse duplicate samples at one timestamp before range functions

A read path that does not deduplicate, such as an append-mode region, can
hand PromQL two rows for one series at one timestamp. Every range function
reads its samples from RangeManipulate, so the duplicate entries were
treated as independent samples: count_over_time counted both, idelta and
irate returned NULL because the last two entries shared a timestamp, and
delta/increase/rate deflated because the average sampling interval divides
the observed span by sample_count - 1.

Prometheus sample identity is one sample per series per timestamp, so its
range functions can never see this input. Restore that identity once, at
the operator every range function shares, instead of special-casing each
function's arithmetic.

The collapse compares native ticks, never the millisecond payload, so
distinct sub-millisecond samples that share a millisecond stay distinct as
calculate_range documents and tests. Batches without duplicate timestamps
take a single adjacency scan and are returned untouched; only when
duplicates exist is a take built, keeping tag and value columns aligned.
Keeping the last row of a group matches MergeMode::LastRow.

The normalize test that fed three rows at one timestamp now uses distinct
timestamps: it exercises histogram start-timestamp offsetting, and its
repeated timestamp was incidental, not asserted behavior.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover duplicate samples end to end and keep the newer value

The operator now keeps the first row of a tied timestamp group, which is the row a
deduplicating read keeps: storage orders equal primary key and timestamp by sequence
descending, and FlatDedupReader takes the first row of each group. The end-to-end case
pins that choice by writing 100 and then 101 at the same timestamp: the window reports
increase 6, while keeping the stale 100 would report 8.

The sqlness case writes duplicate rows into an append_mode region, which is the read
path that does not deduplicate, and asserts every range function plus the unaffected
ordinary table in both the standalone and distributed environments.

Signed-off-by: discord9 <discord9@163.com>

* docs(promql): qualify duplicate-timestamp conflict selection

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): avoid copying tags when collapsing duplicate samples

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-02 04:15:19 +00:00
shuiyisong a57ab18eaa feat(ingest): report confirmed progress for batched writes (#9921)
* feat(ingest): track confirmed writes through batching and retries

Batched writes can partially reach storage before another partition fails. Callers need the successful row count and the failure together to report accurate ingestion results. Add WriteProgress and a confirmed-write context, carry row ownership through partitioning and retry rebuilds, and retain completions through ordinary and logical-table batchers.

Existing callers retain the completion/result wrappers. Tests cover partial progress, request ownership, batching and retries. This supplies the accounting required by Remote Write exemplar written-count headers.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(servers): coalesce flow notifications

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(ingest): reduce write progress overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): drain confirmed writes on cancellation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(batcher): reduce row owner allocations

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(batcher): restore per-flush metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* revert: scope retry-hint fix to storage writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): restore flush metrics and benchmark

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): propagate sync batch write errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve peer batching for confirmed writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: avoid unnecessary row owner copies

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: reuse IPC buffers in tracked retries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* style: order write dispatcher imports

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: rebase main

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-02 03:09:13 +00:00
jeremyhi c40e86c653 fix(datanode): pick a node epoch above the recorded one (#9984)
* fix(datanode): pick a node epoch above the recorded one

The metasrv fences a datanode whose node epoch is lower than the
highest one it has recorded for the node id. A datanode used its start
time as its epoch, so after a restart on a host whose clock is behind
the previous process's, or after the clock stepped back, the new
process picked a lower epoch and was fenced for good: no region lease,
no keep-alive.

Before it creates the heartbeat task, the datanode now reads the
recorded epoch through its metadata backend and uses one more than it
if its start time is not higher. The node info still reports the local
start time. A failed read fails the startup, like the other metadata
reads.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(datanode): tighten the node epoch tests

Use a lower record that leaves room below the start time, so the test
fails if the start time is ignored, and a nonzero node id, so it fails
if the node id is ignored. Clarify the doc comment of the epoch choice.

This assumes the previous process of a node id has stopped before a new
one starts.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-31 16:25:49 +00:00