Commit Graph
497 Commits
Author SHA1 Message Date
dennis zhuang 6e025eef61 fix(auth): secure MySQL and PostgreSQL database selection (#10382)
* fix(mysql): authorize the default database at login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): validate MySQL handshake scope and SQL USE

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): authorize PostgreSQL schema selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): defer PostgreSQL database lookup until authorized

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(servers): propagate catalog lookup errors in protocol tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(postgres): skip unusable search path candidates

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): log database authorization and selection outcomes

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): preserve search path responses and schema boundaries

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): clarify database selection diagnostics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): format handshake username in structured logs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): secure database-less sessions and failed startup

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): retain authorized default database selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(auth): separate database selection from request dispatch

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(mysql): align schema validation with database-less login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(auth): explain database-less session restrictions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 08:07:57 +00:00
Weny Xu 65dc999180 feat: support cancelling compaction tasks by task ID (#10229)
* feat: add ADMIN compaction task cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject metadata region compaction cancellation before routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin greptime-proto to merged cancellation commit

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 07:38:08 +00:00
fys 3051cb41db feat(json2): support fulltext indexes on JSON paths (#10064)
* feat(json2): support fulltext indexes on JSON paths

Signed-off-by: fys <fengys1996@gmail.com>

* chore: use rebased proto for JSON path fulltext

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: reuse index target memory estimate for fulltext cache

Signed-off-by: fys <fengys1996@gmail.com>

* fix: filter disabled fulltext options for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* docs: clarify fulltext intermediate path uniqueness

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: name index targets consistently

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): apply JSON fulltext indexes per compatible target

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): skip invalid JSON fulltext index targets

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): require enabled fulltext indexes for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: simplify JSON fulltext index construction and pruning

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): unify fulltext target compatibility and creator APIs

Signed-off-by: fys <fengys1996@gmail.com>

* perf(function): preserve string layouts when evaluating matches

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): centralize fulltext predicate extraction errors

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: remove enable flag from JSON fulltext proto options

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: clarify JSON fulltext index helpers and imports

Signed-off-by: fys <fengys1996@gmail.com>

* fix: allow rebuilt fulltext indexes on ordinary columns

Signed-off-by: fys <fengys1996@gmail.com>

* test: spell out false positive rate in fulltext tests

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto to upstream merged revision

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): initialize fulltext option in multi-region tests

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-06 10:33:10 +00:00
shuiyisong fd74b9239e feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars

OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.

Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(otlp): bind exemplars to source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): bound exemplar memory and preserve series

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars without metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars for ordinary tables

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(exemplar)!: prefix the shared table name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): avoid waiting with body permits held

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:09:57 +00:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
shuiyisong a57ab18eaa feat(ingest): report confirmed progress for batched writes (#9921)
* feat(ingest): track confirmed writes through batching and retries

Batched writes can partially reach storage before another partition fails. Callers need the successful row count and the failure together to report accurate ingestion results. Add WriteProgress and a confirmed-write context, carry row ownership through partitioning and retry rebuilds, and retain completions through ordinary and logical-table batchers.

Existing callers retain the completion/result wrappers. Tests cover partial progress, request ownership, batching and retries. This supplies the accounting required by Remote Write exemplar written-count headers.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(servers): coalesce flow notifications

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(ingest): reduce write progress overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): drain confirmed writes on cancellation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(batcher): reduce row owner allocations

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(batcher): restore per-flush metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* revert: scope retry-hint fix to storage writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): restore flush metrics and benchmark

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): propagate sync batch write errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve peer batching for confirmed writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: avoid unnecessary row owner copies

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: reuse IPC buffers in tracked retries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* style: order write dispatcher imports

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: rebase main

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-02 03:09:13 +00:00
shuiyisong f2d74650d9 fix: optimize Prometheus remote write v2 ingestion (#9928)
* fix: optimize Prometheus remote write v2 ingestion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: correct remote write v2 memory accounting and response counts

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-30 08:56:34 +00:00
fys d33c664183 feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* chore: remove unused code files

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): pass build type to bloom filter indexer

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): share bloom filter field and JSON update loop

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: better name

* refactor(mito2): simplify JSON path bloom index handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): expose index targets from creators

Signed-off-by: fys <fengys1996@gmail.com>

* docs(mito2): note JSON path index rebuild risk

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto

* chore: update greptime-proto

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-29 14:29:00 +00:00
discord9 bd77b599cc refactor(flow): remove tracked-write protocol (#9992)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-28 07:27:04 +00:00
LFC b369d35d8b feat: enable JSON2 columns on non-append-only tables (#9985)
feat(mito2): support JSON2 in non-append-only tables

Signed-off-by: luofucong <luofc@foxmail.com>
2026-10-28 02:43:29 +00:00
dennis zhuang 2e84833ac8 feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): reject partition column type changes and quote rule identifiers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(table): keep time index widening for partition columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(partition): replace PartitionFunction with IcebergHash

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(partition): describe iceberg_hash in the function module

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: quote partition rule identifiers in SHOW CREATE TABLE expectations

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(partition): check hash rules in information_schema and repartition target columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): coerce hash partition rules against the evaluation schema

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): keep quotes in partition column names when building filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 09:23:00 +00:00
Chenglong Zheng 2117f8b059 feat: Support procedure_state() using the last procedure ID (#9876)
* feat: Support procedure_state() using the last procedure ID

Signed-off-by: hahaha <a310608318@gmail.com>

* feat: Fix and add tests according to review comments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: the zero-argument signature in ADMIN swallows explicit arguments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: fix the CI failures caused by SQL version issues

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: apply the changes based on the review

Signed-off-by: hahaha <a310608318@gmail.com>

---------

Signed-off-by: hahaha <a310608318@gmail.com>
2026-10-27 07:27:55 +00:00
shuiyisong 2eff52c757 feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): guard point accounting and benchmark metrics ingestion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove otlp_metric_compat logic

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(otlp): cache physical schema lookups during metric preflight

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: minor rename

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(otlp): batch metric rows and reuse schema validation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: replace super in import path

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-22 13:36:30 +00:00
fys f35057d9c2 feat(json2): support inverted index (#9324)
feat: build inverted indexes for JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-22 11:42:42 +00:00
Weny Xu 9e9346d2ff fix: preserve datanode write batches when retrying (#9720)
* fix: preserve datanode write batches when retrying

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: isolate invalid entries in metric write batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: adapt batch writes to the unified metric API

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: explicitly accept partial results for batched retries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: assert terminal retry hint instead of masked error text

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: preserve application status in region failure fixtures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: refresh routes after region close

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-22 03:36:02 +00:00
Chenglong Zheng 9f9b35c270 feat: support wait and timeout options for TRUNCATE TABLE (#9753)
* feat: support wait and timeout options for TRUNCATE TABLE

* fix: address review feedback for truncate DDL options

* fix: address review feedback for truncate DDL options
2026-10-21 07:57:16 +00:00
Weny Xu f866ef2bd9 feat: return per-entry region write failures (#9728)
* feat: return per-entry region write failures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints for single-entry region writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve region write error compatibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify region partial result handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align partial failure summaries with input order

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: separate region write response handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: fix partial result expectations and region map guard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: limit lockfile changes to greptime proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: make partial region results an explicit client opt-in

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin greptime-proto to merged partial-result changes

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-21 05:51:15 +00:00
dennis zhuang 672e94f61d feat: add column encoding option and use it for trace span ids (#9629)
* feat: add column encoding option and set it for trace span ids

COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'.
Either disables the dictionary for the column; delta_binary_packed is
limited to integer and timestamp columns.

OTLP trace v1/v2 tables now create span_id and parent_span_id with plain
encoding and 64KiB pages, and timestamp_end with delta_binary_packed.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: add compatibility cases for column encoding options

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: move column encoding downgrade checks into downgrade_compatibility

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-16 09:40:54 +00:00
discord9 778f4556a8 feat: unify query options across SQL, HTTP and gRPC (#9494)
Isolated rebase verification of PR #9494. Preserve the reviewed net change; original PR branch remains untouched.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-16 04:48:32 +00:00
discord9 c3469fad51 fix(flow): retain tracked write completion across transient failures (#9525)
* fix(flow): retain tracked write completion across transient failures

Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths.

Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities.

Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preserve upstream dependencies after rebase

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): fail fast on definitive tracked-write open rejection

A tracked write treats every application-level error from the frontend as
an ambiguous transport failure and retries the same pinned frontend
forever. During a rolling upgrade an older frontend does not recognize the
tracked-write command extension and processes Open as an ordinary empty
SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal
endpoint rejects the extension outright. Both are definitive rejections
that no same-frontend retry can ever resolve, so the flow stalls.

Classify the Open-phase errors by status code: a definitive rejection
(Unsupported / InvalidArguments / PermissionDenied) fails fast with a
terminal pre-write error, while genuinely ambiguous transport failures
(Unavailable, timeout, connection reset, ...) keep the same-capability
retry. Write/Close rejections are deliberately NOT failed fast: once a
write may have been admitted, a same-code rejection (e.g. WriterLost, or
UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome
ambiguous, and failing fast would let the caller replay and duplicate
rows.

Add a regression test that a legacy empty-SQL Open rejection fails fast
with a single attempt and surfaces as Error::External carrying a
TerminalError.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preflight tracked writes without submitting data

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): retain tracked command extensions while decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 10:49:09 +00:00
Weny Xu 36423a95c6 fix(operator): reuse catalog table resolution for write retries (#9574)
* fix(operator): inherit logical table partitions when retrying writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): resolve retried writes through catalog manager

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-14 08:42:51 +00:00
Ning Sun f41e86f43a feat(meta-client): forward actor and event context for region follower RPCs (#9568)
The manage-region-follower chain submitted its procedures with an empty
executor context: `ProcedureServiceHandler::manage_region_follower`
accepted no query context, the operator built an
`ExecutorContext::default()`, and `MetaClient` dropped the context before
delegating to `RegionFollowerClient`. The submitting user (actor) and the
procedure event metadata therefore never reached the submitted
procedures.

- `ProcedureServiceHandler::manage_region_follower` now takes a
  `QueryContextRef`, consistent with `migrate_region`, `gc_regions` and
  `gc_table`.
- `ProcedureServiceOperator` derives the `ExecutorContext` (actor and
  `TriggerReason::Manual` event input) from the query context.
- The `RegionFollowerClient` trait methods now accept `&ExecutorContext`
  and `MetaClient` forwards it, so client implementations can transport
  the actor and event context on the wire.
- `procedure_event_context`/`procedure_actor` are now public so client
  implementations can reuse the trusted protocol derivation.
2026-10-14 04:11:53 +00:00
Weny Xu 863eebdecb feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager

Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): preserve retry policy through error hints

Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): restore sparse partition tags when rebuilding writes

Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): stop detached write retries when callers cancel

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(frontend): configure retries across distributed write paths

Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 14:51:35 +00:00
Lei, HUANG 4bd36cfd20 feat: support compaction options in HTTP hints (#9560)
* feat: support compaction options in HTTP hints

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: share TWCS compaction option prefix

Use the store-api constant for both HTTP hint inference and Mito region option validation so the two paths cannot drift.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-13 12:28:36 +00:00
dennis zhuang 2a70194930 feat: add per-column COMPRESSION option for SST columns (#9426)
* feat: add per-column COMPRESSION option for SST columns

CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N,
page_size = '<size>') on a column. The options are stored in the column
metadata, carried through gRPC column options, shown by SHOW CREATE
TABLE, and applied by the mito SST writer and bulk part encoder as
parquet column properties. Unset fields keep the SST defaults. Nested
types are rejected.

A page size below the dictionary page limit also lowers the column's
dictionary page limit: parquet sizes write batches by the dictionary
budget and keeps that size after falling back to plain encoding, which
would otherwise ignore the smaller data page limit.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: validate column compression options before persistence

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: reject compression overrides on metric logical tables

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-13 08:03:49 +00:00
Weny Xu d86741d0ad feat(operator): retry failed writes with refreshed partitions (#9542)
* feat(operator): retry failed writes with refreshed partitions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(operator): simplify retry test type references

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): clarify write request rebuilding names

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): encapsulate write retry attempts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(operator): group inserter configuration methods

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): share routed peers and separate retry decisions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): avoid temporary write routing allocations

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 06:02:43 +00:00
Weny Xu a2d28e6060 refactor(operator): centralize write partitioning and dispatch (#9541)
* refactor(operator): centralize write partitioning and dispatch

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): preserve write dispatch behavior during refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(operator): import write dispatch error paths

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(operator): align dispatch imports and test utilities

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(operator): use imported types in partition tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 02:30:14 +00:00
jeremyhi 18726c7602 fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory

Compact unused tag dictionary values before merge retention and avoid splitting
a single scan source into redundant batches. Include regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(snapshot): bound writes and reduce logical export overhead

Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible
string dictionaries, bound conversion work, combine Parquet row groups and
filter single-table exports before routing. Include resource-boundary and
packed-restore regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: clarify metric snapshot acceptance criteria

Specify populated scale fixtures, controlled S3 request delay, native macOS
validation, paging exclusions and alternating comparison rounds.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): split batches only for merged readers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(snapshot): cover supported types in both layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: consolidate metric snapshot experiment conclusions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): honor series statistics when splitting L0 scans

Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics.

Cover short-series overlapping L0 files with an ordered latest-value regression.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito): explain memory risk of splitting short L0 series

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address snapshot export review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): accept source errors in lazy snapshot contexts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-12 06:17:03 +00:00
shuiyisong a907cbfbe4 perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues for tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: bound asynchronous trace auxiliary writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: bound trace auxiliary cache by estimated memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: size trace auxiliary cache from host or pod memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add metrics for cache hit and miss

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: unify frontend cache metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: merge and reduce

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add init for the metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-10 02:45:43 +00:00
fys fe8a647bc8 fix(json2): support settings in ALTER TABLE ADD COLUMN (#9472)
* fix(json2): support settings in ALTER TABLE ADD COLUMN

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(sql): reuse Column for ADD COLUMN definitions

Signed-off-by: fys <fengys1996@gmail.com>

* chore: better method name

* refactor(sql): share column parsing between CREATE TABLE and ADD COLUMN

Signed-off-by: fys <fengys1996@gmail.com>

* chore: simplify tests

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): accept trailing commas in ALTER TABLE ADD COLUMN

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): preserve ALTER TABLE ADD COLUMN compatibility

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-09 03:17:22 +00:00
discord9 198d92407c fix(operator): bound FlowMirrorTask pending rows to prevent frontend OOM (#9421)
* fix(operator): bound FlowMirrorTask pending rows to prevent frontend OOM

When flownode is slow or unreachable, FlowMirrorTask::detach spawns
unbounded tasks that each pin a full insert batch for ~10s (gRPC
timeout). With continuous writes, in-flight memory grows unboundedly
(throughput x 10s x peers), causing frontend-write OOMKill and a
million-per-hour ERROR storm.

Add a MAX_MIRROR_PENDING_ROWS cap (1M rows) on Inserter: detach
fetch_adds the batch's row count before spawning; if the cap is
exceeded the reservation is rolled back, the batch is dropped, a
DIST_MIRROR_DROPPED_ROW_COUNT metric is bumped, and a warn is logged
instead of spawning. On normal completion the reservation is released
in both success and error paths.

This keeps core writes alive while degrading flow mirror writes to
best-effort, which is the intended trade-off: protect the primary
insert path at the cost of possible flow aggregation gaps.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address review - balanced reservation, gauge race, log rate limit

P1: Compute pending_rows() from actual per-peer payloads in
self.requests instead of the pre-fanout num_rows field. This fixes
under-reservation for same-table multi-region batches and over-release
in multi-flownode fanout. Each peer task now releases exactly its own
share so reservations and releases balance.

P2: Replace load+set gauge publication with add/sub symmetry.
reserve_mirror_pending_rows does fetch_add + gauge.add, release does
fetch_update + gauge.sub, eliminating the stale-value race.

P3: Rate-limit the dropped-batch warning to at most once per 10s via
MirrorDropLog (atomic compare_exchange), accumulating batch/row counts
between reports.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix race in dual-peer test - wait for gauge release before unlocking

The dual-peer test previously only waited for the pending budget to
return to its start value before releasing MIRROR_METRIC_TEST_LOCK.
But release_mirror_pending_rows moves the budget before the gauge, so
a worker's gauge.sub could still be in flight when the next test
acquires the lock and records its baseline, polluting the result.

Now waits for both pending == start AND gauge == gauge_before.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(operator): preserve instant writes and limit mirror failure logs

Reject saturated requests containing instant-TTL rows with the existing retryable write-admission error before mirror or datanode dispatch. Keep normal source-table inserts independent of best-effort mirror drops.

Reuse a smaller process-wide event limiter for both dropped batches and failed peer tasks, reporting at most once per 10 seconds with failure count, peer context, and error chain. Cover actual mirror construction, mixed and instant admission, fast/slow peer shares, failed-task release, and log concurrency.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(operator): reject oversized instant mirror batches permanently

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-09 02:28:32 +00:00
fys c46fbde9ac fix(json2): preserve metadata when adding columns via alter table (#8696)
* fix(json2): support adding columns via alter table

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): scope missing-column null handling to JSON2

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-08 09:23:25 +00:00
dennis zhuang 90f117b957 fix(partition): preserve region mapping during repartition (#9458)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-08 07:30:12 +00:00
jeremyhi c82da8308b feat: add bounded Metric logical-table batch DDL (#9427)
* feat: add bounded Metric logical-table batch DDL

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: enforce logical batch DDL interceptor boundaries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: bound rewritten DDL size through shared SQL formatting

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve logical DDL error diagnostics

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: verify logical DDL error logging

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-08 06:33:56 +00:00
Lei, HUANG 5ccfcd4644 feat: support automatic column addition for Flight bulk inserts (#9285)
* fix: auto-add columns when initializing bulk insert streams

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject nested columns in bulk schema auto-add

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject unknown columns in non-empty bulk inserts

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-29 06:47:06 +00:00
jeremyhi 3c0e2a8d55 feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject empty packed export time ranges

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover cancellation during packed export I/O

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 15:22:02 +00:00
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-28 12:35:47 +00:00
Ning Sun fedce5c5ec feat: support non-millisecond time index units in the logical batcher (#9346)
* feat: allow customized time index unit for metric engine table

* test: provide query tests

* refactor: revert unnecessary change

* refactor: share timestamp unit conversions in api helper

Address review feedback on the time index unit changeset:

- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
  (the only crate that sees both proto ColumnDataType and TimeUnit due
  to layering; common-time and datatypes have no greptime-proto dep).
  This removes the ColumnDataType -> TimeUnit match duplicated between
  operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
  convert_timestamp_value_data by reusing api::helper::to_grpc_value
  for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
  values, so an overflow mid-batch leaves the request half-converted;
  harmless because the error aborts the whole insert request.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: align time units per destination table and floor remote-read timestamps

Address review feedback on PR #9236:

- Align each metric insert request to the unit of the table it actually
  targets: an existing logical table keeps its own unit (it may be bound
  to a different physical table than the one selected by the request),
  and only new tables use the selected physical table's unit. The
  previous blanket conversion rewrote valid millisecond samples to the
  selected physical table's unit and the engine rejected them.
  Regression test: writing an existing millisecond logical table and a
  new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
  negative infinity (div_euclid), consistent with
  Timestamp::convert_to on the ingestion path; arrow's cast truncates
  towards zero and returned -1ms for a stored -1001us. Widening
  (second -> millisecond) keeps the exact arrow cast. Regression test:
  a negative, non-aligned timestamp round-trips as -2ms.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: fold time unit alignment into existing table lookups

Address review feedback on PR #9236:

- The per-destination unit alignment no longer runs its own pass of
  table lookups: create_or_alter_tables_on_demand gains an
  align_time_index_unit parameter (metric engine path only) and
  converts each request inside the lookups it already performs —
  existing tables to their own unit, new tables to the selected
  physical table's. Default ingest paths now issue zero additional
  catalog lookups compared to main; the separate alignment pass remains
  only in the opt-in logical batcher pre-gate, next to the eligibility
  check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
  (validate_column_count_match guarantees row widths) instead of
  Optional get_mut; the gate-side alignment validates widths itself.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: resolve the batcher time index guard once per write target

All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.

Signed-off-by: Ning Sun <sunning@greptime.com>

* feat: support non-millisecond time index units in the logical batcher

Make the logical table batcher's bulk encode path unit-aware so physical
metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can
use logical batching instead of falling back to the ordinary insert path.

- rows_to_aligned_record_batch builds the time index column in the
  TARGET schema's unit, converting any timestamp encoding via
  Timestamp::convert_to (flooring on narrowing, consistent with the
  ordinary insert path).
- New tables created by the batcher use the selected physical table's
  time index unit (resolved once per submit; a missing physical table
  keeps the millisecond auto-create default).
- columns_taxonomy and the can_batch_metric_rows schema whitelist accept
  any timestamp unit; the prometheus remote write v1/v2 batcher gates
  and the OTLP pre-gate alignment are removed together with
  Inserter::align_metric_row_inserts_time_unit, as the batcher now
  converts internally.

Closes #9342

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

* fix: address review issue

* refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path

The main merge of #9236 (squash) resurrected the OTLP pre-gate
alignment and Inserter::align_metric_row_inserts_time_unit, which this
branch had removed. Drop them again:

- The pre-gate existed because the #9236-era bulk eligibility gate only
  accepted millisecond schemas, so nanosecond-encoded OTLP requests had
  to be converted before the check. This branch makes the bulk path
  unit-aware (the gate accepts all time index units and batch alignment
  converts each request to its destination's unit), so the pre-gate is
  redundant and only added N+1 catalog lookups per batched request —
  the very lookup-count overhead raised in the #9236 review.
- The per-destination unit semantics it implemented remain enforced in
  the two paths that need them: the ordinary insert path
  (create_or_alter_tables_on_demand converts inside its existing table
  lookups) and the batched path (batch alignment resolves each
  destination schema and converts to it).

test_otlp_logical_batcher_alignment (the test the pre-gate originally
fixed) and the mixed-physical-table regression both pass without it.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: cover OTLP batcher cross-physical fallback and nanosecond physical

Extend the logical batcher integration coverage for the cases
previously guarded by the removed OTLP pre-gate alignment:

- test_otlp_logical_batcher_fallback_for_cross_physical_destination:
  with the batcher enabled, an OTLP request targeting an existing
  logical table bound to another physical table must NOT enter the
  batcher (the bulk eligibility check rejects the destination binding)
  and the ordinary insert path must convert it to the destination's
  unit (60s -> 60_000_000us).
- test_otlp_logical_batcher_non_millisecond_physical_table now covers
  both microsecond and nanosecond physical tables (parameterized),
  asserting batcher submissions and unit-precise stored values.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: validate per-batch physical bindings and avoid intermediate timestamp buffers

Address review feedback on PR #9346:

- accepts_bulk_destinations dedupes on (schema, table, selected physical)
  instead of (schema, table): one request can select different physical
  tables per batch (per-series x_greptime_physical_table labels), and the
  old key let a second selection skip validation and flush rows through
  the wrong physical's regions. Missing tables additionally reject
  conflicting physical selections within the same request. Regression
  test covers an existing destination, a missing destination, and a
  consistent selection (which must still batch).
- The timestamp column builder appends each value directly into the
  target-unit Arrow builder; values already in the target unit (the
  unchanged millisecond fast path) are appended without conversion, so
  the default millisecond physical pays no Timestamp construction or
  intermediate Vec allocation.
- The non-millisecond batching tests assert the submit_build_and_align
  counter, which increments on every batcher submission in both
  acknowledgement modes, so a silent fallback to ordinary insertion
  fails the tests instead of passing on stored values alone.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-28 12:27:36 +00:00
Ning Sun 55b7e08a1a feat: allow customized time index unit for metric engine table (#9236)
* feat: allow customized time index unit for metric engine table

* test: provide query tests

* refactor: revert unnecessary change

* refactor: share timestamp unit conversions in api helper

Address review feedback on the time index unit changeset:

- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
  (the only crate that sees both proto ColumnDataType and TimeUnit due
  to layering; common-time and datatypes have no greptime-proto dep).
  This removes the ColumnDataType -> TimeUnit match duplicated between
  operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
  convert_timestamp_value_data by reusing api::helper::to_grpc_value
  for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
  values, so an overflow mid-batch leaves the request half-converted;
  harmless because the error aborts the whole insert request.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: align time units per destination table and floor remote-read timestamps

Address review feedback on PR #9236:

- Align each metric insert request to the unit of the table it actually
  targets: an existing logical table keeps its own unit (it may be bound
  to a different physical table than the one selected by the request),
  and only new tables use the selected physical table's unit. The
  previous blanket conversion rewrote valid millisecond samples to the
  selected physical table's unit and the engine rejected them.
  Regression test: writing an existing millisecond logical table and a
  new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
  negative infinity (div_euclid), consistent with
  Timestamp::convert_to on the ingestion path; arrow's cast truncates
  towards zero and returned -1ms for a stored -1001us. Widening
  (second -> millisecond) keeps the exact arrow cast. Regression test:
  a negative, non-aligned timestamp round-trips as -2ms.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: fold time unit alignment into existing table lookups

Address review feedback on PR #9236:

- The per-destination unit alignment no longer runs its own pass of
  table lookups: create_or_alter_tables_on_demand gains an
  align_time_index_unit parameter (metric engine path only) and
  converts each request inside the lookups it already performs —
  existing tables to their own unit, new tables to the selected
  physical table's. Default ingest paths now issue zero additional
  catalog lookups compared to main; the separate alignment pass remains
  only in the opt-in logical batcher pre-gate, next to the eligibility
  check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
  (validate_column_count_match guarantees row widths) instead of
  Optional get_mut; the gate-side alignment validates widths itself.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: resolve the batcher time index guard once per write target

All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

* fix: address review issue

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-28 07:15:36 +00:00
jeremyhi bf67246645 perf: bound concurrent Metric export writers (#9296)
* fix: preserve objects after conditional Metric export collisions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: share Metric export writer and payload budgets

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover bounded Metric export conversion and mixed roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve export output after ambiguous close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align HTTP export timeout with managed cancellation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: trim redundant Metric export roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: bound pending Metric export writer handles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: protect export files on failed writes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve ordinary export cleanup and large rows

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: report Metric export collisions as invalid arguments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed conditional export closes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support view arrays in managed ordinary export

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: expect successful cleanup after failed close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up unsynced overwrite after close failure

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: scope export JSON expansion estimates to JSON columns

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: rechunk ordinary export batches within the write budget

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 03:10:21 +00:00
Yingwen 3c2aac0a55 feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(storage): route series index build requests

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(admin): add BUILD_SERIES_INDEX

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: specify compaction type in series index fixtures

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: correct series index SQL fixtures and error assertions

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(compat): preserve legacy index rebuild across upgrades

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(sql): cover series index admin validation

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): bound series index maintenance queue

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: remove series index how-to guide

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: remove index build upgrade compatibility case

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: address series index reconciliation review feedback

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: bound manual series index reconciliation admission

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update greptime-proto to merged index build options

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-24 09:39:21 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
Weny Xu 723da69b21 feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-22 14:38:17 +00:00
fys 045441e3cc feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-22 12:56:26 +00:00
LFC 16978cf6c2 feat(trace): support Semantic Graph for Trace V2 (follow-up to #9192) (#9278)
* feat(trace): support Semantic Graph for Trace V2

Signed-off-by: luofucong <luofc@foxmail.com>

* fix: remove unused annotation context import

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-22 02:39:36 +00:00
jeremyhi 19c127d9c3 feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: restore packed metric snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: stream large packed parquet entries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: allow default S3 endpoint in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: skip unconfigured S3 in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use portable file URLs in packed restore fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate packed snapshot structure before restore

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve strict manifest decoding and verify fixture

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject packed layout on both database export paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: implement file size in coordinator test storage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use expect_err for rejected export layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 00:07:04 +00:00
jeremyhi 263e229103 feat: add experimental Metric export to V2 snapshots (#9233)
* feat: add experimental Metric export to V2 snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate the complete Metric export capability response

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: construct portable file URLs for Metric export fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address Metric export review nits

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-21 09:10:01 +00:00
shuiyisongandLei, HUANG 66d38e8e1c feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add `InsertLimitInterceptor` hook to `Inserter`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: attribute insert limit checks to the target table's database

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: unify write admission through metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce write admission for pending row batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: distinguish internal requests for ingestion metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: admit split ingestion requests once per database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: OpenTSDB throws error reason

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use meter crate main rev

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude database ingest rate limit from table options

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: reserve channel 255 for internal requests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 07:27:22 +00:00
LFC 697cc5fa29 fix(json): fix JSONPath panic with jsonb 0.5.6 (follow-up to #9192) (#9228)
* fix(json): upgrade jsonb to fix unterminated JSONPath panic

Signed-off-by: luofucong <luofc@foxmail.com>

* fix(json): align integer extraction with JSON2 conversions

Signed-off-by: luofucong <luofc@foxmail.com>

* style: format jsonb dependency declaration

Signed-off-by: luofucong <luofc@foxmail.com>

* fix(otlp): report concrete unsupported JSONB type names

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-18 04:55:02 +00:00
discord9 6fa375021f feat(flow): expose extension-owned batching execution hooks (#9171)
* fix: preserve structured query errors through distributed execution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: use exact sequence ranges for capable incremental flow sources

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update SQL expectations for preserved query error codes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: format exact sequence recovery tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: merge aggregate states in incremental flows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: exercise dispatched exact delta failure recovery

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: record flushed exact sequence flow results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: pass query engine to state merge execution tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(flow): expose extension-owned batching execution hooks

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): correct extension matcher borrowing and regression assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: prove executed empty exact deltas keep incremental mode

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: pass query engine to empty exact delta regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover aggregate state merge through the full flow path

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: pass query engine to flow batching task regression calls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-18 04:35:29 +00:00