948 Commits
Author SHA1 Message Date
Yingwen 3a4c24a863 fix: temporarily disable v2 series scan by default (#9436)
* fix: disable v2 series scan by default temporarily

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: explicitly enable v2 series scan in sqlness configs

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: regenerate sqlness results for legacy series scan default

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-02 12:09:16 +00:00
Dragon RoarandDennis Zhuang d7f5331876 feat: add CORS support for gRPC-Web on the frontend gRPC server (#9374)
* feat: add CORS support for gRPC-Web on the frontend gRPC server

A browser-based gRPC-Web client cannot read a cross-origin response. The
request carries a non-simple content type, so the browser sends an
`OPTIONS` preflight first, and it drops the call when the response has no
`Access-Control-Allow-Origin`.

Add `enable_cors` and `cors_allowed_origins` to `[grpc]`, threaded from
`GrpcOptions` to `GrpcServerConfig` the same way `tls` is.

The CORS layer sits outside `tonic_web::GrpcWebLayer`, so it answers the
preflight before the request reaches the gRPC routes. It exposes
`grpc-status`, `grpc-message` and `grpc-status-details-bin`, which a
browser cannot read otherwise. An empty `cors_allowed_origins` allows any
origin.

`enable_cors` defaults to false, unlike `[http]`. `GrpcOptions` is shared
by the public `[grpc]` section and the internal `[internal_grpc]` one, and
serde cannot tell them apart, so a true default would also turn CORS on
for the internal gRPC listeners: the frontend internal gRPC server, and
the datanode and flownode gRPC servers. None of them authenticate callers,
and a browser on the host or in the cluster network can reach all of them.
Set `enable_cors = true` to turn it on.

Tests start a real server on an ephemeral port and cover the preflight, a
custom origin list, and the disabled case.

Update the example TOMLs and regenerate config/config.md.

Signed-off-by: lczllx <2181719471@qq.com>

* fix: drop the redundant OPTIONS from the gRPC CORS allow_methods

Signed-off-by: lczllx <2181719471@qq.com>

* test: assert the allow-headers and allow-methods headers in the gRPC CORS preflight

The preflight answers with `access-control-allow-headers: *` from
`AllowHeaders::any()`, and with `access-control-allow-methods: post` from the
POST-only `allow_methods` list. Pin both, the way the HTTP CORS test pins its
own headers: a missing allow-headers header would let the preflight pass the
origin check and still have the browser block every gRPC-Web call, which a
non-browser client would never notice.

Signed-off-by: lczllx <2181719471@qq.com>

* docs(config): stop the example configs from setting the new gRPC CORS keys

Signed-off-by: lczllx <2181719471@qq.com>

* test(servers): cover the exposed gRPC CORS headers and pin the option plumbing

Signed-off-by: lczllx <2181719471@qq.com>

* docs(config): document the gRPC CORS origin example with `#+`

Signed-off-by: lczllx <2181719471@qq.com>

* fix(frontend): keep CORS off the internal gRPC server

Signed-off-by: lczllx <2181719471@qq.com>

* docs(grpc): document that the internal gRPC server never serves CORS

Signed-off-by: lczllx <2181719471@qq.com>

* Update src/servers/src/grpc.rs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: trim redundant comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: share CORS origin parsing and simplify gRPC-Web CORS tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: enable gRPC CORS only on the frontend public server

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(config): clarify the gRPC CORS options

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-30 10:40:17 +00:00
jeremyhi 78eefbe4c4 perf: batch schema export requests (#9408)
* perf: batch schema export requests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: propagate legacy schema export failures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep credentials out of SQL response errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use a typo-safe dotted catalog name

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(cli): address schema export review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-30 06:23:33 +00:00
dennis zhuang 78a7b93292 fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics

- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): align label_replace and label_join edge cases with Prometheus

- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): join absent labels as empty strings in label_join

concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): skip rows without a sample in timestamp()

timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-30 01:42:11 +00:00
Ning Sun 3131cdbcb6 fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests

The request-memory accounting only bounds and charges the encoded bytes on
the wire, but a compressed body can expand far beyond that during
decompression, so tiny requests could allocate disproportionate frontend
memory before any protobuf validation or quota charge.

- Handler-level decompression (Prometheus remote read/write v1+v2, Loki)
  now enforces a hard 512 MiB decoded-size cap, checked before any output
  buffer is allocated, and charges the decoded bytes to the shared
  ServerMemoryLimiter, holding the permits for the lifetime of the
  decompressed buffer.
- gRPC requests with transport compression reserve the configured
  max_recv_message_size before tonic decompresses, so the decoding phase
  is admitted against max_in_flight_write_bytes; the later per-message
  charge is skipped to avoid double accounting.
- The HTTP memory-limit middleware keeps its upfront Content-Length charge
  but now also accounts the bytes actually streamed beyond it, so chunked
  requests and understated Content-Length headers no longer bypass the
  aggregate quota.
- Routes that decompress request bodies via RequestDecompressionLayer
  (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards)
  now charge the decompressed bytes as handlers consume them: the global
  middleware marks Content-Encoding requests, and a route-local
  accounting layer inside the decompression layer charges the decoded
  stream. Plain requests are skipped to avoid double-counting.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on memory admission guard lifetimes and 429 mapping

- Retain the gRPC pre-decode reservation for the whole request: the
  extensions holding the guard were dropped when the request was consumed
  (into_inner), releasing the reservation while per-message charges
  stayed skipped. Both the unary and streaming handlers now clone and
  hold the reservation marker for the duration of request handling.
- Retain the HTTP body permits across the handler: the AccountedBody
  wrapper and the request extensions are dropped once the extractors
  finish collecting the body, before the handler is done with the decoded
  data. Both accounting middlewares now keep their own accounting handle
  alive across next.run(req).await.
- Return a ChargedBuffer from the Loki snappy decompressor so the
  reservation outlives the decompressed bytes through the caller's
  protobuf decoding, matching the Prometheus path.
- Map mid-stream quota exhaustion to 429 instead of the generic body
  error (400): the accounting flags exhaustion and the middlewares
  rewrite the extractor rejection, so clients can distinguish
  backpressure from malformed input.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: resolve the in-flight body acquisition before trying a new one

A parked acquisition in AccountedBody::charge always belongs to the
currently buffered frame, so it must be resolved before any new
acquisition is tried. Letting the fast-path try_acquire succeed while a
waiter is parked left the waiter alive, and its late completion would
then be credited with a later frame's byte count, under-reserving memory
relative to what was marked charged.

Adds a regression test that reproduces the misattribution: with the
buggy ordering the test delivers a body the quota cannot cover; with the
fix the over-quota frame waits for its own acquisition and times out.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-29 10:07:18 +00:00
discord9 abf1396c28 fix(client): yield Flight batches and affected rows without waiting for next message (#8918)
* fix(client): avoid Flight metrics lookahead stalls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(client): extract trailing Flight metrics task

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(client): synchronize trailing metrics and bound cancellation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-29 08:05:50 +00:00
Lei, HUANG 5ccfcd4644 feat: support automatic column addition for Flight bulk inserts (#9285)
* fix: auto-add columns when initializing bulk insert streams

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject nested columns in bulk schema auto-add

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject unknown columns in non-empty bulk inserts

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-29 06:47:06 +00:00
jeremyhi 3c0e2a8d55 feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject empty packed export time ranges

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover cancellation during packed export I/O

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 15:22:02 +00:00
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-28 12:35:47 +00:00
Ning Sun fedce5c5ec feat: support non-millisecond time index units in the logical batcher (#9346)
* feat: allow customized time index unit for metric engine table

* test: provide query tests

* refactor: revert unnecessary change

* refactor: share timestamp unit conversions in api helper

Address review feedback on the time index unit changeset:

- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
  (the only crate that sees both proto ColumnDataType and TimeUnit due
  to layering; common-time and datatypes have no greptime-proto dep).
  This removes the ColumnDataType -> TimeUnit match duplicated between
  operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
  convert_timestamp_value_data by reusing api::helper::to_grpc_value
  for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
  values, so an overflow mid-batch leaves the request half-converted;
  harmless because the error aborts the whole insert request.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: align time units per destination table and floor remote-read timestamps

Address review feedback on PR #9236:

- Align each metric insert request to the unit of the table it actually
  targets: an existing logical table keeps its own unit (it may be bound
  to a different physical table than the one selected by the request),
  and only new tables use the selected physical table's unit. The
  previous blanket conversion rewrote valid millisecond samples to the
  selected physical table's unit and the engine rejected them.
  Regression test: writing an existing millisecond logical table and a
  new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
  negative infinity (div_euclid), consistent with
  Timestamp::convert_to on the ingestion path; arrow's cast truncates
  towards zero and returned -1ms for a stored -1001us. Widening
  (second -> millisecond) keeps the exact arrow cast. Regression test:
  a negative, non-aligned timestamp round-trips as -2ms.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: fold time unit alignment into existing table lookups

Address review feedback on PR #9236:

- The per-destination unit alignment no longer runs its own pass of
  table lookups: create_or_alter_tables_on_demand gains an
  align_time_index_unit parameter (metric engine path only) and
  converts each request inside the lookups it already performs —
  existing tables to their own unit, new tables to the selected
  physical table's. Default ingest paths now issue zero additional
  catalog lookups compared to main; the separate alignment pass remains
  only in the opt-in logical batcher pre-gate, next to the eligibility
  check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
  (validate_column_count_match guarantees row widths) instead of
  Optional get_mut; the gate-side alignment validates widths itself.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: resolve the batcher time index guard once per write target

All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.

Signed-off-by: Ning Sun <sunning@greptime.com>

* feat: support non-millisecond time index units in the logical batcher

Make the logical table batcher's bulk encode path unit-aware so physical
metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can
use logical batching instead of falling back to the ordinary insert path.

- rows_to_aligned_record_batch builds the time index column in the
  TARGET schema's unit, converting any timestamp encoding via
  Timestamp::convert_to (flooring on narrowing, consistent with the
  ordinary insert path).
- New tables created by the batcher use the selected physical table's
  time index unit (resolved once per submit; a missing physical table
  keeps the millisecond auto-create default).
- columns_taxonomy and the can_batch_metric_rows schema whitelist accept
  any timestamp unit; the prometheus remote write v1/v2 batcher gates
  and the OTLP pre-gate alignment are removed together with
  Inserter::align_metric_row_inserts_time_unit, as the batcher now
  converts internally.

Closes #9342

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

* fix: address review issue

* refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path

The main merge of #9236 (squash) resurrected the OTLP pre-gate
alignment and Inserter::align_metric_row_inserts_time_unit, which this
branch had removed. Drop them again:

- The pre-gate existed because the #9236-era bulk eligibility gate only
  accepted millisecond schemas, so nanosecond-encoded OTLP requests had
  to be converted before the check. This branch makes the bulk path
  unit-aware (the gate accepts all time index units and batch alignment
  converts each request to its destination's unit), so the pre-gate is
  redundant and only added N+1 catalog lookups per batched request —
  the very lookup-count overhead raised in the #9236 review.
- The per-destination unit semantics it implemented remain enforced in
  the two paths that need them: the ordinary insert path
  (create_or_alter_tables_on_demand converts inside its existing table
  lookups) and the batched path (batch alignment resolves each
  destination schema and converts to it).

test_otlp_logical_batcher_alignment (the test the pre-gate originally
fixed) and the mixed-physical-table regression both pass without it.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: cover OTLP batcher cross-physical fallback and nanosecond physical

Extend the logical batcher integration coverage for the cases
previously guarded by the removed OTLP pre-gate alignment:

- test_otlp_logical_batcher_fallback_for_cross_physical_destination:
  with the batcher enabled, an OTLP request targeting an existing
  logical table bound to another physical table must NOT enter the
  batcher (the bulk eligibility check rejects the destination binding)
  and the ordinary insert path must convert it to the destination's
  unit (60s -> 60_000_000us).
- test_otlp_logical_batcher_non_millisecond_physical_table now covers
  both microsecond and nanosecond physical tables (parameterized),
  asserting batcher submissions and unit-precise stored values.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: validate per-batch physical bindings and avoid intermediate timestamp buffers

Address review feedback on PR #9346:

- accepts_bulk_destinations dedupes on (schema, table, selected physical)
  instead of (schema, table): one request can select different physical
  tables per batch (per-series x_greptime_physical_table labels), and the
  old key let a second selection skip validation and flush rows through
  the wrong physical's regions. Missing tables additionally reject
  conflicting physical selections within the same request. Regression
  test covers an existing destination, a missing destination, and a
  consistent selection (which must still batch).
- The timestamp column builder appends each value directly into the
  target-unit Arrow builder; values already in the target unit (the
  unchanged millisecond fast path) are appended without conversion, so
  the default millisecond physical pays no Timestamp construction or
  intermediate Vec allocation.
- The non-millisecond batching tests assert the submit_build_and_align
  counter, which increments on every batcher submission in both
  acknowledgement modes, so a silent fallback to ordinary insertion
  fails the tests instead of passing on stored values alone.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-28 12:27:36 +00:00
Ning Sun 55b7e08a1a feat: allow customized time index unit for metric engine table (#9236)
* feat: allow customized time index unit for metric engine table

* test: provide query tests

* refactor: revert unnecessary change

* refactor: share timestamp unit conversions in api helper

Address review feedback on the time index unit changeset:

- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
  (the only crate that sees both proto ColumnDataType and TimeUnit due
  to layering; common-time and datatypes have no greptime-proto dep).
  This removes the ColumnDataType -> TimeUnit match duplicated between
  operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
  convert_timestamp_value_data by reusing api::helper::to_grpc_value
  for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
  values, so an overflow mid-batch leaves the request half-converted;
  harmless because the error aborts the whole insert request.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: align time units per destination table and floor remote-read timestamps

Address review feedback on PR #9236:

- Align each metric insert request to the unit of the table it actually
  targets: an existing logical table keeps its own unit (it may be bound
  to a different physical table than the one selected by the request),
  and only new tables use the selected physical table's unit. The
  previous blanket conversion rewrote valid millisecond samples to the
  selected physical table's unit and the engine rejected them.
  Regression test: writing an existing millisecond logical table and a
  new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
  negative infinity (div_euclid), consistent with
  Timestamp::convert_to on the ingestion path; arrow's cast truncates
  towards zero and returned -1ms for a stored -1001us. Widening
  (second -> millisecond) keeps the exact arrow cast. Regression test:
  a negative, non-aligned timestamp round-trips as -2ms.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: fold time unit alignment into existing table lookups

Address review feedback on PR #9236:

- The per-destination unit alignment no longer runs its own pass of
  table lookups: create_or_alter_tables_on_demand gains an
  align_time_index_unit parameter (metric engine path only) and
  converts each request inside the lookups it already performs —
  existing tables to their own unit, new tables to the selected
  physical table's. Default ingest paths now issue zero additional
  catalog lookups compared to main; the separate alignment pass remains
  only in the opt-in logical batcher pre-gate, next to the eligibility
  check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
  (validate_column_count_match guarantees row widths) instead of
  Optional get_mut; the gate-side alignment validates widths itself.

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf: resolve the batcher time index guard once per write target

All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

* fix: address review issue

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-28 07:15:36 +00:00
discord9 678aa81cae fix(client): complete transport lane isolation (#9030)
* fix(client): complete transport lane isolation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(client): deprecate legacy single-manager constructors

Per review: mark the legacy single-manager constructors and helper as
deprecated so callers move to the isolated query/control manager pair.
Tests intentionally exercising the legacy path are annotated with
`#[allow(deprecated)]`.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(client): update deprecated constructor callers for CI

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 04:39:20 +00:00
jeremyhi bf67246645 perf: bound concurrent Metric export writers (#9296)
* fix: preserve objects after conditional Metric export collisions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: share Metric export writer and payload budgets

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover bounded Metric export conversion and mixed roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve export output after ambiguous close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align HTTP export timeout with managed cancellation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: trim redundant Metric export roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: bound pending Metric export writer handles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: protect export files on failed writes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve ordinary export cleanup and large rows

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: report Metric export collisions as invalid arguments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed conditional export closes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support view arrays in managed ordinary export

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: expect successful cleanup after failed close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up unsynced overwrite after close failure

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: scope export JSON expansion estimates to JSON columns

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: rechunk ordinary export batches within the write budget

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 03:10:21 +00:00
Yingwen 3c2aac0a55 feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(storage): route series index build requests

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(admin): add BUILD_SERIES_INDEX

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: specify compaction type in series index fixtures

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: correct series index SQL fixtures and error assertions

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(compat): preserve legacy index rebuild across upgrades

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(sql): cover series index admin validation

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): bound series index maintenance queue

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: remove series index how-to guide

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: remove index build upgrade compatibility case

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: address series index reconciliation review feedback

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: bound manual series index reconciliation admission

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update greptime-proto to merged index build options

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-24 09:39:21 +00:00
dennis zhuang b9f991502c refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 05:38:54 +00:00
Yingwen c7fa48ef95 feat(mito): limit approximate series index disk usage (#9313)
* feat(mito): limit series index disk usage

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): enforce series index quota during reconciliation

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): remove series index disk budget layer

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): simplify series index limit to estimated usage

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): trust index catalogs when loading snapshots

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): minimize series index disk limit changes

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): keep series index cleanup running at capacity

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): avoid no-op index clones and stabilize capacity tests

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: update config API expectation and stabilize index build tests

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-23 14:18:42 +00:00
shuiyisong 20dde2601f feat: enable native histogram ingestion by default (#9301)
* feat: enable native histogram ingestion by default

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-09-23 13:03:51 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
Weny Xu 953d01ac54 feat: support pending rows batching for MySQL and PostgreSQL (#9302)
* feat: support pending rows batching for MySQL and PostgreSQL

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style: group batcher imports before item definitions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: use 65536 as the default batcher worker channel capacity

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: use a distinct custom worker channel capacity

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: complete Prom config in worker capacity override case

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-23 04:41:13 +00:00
Ning Sun 0f118bc5ba fix: serialize struct to json in postgres (#9170)
* feat: serialize struct to json in postgres

* fix: support view scalars and preserve null structs in scalar-to-value conversion

Address PR review:
- Utf8View/BinaryView ScalarValues now convert like their non-view forms
  instead of failing row extraction for struct columns
- a null struct scalar converts to Value::Null so a null struct inside a
  list stays null in the serialized JSON

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: return errors instead of panics for unsupported arrow field types

Struct-typed query results with arrow field types greptimedb cannot
represent (e.g. Decimal256) used to panic during schema conversion and
row extraction, dropping the client connection. They now surface as
query errors:

- ConcreteDataType::try_from builds struct types fallibly via the new
  StructType::try_from_arrow_fields
- Value::try_from(ScalarValue::Struct) uses the same fallible path
- new try_value_from_array converts an arrow element to Value with
  error propagation, used by the postgres struct encoding

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-23 01:55:46 +00:00
dennis zhuang f3eb8e6c72 test: cut integration test time and make the storage matrix meaningful (#9308)
* test: cut integration test time and make the storage matrix meaningful

tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.

- Point the PR CI object-store matrix at the MinIO already started by
  tests-integration/fixtures. Three GT_S3_* consumers did not read
  GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
  they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
  release integration-test container. The release previously ran every
  remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
  /tmp/greptimedb_cache, which the datanode wipes on startup, so a
  starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
  non-trivial SST representation: JSON/JSON2 columns, native histograms,
  metric-engine logical tables, and tables carrying fulltext or skipping
  indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
  shorten it in the event tests, which otherwise wait a 5s window per DDL
  they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
  test_grpc_message_size_ok and is now rewritten to assert the negotiated
  grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
  prefixes of their copy_from siblings; two standalone/distributed event
  test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
  in a parallelism-dependent order, so its last digits are unstable; the
  test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
  ttl=7d with 2025 timestamps, so its rows were only readable as long as
  they stayed in the memtable.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: address review — wire nightly real-S3 job into check-status, keep the short event interval

The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.

In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-23 01:52:52 +00:00
Weny Xu 723da69b21 feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-22 14:38:17 +00:00
dennis zhuang aa36f74feb fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-22 14:38:22 +00:00
Dhruv Vaishnav f1e9a74f00 fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
2026-09-22 07:35:07 +00:00
jeremyhi 9dfe057199 test: make export chunk deletion failure deterministic (#9291)
* test: make export chunk deletion failure deterministic

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify export chunk deletion failure fixture

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: guide deterministic storage failure tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 04:58:58 +00:00
jeremyhi 19c127d9c3 feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: restore packed metric snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: stream large packed parquet entries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: allow default S3 endpoint in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: skip unconfigured S3 in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use portable file URLs in packed restore fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate packed snapshot structure before restore

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve strict manifest decoding and verify fixture

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject packed layout on both database export paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: implement file size in coordinator test storage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use expect_err for rejected export layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 00:07:04 +00:00
jeremyhi 263e229103 feat: add experimental Metric export to V2 snapshots (#9233)
* feat: add experimental Metric export to V2 snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate the complete Metric export capability response

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: construct portable file URLs for Metric export fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address Metric export review nits

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-21 09:10:01 +00:00
shuiyisongandLei, HUANG 66d38e8e1c feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add `InsertLimitInterceptor` hook to `Inserter`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: attribute insert limit checks to the target table's database

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: unify write admission through metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce write admission for pending row batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: distinguish internal requests for ingestion metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: admit split ingestion requests once per database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: OpenTSDB throws error reason

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use meter crate main rev

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude database ingest rate limit from table options

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: reserve channel 255 for internal requests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 07:27:22 +00:00
Ning Sun 4df557bb9e test: exclude testing feature completely (#9072)
* test: exclude testing feature completely

* chore: fmt
2026-09-21 06:45:04 +00:00
Weny Xu 941193e9c1 refactor: reorganize logical table batching and isolate encoding (#9210)
* refactor: relocate the logical table batcher

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: isolate logical batch conversion and region writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-21 03:37:47 +00:00
Lei, HUANG e9dd79d1d2 fix(mito2): compare primary key ranges across schema versions (#9205)
* fix(mito2): compare primary key ranges across schema versions

Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes.

Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): report invalid primary key ranges

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): scope primary key ranges to comparisons

Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts.

Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle.

Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): cache aligned primary key ranges per file

Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes.

Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): clarify primary key mapper schema snapshot

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): align primary key range fixtures with table contract

Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario.

Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): enable filesystem GC in release unit tests

Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito-codec): skip release value decoding in PK prefix counts

Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build.

Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito-codec): remove redundant prefix integration tests

Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: wait for MySQL to accept authenticated TCP queries

Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed.

Verify readiness with SELECT 1 over TCP rather than the initialization socket.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-20 14:30:12 +00:00
LFC fcb165cf77 feat(trace): support Jaeger queries for Trace V2 and optimize writes (follow-up to #9192) (#9257)
feat(trace): support Jaeger queries for v2 and optimize fixed-column writes

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-20 08:53:00 +00:00
jeremyhi 33cfb72a43 fix: make database export assertions portable on Windows (#9256)
fix: compare database export paths portably on Windows

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-20 04:23:37 +00:00
Yingwen 798adb64e7 feat(mito2): make range index reads and builds opt-in (#9219)
* feat: add config flag to control range index reads

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: disable range index builds when configured and default to off

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-20 03:13:52 +00:00
dennis zhuang 007836ba7a fix(prometheus): honor label matchers in __name__ values query (#9134)
* fix(prometheus): honor label matchers in __name__ values query

`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.

Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.

Only metric engine tables are covered; other engines share no column space
to scan.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(prometheus): batch-resolve metric names by table id

Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.

The catalog walk stays, but only to find the physical tables to scan.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): read an absent label as the empty string

A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.

Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.

This is the Prometheus compatibility fix #8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): don't panic on pre-epoch label value bounds

`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.

Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(prometheus): drop applicable_matchers, share the distinct scan

With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.

Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 03:11:06 +00:00
LFC 09a9d9088d feat(otlp): preserve trace v2 events and links as JSON (follow-up to #9192) (#9232)
feat(otlp): preserve trace v2 events and links as JSON

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-18 08:02:34 +00:00
jeremyhi dca01654bc feat: prepare and execute database Metric exports (#9180)
* feat: prepare and authorize captured database exports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: bound database export jobs and drain failures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: validate database export identity and restore equivalence

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate database export directory URLs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve Windows export directory paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject local database export filename aliases

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: consolidate database export planning policies

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: restore escaped database export filenames

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align database restore assertions with shared policies

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify database export boundaries and names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-17 12:02:39 +00:00
LFC aacf04cf6e feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-17 10:08:45 +00:00
Weny Xu be15c88e92 feat: batch ordinary table writes across HTTP protocols (#9115)
* feat: integrate table batching across HTTP protocols

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: skip empty prepared writes before batch admission

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: load batching protocols from environment and document frontend wiring

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove experimental prefix from pending rows batcher config

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style: sort frontend test dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: count batched ingestion once and update config snapshot

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-17 09:32:12 +00:00
discord9 7c7132ea65 refactor(flow): execute streaming flows with DataFusion (#8976)
* test(mito2): cover regex inverted index pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): execute streaming flows with DataFusion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): remove legacy streaming runtime

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): avoid retrying stateless sink inserts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): align stateless writes with sink schema

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): reject stale stateless source schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): validate stateless flow routing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* Revert "test(mito2): cover regex inverted index pruning"

This reverts commit 79e96ac745.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preserve source timestamps in stateless flows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): address stateless review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): recover stateless flows after schema changes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): serialize schema rebuilds and validate retained sources

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): verify streaming recovery and schema changes through SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): cool down failed schema rebuilds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): explain legacy aggregate recreation requirements

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): adapt stateless provider downcast to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-17 06:44:01 +00:00
discord9 f5428f8a6a test: regenerate expired TLS certificates for integration fixtures (#9196)
* test: regenerate expired TLS certificates for integration fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore root.srl serial file for TLS fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(ci): update compatibility test window to v1.2.1

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add proper TLS extensions to regenerated integration certificates

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-16 13:48:30 +00:00
discord9 8af3a04ed7 fix: preserve structured query errors through distributed execution (#9161)
* fix: preserve structured query errors through distributed execution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update SQL expectations for preserved query error codes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-16 06:28:15 +00:00
jeremyhi d7ada1761d feat: export logical tables from Metric physical scans (#9159)
* feat: add physical Metric table exporter

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: drain Metric export writes before cancellation cleanup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: construct Metric export error context lazily

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: name the logical table export entry point

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: share Parquet writer for logical table exports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve Parquet destinations and cancellation boundaries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify logical table export field names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify logical table export helper responsibilities

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate logical export membership by table ID

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify logical table export coverage and strengthen assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-16 02:48:47 +00:00
discord9andNing Sun 94d7e2c7fc feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: migrate DataFusion 55 APIs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve table function planning behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve PostgreSQL query compatibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve distributed execution plan behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover DataFusion 55 behavior regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update DataFusion 55 SQLness expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DataFusion 55 test API migration

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address DataFusion 55 CI regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address remaining DataFusion 55 regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: adapt latest base code to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: normalize environment-specific DataFusion 55 plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align final DataFusion 55 expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: isolate DataFusion 55 regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve empty result schema in timestamp widening

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve JSON source column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: use released DataFusion 55 integrations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: adapt latest execution plan mock to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: pin DataFusion recursive schema and date repairs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): align dictionary temporality match keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: retain Greptime DataFusion fork behaviors on version 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore ordinary function error expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh distributed count compatibility plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt last-row cast hint to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh instant last-row empty results for Arrow 59

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: simplify DataFusion expression visitor imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore sorting and PostgreSQL column-order assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): restore primitive numeric coercion signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(function): share geo integer signature types

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover timestamp widening overflow boundaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix decimal coercion regression imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): preserve scalar count_hash NULL state semantics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: simplify decimal clamp case type inference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: retain historical count_hash wrapper result

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restore timestamp widening equality and IN pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream aggregate dynamic filter correctness fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream null and predicate simplification fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore baseline JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore histogram JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh empty PromQL range result schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align native timestamp plan with DF55 decimal display

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh native timestamp SQLness results for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regenerate NULL sample empty result headers for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use DF55 child replacement API in timestamp regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: expose pushed scan dynamic filters to DF55 producers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: encode string-backed PostgreSQL OID aliases in binary results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify REGPROC binary and text over PostgreSQL protocol

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: register real PostgreSQL catalogs in server fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DF55 expression inventories for custom query plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: correct RangeSelect expression fixture and column identities

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* ci: wait for Kafka WAL helper deployment rollout

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update custom storage empty result headers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: require exact row counts in scan statistics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: suppress deprecated partition_statistics warning in test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-09-15 11:42:38 +00:00
LFC 9f6a79da30 fix: support native JSON2 row inserts over gRPC (#9145)
* fix: support native JSON2 row inserts over gRPC

Signed-off-by: luofucong <luofc@foxmail.com>

* test: cover unknown JSON2 schema compatibility

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-15 10:19:48 +00:00
Yingwen 7c85798ad0 feat(mito2): reconcile series indexes in background (#9086)
* feat(mito2): reconcile series indexes in background

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): clean up series indexes published during region drop

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(config): align series index examples with upstream enable flag

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): run series index tasks on compaction runtime

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-15 08:44:18 +00:00
Weny Xu d9b97796a6 fix: preserve count correctness after repartition (#9154)
* fix: preserve count correctness after repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: restore full predicate guard for count statistics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: update count plans for full predicate guard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve count statistics for safe partition scans

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: retain repartition home guard and cover staging flush

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-15 08:26:57 +00:00
Weny Xu 737025760e feat: support request-level WAL skipping for bulk inserts (#9110)
* feat: support request-level WAL skipping for bulk inserts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover bulk insert WAL skipping across protocols

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: align WAL snapshot naming with sequence watermarks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: bump proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-14 09:38:05 +00:00
Yingwen a23fe1c2dc fix: configure series indexes with an enable flag (#9141)
* fix: use join_dir for series index config path

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: configure series indexes with an enable flag

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: omit experimental series index from example configs

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: preserve legacy cache cleanup path behavior

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: remove trivial path joining tests

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: isolate worker group WAL directories on Windows

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-14 09:21:31 +00:00
discord9 13c69cdaee perf(gc): pack file reference exchange (#9009)
* perf(gc): pack file reference exchange

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): address packed reference review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): stop without retry when maintenance is enabled

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(meta): avoid logging malformed mailbox payloads

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 07:30:39 +00:00