Commit Graph
6509 Commits
Author SHA1 Message Date
Weny Xu 8ecf49dca0 ci: authenticate ECR Public pulls in benchmark workflows (#10056)
* ci: authenticate agent benchmark pulls to ECR Public

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: authenticate remaining ECR Public benchmark pulls

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-29 19:16:19 +00:00
fys d33c664183 feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* chore: remove unused code files

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): pass build type to bloom filter indexer

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): share bloom filter field and JSON update loop

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: better name

* refactor(mito2): simplify JSON path bloom index handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): expose index targets from creators

Signed-off-by: fys <fengys1996@gmail.com>

* docs(mito2): note JSON path index rebuild risk

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto

* chore: update greptime-proto

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-29 14:29:00 +00:00
Weny XuandLei, HUANG ee55c0c2fa feat: share concurrent SST metadata loads (#9729)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-10-29 10:03:35 +00:00
Lei, HUANG c7ea80558f fix(mito2): preserve live index files across remote compactions (#10046)
fix(mito2): reuse executor-owned compaction index resources

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-29 17:58:37 +08:00
Weny Xu 694ca48517 feat(meta): add drain migration batch operations (#10035)
* feat(meta): add drain migration batch operations

Build migrations through the shared manager and collect batch results with route reconciliation and failure strategies. Keep the existing fence-and-plan flow while preparing tested operations for drain orchestration.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): gate procedure receiver helpers to unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-29 16:34:41 +08:00
7d42feb023 chore(ci): update compatibility versions (#10051)
* chore(ci): update compatibility versions

Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>

* test(compat): bound the fulltext bloom case below v1.3.0

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit ea570c510e)
Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
Co-authored-by: discord9 <discord9@163.com>
2026-10-29 01:24:50 -07:00
shuiyisong ec240ad7e6 feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables

Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions.

Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): preserve reserved sparse key semantics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): clarify sparse errors and avoid panics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito): reduce sparse key validation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito-codec): reduce sparse validation cost

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(mito): remove ordinary sparse support

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-29 05:45:07 +00:00
Weny Xu e87766d7ef ci: reduce repeated Windows and fuzz tests on pull requests (#10043)
* ci: reduce repeated Windows and fuzz tests on pull requests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: use the nightly merge label as the enqueue opt-in

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: log nightly merge decisions for dry-run verification

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: gate expensive jobs on GitHub script tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: grant pull request write permission for nightly label sync

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: run platform checks on schedule or manual dispatch

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(ci): update PR gate count for scheduled platform checks

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-29 00:00:07 +00:00
shuiyisong efed847ca6 fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce sparse primary key schema

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: fix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-28 17:09:33 +00:00
Dragon Roar b21cd16f1d feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup

Signed-off-by: lczllx <2181719471@qq.com>

* fix(datanode): harden the startup object storage self-test

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): cover the File store skip in the object store self-test

Signed-off-by: lczllx <2181719471@qq.com>

* refactor(datanode): rename the object storage self-test to probe

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): disable object store self-test in sqlness and integration tests

Signed-off-by: lczllx <2181719471@qq.com>

* docs(datanode): correct what the object storage probe claims to cover

Signed-off-by: lczllx <2181719471@qq.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
2026-10-28 17:09:29 +00:00
Lei, HUANG 045cf74c3f perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task

Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.

The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.

One output of 4.8M rows took 1.28s instead of 1.65s.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): parallelize decoding and encoding of compaction outputs

Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.

The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.

The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): cancel compaction producers before consumer polling

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): support per-table compaction output parallelism

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoding tasks before returning failures

Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.

Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoders on SST write errors

Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.

Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain compaction outputs on cancellation

Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.

Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-28 17:09:26 +00:00
Weny Xu b83ceb5e7b feat: add reusable paged byte cache (#9791)
* feat: add reusable paged byte cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): include common cache in scope rules

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 17:09:23 +00:00
Yvan Wang 4c37ed4a0e feat(pipeline): add logfmt processor (#9586)
Add a built-in pipeline processor that parses logfmt lines
(https://brandur.org/logfmt) into structured objects, with support for
quoted values, escapes, bare-key flags, and ignore_missing.

Closes #5049

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
2026-10-28 17:09:21 +00:00
jeremyhi 8e16710425 fix(log-store): accept a retried first create of the conditional create probe (#10034)
* fix(log-store): accept a retried first create of the conditional create probe

The startup probe of the object store WAL wrote its first create by hand
and failed on any error. When that create is stored but its response is
lost, the retry layer repeats it, the repeat meets the object, and the
probe reported a failed write, so the WAL refused to open.

Move the create of put_if_absent, which reads the object back and accepts
identical content, into a path-based core and use it for the probe's
first create too. The second create, which the store must refuse, is
unchanged.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): reuse LosingWriter for the retried probe create test

Move LosingWriter from the fault matrix into the test module and use it,
armed with the lose_next pattern, in place of RetriedWriter. Say why the
probe's second create bypasses put_path_if_absent: a read-back would find
the first create's object and let an overwriting store pass.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-28 13:33:39 +00:00
Weny Xu f646f06430 ci: enqueue eligible pull requests during the nightly window (#10040)
* ci: enqueue eligible pull requests during the nightly window

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: hide nightly merge instructions in PR template

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 13:32:23 +00:00
Weny Xu ebb7e22566 chore(ci): gate fuzz, E2E and KV backends by diff scope (#9927)
* ci: gate the integration CI long tail by diff scope

`integration.yml` starts its long tail on every pull request: the three fuzz
families, the chaos legs, the export/import E2E job and the two sqlness
KvBackend legs. Most diffs cannot affect them, so a `scope` job now classifies
the pull request and tells those jobs whether to start.

The classifier is deterministic and fail-open. Forced modes (merge queue,
`schedule`, `workflow_dispatch`, `ci:full`, `CI_SCOPE_FORCE_ALL`) and three
floors -- always-run paths, an uncovered `src/` change, and a PromQL/flow
filter -- can only widen the result; everything else is a glob match against the
suite catalog. Every step of the job is `continue-on-error` and every gate reads
`needs.scope.outputs.<key> != 'false'`, so a classifier crash runs the jobs
instead of skipping them. The rules are checked out from the pull request base,
so a pull request cannot relax its own gate.

The policy is data: `.github/ci/scope-rules.toml` holds the path groups, the
suite catalog and the always-run list, with a `reason` on every exclusion.
`ci_scope_rules.py` loads it with the stdlib `tomllib` and validates the shape
(unknown or cyclic `@group` references, a `family` that disagrees with the suite
id, an empty reason) and that every glob's literal prefix still exists, so a
renamed crate cannot leave a rule that matches nothing. `make check-ci-scope`
runs that check locally; the `CI Scope Rules` job runs it on `ubuntu-slim`, the
same image the classifier uses. `.github/ci/scope-cases.toml` holds golden cases
that `test_ci_scope.py` replays through the real classifier.

The PostgreSQL and MySQL sqlness legs move into their own `sqlness-kvbackend`
job so they can be gated: a job-level `if:` cannot read the matrix, and gating
them inside `sqlness` would skip required checks and block the merge. They run
the same cases as the required Basic leg, so only the backend itself matters.

The verdict is visible where triage looks: one log line per suite, the exact
gate values, `::warning::` when a lookup fails and the gate widens, and a job
summary that lists each suite with the rule that decided it and the changed
files that matched.

Measured on 200 merged and open pull requests: 27.5% always run, 41% decided by
path rules, 31.5% skip every gated job; 995 of 2600 suite selections are
skipped, with no selection that a path rule makes. The classifier itself runs
in about ten seconds on a single-vCPU runner.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): let the scope job read pull requests

The `scope` job declared `permissions: contents: read`, and an explicit block
sets every unlisted permission to `none`. `GET /pulls/{n}` therefore answered
403, `build_plan` widened to a full run as designed, and every gate read an
absent output as "run": the gate could never skip anything.

- `pull-requests: read` on the `scope` job.
- `tests/runner/**` joins `@kv-backend`: the runner implements
  `--setup-pg`/`--setup-mysql`, which only the KV-backend legs pass, so a
  change there must not skip the only job that exercises it. The
  `UNMAPPED_CRATES` excuse is gone with it.
- The README no longer claims a queue failure blocks the merge: the gated jobs
  are not required checks.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): run every suite when the kind registry script changes

The distributed and chaos fuzz jobs start their cluster through
`.github/actions/setup-kind`, which runs `.github/scripts/kind-with-registry.sh`.
`.github/actions/**` is in `ALWAYS_RUN` but the script is not, so a change
confined to it selected no suite at all and every gate stayed `false`: the
jobs that exercise the changed cluster setup were skipped.

The script joins `ALWAYS_RUN`, consistent with the action that calls it, and a
golden case pins it.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): widen the gate for the fuzz corpus seeding script

`fuzz-test/action.yaml` runs `.github/scripts/prepare-fuzz-corpus.sh` before
the cache restore in every fuzz job, but the script was not in `ALWAYS_RUN`.
A change confined to it therefore selected no suite: `any_fuzz` stayed
`false` and every fuzz job that runs it was skipped.

The script joins `ALWAYS_RUN` next to the other scripts the fuzz jobs execute,
with a golden case for it.

Both misses so far (this one and `kind-with-registry.sh`) came through a
composite action, which the workflow never names. `WorkflowContractTests`
now resolves `uses: ./.github/actions/<name>` transitively for the gated jobs
and fails when one of the scripts they run is not in `ALWAYS_RUN`.

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 12:51:52 +00:00
jeremyhi 89a5df3141 fix: ignore stale leader grants after a region is downgraded (#9987)
* fix: ignore stale leader grants after a region is downgraded

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: compare stale leader grants with the longest lease seen

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: treat a lease too long to store as unknown

A lease longer than a u64 of nanoseconds saturated to a shorter one, so the
derived send time could be later than the real one. Now such a lease makes
the send time unknown, which counts as earlier than any gate.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address review comments on stale leader grants

Scope the early send time claim to after a lease has been seen, drop the
unused role parameter of demote, clarify a test comment and check that a
fresh grant still makes a reopened region writable.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: compare stale leader grants with the heartbeat send time

Pass the send time of the heartbeat behind a lease to set_region_role_with_lease,
and return whether the role was applied. Mito compares the send time with the
grant gate directly, so it no longer derives it from the longest lease seen.
The datanode finalizes leadership only when the grant was applied.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover stale grants through the datanode and repeated downgrades

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: keep leader grant rules in one region method

Apply a role grant in MitoRegion::apply_role_grant, which renews the lease
and decides whether the role changes; the engine only looks up the region.
Rename the gate to downgrade_started_at and demote to begin_downgrade, keep
the field private, and copy it on reopen through
inherit_downgrade_state_from.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: drop a needless borrow

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: keep lease concepts in the engine and role ordering in the region

ManifestContext, which owns the role, records when the role was last
demoted and promotes a region only for a promotion issued after that. The
time starts when the context is created, so a reopened region ignores
grants for heartbeats sent before the reopen, without copying state. The
engine translates leases: it renews the deadline and promotes with the
heartbeat send time. MitoRegion keeps no grant state.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep the data region of a metric region off a grant its metadata region ignores

The metric engine applies a grant to the metadata region first and stops
if it is ignored, so the data region, whose role the engine reports, never
becomes a leader that skips finalization. Also describe what promoting
returns.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: describe when a lease grant is ignored and flatten the finalize check

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: move lease types out of the region and the Leader deadline into ManifestContext

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: name the promotion fence after the downgrade it records

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-28 11:18:25 +00:00
jeremyhi 05536cd7f5 fix(meta): register failure detectors for regions switched before a rollback (#9989)
* fix(meta): register failure detectors for regions switched before a rollback

A region migration switches the routes of its tables one table at a
time. If it aborts after some tables were switched, those regions are
already served by the target, but the rollback deregistered the
target's failure detectors for every region and registered only the
source's. The target's next heartbeat registers them again, so if the
target stopped before sending one, nothing detected its failure and
the switched regions stayed unavailable until the metasrv leader
changed or the target came back.

The rollback now collects, from the routes it already reads, the
regions whose leader is the target and resets the target's failure
detectors for them at the end. It also skips a table whose route no
longer exists, so a table dropped meanwhile no longer fails the
rollback before the detectors are handled.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(meta): roll back a downgrading leader in the rollback detector test

The table that was not switched yet keeps a downgrading leader in a real
abort, so the test now builds it that way and checks that the rollback
clears it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): monitor regions switched before a rollback only on the target

The rollback registered the source's failure detectors for every region
of the migration, including the regions already switched to the target.
Those regions are served by the target after the rollback, so their
source detectors watched the wrong peer: when one fired, the failover
found the leader had changed and only removed it. The rollback now
registers the source's detectors only for the regions that were not
switched, and keeps resetting the target's detectors for the others.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(meta): rename reset_failure_detectors_for_candidate_regions to reset_failure_detectors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-28 09:39:45 +00:00
Weny Xu df33737e9d feat: add shared asynchronous cache primitives (#9790)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 08:45:10 +00:00
Ning Sun c818b6ac2a feat(postgres): influxdb line protocol ingest via COPY FROM STDIN (#9982)
* feat(postgres): influxdb line protocol ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT influxdb [, precision 'ns']) ingests
InfluxDB line protocol over the postgres wire protocol, reusing the
HTTP ingestion pipeline wholesale for correctness and performance
parity: the whole payload is buffered and completed once at CopyDone
through the frontend's InfluxdbLineProtocolHandler::exec — conversion,
permissions, line-protocol interceptors, timestamp alignment to table
time-index units, merge mode and inserts are the same path an HTTP
write of the same bytes takes, under the connection's regular postgres
query context.

Formats are an explicit, closed enum (CopyFormat::Csv |
CopyFormat::Influxdb(options)); each variant states its own rules in
the copy lifecycle's match arms — csv requires a target table and
parses incrementally, influxdb forbids one (each line names its own
measurement) and buffers one bounded request body. The COPY layer owns
only the buffering: record boundaries, including newlines inside quoted
field values and after escapes, belong to the line-protocol parser
alone — one COPY statement is one bounded ingestion request (64 MiB;
larger imports are multiple COPY statements).

- Wiring: PostgresServer::with_influxdb_handler attaches the shared
  line-protocol handler; unwired servers reject the format when a copy
  starts.
- Grammar: COPY FROM STDIN without a table is scanned (an unquoted
  FROM cannot be a table name); table-bound formats reject it with
  guidance, influxdb rejects a named table.
- Options: precision accepts the HTTP endpoint's values via the shared
  parse_time_precision (moved to servers::influxdb), defaulting to
  nanoseconds.

Review findings (fengjiachun) are fixed at the root by removing the
pre-splitter rather than repairing it: the retired naive line splitter
broke quoted multiline records at chunk/flush boundaries (found by
removing record splitting entirely), drained the buffer quadratically
(found by append-only buffering), and capped nothing (found by an
explicit payload limit).

e2e: multi-measurement ingest with chunked feeds; a 32,768-physical-line
flush-threshold regression with a multiline quoted record split across
CopyData chunks; precision 'ms' epoch verification; invalid UTF-8 and
trailing-record-without-newline handling; payload limit; table-named
rejection; malformed-line abort with connection recovery.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: typo

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-28 08:00:41 +00:00
discord9 4135e65a76 fix(query): keep requested sort when filtering on a lossy cast (#10026)
fix(query): pin DataFusion cast constraint propagation fix

Update the pinned GreptimeTeam/datafusion revision from e3221b8 to
cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger
regression test).

CastExpr::propagate_constraints used to cast a filter's output interval
straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton
output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so
FilterExec advertised k as constant, the equivalence classes dropped the
ordering required by ORDER BY k, and rows returned in scan order.

The sqlness case keeps every #9600 assertion and the explicit ORDER BY,
and adds ascending and descending checks for the partitioned table, the
control table and a table whose matching rows are stored in neither
ascending nor descending k order.

Cargo.lock only rewrites the 35 datafusion source lines.

Signed-off-by: discord9 <discord9@163.com>
2026-10-28 10:40:43 +00:00
discord9 bd77b599cc refactor(flow): remove tracked-write protocol (#9992)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-28 07:27:04 +00:00
Weny Xu cdc0d3a055 feat(procedure): support detached subprocedure recovery (#10009)
Select Discard or Detach when submitting children via Status::Suspended.
Persist the policy from the child's first normal or rollback checkpoint.
Recover detached children as independent roots with their original id and
context, using existing execution and cleanup without rebuilding the graph.

Keep fresh root admission serialized while allowing accepted procedures to
recover above the submission limit. Cover first persistence, repeated
recovery, rollback, cleanup, defaults, and concurrent admission.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 07:26:22 +00:00
Chenglong Zheng 724072bb33 feat: Add last_sequence for max allocated sequence and add tests for … (#9951)
* feat: Add last_sequence for max allocated sequence and add tests for the reproduced issue cases

Signed-off-by: hahaha <a310608318@gmail.com>

* test: Add a regression test for Kafka remote catchup

Signed-off-by: hahaha <a310608318@gmail.com>

---------

Signed-off-by: hahaha <a310608318@gmail.com>
2026-10-28 06:56:11 +00:00
Dragon Roar 5f0e22171e fix(log-store): skip records with unparsable keys instead of aborting the read (#9749)
* fix(log-store): skip records with unparsable keys instead of aborting the read

Signed-off-by: lczllx <2181719471@qq.com>

* fix(log-store): keep undecodable RecordMeta keys fatal

Signed-off-by: lczllx <2181719471@qq.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
2026-10-28 06:55:55 +00:00
B. Naut 3d2b73762a fix(catalog): propagate frontend process-list failures (#9999)
* fix(catalog): propagate frontend process-list failures

Return the existing invocation error when a selected frontend cannot list processes, so a successful response cannot omit that peer.

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>

* test(frontend): cover remote process-list errors and aggregation

Exercise real metadata discovery and gRPC listeners for a missing management service and successful catalog-scoped aggregation.

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>

---------

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
2026-10-28 03:47:17 +00:00
LFC b369d35d8b feat: enable JSON2 columns on non-append-only tables (#9985)
feat(mito2): support JSON2 in non-append-only tables

Signed-off-by: luofucong <luofc@foxmail.com>
2026-10-28 02:43:29 +00:00
Dhruv Vaishnav 3c490bd2dc ci: reuse fuzz corpora across integration runs (#9979)
ci: persist fuzz corpora between runs

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
2026-10-28 02:37:49 +00:00
Yvan Wang 180e2716fc fix(log-store): make raft-engine periodic WAL sync actually fsync (#9715)
* fix(log-store): make raft-engine periodic WAL sync actually fsync

On raft-engine 0.4.2, Engine::sync() writes an empty LogBatch and
write() returns before fsync (tikv/raft-engine#395). The periodic
sync task therefore never durable-flushed acknowledged WAL data.

Work around by writing a minimal non-empty marker with sync=true so
the fsync path is reached until a crates.io release includes
tikv/raft-engine#400.

Fixes #9712

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* perf(log-store): reuse LogBatch across periodic WAL syncs

Keep the periodic sync task's LogBatch across ticks instead of
allocating a fresh one (4 KiB encode buffer plus item vec) every
period. A successful Engine::write drains the batch but retains its
buffers; after a failed write or a panicked blocking task the batch is
discarded and recreated on the next tick, since it may be left
partially encoded.

Keep the marker put rather than deleting an empty key: the put
overwrites a single key, whereas every delete would become a recovery
tombstone that raft-engine keeps in memory while replaying append
files.

Also document that the marker value must not decode as a NamespaceImpl
(the marker key lies inside the range scanned by list_namespaces), add
a test covering batch reuse and namespace listing, and apply rustfmt.

Refs #9712

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

---------

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
2026-10-28 02:34:56 +00:00
LFCandgreptimedb-ci 9fc396c148 ci(query-regression): bump RUNNER_IMAGE_EPOCH to 8 (#10019)
ci(query-regression): bump RUNNER_IMAGE_EPOCH to 8 for image m-0xihwmu1dxusb9vuf13q

Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com>
v1.4.0-nightly-20261028
2026-10-28 01:04:44 +00:00
jeremyhi 62ebf71f90 fix(meta): exclude route followers from failover targets (#10001)
* fix(meta): exclude route followers from failover targets

Failover target selection did not exclude the region's route followers.
The selector could keep picking a follower, the submit rejected it as a
peer conflict, and the region never failed over.

Group the failed regions by their followers and exclude them per group.
A group without a candidate is skipped instead of failing every region of
the datanode.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: check failover continues after a group without candidates

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep failover going when follower routes can't be read

Fall back to selecting without follower exclusion when the routes can't
be read, log skipped groups with their followers at warn level, and test
a region without a route.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(meta): select failover targets per exclusion set in select_peers

generate_failover_tasks now only reads the route followers of the regions.
select_peers resolves the affinity and the unschedulable datanodes once,
then selects per group of regions that exclude the same datanodes.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(meta): drop a comment that repeats the code

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(meta): tighten failover target selection

Name the route followers after the parameter they fill, read the routes
with unwrap_or_else, and log a group without a candidate as an error, as
the caller did before.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): let the region-stat-aware selector balance the whole batch

The region-stat-aware selector now receives each region's excluded peers
along with the whole batch, so it can still balance load across the
regions while keeping a region off its own followers. A region without a
candidate is left out of its output instead of failing the others. The
selectors driven by SelectorOptions keep selecting per group of regions
that exclude the same datanodes.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: name the batch type of the stat-aware recorder

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(meta): treat a selector without candidates the same in both arms

Both selector arms now place what they can and treat NoEnoughAvailableNode
as nothing placed; the stat-aware selector's contract says so. Rename its
parameter to exclude_peer_ids_by_region, check for unplaced regions in
linear time, and describe what the code does in the comments.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: keep the stat-aware recorder's placements off excluded peers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-27 13:12:52 +00:00
Lei, HUANG 44df6e5ded fix(mito2): coordinate range cache invalidation and singleflight reads (#9935)
* fix(mito2): invalidate retired SST range results and coalesce loads

Coordinate decoded range loads by their full scan key, bound shared work, and prevent late fills after SST retirement. Invalidate range results for local purge and object-store GC while preserving old snapshot reads.

Fixes #9934

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito): preserve SST lifetime across index replacement

Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.

Fixes #9944

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): list-based SST purge and single FilePurger::remove_file

The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.

- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
  runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
  is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
  is_delete) method; is_delete means the file was retired AND this is the
  last live view.
- LocalFilePurger deletes the SST and all its index files via the new
  delete_file_and_all_indexes helper (list before delete, so a listing
  failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
  callers in the entire repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito): elect final SST view release atomically

Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.

Add a deterministic concurrent-drop regression covering retired and live files.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): restore Arc-owned SST cleanup and clarify reference release

Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing.

Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): track SST references by file lifetime

Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling.

Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers.

Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): release SST references directly on lifetime drop

Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge.

Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito): verify SST cleanup with real storage and isolate GC policy cases

Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings.

Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito): format SST lifetime changes

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): use crate-rooted range cache imports

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): account for range cache delivery lifetime

Keep completed range results charged while watch subscribers or replay streams retain them. Cache entries keep weak delivery references, allowing cache-only ownership to release the delivery reservation. Cache hits reuse or nonblockingly acquire the reservation, falling back when the budget is exhausted.

Resize input reservations to completed-result estimates and cover successive slow consumers, cache-only handoff, hit sharing, invalidation, admission rejection, and cancellation with regression tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-27 13:11:44 +00:00
discord9 984f80e277 fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit- or timezone-changing time casts

A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.

The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit-changing timestamp wrappers

to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.

The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regen windowed sort sqlness after wrapper guard

The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): simplify windowed sort cast regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 11:51:43 +00:00
dennis zhuang 5b4efd908c perf(mito2): index encoded bulk parts from memory during flush (#9913)
* perf(mito2): build encoded bulk part indexes from in-memory bytes during flush

Sync flush uploaded encoded bulk parts without indexes and rebuilt them after
the flush, which read every encoded SST back from the object store and needed a
second manifest commit. Build the indexes from the encoded bytes still in memory
before the flush commits, the same way the regular SST writer does, and drop the
post-flush rebuild for these files.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): keep flush going when indexing encoded bytes fails

Match ParquetWriter: a failure while building indexes from the encoded bytes
logs and leaves the SST unindexed instead of failing the flush after the SST
was uploaded.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): own encoded SSTs for cleanup before uploading them

Index creation now runs after an encoded SST is uploaded, so a truncate or drop
could cancel the flush after the upload but before UncommittedSsts tracked the
file, leaving the SST and its index behind. Track the file before the upload,
and fill the index metadata on the caller's SstInfo instead of a partial one.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(mito2): name encoded SST index build params explicitly

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): drop index metadata when the puffin file fails and size index decode batches

Indexer::do_finish kept the per-index metadata when finishing the puffin file
failed, so callers could publish indexes that were not written. Return an
empty output instead.

Indexing encoded bulk parts now decodes with the batch size estimated from the
largest row group, the same estimate bulk merges use, instead of a fixed 8192
rows.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(config): document the mito index build_mode option

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): decode encoded SSTs for indexing on the blocking pool

Indexing encoded bulk parts decoded Parquet batches inside the flush task on
the global async runtime. Decode each batch with spawn_blocking_global and keep
the indexer update on the async task, with one batch in flight.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 10:49:16 +00:00
Lei, HUANG 17ed988f49 fix(mito): preserve SST lifetime across index replacement (#9950)
* fix(mito): preserve SST lifetime across index replacement

Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.

Fixes #9944

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): list-based SST purge and single FilePurger::remove_file

The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.

- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
  runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
  is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
  is_delete) method; is_delete means the file was retired AND this is the
  last live view.
- LocalFilePurger deletes the SST and all its index files via the new
  delete_file_and_all_indexes helper (list before delete, so a listing
  failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
  callers in the entire repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito): elect final SST view release atomically

Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.

Add a deterministic concurrent-drop regression covering retired and live files.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): restore Arc-owned SST cleanup and clarify reference release

Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing.

Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): track SST references by file lifetime

Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling.

Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers.

Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): release SST references directly on lifetime drop

Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge.

Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito): verify SST cleanup with real storage and isolate GC policy cases

Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings.

Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito): format SST lifetime changes

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito): reclaim obsolete indexes by reader version

Track exact index-version references per metadata view and restore version-specific Puffin liveness in both GC paths. Keep file-level Parquet protection and shared SST lifetime cleanup, without expanding snapshots into unreferenced index generations.

Include old readers of imported files even when the source region is not open locally. Add real-storage GC coverage for reader release and lingering while retaining Parquet and the current index.

Validated workspace checks with default and all features, all-target all-feature Clippy with warnings denied, and all 1853 mito2 tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-27 10:20:45 +00:00
dennis zhuang 2e84833ac8 feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): reject partition column type changes and quote rule identifiers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(table): keep time index widening for partition columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(partition): replace PartitionFunction with IcebergHash

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(partition): describe iceberg_hash in the function module

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: quote partition rule identifiers in SHOW CREATE TABLE expectations

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(partition): check hash rules in information_schema and repartition target columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): coerce hash partition rules against the evaluation schema

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): keep quotes in partition column names when building filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 09:23:00 +00:00
Chenglong Zheng 2117f8b059 feat: Support procedure_state() using the last procedure ID (#9876)
* feat: Support procedure_state() using the last procedure ID

Signed-off-by: hahaha <a310608318@gmail.com>

* feat: Fix and add tests according to review comments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: the zero-argument signature in ADMIN swallows explicit arguments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: fix the CI failures caused by SQL version issues

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: apply the changes based on the review

Signed-off-by: hahaha <a310608318@gmail.com>

---------

Signed-off-by: hahaha <a310608318@gmail.com>
2026-10-27 07:27:55 +00:00
dennis zhuang e6b60644b3 fix(mito2): stop the series distributor from stalling on unread partitions (#9991)
When every partition channel is full, the distributor walked the senders
round-robin with a 100us send timeout each. Tokio timers fire on a 1ms
tick, so a consumer that leaves some partitions unpolled costs about one
millisecond per unpolled partition for every series batch.
SeriesScan::build_stream and scan_all_partitions drain partitions one
after another, and SortPreservingMergeExec above a multi-partition
SeriesScan only pulls the partition holding the current minimum key.
With 16 partitions and one batch per series after a merge, a 40-series
scan read partition by partition took ~480ms (measured on the native
PromQL two-input route).

Wait on all senders at once and send to the first one that frees a slot.
The send-timeout metric goes away with the timeout.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 03:42:27 +00:00
shuiyisong 5e9bb3c689 feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align native histogram ingestion with remote read validation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-27 03:09:13 +00:00
Weny Xu 2d00ae25c8 feat(meta): fence a draining datanode and plan the drain (#9960)
Introduce the metasrv side procedure that fences a datanode before it is
drained, and collect the regions placed on it:

- mark the datanode unschedulable through the node scheduling manager
- build the drain plan from the datanode table index and the authoritative
  table routes, splitting leader regions from follower regions
- keep the plan in the volatile context: it is derived from the table
  routes, so a restarted procedure rebuilds it instead of reading a
  persisted copy

The leader regions are not migrated yet: the migration state machine lands
separately. The procedure has no caller yet, so it stays inert.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-26 11:52:41 +00:00
Weny Xu 9f0e58e2af fix(meta): share peer discovery with plugin hooks (#9977)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-26 11:34:45 +00:00
discord9 0dab3aef21 fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601)
* fix(query): skip LastRow pushdown for IGNORE NULLS

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep IGNORE NULLS aggregates above remote scans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): record IGNORE NULLS results before and after flush

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): share IGNORE NULLS coverage across deployments

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh last-value plans after rebase

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 10:25:25 +00:00
discord9 df8f6848cc fix(query): avoid lossy casts in partition pruning (#9600)
* fix(query): keep lossy casts out of partition pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): align cast pruning expectations with conservative fallback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): record lossy cast partition pruning regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain safe pruning conjuncts beside unsupported casts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh cast pruning plans after rebase

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 09:53:32 +00:00
jeremyhi 91a59b82fb fix: acknowledge writes only within the region's leader lease (#9904)
* fix: acknowledge writes only within the region's leader lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: drop cached logical columns after a failed logical drop

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep the prepared region for the lease check after the WAL append

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: serialize a logical drop with fills of the logical column cache

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: check the drop lock and grant long leases deterministically

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover a bulk insert that loses its lease during the WAL append

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-26 09:49:13 +00:00
Weny Xu 8170d0458a fix(ci): scope Aliyun janitor sweeps to the owning repository (#9966)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-26 09:36:49 +00:00
jeremyhi 1e97a0e7c3 fix(log-store): refuse an object store that ignores conditional create (#9909)
* fix(log-store): refuse an object store that ignores conditional create

The object store WAL checks only the write_with_if_not_exists capability
before it opens, and OpenDAL declares that capability per service. A store
behind the S3 service can accept If-None-Match and overwrite the object
anyway, so the WAL could overwrite its own objects and lose acknowledged
writes without an error.

Before recovery, the store now creates a probe key, unique to the open,
twice under <node prefix>/probes/, outside the objects it lists, and
deletes it. The second create must be refused with a condition failure,
otherwise the open fails with an error naming the object store. The
probe costs two writes and one delete per open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): drop a comment that restates the keys helper

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): accept AlreadyExists from the second probe create

A backend may report an existing conditional-create target as AlreadyExists. The first create stored the probe key, so that error can only refer to it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-26 09:08:13 +00:00
jeremyhi 6f5c064f54 fix(meta): fence datanode processes replaced by a newer process (#9878)
* fix(meta): fence datanode processes replaced by a newer process

Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.

The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.

Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache load and pusher updates coherent

A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.

A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.

The epoch check moves into the heartbeat service.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache coherent with writes while it is disabled

A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(meta): pin the epoch 0 exemption and pusher order, and update docs

The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): register a datanode pusher only after its first request passes

A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.

Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): check a datanode epoch again before replying

A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): handle a datanode's requests one at a time

A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.

The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-26 09:02:40 +00:00
Weny Xu c96cdd0026 perf: stream and prune SST manifest scans (#9931)
* refactor: add reusable metadata pruning and projected SST batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): stream and prune manifest SST metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(datanode): stream inspection queries through reusable providers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(catalog): push manifest filters through inspection queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: align manifest column constants with module conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: reduce manifest inspection allocations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: narrow manifest inspection API visibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: retain residual filters for manifest inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-26 08:54:02 +00:00
shuiyisong 513b9cee58 feat(catalog): add public-global information schema scope (#9957)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-26 07:23:30 +00:00
B. Naut e87573e286 test(sqlness): cover query options in merge-scan regressions (#9956)
* test(sqlness): cover query options on persistent SQL sessions

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>

* test(query): cover merge-scan joins with query option controls

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>

---------

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
2026-10-26 07:22:04 +00:00