I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Target: `narthex`, which now contains merged #9598 and #9599. Their
landed changes have been synchronized without rewriting feature history,
so this PR remains the histogram bridge increment.
## What's changed and what's your intention?
Connect the existing native-histogram representation to the shared
columnar sample containers. The incremental diff is limited to
`common-query/src/native_histogram.rs`.
- Re-export the shared `CounterResetHint` at its existing query-module
path, preserving numeric conversions and unknown values. Add an
owned-histogram adapter to `HistogramBatchBuilder`.
- Add a quantile entrypoint for borrowed `HistogramView` values. It
shares the existing quantile calculations and preserves early-return
precedence, custom/exponential bucket behavior and NaN information.
Borrowed-view validation does not tighten the legacy owned histogram's
malformed-layout behavior.
- Normalize persisted Arrow histogram structs directly into an owning
`Samples` value, without reconstructing a per-row owned
`NativeHistogram`. Preserve row positions, parent nulls, complete
payload bits, both bucket sides, reset/start metadata, and sliced-parent
offsets. Float payloads take precedence when present; stored integer
bucket counts are already absolute and are converted directly, with a
missing zero count defaulting to zero.
- Reject missing required scalar values, null list items and invalid
span layouts. The existing `read_histogram` decoder and persisted struct
schema remain unchanged.
Normalization borrows source values while reading and then copies them
into the canonical owning batch; integer conversion uses temporary
buffers. The borrowed quantile path still derives bucket/index buffers.
This is not an allocation-free or measured-performance claim. Scanner,
frame/Flight, execution-operator and public HTTP/TQL/SQL integration are
not included, and no persisted or network format changes.
### Validation
Ten new focused tests cover borrowed quantiles and NaN information,
malformed/overflow spans and early-return precedence, legacy
owned-layout compatibility, complete payload bit preservation, nonzero
batch/parent slices after original owners drop, empty/all-null input,
integer counts and float-family selection, and
required-field/list-item/span errors.
- Rust 1.98.1: `cargo check --locked -p datatypes -p common-query -p
promql --all-targets` — passed.
- Rust 1.98.1: `cargo nextest run --locked -p datatypes -p common-query
-p promql --retries 0 --no-fail-fast` — 707 passed, 0 skipped, both
before and after the final test-noise cleanup. The actual listing
includes all new tests and the parent buffer-capacity regression.
- The three cleanup-affected payload/lifecycle/empty tests were also run
explicitly — 3 passed; the complete unfiltered suite then passed all
707.
- Rust 1.99.0: `rustfmt --edition 2024 --config skip_children=true
--check src/common/query/src/native_histogram.rs` — passed.
- Scoped `hawkeye format` and `hawkeye check` on a byte-identical source
copy, plus `git diff --check` — passed. A missing-header negative
control confirmed that the scoped license check inspects the file.
- Rust 1.99.0: `cargo clippy --locked --no-deps -p common-query -p
promql --all-targets -- -D warnings` — blocked by
`async_trait`-generated `clippy::double_must_use` at unchanged
`common-query/src/logical_plan.rs:129`. The exact same command on the
exact parent commit `7e5acfb9a9c` reproduces that failure. Pinned 1.98.1
Clippy is unavailable in the local read-only toolchain. No lint or CI
policy was weakened; this is not a passing Clippy or full-workspace
validation claim.
### Diff-scoped cleanup
No whole tests were deleted. Removed only an empty-result assertion that
accepted every dense-storage representation after exact length was
already verified; the same named test retains empty/two-null conversion,
length and missing-position checks. Simplified payload assertions to
explicit histogram bindings and a shared complete bitwise assertion
helper, and renamed the payload test to describe its explicit
expected-value oracle. Public API rustdoc, ownership rationale and
compatibility/precedence contracts remain. No scanner, protocol or
unrelated cleanup is bundled.
### Verified review follow-up
- Current tested/pushed head:
`1164877b5a5982d0cea2860e01e1c28714299e9e`, including the merged #9599
parent. The only query-file conflict resolution relocates the same enum
re-export and retains the exact landed reset-hint identity regression;
no bridge/helper code was dropped. Independently reviewed after
resolution.
- Owned `all_buckets` and borrowed-view quantile execution now share one
private `assemble_bucket_sides`; no duplicate negative reversal or
zero-bucket rule remains. Fallible side builders, validation and
custom-bound handling are unchanged.
- A focused expected-sequence regression independently checks bounds,
counts and boundary rules, including insertion for nonzero count and
omission for both signed zeros.
- On the final synchronized head, Rust 1.98.1 all-target check and
complete `common-query / promql / datatypes` nextest: **712/712, 0
skipped** (76 + 301 + 335). Strict datatypes-only Rust 1.99 Clippy
passes. The earlier strict three-package run reports the existing
common-query macro diagnostic; the final known-category comparison
passes, not strict query lint. Normal formatting/TOML/license hooks
pass.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Stack base: `narthex` (includes merged #9598). This PR targets
`feat/narthex-series-foundation`; review and merge the foundation first,
then retarget this increment to `narthex`.
## What's changed and what's your intention?
Add the typed sample and field containers that complete the next
in-memory series increment. The incremental diff is limited to the
`datatypes` samples module, its focused tests, and its module export.
- `Samples` retains typed Arrow float, histogram, mixed, or chunked
storage. Null means missing; ordinary NaNs and float payload bits remain
values. Histogram indices retain their immutable owning batch, and mixed
storage rejects overlapping float/histogram validity.
- Clones and checked nonzero slices share numeric buffers, validity
bitmaps and histogram tables. Chunk construction flattens descriptors
without flattening sample buffers; sequential iteration reports exact
remaining length.
- Explicit `into_dense()` concatenates only when requested. When every
histogram-bearing chunk references the same `Arc<HistogramBatch>`, it
reuses that complete table and joins only values and indices, including
mixed and all-null histogram chunks. With different table owners, it
preserves and concatenates the complete participating tables and
checks/remaps their indices; it does not perform per-call owner or
content deduplication in that fallback. Numeric builders use the checked
local result length rather than Arrow's 1,024-slot default reservation.
Pure-histogram densification does not create a float builder; a float
payload is allocated only when a Float or Mixed representation requires
it, including an all-null Mixed float component.
- `SeriesFields` stores one field inline or multiple fields in a boxed
slice, rejecting empty or unequal-length fields. `SeriesSchema` shares
field descriptions and an optional evaluation grid.
- `Series::try_new` validates field count, local time/field alignment
and timestamp order. A grid axis requires a value-equal schema grid;
explicit and explicit-chunk axes require no schema grid. A short series
can hold 20 values on `[800,820)` of a 2,017-point grid without padding
its fields to the full grid.
This only adds in-memory types. It does not switch HTTP/TQL/SQL queries
to a new execution path or change persisted data or network formats.
Scanning, frame/Flight integration, execution operators and
producer-side synchronized value trimming are not included. Supporting a
short range here does not prove that a future computation producer
avoids allocating full-grid values; that producer needs its own
acceptance tests.
### Focused validation
Current verified head: `cb0a75f2c0a7` (the test cleanups plus the
histogram-only temporary-allocation fix). The allocation follow-up
changes only builder creation, append and finish conditions; table
reuse/remapping and sample semantics remain unchanged.
Thirteen samples tests cover valid/missing/NaN/stale values, malformed
indices and mixed validity, nested/empty chunks, nonzero shared slices,
complete histogram payload and whole-table remapping, single/multiple
fields, grid policy, invalid explicit ordering, and actual Arrow
allocation retention/release. The shared-table regressions cover
histogram-only, mixed, and all-null indices; different-owner cases check
every logical value and missing position without prescribing table
deduplication or index layout.
- Rust 1.98.1: `cargo check --locked -p datatypes --all-targets` —
passed.
- Rust 1.98.1: `cargo nextest run --locked -p datatypes --retries 0
--no-fail-fast` — 335 passed, 0 skipped, both before and after this
review cleanup. This includes the parent capacity regression and all
thirteen samples tests; the renamed candidate tests were checked in the
actual test listing. The five affected or explicitly retained samples
tests also passed in a targeted run.
- The added numeric-buffer assertion in
`densification_preserves_values_nulls_nan_and_remaps_multitable_histograms`
fails on the preceding implementation at 12,544 retained bytes and
passes with the fix. A temporary measurement of the fixed fixture
reported 212 bytes across its float/index buffers, excluding
histogram-table storage. These are retained Arrow buffer capacities, not
RSS.
- A separate local allocator probe around the actual `into_dense()` call
used 1,048,576 histogram positions, 64 segments, one shared table and
alternating values/missing positions. Requested allocation bytes fell
from **12,845,472 to 4,325,584** (10 allocation requests to 5); the one
8,388,608-byte temporary allocation and its in-call free disappeared.
Histogram representation, shared table identity and all positions were
unchanged. These are requested allocations during this fixture, not RSS,
final retained bytes or a throughput benchmark. The probe is not added
to the production or unit-test allocator.
- Rust 1.99.0: `cargo fmt --all -- --check` — passed.
- Rust 1.99.0: `cargo clippy --locked --no-deps -p datatypes
--all-targets -- -D warnings` — passed. The pinned 1.98.1 Clippy
component is unavailable in the local read-only toolchain;
dependency-inclusive 1.99 linting is still blocked by pre-existing
`double_must_use` diagnostics in `common-base/src/range_read.rs`. No
lint level or CI rule was weakened.
- `hawkeye format`, `hawkeye check`, and `git diff --check` — passed.
### Diff-scoped cleanup
The earlier cleanup removed an unused crate-private chunk accessor and a
single-use generic parameter from the private field container. Public
API rustdoc, rationale, SAFETY comments, and all whole tests remain.
This review makes only these additional local changes:
| Changed test | Removed constraint or duplicate | Retained behavior and
actual evidence |
| --- | --- | --- |
| `densification_preserves_distinct_shared_table_owner_values` (renamed)
| Exact output-table cardinality, prescribed remapped indices, and
forced distinct output allocation | Genuinely different histogram
tables, including repeated owners and missing positions; every output
value is checked by the existing complete histogram/float-bit comparator
and every non-null index is in bounds. The renamed test passed. |
|
`short_multifield_series_uses_its_grid_range_and_checks_explicit_chunk_order`
| Only the final duplicate single-axis `[2, 1]` rejection block |
`fields_clone_buffers_and_series_validate_schema_alignment_and_time_order`
retains that rejection with valid matching fields and an ordered
positive control. Both tests passed. Short-grid, multifield and
cross-chunk checks remain. |
The two same-`Arc` sharing regressions retain their pointer-equality
guarantees and passed. The mixed regression additionally checks that an
all-null float component does not erase the Mixed representation. Owner
retention/release, malformed-input, grid-boundary and histogram
regressions are not removed. The earlier shared-table memory fix was
also checked red/green: the pre-fix implementation retained 64 or 2
table rows where the shared-table fixtures require one; the fixed
implementation passes without changing fallback semantics.
### Verified review follow-up
- Retargeted to `narthex` after #9598 merged; synchronized its merge
commit without rewriting feature history. Current tested/pushed head:
`ad1d1ee0fb748dca664726a01508d3fdf499ffe8`.
- Consolidate ordering and checked length in crate-private
`TimeAxis::try_len`, used by both constructors and `Series::try_new`;
remove the duplicated Series helper. Preserve direct-public-payload
validation, chunk sharing, error ordering, grids and slicing.
- Strengthen the existing clone-owner regression with nonempty
value-buffer pointer identity, actual values, and release only after
both Series and clone are dropped.
- Rust 1.98.1 all-target checking for `datatypes / common-query` passes;
complete nextest **400/400, 0 skipped** (335 + 65). Strict
datatypes-only Rust 1.99 Clippy and ordinary fmt/TOML/license
commit/push hooks pass. No expanded two-package strict-lint claim.
- The previous hosted MySQL SQLness job's durable failure annotation
reports that the runner lost communication; its raw log is unavailable
(`BlobNotFound`). This is not presented as a successful SQLness run, and
no SQL semantics or CI settings were changed to mask it.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
First Narthex foundation PR, targeting the `narthex` development branch.
## What's changed and what's your intention?
Introduce the first shared in-memory building blocks for native series
execution, extracted from the earlier local implementation and updated
to use shared evaluation grids and per-series ranges.
- Add `Label` and `MetricName`: shared strings and sorted labels,
content-based identity, separate metric-name handling, and copy-on-write
label changes.
- Add `GridSpec`, `EvalGrid`, and `GridRange`: validate numeric and
range bounds, materialize a grid once, and let short results reference
their absolute range without copying timestamps.
- Add `TimeAxis` support for grid ranges, explicit timestamps, and
explicit chunks, including checked zero-copy slicing.
- Add Arrow-backed `HistogramBatch`, borrowed histogram/bucket views,
and a builder with structural validation. Preserve all histogram
components, float bits, and unrecognized reset-hint codes.
- Cover label isolation, numeric extremes, malformed inputs,
nonzero/nested slices, and real Arrow-allocation retention and release.
- Start histogram primitive builders at zero capacity rather than
Arrow's 1,024-slot default. A one-row retained-buffer regression rejects
the old large reservation; no finish-time shrinking or new capacity API
is added.
This PR adds types to `datatypes` and re-exports the shared reset-hint
enum from `common-query` instead of defining it twice. It does not
enable a new query path or change existing HTTP/TQL/SQL results, storage
formats, or network protocols. `Samples`, `SeriesFields`, and
`SeriesSchema` follow in the next structural increment; producer-side
synchronized value trimming, scanning, Flight integration, and execution
operators are not included.
### Verification at `bdc12034e559cbabc227ce3e53405928f568a22a`
- Rust 1.98.1: `cargo check --locked -p datatypes -p common-query
--all-targets` — passed.
- Rust 1.98.1: `cargo nextest run --locked -p datatypes -p common-query
--retries 0 --no-fail-fast` — **387 passed, 0 skipped** (`datatypes`
322; `common-query` 65).
- Rust 1.98.1: `cargo test --locked -p datatypes -p common-query --doc`
— command passed; both crates currently have zero doctests, so this is
not additional behavior coverage.
- Rust 1.99.0: scoped Rust formatting, workspace formatting through
ordinary commit/push hooks, TOML checks, Hawkeye format/check, and `git
diff --check` — passed.
- Rust 1.99.0: `cargo clippy --locked --no-deps -p datatypes -p
common-query --all-targets -- -D warnings` — **not passed**: one
existing `async_trait` `double_must_use` diagnostic in
`common-query/src/logical_plan.rs` is emitted for both lib and lib-test
targets. The identical command on immutable parent `78e94ced920` reports
the same diagnostics. A diagnostic-only comparison allowing that
category passes; no lint setting changed. The earlier datatypes-only
strict result below does not certify the expanded two-package command.
### Review follow-up
- Keep public `TimeAxis` payloads and clarify the validation boundary.
The next increment #9599 already validates directly constructed explicit
and chunked axes inside `Series::try_new`, including strict ordering and
checked total length. Its direct-construction and cross-chunk-order
regressions were run again at published head `cb0a75f2c0a7`: **2
passed**.
- Remove `TimeAxis`'s unused representation-derived equality rather than
add a timestamp-comparison implementation.
- Remove the discarded chunk-total accumulator; ordering validation,
empty-chunk removal and zero-copy storage/slicing remain unchanged. The
Series boundary keeps its checked aggregate-length validation.
- Use the datatypes-owned `CounterResetHint` through a public
query-module re-export. Existing query imports and known/unrecognized
`i32` conversions remain unchanged. A new focused test checks direct
cross-crate type identity and all known codes, unrecognized codes, and
both integer extremes; the existing Arrow unknown-hint round trip
remains covered.
### Diff-scoped cleanup
- Use ordinary `mod tests;` discovery; remove the three redundant
`#[path]` attributes without moving or dropping tests.
- Remove one duplicate ordinary-label miss assertion from
`src/datatypes/src/series/tests.rs`. No whole tests are deleted; all 18
original new tests remain, with one additional retained-capacity
regression. Keep invalid-input, boundary and allocation-lifetime
coverage and the public API, invariant and SAFETY comments.
| Removed assertion | Protected behavior | Surviving coverage | Actual
verification |
| --- | --- | --- | --- |
| `one.get("__name_missing") == None` in
`metric_identity_uses_contents_and_distinguishes_absent_from_empty` |
Absent ordinary-label lookup | `one.get("missing") == None` in the same
test; empty-label and exact `__name__` checks also remain | Named
nextest run: 1 passed; full suite before and after: 321 passed, 0
skipped |
At the prior checkpoints, the datatypes all-target check, formatting,
license and datatypes-only strict primary-crate Clippy checks passed
after cleanup and after the capacity fix (322 passed, 0 skipped). A
redundant comment restating the new capacity test was also removed
before the final checks. No skip, lint level, CI policy or production
behavior was changed.
### Small histogram retained-buffer regression
The exact new test failed against the pre-fix source: a one-row fixture
retained 90,352 bytes of Arrow buffers. With the final production fix,
the same fixture retained 528 bytes. These are
`get_buffer_memory_size()` capacity measurements, including offsets and
validity where present, not RSS measurements. The committed test
requires less than 4,096 bytes. Its failure on the original builder and
success on the candidate were both run locally.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
Adapt the existing series reader through an optional scanner entrypoint with
owned snapshot-bound label metadata and complete encoded series keys. Preserve
selected field order and Arrow segment views without eager label decoding.
Keep ordinary storage pushdown and MVCC semantics. Inexact field predicates
can leave false-positive candidates; callers must retain residual columns and
evaluate the ordinary residual filter before final projection. This adds no
public native query consumer or replacement filter evaluator.
Exercise both internal batch forms, stream finality/time/error/drop boundaries,
real Mito candidate parity, and logical metric routing with immutable binding.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
The flow and nested-MergeScan ingress tests hardcoded "follower" among
the read-preference values that must be rejected, coupling them to the
current leader-only ReadPreference enum. Assert only genuinely invalid
values there, matching the wire-values test which already limits itself
to ["", "unknown"]. This keeps the ingress tests valid for builds that
extend ReadPreference with more variants.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(mito): decouple compaction unit execution and publication
Execute shared-input dependency groups as independent local compaction units. Publish and acknowledge each unit separately, retain successful sibling results, and release each unit's input leases after worker apply.
Preserve picker priority, panic notifications, snapshot-free execution context, and indexed input leases. Include regression coverage for independent visibility, purge, cancellation, partial failure, and restart recovery.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): use crate visibility for compaction unit items
Replace the newly introduced pub(super) declarations with pub(crate), and make CompactingFiles::try_reserve private since it is only used within its module, per docs/style-guide.md.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): simplify compaction unit state and test fixtures
Keep local input leases in the task and derive running units from tracked and pending work. Replace the single-implementation task trait with an inherent async method and align exposed state type visibility.
Use snapshot-free region fixtures and named test scenarios, and cover independent input purging across three units with bounded parallelism.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): align compaction task and test module names
Rename CompactionTaskImpl to CompactionTask now that the CompactionTask
trait has been removed and the Impl suffix no longer marks a trait
implementation. Rename the state test module from lease_tests to tests
to follow the test module naming convention.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): align compaction task debug label
The Debug impl reported "TwcsCompactionTask" while the type is now
CompactionTask. Report the type name for consistency.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): label compaction elapsed metric per published unit
Record greptime_mito_compaction_total_elapsed once per published unit and
label it by type: a local dependency unit or a remote job. Update the
metric docs and group the Grafana compaction OPS/latency panels by type.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito): clarify compacting file lease sharing
`CompactingFiles::for_inputs` clones the shared per-file leases instead of
transferring them. Document that the original collection keeps its leases,
so an input is released only after every owner drops its lease.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito): document local compaction unit map key
Document that `LocalCompaction::units` is keyed by each unit's execution
plan ID, which matches `CompactionExecution::plan_id()` and the `plan_id`
carried by unit notifications.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): rename submit_prepared_compaction
The function submits remote jobs but only prepares local unit tasks; the
caller dispatches them after installing the returned phase. Rename it to
`prepare_or_submit_compaction` and document that ordering.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(hdfs): clean up temporary file when an atomic write fails
Signed-off-by: fengyue-xve <1759519137@qq.com>
* style: apply rustfmt to hdfs layer test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(hdfs): report earlier write failures and log cleanup failures
- Later `write`/`close` calls on a writer whose failed write was
cleaned up now mention the original failure instead of the bare
"unavailable after close or abort" message.
- `abort_and_delete` logs abort and deletion failures with the
temporary path instead of discarding them; the original write error
is still returned to the caller.
- The temporary-file cleanup test now lists the directory holding the
temporary file, so it fails if the cleanup is reverted.
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
* fix: align fulltext bloom tokens with matches_term and use them for LIKE
The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.
Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop unused lazy_static from index and avoid typo-flagged test tokens
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name probe extraction with verbs and document CharClass
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: rebuild legacy fulltext bloom indexes and probe lower(col)
The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.
Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover legacy bloom index rewrite by compaction in the compat case
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: log outdated fulltext bloom indexes in the consistency check
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: explain the unrecorded fulltext bloom version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): centralize PromQL function registration in promql::functions
Move the PromQL scalar UDF and UDAF registrations out of the query plan
decoder into a single `promql::functions::register_promql_functions`
entry point, so plan decoding no longer has to enumerate every PromQL
function implementation. The registered function set, aliases (e.g.
`prom_holt_winters`) and registration order are unchanged.
Add a registration test pinning the full set of `prom_*` scalar
functions and the `prom_holt_winters` alias, plus a decode regression
test covering the centralized registration path.
Fixes#9493
Signed-off-by: fengyue-xve <1759519137@qq.com>
* refactor(promql): register Ieee UDFs centrally and address review
Rebase onto main picked up #9479, which registered the Ieee comparison
and sqrt UDFs from the decoder directly. Register them from
`register_promql_functions` instead so the single entry point stays
complete, and add the seven `prom_eq`..`prom_sqrt` names to the
registration test.
Also per review: drop the doc paragraph that explains why the function
was moved, drop the redundant `test_serializer_decode_centralized_promql_registration`
test and the registration comment in the decoder, move `use std::sync::Arc`
up with the imports, and simplify the registration test assertions.
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
* fix(log-store): keep the enqueued backlog on permanent object store errors
On object storage the WAL's object store sits behind OpenDAL's retry
layer, which marks every error it returns as persistent. The enqueued
mode classified create failures by that status, so a 403 or a missing
bucket was repeated every 100 ms on every create in flight, logged
nothing, and made appends hang once the sealed batches were full.
Object store errors are now classified by kind: denied access, an
invalid configuration (how OpenDAL reports a missing S3 bucket) and a
write to a missing location are non-retriable. In the enqueued mode such
an error makes the store failing instead of poisoning it: the
acknowledged backlog is kept under its sequences and bytes, its lowest
pending batch is attempted alone with a delay that doubles from 100 ms
to 30 s, appends are rejected at once with a retryable error that
carries the last object store error, and the first create that succeeds
ends the state. Stop uploads the backlog at once and reports a failure
as before. The state is logged on entry, at most once a minute while it
lasts and on exit, and exported as the `failing` gauge and the
`rejected_appends_total` counter. Conflicts, encoding and catalog errors
still poison the store; the durable mode is unchanged.
An append held back because the sealed batches are full is now counted
and timed by the stall metrics in both modes.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe how the enqueued object store WAL handles refused creates
Describe the enqueued failure contract as the store now implements it:
non-retriable object store errors are told apart by kind because
OpenDAL's retry layer under the store marks every error persistent, a
failing store keeps its backlog and attempts it with backoff while it
rejects appends, and stop uploads the backlog at once. Add the failure
matrix row, the failing gauge and the rejected appends counter, the
sealed-batch admission limit and the stall metrics that now count it,
and the fault matrix case counts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): time the failing attempts where they start and fix two statements
The test of the attempts of a failing store now records when each
refused write starts and checks the gaps between them, instead of
sampling a counter after a sleep of the test task. The delay comment and
the RFC describe 30 seconds as the longest delay between two attempts,
not as a bound on when appends resume, and the RFC states that the next
append is held once seven sealed batches wait.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): drop the wait_until helper that main now defines
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): name errors that need an operator and trim the RFC contract
Rename is_non_retriable to requires_intervention: such a create is still
repeated, only slowly, until someone fixes the credentials, configuration
or bucket. Name the backoff test after what it asserts, cut the test of
the lowest pending batch to two failed rounds since the cadence is
covered by the backoff test, and shorten the RFC failure contract to
the contract and its rationale, leaving error mappings, request counts
and logging cadence to the code comments.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): tie the slow repeat to errors requiring intervention
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
The manage-region-follower chain submitted its procedures with an empty
executor context: `ProcedureServiceHandler::manage_region_follower`
accepted no query context, the operator built an
`ExecutorContext::default()`, and `MetaClient` dropped the context before
delegating to `RegionFollowerClient`. The submitting user (actor) and the
procedure event metadata therefore never reached the submitted
procedures.
- `ProcedureServiceHandler::manage_region_follower` now takes a
`QueryContextRef`, consistent with `migrate_region`, `gc_regions` and
`gc_table`.
- `ProcedureServiceOperator` derives the `ExecutorContext` (actor and
`TriggerReason::Manual` event input) from the query context.
- The `RegionFollowerClient` trait methods now accept `&ExecutorContext`
and `MetaClient` forwards it, so client implementations can transport
the actor and event context on the wire.
- `procedure_event_context`/`procedure_actor` are now public so client
implementations can reuse the trusted protocol derivation.
discord9
discord9
Ning Sun
Lei, HUANG
fengyue-xve
Weny Xu
dennis zhuang
Yingwen
sun
jeremyhi
NightG1owZandNightG1owZ