`native/wire.rs` encoded `NativeTree` by hand: fixed node tags, fixed field
layouts, both directions written out, and a `check_shape` that narrowed the
transport to `Input -> Instant|Window -> Map(Math*)`. The serialization layer
therefore decided which plans could exist. `abs(ceil(x))` compiled but could
not be transported, and every new node kind cost a second codec path plus a
new whitelist entry. Widening the whitelist to nested maps maintained that
interpreter rather than removing it.
The file is deleted. The plan types derive serde and the codec is postcard
1.1.3 -- bincode is unusable here because RUSTSEC-2025-0141 marks the whole
crate unmaintained with no patched version. postcard is maintained, has a
documented stable wire format since 1.0, and writes an f64 as its bit pattern
in eight little-endian bytes, so NaN payloads, signed zero, infinities and
subnormals survive exactly.
What remains hand-written is framing only: magic `NPRQ`, a version, a payload
length. The version is rejected before the payload is deserialized, so an
unknown version is a clean error instead of a silent misparse, and oversize,
trailing bytes and truncation are separate errors. Decoding ends at
`NativeTree::try_new`, so a tree the compiler would not build cannot be
decoded.
Three types keep written impls because they hold invariants. `EvalGrid`
serializes its `GridSpec` and rebuilds through `try_deferred` with an empty
cache -- the grid is never serialized and never materialized on decode.
`SeriesLabels` rebuilds through `try_new`, keeping the sorting and the
duplicate / `__name__`-as-a-tag rejection. `LabelRewrite` travels as its
parameters with the regex recompiled, and fails to decode if the pattern no
longer compiles; compiled internals never travel.
`FusedSeries` now carries its output schema and accepts `PlainRows` and
`partial_fold`, which it used to refuse. A carried schema is only honest if it
is checked, so decode validates every column role, the exact time type and
nullability, missing or extra value columns, and the scalar-root tag rule; a
`SeriesFrame` schema must equal the frame layout the tree derives, and a
partial export must match `state_schema(binding.tags)` including tag order.
Identity stops being display text. `FusedSeries` compared `tree.to_string()`
with a field patch, `AggregateFinalize` hashed the rendering, and
`LabelRewrite` compared its own output; all three are structural now, with
floats compared by `to_bits()` so that signed zero and NaN payloads stay
reflexive under `Eq`. `same_binding` compared `Arc::ptr_eq`, which put an
allocation address into a transportable plan; it compares the ordered fields
and the optional grid spec now, distinguishing raw from evaluated.
Tests assert the properties rather than the byte layout: a non-canonical NaN
payload round-trips by bits, an unknown version is rejected before decode even
when the payload is garbage, non-prefix trees (binary, set, aggregate, scalar,
evaluated input) round-trip, thirteen malformed schemas are each rejected with
their own message, and a multi-chunk chain over a real region is checked
against its exact sample count and contiguous timestamps.
The multi-partition integration test is split in two and labelled honestly. It
used to assert a failure, on the theory that coalescing interleaved multi-chunk
chains; its fixture's grid has one position, so no chain has more than one
chunk and that assertion was already failing at the base commit. One test now
covers the single-position shape and says it does not exercise interleaving;
the other covers a 8192-position grid that does.
Deserialization is bounded by depth, not only by bytes. The derived recursion
walks boxed nodes, and a payload under the 1 MiB limit can nest far deeper than
the stack allows, which aborts the process instead of returning an error. A
thread-local budget guards `NativeNode`'s deserializer, so a hostile payload is
rejected before a complete tree exists. `NativeTree::Deserialize` also routes
through `try_new`, so no path can materialize a tree the constructor would
refuse. Measured under postcard with no `RUST_MIN_STACK` override: 128 nodes
decode, 129 return a decode error. Postcard collapses custom serde errors to
its unit variant, so that rejection does not name the nesting limit; the other
envelope failures (version, size, trailing bytes, truncation) each keep their
own message.
Signed-off-by: discord9 <discord9@163.com>
A multi-input FusedSeriesExec drained every partition of its first input
before it touched the second, and RawSeriesSourceExec drained its scan
partitions one after another. The series distributor feeding those
partitions then waited on the one partition being read, the two inputs'
scans never overlapped, and all decoding ran on the one consuming task.
Read every partition through SeriesGather instead. It polls all of them
and hands a chunk on unless another partition of the same input has a
series open; such a chunk is held until that series closes, so each
input still sees contiguous series. A partition holding a closed series
is not polled again until it is handed over, which bounds what is held
to one series per partition, and that is charged to a memory
reservation. Inputs do not constrain each other.
When there is more than one partition, each is driven on its own query
task with a two-chunk channel, so the scan conversion, frame codec and
label decode of different partitions run in parallel. Dropping the
stream aborts those tasks, and a task that panics ends its partition in
an error rather than a silent end.
On the o11ybench vmlite/vmlong data (standalone, native ON, median of 10
alternating runs; results byte-equal to the sequential reader):
sequential* this change
vmbench-lite-05 4.4ms 3.0ms
vmbench-lite-06 7.3ms 4.2ms
vmbench-lite-08 5.0ms 3.3ms
vmbench-long-range-03 54.4ms 39.6ms
cpu same-table pair 27.6ms 15.3ms
long fs same-table pair 104.2ms 83.6ms
* with the series distributor fix; without it the sequential reader
stalled for up to 12.8s on these queries, and this change does not
need that fix.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Nothing called `native_compile::compile` or built a `FusedSeries`, so the native
tree and its node were library code with tests and no query behind them. This
adds the route, behind a per-query flag.
The statement is offered to the compiler at the root, where the expression the
reader wrote still exists. A shape the compiler refuses - `NotImplemented` - is
this route declining the query, and planning continues exactly as it would with
the flag off. Anything else is a real error.
The physical half is an arm in `PromExtensionPlanner`. Its children are already
built by then, so a metric scan arrives as the region scan, and the node asks that
scan for its named raw-series binding: `try_to_named_series_scan` resolves the
label source and ordering from the scanner itself, which is the only thing that
can name them, and returns the leaf the node reads. A scanner that cannot serve a
raw frame is reported, not silently re-planned.
Two things this needed that the route does not get for free:
- The region scan has to be prepared as one partition. `resolve_named_series_binding`
refuses a scan whose partition count is not one, and a fresh scan has not set
its partitions at all: `ParallelizeScan` runs after extension planning and would
spread it, so the node would run one evaluator per partition and split a logical
series. Preparing the ranges as one partition is what keeps `FusedSeriesExec`
over one series stream, and it is why this route does not scan in parallel yet.
- The scan needs the per-series distribution hint. That is what makes a region
prepare its series scan, and only that path packs a series' chunks into the frame
this node reads. The existing planner asks for it through a sort that
`ScanHintRule` turns into the hint; this node has no such sort below it, so the
hint is attached to the scan directly.
A query the route cannot bind faithfully keeps the existing plan rather than a
wrong one: more than one selector, a label matcher that would survive
preprocessing, an `__field__` matcher, a native histogram field, a missing table,
or a shape whose grid the compiler's statement cannot express.
Signed-off-by: discord9 <discord9@163.com>
Compile range functions over explicit/default-step, offset, anchored, and nested subqueries using the shared range-evaluator grid calculation. Retain complete sampled coverage and reject empty child grids explicitly.
Keep compilation allocation proportional to plan structure, not timestamp count: validate deferred root and child grids; preserve eager EvalGrid::try_new; materialize shared timestamps fallibly only when a consumer requests them. Migrate Rust timestamp-buffer callers to Result without changing wire or persisted formats.
Regression coverage includes grid phase/coverage, rounded sampled endpoints, offset overflow, huge and nested compile-only grids, sparse gathering, sharing, and allocation-error propagation. Deferred compilation does not bound execution memory. No public Fused entrypoint or Scalar runtime is added here.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Add time(), the date functions and label_replace/label_join for
evaluated series, following Prometheus funcTime, dateWrapper,
evalLabelReplace and evalLabelJoin, and merge series that end up with
one label set as mergeSeriesWithSameLabelset does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: align fulltext bloom tokens with matches_term and use them for LIKE
The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.
Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop unused lazy_static from index and avoid typo-flagged test tokens
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name probe extraction with verbs and document CharClass
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: rebuild legacy fulltext bloom indexes and probe lower(col)
The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.
Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover legacy bloom index rewrite by compaction in the compat case
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: log outdated fulltext bloom indexes in the consistency check
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: explain the unrecorded fulltext bloom version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(operator): route shared requests through NodeManager
Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): preserve retry policy through error hints
Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): restore sparse partition tags when rebuilding writes
Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): stop detached write retries when callers cancel
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(frontend): configure retries across distributed write paths
Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(log-store): fetch object store WAL replay segments concurrently
A region read fetched each segment only after the previous object's
entries were yielded, so a replay paid the full request latency once per
object. Fetch up to READ_PREFETCH_DEPTH (8) segments ahead, each on its
own task, and handle their outcomes in object sequence order: entries,
collected-object skips, corrupted-segment holes and errors surface
exactly as before, and dropping the stream aborts the fetches in flight.
A read holds at most eight decoded segments besides the one it yields.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): raise object store WAL recovery concurrency to 64
On S3, recovery reads the header and tail of every object, about 57 to
91 ms per object per lane on 4, 8 and 16 vCPUs alike, and its time
scales with objects divided by concurrency up to 128 lanes without
throttling: about 9,500 objects recovered in 99 s at 8 and 10.5 s at
64. At 64 recovery issues about 1,800 reads per second, a third of
S3's request rate per prefix, and holds one 64 KiB tail window per
lane, 4 MiB in all.
The recovery concurrency tests now scale with the constant.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): share one prefetch byte budget among object store WAL reads
Each read fetched up to eight segments ahead on its own, so regions
opened in parallel (16 by default) could together hold 128 segments of
up to max_batch_bytes each. A read now always fetches the segment it
yields next, as before, and fetches a later one only while its length
fits in a 64 MiB budget shared by every read of the store. Regions
replayed in parallel thus hold one segment each plus at most 64 MiB,
however many open at once. Nothing waits on the budget: a read that
cannot reserve stops fetching ahead until its next poll.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): fail an object store WAL read whose fetch task panics
A fetch task that panicked was re-raised in the reader with
resume_unwind, which unwinds through the caller, such as Mito's region
opener task, past its cleanup. Return it as a WalSegmentFetch error
instead, so the read ends at that object like any other failed fetch.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): bound replay prefetch by decoded size held by fetch tasks
The prefetch reservation sat in the read's queue, so dropping a read
released it at once while an aborted fetch could still hold its
buffers until it next yielded. The fetch task now holds the reservation
and returns it with its outcome, so it is released with the buffers.
A reservation also counted only the encoded segment length, while a
fetched segment is held decoded. It now counts the length plus one
Record per entry, and the comment states the whole bound.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): round the measured figures in the recovery concurrency comment
Keep the magnitudes in the comment; the exact S3 figures belong to the
measurement run rather than the code.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): abort replay fetch tasks with tokio-util's AbortOnDropHandle
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): rename the replay prefetch reservation to try_reserve_prefetch_bytes
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): share the bounded polling helper across the object store WAL tests
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(datanode): execute nested MergeScan plans on the datanode
Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.
Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
engine catalog manager, and decodes a MergeScan payload's tables by
(catalog, schema, table) name through the engine catalog rather than
the region-aware request catalog list (which binds every table to the
request's region and breaks on mismatched columns).
Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
for the MergeScan nodes of a received plan, resolving region leaders
via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
the datanode's catalog/cache registry (with_dist_planner stays false:
a datanode never rewrites its own plan).
Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
partition_info caches the datanode's catalog manager and partition
rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
on a current-thread runtime from within another executor (spawn a
dedicated thread + runtime when not on a multi-thread runtime).
Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): keep GreptimeDB function bindings after rebuilding decode state
SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.
Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(datanode): address review on nested MergeScan capability
Three follow-ups from review of the nested MergeScan execution capability:
1. Layer the datanode caches (P2b). table_cache derives from
table_info_cache + table_name_cache, and partition_info_cache derives
from table_route_cache, but they were all registered in a single flat
CacheRegistry whose invalidate() runs concurrently with no ordering.
A derived cache could be invalidated and refilled from a base cache
that had not been invalidated yet, leaving stale metadata. Build the
base caches and the derived caches as separate layers of a
LayeredCacheRegistry (reusing the existing layering mechanism) so base
caches finish invalidating before derived ones. Both the production
path (cmd) and the test scaffolding now build the same layered
registry; the flat build_datanode_cache_registry is removed.
2. Don't let a nested MergeScan's internal region query compete for the
concurrency permit (P2a). An outer query holds its permit until its
stream is fully consumed, and a nested MergeScan makes the datanode
issue region queries to peer datanodes as execution stages of that
same query. If such an inner stage also took a permit, the outer query
waiting on it would deadlock against its own held permit (or time out
when max_concurrent_queries > 0). Mark inner DN->DN region queries as
internal stages: QueryRequest gains an flag carried over
the wire in the request header's query_context extensions (reserved
key, not injectable via hints), and the region server skips acquiring
a concurrency permit for internal-stage requests on both acquire
sites. Outer queries still acquire normally.
3. Add an end-to-end capability test
(tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
Join(<local probe region>, MergeScan(build regions on both datanodes))
with mismatched column names and duplicate build keys, ship it to a
datanode over a real TCP Flight endpoint, consume the stream fully and
compare the row multiset against the frontend's reference join result,
assert a real cross-datanode DoGet via an RPC counter, and assert an
inner region failure fails the whole query. Also migrate the
with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): adapt date_format regression test to the updated DataFusion fork
ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(datanode): cover the internal stage under the concurrency limit
Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.
Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(datanode): cover cold route loads under unrelated invalidations
The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(query): trim nested plan decoder commentary
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor: remove redundant nested scan support plumbing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): decode nested MergeScan payloads asynchronously
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): avoid unnecessary extension name allocation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): retain extension decoding for manual session states
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(datanode): keep nested scan query admission unchanged
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(datanode): honor query handler factories and trim nested scan fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(datanode): retain required cache error import
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(query): reuse registered state for nested scan decoding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): verify payload information schema table functions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): retain nested scan runtime and cold-route guards
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(query): address nested scan review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(session): preserve read preference across region query hops
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(session): verify strict context rejection at Flow ingress
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): remove redundant serializer error case
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Update opensrv-mysql from f3135e5 to 1f3d88c, which replies a proper EOF packet to COM_FIELD_LIST instead of a malformed OK packet that MariaDB 12.3 misparses and hangs on USE information_schema.
Fixes#9158
Signed-off-by: tian1220A <1573612521@qq.com>
* fix: bound decompressed request size and close memory-admission gaps for compressed requests
The request-memory accounting only bounds and charges the encoded bytes on
the wire, but a compressed body can expand far beyond that during
decompression, so tiny requests could allocate disproportionate frontend
memory before any protobuf validation or quota charge.
- Handler-level decompression (Prometheus remote read/write v1+v2, Loki)
now enforces a hard 512 MiB decoded-size cap, checked before any output
buffer is allocated, and charges the decoded bytes to the shared
ServerMemoryLimiter, holding the permits for the lifetime of the
decompressed buffer.
- gRPC requests with transport compression reserve the configured
max_recv_message_size before tonic decompresses, so the decoding phase
is admitted against max_in_flight_write_bytes; the later per-message
charge is skipped to avoid double accounting.
- The HTTP memory-limit middleware keeps its upfront Content-Length charge
but now also accounts the bytes actually streamed beyond it, so chunked
requests and understated Content-Length headers no longer bypass the
aggregate quota.
- Routes that decompress request bodies via RequestDecompressionLayer
(InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards)
now charge the decompressed bytes as handlers consume them: the global
middleware marks Content-Encoding requests, and a route-local
accounting layer inside the decompression layer charges the decoded
stream. Plain requests are skipped to avoid double-counting.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on memory admission guard lifetimes and 429 mapping
- Retain the gRPC pre-decode reservation for the whole request: the
extensions holding the guard were dropped when the request was consumed
(into_inner), releasing the reservation while per-message charges
stayed skipped. Both the unary and streaming handlers now clone and
hold the reservation marker for the duration of request handling.
- Retain the HTTP body permits across the handler: the AccountedBody
wrapper and the request extensions are dropped once the extractors
finish collecting the body, before the handler is done with the decoded
data. Both accounting middlewares now keep their own accounting handle
alive across next.run(req).await.
- Return a ChargedBuffer from the Loki snappy decompressor so the
reservation outlives the decompressed bytes through the caller's
protobuf decoding, matching the Prometheus path.
- Map mid-stream quota exhaustion to 429 instead of the generic body
error (400): the accounting flags exhaustion and the middlewares
rewrite the extractor rejection, so clients can distinguish
backpressure from malformed input.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: resolve the in-flight body acquisition before trying a new one
A parked acquisition in AccountedBody::charge always belongs to the
currently buffered frame, so it must be resolved before any new
acquisition is tried. Letting the fast-path try_acquire succeed while a
waiter is parked left the waiter alive, and its late completion would
then be credited with a later frame's byte count, under-reserving memory
relative to what was marked charged.
Adds a regression test that reproduces the misattribution: with the
buggy ordering the test delivers a body the quota cannot cover; with the
fix the over-quota frame waits for its own acquisition and times out.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(index): build bloom filters from element hashes without per-token allocation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(index): speed up inverted index building with hashed buffers and sync pushes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(index): use BuildHasher::hash_one for element hashes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(index): require callers to act on spill requests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(index): insert bloom hashes one by one to keep segment set capacity bounded
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(index): add index build and bloom search benchmarks
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(index): keep applier setup out of the bloom search benchmark
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(index): skip empty spills and keep the old inverted sort memory estimate
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(index): stream fulltext token hashes and test spill dispatch
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(index): note non-ASCII tokens still allocate in analyze_text_hashes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(index): narrow the allocation note to case-insensitive non-ASCII tokens
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): preserve global fetch through physical-optimizer repartitioning
Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch)
below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The
DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then
dropped or rewrote that fetched coalesce, losing the global limit. Root-caused
and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo
pins to that fix commit and adds regression coverage.
- pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix)
- global_limit.rs: adapt to new DF API (input_distribution_requirements,
child_distribution, replace_children+Recompute); accept HashPartitioned and
KeyPartitioned as partitioning to restore; add regression unit tests
- sqlness: extend standalone common ssts and refresh distributed ssts_limit
result for the preserved fetch plan
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor: drop dead HashPartitioned compat arms and test wrapper
- remove HashPartitioned compatibility matches (pinned DF only emits
KeyPartitioned); drop their #[expect(deprecated)] attributes
- remove single-use agg_with_limit helper whose seed limit is always
overwritten by the soft-limit transform
- restore Cargo.lock dependency edges unrelated to the DataFusion pin
bump (cargo update --precise had downgraded 26 unrelated packages)
Addresses oracle ablation review.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(deps): pin DataFusion to merged fetch preservation fix
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks
cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.
- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
misplaced deps to the correct sections, drop orphaned
[workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
for dependencies that are structurally required despite no textual
reference: sqlparser (required by sqlparser_derive expansions in
datatypes, common-query), common-error (required by common-macro's
stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
(feature-pinning for the transitive kube dependency in tests-fuzz),
tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
common-mem-prof), protobuf (required by build.rs-generated bindings in
log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections
Part of #9289
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test(meta): verify physical metric column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
---------
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* fix(postgres): return empty responses for comment-only SQL (#9295)
fix(postgres): handle parsed empty queries in both protocols
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
* ci: create docs follow-up issue on PR merge instead of on label (#9237)
* ci: create docs follow-up issue on PR merge instead of on label
The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.
Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
carries the docs-required label
This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on docs issue creation timing
- Only touch docs labels when the docs checkbox state actually changed
in an edit. Previously, editing any other part of the PR body while
the checkbox stayed checked removed the docs-required label, silently
dropping the docs follow-up now that issue creation happens at merge.
Unchanged checkbox now leaves labels untouched, which also preserves
manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
re-read the live PR via the API and create the docs issue if the
docs-required label is present OR the checkbox is ticked in the
current body.
- Make the workflow concurrency group action-aware so a merge run does
not cancel an in-flight label update from an edit run.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: make docs-required label the single source of truth at merge
The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.
At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): check matcher safety on the receiving operand
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): spell out the shapes a filter may cross
`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.
Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.
No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): assert the new pushdowns reach the scan
The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.
Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns
Signed-off-by: fys <fengys1996@gmail.com>
* fix(json2): preserve rows on type hint mismatch during compaction
* refactor(json2): simplify alter settings handling
* fix(json2): preserve coerced values during compaction
* chore: remove unnecessary clone
* chor: reduce memory allocations
* fix: cargo clippy
* chore: update greptime-proto to main branch
* refactor(datatypes): unify string handling with other JSON type hints
* fix: cr
---------
Signed-off-by: fys <fengys1996@gmail.com>
* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
* fix: run compaction pruner tasks on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep compaction metadata and index work on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev
Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add Jev choice and score functions
Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: use generic AI SQL function names
Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: share constant AI criteria within each batch
Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: preserve AI score uncertainty in JSONB results
Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: explain reuse of volatile AI evaluations
Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: unify pending rows batch acknowledgement policy
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align logical batcher example configuration expectations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align batcher worker channel defaults to 65536
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: sync lock
* fix: shear and check issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* feat: add experimental Jev SQL filtering
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: gate Jev filtering behind an opt-in Cargo feature
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: verify Jev registration with default features
Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: clarify Jev concurrency scope and stabilization work
Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: enable renamed ai-functions feature by default
Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: remove extra AI feature-off checks
Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: rename AI feature to ai_functions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(telemetry): remove the shared lock from TraceLayer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(telemetry): stop collecting span data while tracing is disabled
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Rebase the GreptimeDB DataFusion fork from official 55.0.0 to 55.1.0.
DataFusion 55.1.0 is a patch release on branch-55 containing eleven
cherry-picked fixes (schema-adaptation struct filters, cast/projection
metadata propagation, nested-nullability aggregation adaptation,
UnnestExec batch_size, RightMark join ordering panic, empty-struct
ScalarValue, and FFI codec fixes). All twenty GreptimeDB fork patches
rebase onto it with no textual or semantic overlap; none of them is
absorbed upstream, so all are retained.
Fork pin moves to discord9/datafusion branch greptimedb-55.1.0,
commit 2aa87d52cdce7006af492330064738f33ed294c1 (55.1.0 +
20 patches).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations
The new revision (v0y4g3r/memcomparable main, PR #1) includes:
- Deserializer::read_bytes_into for reusable-buffer decoding
- read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain)
- bulk put_slice in both normal and reverse modes
- Error::Eof instead of panic on truncated input
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): extract sparse tag values lazily without full PK decode
Replace the full primary-key decode in two hot consumers of
SortField::deserialize with column-lazy extraction over
SparsePrimaryKeyView, eliminating the per-label owned values
(Vec<Value>, one heap allocation per string) for every distinct key:
- flat_format: DecodedPrimaryKeys keeps sparse keys encoded and
extracts each tag column directly from the raw dictionary bytes into
a reusable buffer. FlatConvertFormat (flush/compaction write path)
builds all projected tag columns in one pass per key, sharing offset
discovery between columns; the read path (file_range filters) keeps
per-column lazy extraction.
- series_index writer: builds rows from the reserved prefix and only
the indexed tags via a shared offsets cache and buffer, instead of
decoding all labels and copying out a few strings.
- mito-codec: add SparsePrimaryKeyView::reserved_value for typed
table_id/tsid extraction.
Persisted formats are unchanged; values pushed to column builders are
identical (NULL vs empty string, reserved column types, duplicate
dictionary entries and consecutive-run order are preserved).
Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion
30 samples): old = full decode + per-column builds, new = lazy
extraction (+ one-pass for multi-column projection).
| Workload | Before | After | Change |
| --- | ---: | ---: | ---: |
| 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% |
| 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% |
| 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% |
| 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% |
| 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% |
Also adds a decode_sparse benchmark group to mito-codec's
bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): share sparse tag scans across filters
Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics.
Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches.
Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): benchmark sparse tag predicate count scaling
Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
discord9
Dennis Zhuang
discord9
Yingwen
Weny Xu
jeremyhi
fys
shuiyisong
zhaiAohan
dependabot[bot]
yihong
Ning Sun
LFC