The node still framed its fields by hand: the grid spec's integers, a 0/1
marker for the metric name, a length prefix before every string, the tag
count, a format tag, and separate length-prefixed blobs for the partial fold
and the output schema, with a matching Reader on the way in. That is
per-field serialization sitting inside a change whose point was to stop
writing per-field serialization, and it kept the maintenance cost: any new
field touched both directions, and a test located a byte by computing
8 + tree.len() + 8 + 8 + 8.
One derived payload replaces it. The envelope is now a version, a payload
length, and a single Postcard-encoded `FusedSeriesPayload` holding the tree,
the grid spec, the binding's name/time index/tags, the output format, the
statement text, the optional partial fold and the output schema. Substrait
still carries the children, which are exactly the fields this format should
not be inventing. `text`, `Reader` and `MAX_TAG_COUNT` are gone; the tag bound
existed only because a hand-written length prefix needs one.
Every carried-schema check stays: the tag-order-sensitive comparison against
`state_schema(binding.tags)` for a partial export, exact equality with
`series_frame_output_schema` for a frame node, and the full column-role
validation for plain rows. Decode also requires the payload to be consumed
exactly, so a byte appended after a valid payload is rejected rather than
ignored -- `postcard::from_bytes` would have accepted it, which is why the
sub-payload framing had that hole.
Two tests were reading the format rather than the behaviour:
The depth-limit test asserted that `RUST_MIN_STACK` was unset, but CI sets it
to 8 MiB, so the assertion failed before the decoder ran and the test could
never do its job there. It now runs on an explicit 2 MiB worker stack, which
is the stack a normal worker has, and asserts from inside that thread that 128
nodes decode and 129 are rejected.
The integration test located the embedded version by assuming an offset and
comparing raw bytes. The framing change made that offset stale and it failed.
It now finds the envelope, decodes the version word and asserts the number,
checks the length word against the payload, and prints the observed bytes when
either assertion trips.
Signed-off-by: discord9 <discord9@163.com>
`native/wire.rs` encoded `NativeTree` by hand: fixed node tags, fixed field
layouts, both directions written out, and a `check_shape` that narrowed the
transport to `Input -> Instant|Window -> Map(Math*)`. The serialization layer
therefore decided which plans could exist. `abs(ceil(x))` compiled but could
not be transported, and every new node kind cost a second codec path plus a
new whitelist entry. Widening the whitelist to nested maps maintained that
interpreter rather than removing it.
The file is deleted. The plan types derive serde and the codec is postcard
1.1.3 -- bincode is unusable here because RUSTSEC-2025-0141 marks the whole
crate unmaintained with no patched version. postcard is maintained, has a
documented stable wire format since 1.0, and writes an f64 as its bit pattern
in eight little-endian bytes, so NaN payloads, signed zero, infinities and
subnormals survive exactly.
What remains hand-written is framing only: magic `NPRQ`, a version, a payload
length. The version is rejected before the payload is deserialized, so an
unknown version is a clean error instead of a silent misparse, and oversize,
trailing bytes and truncation are separate errors. Decoding ends at
`NativeTree::try_new`, so a tree the compiler would not build cannot be
decoded.
Three types keep written impls because they hold invariants. `EvalGrid`
serializes its `GridSpec` and rebuilds through `try_deferred` with an empty
cache -- the grid is never serialized and never materialized on decode.
`SeriesLabels` rebuilds through `try_new`, keeping the sorting and the
duplicate / `__name__`-as-a-tag rejection. `LabelRewrite` travels as its
parameters with the regex recompiled, and fails to decode if the pattern no
longer compiles; compiled internals never travel.
`FusedSeries` now carries its output schema and accepts `PlainRows` and
`partial_fold`, which it used to refuse. A carried schema is only honest if it
is checked, so decode validates every column role, the exact time type and
nullability, missing or extra value columns, and the scalar-root tag rule; a
`SeriesFrame` schema must equal the frame layout the tree derives, and a
partial export must match `state_schema(binding.tags)` including tag order.
Identity stops being display text. `FusedSeries` compared `tree.to_string()`
with a field patch, `AggregateFinalize` hashed the rendering, and
`LabelRewrite` compared its own output; all three are structural now, with
floats compared by `to_bits()` so that signed zero and NaN payloads stay
reflexive under `Eq`. `same_binding` compared `Arc::ptr_eq`, which put an
allocation address into a transportable plan; it compares the ordered fields
and the optional grid spec now, distinguishing raw from evaluated.
Tests assert the properties rather than the byte layout: a non-canonical NaN
payload round-trips by bits, an unknown version is rejected before decode even
when the payload is garbage, non-prefix trees (binary, set, aggregate, scalar,
evaluated input) round-trip, thirteen malformed schemas are each rejected with
their own message, and a multi-chunk chain over a real region is checked
against its exact sample count and contiguous timestamps.
The multi-partition integration test is split in two and labelled honestly. It
used to assert a failure, on the theory that coalescing interleaved multi-chunk
chains; its fixture's grid has one position, so no chain has more than one
chunk and that assertion was already failing at the base commit. One test now
covers the single-position shape and says it does not exercise interleaving;
the other covers a 8192-position grid that does.
Deserialization is bounded by depth, not only by bytes. The derived recursion
walks boxed nodes, and a payload under the 1 MiB limit can nest far deeper than
the stack allows, which aborts the process instead of returning an error. A
thread-local budget guards `NativeNode`'s deserializer, so a hostile payload is
rejected before a complete tree exists. `NativeTree::Deserialize` also routes
through `try_new`, so no path can materialize a tree the constructor would
refuse. Measured under postcard with no `RUST_MIN_STACK` override: 128 nodes
decode, 129 return a decode error. Postcard collapses custom serde errors to
its unit variant, so that rejection does not name the nesting limit; the other
envelope failures (version, size, trailing bytes, truncation) each keep their
own message.
Signed-off-by: discord9 <discord9@163.com>
Lower MAX_DEPTH to 32. The old 128 was validated recursively, and a
debug build overflows a default test-thread stack well before reaching
it, so the depth guard itself could abort instead of refusing.
Report the refused node by its own name instead of "unsupported", which
is what every kind used to say.
Add the multi-partition datanode case. It asserts the failure the review
reproduced - a series chain cut by CoalescePartitionsExec no longer
decodes - so the limitation stays visible and turns red when the region
merge becomes chain-atomic.
Reconcile the retained-rate comment with the values the test asserts.
Signed-off-by: discord9 <discord9@163.com>
A payload the encoder accepts has to be one this version can decode. Three
places disagreed.
An `Input`'s field count was bounded only on the way in, so a 129-field
`Input → Instant` encoded to 983 bytes and then failed to decode. The
decoder's limit was `MAX_COUNT`, a constant that also bounds nesting depth
and `Map` operation count, which is why the missing half was easy to miss;
each limit now has its own name and both sides check it.
`FusedSeries::serialize` only checked that the tag count fit in a `u32`
while `deserialize` bounded it at 1024. Both sides now share `MAX_TAG_COUNT`.
`FusedSeries::deserialize` passed an empty schema to `try_new_with_format`,
which checks a frame node's declared schema against the layout derived from
its tree rather than overwriting it. Every serialized frame node was
therefore impossible to rebuild. The schema is derived from the decoded tree
now, which is where it comes from in the first place, so nothing is added to
the wire.
Encoding refuses an over-limit count instead of writing a payload it cannot
read back, and a field-count overflow no longer reports "operation count
exceeds maximum".
`rejects_excess_operation_count` hand-built a payload whose operation count
was read from the wrong offset: it placed an `Instant` where the decoder
expects the count, so it exercised the limit with 258 while claiming 129.
It now starts from a tree the codec accepts, asserts the bytes it edits are
the count it means, and checks the error names both the operation limit and
the value. The same is added for field count, tag count and nesting.
The integration test moves to its own file: it was written against
`tests-integration/tests/raw_series.rs`, which the raw-fan-in line creates
and this one does not carry.
Signed-off-by: discord9 <discord9@163.com>
* feat(promql): run info() natively
info(v, {selector}) fell back to the existing plan, which has no info
and refuses the query ("Unsupported expr type: info", or "Cannot accept
multiple vector as function input" when a selector is given).
Compile it into the native tree and run it on the info kernel:
- Plan: NativeExpr::Info { base, info, selector }. The info side is an
instant selection reporting each selected sample's own time in
milliseconds (InstantValue::InfoTimestamp, which replaces the
BoundInstant.timestamp flag); validation refuses any other info side.
- Compiler: info_input derives the info selector the way Prometheus
does: target_info or one __name__="..." matcher, the data matchers,
and the @ and offset every selector of the first argument shares
(subquery offsets added until an @ anchors them, none when they
differ). Regex or negative name matchers, which can select several
tables, __field__/__schema__/__database__ matchers and multi-field
info tables are refused. The planner's selector walk uses the same
helper, so both bind the info table as the last input.
- Runtime: BinaryRuntime runs the kernel as PairKernel::Info, with the
retention and pool accounting of binary and set nodes; the kernel
reports its workspace through info_with_reservation.
- Planner: the info table is a second table, so it is read through its
own RawSeriesSource and the node runs on the frontend. The output adds
the info table's tags, or only the data labels the selector names.
Partitioned or non-Mito tables and a missing info table keep the
existing plan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): pin region-local placement of info over its own table
info(rate(target_info[5m])) reads the info table twice. Check that the
planner binds both as direct scans under one FusedSeries node bound to
the table, that the node is placed in the region, and the columns it
reports.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep sample-less base series out of info's identifying patterns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): share identifying label strings in info() instead of copying them
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Reuse pack_evaluated_series for fused Frame output, preserving the single native driver and PlainRows/partial timing. Reject mismatched logical frame schemas and use the reset scan plan in integration tests. Add evaluated-packer lifecycle and invalid-sequence regressions, and address the review follow-ups.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
The workspace denies the default Clippy warnings, and the new tests tripped
three of them without changing what any of them proves:
- an expected value was built through `scale as f64` even though the tuple
already carries an `f64`, so the conversion is spelled in the literal and the
cast is gone;
- the standalone replay printed the executed plan and the per-partition counts
even though the surrounding assertions already require four nonempty
partitions, 24 rows in total and eight closing frames, so the diagnostics
are gone rather than exempted from the workspace lint.
No production code, assertion or fixture changes.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(jaeger): support v3 HTTP query APIs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(jaeger): preserve full traces with v3 time hints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(jaeger): clarify validation of v3 lookup hints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(jaeger): bound v3 searches and preserve v1 binary attributes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(jaeger): match resource attributes in v0 tag filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): return range-vector results for instant queries
An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): keep encode_substrait on a valid range query
A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): plan wrapped range-vector roots and reject a zero subquery step
TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover wrapped range-vector roots and a zero subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): default subquery step and reject range vectors in range queries
A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover root subquery default step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): align root subqueries with subquery_child_window
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): return millisecond timestamps for range-vector results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): keep the time index timezone and name its table in errors
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(log-store): collect object store WAL objects below a persisted reclaim boundary
Every object header carries a reclaim boundary: every object below it was
fully flushed when the object was sealed. The writer computes the boundary
from the catalog and the obsolete watermarks after every obsolete and every
indexed object, and stamps it on every sealed object and on the start
object. The boundary of the last indexed object permits deletion: every
present object below it, indexed, listed by recovery off the chain, or
left by a failed batch, is deleted in the background, at most four at a
time.
Recovery judges every candidate with its own boundary: a link to a
predecessor below it holds whether or not the predecessor is present, and
the chain stops there. A read that listed an object before it was
collected skips it instead of failing.
The header grows from 46 to 54 bytes; the format version stays 1.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): scan the reclaim boundary from the previous boundary
The boundary was recomputed by scanning every region's indexed objects
from the first one, so the flushed objects of busy regions were scanned
again on every indexed object while another region held the boundary.
Scan the objects in sequence order from the previous boundary instead
and stop at the first one holding a live segment. Watermarks never move
down and a region first appears above the boundary, so the result is
the same and an object is not scanned again once the boundary passed it.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): name the parked operation helpers after both their uses
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(log-store): skip corrupted object store WAL segments per region
A segment that does not decode, because its checksum does not match or
its content disagrees with its footer entry, is fetched once more. If it
still does not decode, the read follows `on_corrupted_segment`: `skip`
records a WAL hole for the region with the object key, sequence and the
footer's entry id range, increments
`greptime_logstore_object_store_wal_skipped_segments_total`, logs a
warning and continues with the next object; `fail` fails the read. A
failed fetch fails the read in both modes. Holes are kept in memory per
region and can be listed from the store.
The standalone test builder allocates WAL options through
`build_standalone_wal_provider`, and integration tests run a standalone
instance on the object store WAL across restarts, on S3 when the
`GT_S3_*` variables are set. The standalone shutdown runs every step and
returns the first error.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(cmd): cover the best-effort standalone stop
Build a standalone instance whose leader services fail to stop and whose
datanode servers fail afterwards, and check that `Instance::stop` still
stops the object store WAL and returns the leader services error.
Rename `skip_segment` to `record_skipped_segment` and the file store
integration test to `test_standalone_object_store_wal_survives_restarts_on_file`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat: add CORS support for gRPC-Web on the frontend gRPC server
A browser-based gRPC-Web client cannot read a cross-origin response. The
request carries a non-simple content type, so the browser sends an
`OPTIONS` preflight first, and it drops the call when the response has no
`Access-Control-Allow-Origin`.
Add `enable_cors` and `cors_allowed_origins` to `[grpc]`, threaded from
`GrpcOptions` to `GrpcServerConfig` the same way `tls` is.
The CORS layer sits outside `tonic_web::GrpcWebLayer`, so it answers the
preflight before the request reaches the gRPC routes. It exposes
`grpc-status`, `grpc-message` and `grpc-status-details-bin`, which a
browser cannot read otherwise. An empty `cors_allowed_origins` allows any
origin.
`enable_cors` defaults to false, unlike `[http]`. `GrpcOptions` is shared
by the public `[grpc]` section and the internal `[internal_grpc]` one, and
serde cannot tell them apart, so a true default would also turn CORS on
for the internal gRPC listeners: the frontend internal gRPC server, and
the datanode and flownode gRPC servers. None of them authenticate callers,
and a browser on the host or in the cluster network can reach all of them.
Set `enable_cors = true` to turn it on.
Tests start a real server on an ephemeral port and cover the preflight, a
custom origin list, and the disabled case.
Update the example TOMLs and regenerate config/config.md.
Signed-off-by: lczllx <2181719471@qq.com>
* fix: drop the redundant OPTIONS from the gRPC CORS allow_methods
Signed-off-by: lczllx <2181719471@qq.com>
* test: assert the allow-headers and allow-methods headers in the gRPC CORS preflight
The preflight answers with `access-control-allow-headers: *` from
`AllowHeaders::any()`, and with `access-control-allow-methods: post` from the
POST-only `allow_methods` list. Pin both, the way the HTTP CORS test pins its
own headers: a missing allow-headers header would let the preflight pass the
origin check and still have the browser block every gRPC-Web call, which a
non-browser client would never notice.
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): stop the example configs from setting the new gRPC CORS keys
Signed-off-by: lczllx <2181719471@qq.com>
* test(servers): cover the exposed gRPC CORS headers and pin the option plumbing
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): document the gRPC CORS origin example with `#+`
Signed-off-by: lczllx <2181719471@qq.com>
* fix(frontend): keep CORS off the internal gRPC server
Signed-off-by: lczllx <2181719471@qq.com>
* docs(grpc): document that the internal gRPC server never serves CORS
Signed-off-by: lczllx <2181719471@qq.com>
* Update src/servers/src/grpc.rs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim redundant comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: share CORS origin parsing and simplify gRPC-Web CORS tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: enable gRPC CORS only on the frontend public server
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(config): clarify the gRPC CORS options
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): align timestamp() and label_join with Prometheus semantics
- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): align label_replace and label_join edge cases with Prometheus
- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): join absent labels as empty strings in label_join
concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): skip rows without a sample in timestamp()
timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: bound decompressed request size and close memory-admission gaps for compressed requests
The request-memory accounting only bounds and charges the encoded bytes on
the wire, but a compressed body can expand far beyond that during
decompression, so tiny requests could allocate disproportionate frontend
memory before any protobuf validation or quota charge.
- Handler-level decompression (Prometheus remote read/write v1+v2, Loki)
now enforces a hard 512 MiB decoded-size cap, checked before any output
buffer is allocated, and charges the decoded bytes to the shared
ServerMemoryLimiter, holding the permits for the lifetime of the
decompressed buffer.
- gRPC requests with transport compression reserve the configured
max_recv_message_size before tonic decompresses, so the decoding phase
is admitted against max_in_flight_write_bytes; the later per-message
charge is skipped to avoid double accounting.
- The HTTP memory-limit middleware keeps its upfront Content-Length charge
but now also accounts the bytes actually streamed beyond it, so chunked
requests and understated Content-Length headers no longer bypass the
aggregate quota.
- Routes that decompress request bodies via RequestDecompressionLayer
(InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards)
now charge the decompressed bytes as handlers consume them: the global
middleware marks Content-Encoding requests, and a route-local
accounting layer inside the decompression layer charges the decoded
stream. Plain requests are skipped to avoid double-counting.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on memory admission guard lifetimes and 429 mapping
- Retain the gRPC pre-decode reservation for the whole request: the
extensions holding the guard were dropped when the request was consumed
(into_inner), releasing the reservation while per-message charges
stayed skipped. Both the unary and streaming handlers now clone and
hold the reservation marker for the duration of request handling.
- Retain the HTTP body permits across the handler: the AccountedBody
wrapper and the request extensions are dropped once the extractors
finish collecting the body, before the handler is done with the decoded
data. Both accounting middlewares now keep their own accounting handle
alive across next.run(req).await.
- Return a ChargedBuffer from the Loki snappy decompressor so the
reservation outlives the decompressed bytes through the caller's
protobuf decoding, matching the Prometheus path.
- Map mid-stream quota exhaustion to 429 instead of the generic body
error (400): the accounting flags exhaustion and the middlewares
rewrite the extractor rejection, so clients can distinguish
backpressure from malformed input.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: resolve the in-flight body acquisition before trying a new one
A parked acquisition in AccountedBody::charge always belongs to the
currently buffered frame, so it must be resolved before any new
acquisition is tried. Letting the fast-path try_acquire succeed while a
waiter is parked left the waiter alive, and its late completion would
then be credited with a later frame's byte count, under-reserving memory
relative to what was marked charged.
Adds a regression test that reproduces the misattribution: with the
buggy ordering the test delivers a body the quota cannot cover; with the
fix the over-quota frame waits for its own acquisition and times out.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: allow customized time index unit for metric engine table
* test: provide query tests
* refactor: revert unnecessary change
* refactor: share timestamp unit conversions in api helper
Address review feedback on the time index unit changeset:
- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
(the only crate that sees both proto ColumnDataType and TimeUnit due
to layering; common-time and datatypes have no greptime-proto dep).
This removes the ColumnDataType -> TimeUnit match duplicated between
operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
convert_timestamp_value_data by reusing api::helper::to_grpc_value
for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
values, so an overflow mid-batch leaves the request half-converted;
harmless because the error aborts the whole insert request.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: align time units per destination table and floor remote-read timestamps
Address review feedback on PR #9236:
- Align each metric insert request to the unit of the table it actually
targets: an existing logical table keeps its own unit (it may be bound
to a different physical table than the one selected by the request),
and only new tables use the selected physical table's unit. The
previous blanket conversion rewrote valid millisecond samples to the
selected physical table's unit and the engine rejected them.
Regression test: writing an existing millisecond logical table and a
new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
negative infinity (div_euclid), consistent with
Timestamp::convert_to on the ingestion path; arrow's cast truncates
towards zero and returned -1ms for a stored -1001us. Widening
(second -> millisecond) keeps the exact arrow cast. Regression test:
a negative, non-aligned timestamp round-trips as -2ms.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: fold time unit alignment into existing table lookups
Address review feedback on PR #9236:
- The per-destination unit alignment no longer runs its own pass of
table lookups: create_or_alter_tables_on_demand gains an
align_time_index_unit parameter (metric engine path only) and
converts each request inside the lookups it already performs —
existing tables to their own unit, new tables to the selected
physical table's. Default ingest paths now issue zero additional
catalog lookups compared to main; the separate alignment pass remains
only in the opt-in logical batcher pre-gate, next to the eligibility
check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
(validate_column_count_match guarantees row widths) instead of
Optional get_mut; the gate-side alignment validates widths itself.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: resolve the batcher time index guard once per write target
All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: support non-millisecond time index units in the logical batcher
Make the logical table batcher's bulk encode path unit-aware so physical
metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can
use logical batching instead of falling back to the ordinary insert path.
- rows_to_aligned_record_batch builds the time index column in the
TARGET schema's unit, converting any timestamp encoding via
Timestamp::convert_to (flooring on narrowing, consistent with the
ordinary insert path).
- New tables created by the batcher use the selected physical table's
time index unit (resolved once per submit; a missing physical table
keeps the millisecond auto-create default).
- columns_taxonomy and the can_batch_metric_rows schema whitelist accept
any timestamp unit; the prometheus remote write v1/v2 batcher gates
and the OTLP pre-gate alignment are removed together with
Inserter::align_metric_row_inserts_time_unit, as the batcher now
converts internally.
Closes#9342
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
* fix: address review issue
* refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path
The main merge of #9236 (squash) resurrected the OTLP pre-gate
alignment and Inserter::align_metric_row_inserts_time_unit, which this
branch had removed. Drop them again:
- The pre-gate existed because the #9236-era bulk eligibility gate only
accepted millisecond schemas, so nanosecond-encoded OTLP requests had
to be converted before the check. This branch makes the bulk path
unit-aware (the gate accepts all time index units and batch alignment
converts each request to its destination's unit), so the pre-gate is
redundant and only added N+1 catalog lookups per batched request —
the very lookup-count overhead raised in the #9236 review.
- The per-destination unit semantics it implemented remain enforced in
the two paths that need them: the ordinary insert path
(create_or_alter_tables_on_demand converts inside its existing table
lookups) and the batched path (batch alignment resolves each
destination schema and converts to it).
test_otlp_logical_batcher_alignment (the test the pre-gate originally
fixed) and the mixed-physical-table regression both pass without it.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cover OTLP batcher cross-physical fallback and nanosecond physical
Extend the logical batcher integration coverage for the cases
previously guarded by the removed OTLP pre-gate alignment:
- test_otlp_logical_batcher_fallback_for_cross_physical_destination:
with the batcher enabled, an OTLP request targeting an existing
logical table bound to another physical table must NOT enter the
batcher (the bulk eligibility check rejects the destination binding)
and the ordinary insert path must convert it to the destination's
unit (60s -> 60_000_000us).
- test_otlp_logical_batcher_non_millisecond_physical_table now covers
both microsecond and nanosecond physical tables (parameterized),
asserting batcher submissions and unit-precise stored values.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: validate per-batch physical bindings and avoid intermediate timestamp buffers
Address review feedback on PR #9346:
- accepts_bulk_destinations dedupes on (schema, table, selected physical)
instead of (schema, table): one request can select different physical
tables per batch (per-series x_greptime_physical_table labels), and the
old key let a second selection skip validation and flush rows through
the wrong physical's regions. Missing tables additionally reject
conflicting physical selections within the same request. Regression
test covers an existing destination, a missing destination, and a
consistent selection (which must still batch).
- The timestamp column builder appends each value directly into the
target-unit Arrow builder; values already in the target unit (the
unchanged millisecond fast path) are appended without conversion, so
the default millisecond physical pays no Timestamp construction or
intermediate Vec allocation.
- The non-millisecond batching tests assert the submit_build_and_align
counter, which increments on every batcher submission in both
acknowledgement modes, so a silent fallback to ordinary insertion
fails the tests instead of passing on stored values alone.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: allow customized time index unit for metric engine table
* test: provide query tests
* refactor: revert unnecessary change
* refactor: share timestamp unit conversions in api helper
Address review feedback on the time index unit changeset:
- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
(the only crate that sees both proto ColumnDataType and TimeUnit due
to layering; common-time and datatypes have no greptime-proto dep).
This removes the ColumnDataType -> TimeUnit match duplicated between
operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
convert_timestamp_value_data by reusing api::helper::to_grpc_value
for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
values, so an overflow mid-batch leaves the request half-converted;
harmless because the error aborts the whole insert request.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: align time units per destination table and floor remote-read timestamps
Address review feedback on PR #9236:
- Align each metric insert request to the unit of the table it actually
targets: an existing logical table keeps its own unit (it may be bound
to a different physical table than the one selected by the request),
and only new tables use the selected physical table's unit. The
previous blanket conversion rewrote valid millisecond samples to the
selected physical table's unit and the engine rejected them.
Regression test: writing an existing millisecond logical table and a
new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
negative infinity (div_euclid), consistent with
Timestamp::convert_to on the ingestion path; arrow's cast truncates
towards zero and returned -1ms for a stored -1001us. Widening
(second -> millisecond) keeps the exact arrow cast. Regression test:
a negative, non-aligned timestamp round-trips as -2ms.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: fold time unit alignment into existing table lookups
Address review feedback on PR #9236:
- The per-destination unit alignment no longer runs its own pass of
table lookups: create_or_alter_tables_on_demand gains an
align_time_index_unit parameter (metric engine path only) and
converts each request inside the lookups it already performs —
existing tables to their own unit, new tables to the selected
physical table's. Default ingest paths now issue zero additional
catalog lookups compared to main; the separate alignment pass remains
only in the opt-in logical batcher pre-gate, next to the eligibility
check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
(validate_column_count_match guarantees row widths) instead of
Optional get_mut; the gate-side alignment validates widths itself.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: resolve the batcher time index guard once per write target
All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
* fix: address review issue
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(client): complete transport lane isolation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(client): deprecate legacy single-manager constructors
Per review: mark the legacy single-manager constructors and helper as
deprecated so callers move to the isolated query/control manager pair.
Tests intentionally exercising the legacy path are annotated with
`#[allow(deprecated)]`.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(client): update deprecated constructor callers for CI
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(mito): limit series index disk usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): enforce series index quota during reconciliation
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): remove series index disk budget layer
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): simplify series index limit to estimated usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): trust index catalogs when loading snapshots
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): minimize series index disk limit changes
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): keep series index cleanup running at capacity
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): avoid no-op index clones and stabilize capacity tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: update config API expectation and stabilize index build tests
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: serialize struct to json in postgres
* fix: support view scalars and preserve null structs in scalar-to-value conversion
Address PR review:
- Utf8View/BinaryView ScalarValues now convert like their non-view forms
instead of failing row extraction for struct columns
- a null struct scalar converts to Value::Null so a null struct inside a
list stays null in the serialized JSON
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: return errors instead of panics for unsupported arrow field types
Struct-typed query results with arrow field types greptimedb cannot
represent (e.g. Decimal256) used to panic during schema conversion and
row extraction, dropping the client connection. They now surface as
query errors:
- ConcreteDataType::try_from builds struct types fallibly via the new
StructType::try_from_arrow_fields
- Value::try_from(ScalarValue::Struct) uses the same fallible path
- new try_value_from_array converts an arrow element to Value with
error propagation, used by the postgres struct encoding
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cut integration test time and make the storage matrix meaningful
tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.
- Point the PR CI object-store matrix at the MinIO already started by
tests-integration/fixtures. Three GT_S3_* consumers did not read
GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
release integration-test container. The release previously ran every
remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
/tmp/greptimedb_cache, which the datanode wipes on startup, so a
starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
non-trivial SST representation: JSON/JSON2 columns, native histograms,
metric-engine logical tables, and tables carrying fulltext or skipping
indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
shorten it in the event tests, which otherwise wait a 5s window per DDL
they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
test_grpc_message_size_ok and is now rewritten to assert the negotiated
grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
prefixes of their copy_from siblings; two standalone/distributed event
test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
in a parallelism-dependent order, so its last digits are unstable; the
test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
ttl=7d with 2025 timestamps, so its rows were only readable as long as
they stayed in the memtable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: address review — wire nightly real-S3 job into check-status, keep the short event interval
The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.
In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
discord9
dennis zhuang
discord9
Yingwen
jeremyhi
shuiyisong
Dragon RoarandDennis Zhuang
Ning Sun
Weny Xu
Dhruv Vaishnav
LFC