* docs(config): align example configs and config.md with actual defaults
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(cmd): compare example defaults inside optional sections
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(log-store): skip corrupted object store WAL segments per region
A segment that does not decode, because its checksum does not match or
its content disagrees with its footer entry, is fetched once more. If it
still does not decode, the read follows `on_corrupted_segment`: `skip`
records a WAL hole for the region with the object key, sequence and the
footer's entry id range, increments
`greptime_logstore_object_store_wal_skipped_segments_total`, logs a
warning and continues with the next object; `fail` fails the read. A
failed fetch fails the read in both modes. Holes are kept in memory per
region and can be listed from the store.
The standalone test builder allocates WAL options through
`build_standalone_wal_provider`, and integration tests run a standalone
instance on the object store WAL across restarts, on S3 when the
`GT_S3_*` variables are set. The standalone shutdown runs every step and
returns the first error.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(cmd): cover the best-effort standalone stop
Build a standalone instance whose leader services fail to stop and whose
datanode servers fail afterwards, and check that `Instance::stop` still
stops the object store WAL and returns the leader services error.
Rename `skip_segment` to `record_skipped_segment` and the file store
integration test to `test_standalone_object_store_wal_survives_restarts_on_file`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(otlp): coerce empty and "-" trace attribute strings to null
Instrumentations fill numeric attributes from absent headers with an empty
string (PSR-7 getHeaderLine returns ""). Coercing that into an existing
numeric column failed and rejected the whole span.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: trim redundant row from empty-string rewrite test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: accept a value for the CLI --auto-create-schema flag
clap derives ArgAction::SetTrue for bool fields, so --auto-create-schema
took no value and could only set true, which is already the default.
Users could not turn automatic schema creation off from the CLI, unlike
metasrv's auto_create_schema = false.
Set action = clap::ArgAction::Set so the flag accepts an explicit
true/false value while keeping true as the default.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* chore: enhance CLI argument for auto_create_schema option
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix: coalesce equal histogram quantile bucket boundaries
Merge numerically equivalent bucket boundaries before monotonicity repair, preserving the allocation-free path for unique boundaries. Share coalescing with histogram_fraction and cover duplicate boundaries with unit and SQL regressions.
Fixes#9443
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: keep duplicate infinity buckets in one histogram group
Confirm classic histogram completion at a series/timestamp boundary or EOF instead of the first positive infinity. Retain lookahead across batches and reject incomplete optimistic groups so equivalent infinity labels reach numeric coalescing together.
Add stream and SQL regressions for duplicate infinity labels, including established bucket counts and EOF. Correct older fixtures to use distinct keys for distinct histogram groups.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: avoid buffering histogram groups with invalid trailing bounds
Discover candidate bucket counts at the first group boundary and let optimistic validation fall back to safe mode for malformed bounds. Document discovery and cover trailing invalid and null labels with a focused streaming regression.
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* ci: make query regression non-blocking for scheduled nightly releases
Scheduled nightly releases no longer gate publishing on the query
regression release test. The test still runs (validation policy stays
'all' for automatic releases) and uploads its report, but any result -
failure, timeout, cancellation, or skip - no longer blocks image
publishing or the GitHub release for schedule-triggered runs.
Since reusable-workflow caller jobs cannot use continue-on-error, the
gate is relaxed at the consumers instead: the if conditions of
release-images-to-dockerhub and publish-github-release now also pass
when github.event_name == 'schedule' regardless of the query-regression
result. All other gates (runner allocation, artifact builds,
prepare-release-validation) are unchanged.
Tag-push and manual-dispatch releases remain fully blocking unless a
skip policy is explicitly chosen via the release_validation input.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: preserve downstream nightly release jobs after regression failures
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
* feat: add CORS support for gRPC-Web on the frontend gRPC server
A browser-based gRPC-Web client cannot read a cross-origin response. The
request carries a non-simple content type, so the browser sends an
`OPTIONS` preflight first, and it drops the call when the response has no
`Access-Control-Allow-Origin`.
Add `enable_cors` and `cors_allowed_origins` to `[grpc]`, threaded from
`GrpcOptions` to `GrpcServerConfig` the same way `tls` is.
The CORS layer sits outside `tonic_web::GrpcWebLayer`, so it answers the
preflight before the request reaches the gRPC routes. It exposes
`grpc-status`, `grpc-message` and `grpc-status-details-bin`, which a
browser cannot read otherwise. An empty `cors_allowed_origins` allows any
origin.
`enable_cors` defaults to false, unlike `[http]`. `GrpcOptions` is shared
by the public `[grpc]` section and the internal `[internal_grpc]` one, and
serde cannot tell them apart, so a true default would also turn CORS on
for the internal gRPC listeners: the frontend internal gRPC server, and
the datanode and flownode gRPC servers. None of them authenticate callers,
and a browser on the host or in the cluster network can reach all of them.
Set `enable_cors = true` to turn it on.
Tests start a real server on an ephemeral port and cover the preflight, a
custom origin list, and the disabled case.
Update the example TOMLs and regenerate config/config.md.
Signed-off-by: lczllx <2181719471@qq.com>
* fix: drop the redundant OPTIONS from the gRPC CORS allow_methods
Signed-off-by: lczllx <2181719471@qq.com>
* test: assert the allow-headers and allow-methods headers in the gRPC CORS preflight
The preflight answers with `access-control-allow-headers: *` from
`AllowHeaders::any()`, and with `access-control-allow-methods: post` from the
POST-only `allow_methods` list. Pin both, the way the HTTP CORS test pins its
own headers: a missing allow-headers header would let the preflight pass the
origin check and still have the browser block every gRPC-Web call, which a
non-browser client would never notice.
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): stop the example configs from setting the new gRPC CORS keys
Signed-off-by: lczllx <2181719471@qq.com>
* test(servers): cover the exposed gRPC CORS headers and pin the option plumbing
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): document the gRPC CORS origin example with `#+`
Signed-off-by: lczllx <2181719471@qq.com>
* fix(frontend): keep CORS off the internal gRPC server
Signed-off-by: lczllx <2181719471@qq.com>
* docs(grpc): document that the internal gRPC server never serves CORS
Signed-off-by: lczllx <2181719471@qq.com>
* Update src/servers/src/grpc.rs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim redundant comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: share CORS origin parsing and simplify gRPC-Web CORS tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: enable gRPC CORS only on the frontend public server
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(config): clarify the gRPC CORS options
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): truncate parquet column index min/max for SST writes
The SST writer and the bulk part encoder disabled column index truncation
together with statistics truncation in #6977. Mito never reads the column
index, but every page of a large string or binary column still stored its
full min and max values, uncompressed. Truncate the column index to the
parquet default of 64 bytes and keep chunk statistics untruncated, since
primary key pruning decodes them.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(mito2): note that the column index must not be used to decode primary keys
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito): wait for WAL durability before publishing a manifest watermark
A log store that acknowledges appends before their entries are durable can
hand out entry ids that a crash loses. Before a flush, a full truncate or a
discard of unflushed data records an entry id in the manifest, Mito now waits
on `LogStore::wait_durable` for that id. The flush waits after its SSTs are
written and observes its cancellation, so a drop or truncate queued behind it
is not held up by the upload.
Add engine tests of the object store WAL across restarts and of the barrier,
and a log store testing hook that stores an object and then reports its create
as failed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito): synchronize the WAL barrier tests on store signals
The barrier tests waited with fixed sleeps, which do not establish that a
flush or truncate has reached its durability wait, and the lost backlog test
could stop the store before the seal was handled. The object store WAL gets
testing hooks that observe parked creates and durability waits, and one that
ends the actor as a crash would; the tests wait on them, crash the store and
the engine before a reopen until nothing holds the old store, and cover a
truncate or discard of unflushed data that completes once its entry is
durable.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito): rename the recovery state and the crash test in the WAL recovery tests
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito): drop a comment that restates the empty recovery state
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(promql): align timestamp() and label_join with Prometheus semantics
- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): align label_replace and label_join edge cases with Prometheus
- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): join absent labels as empty strings in label_join
concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): skip rows without a sample in timestamp()
timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(datanode): wire the object store WAL into the datanode and standalone
A datanode configured with `provider = "experimental_object_store"` now
builds an `ObjectStoreLogStore` instead of failing with the placeholder
"not supported yet" error. The builder resolves `storage_provider`
against the object store manager (empty selects the default store, an
unknown name is rejected as an invalid `storage_provider`), opens the
store under the node id and the standalone generation, stops it if a
later build step fails, and hands it to `Datanode`, whose shutdown stops
it after the region server. Datanode shutdown is now best-effort: every
step runs and the first error is returned. A datanode with a metasrv
client still rejects the provider.
The standalone bootstrap builds `WalProvider::ObjectStore` with the same
derived node prefix before the metasrv WAL conversion, so the WAL
options it allocates match the prefix the store runs under.
`log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's
test utilities can build an engine on the object store WAL over an
in-memory object store, and new engine tests cover create, reopen,
replay isolation between regions, batch open, drop and offline cleanup,
the latest entry id after replay, and the rejection of object store WAL
options on a Raft Engine log store.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(datanode): stop every region engine before the object store WAL
The region server returned at the first engine whose stop failed, so the
engines after it in map order, Mito included, were never stopped before
the datanode stopped the object store WAL. It now attempts every engine,
returns the first error and logs the later ones.
`ObjectStoreLogStore` is also exported from the `log-store` crate root,
and its callers import it from there.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: bound decompressed request size and close memory-admission gaps for compressed requests
The request-memory accounting only bounds and charges the encoded bytes on
the wire, but a compressed body can expand far beyond that during
decompression, so tiny requests could allocate disproportionate frontend
memory before any protobuf validation or quota charge.
- Handler-level decompression (Prometheus remote read/write v1+v2, Loki)
now enforces a hard 512 MiB decoded-size cap, checked before any output
buffer is allocated, and charges the decoded bytes to the shared
ServerMemoryLimiter, holding the permits for the lifetime of the
decompressed buffer.
- gRPC requests with transport compression reserve the configured
max_recv_message_size before tonic decompresses, so the decoding phase
is admitted against max_in_flight_write_bytes; the later per-message
charge is skipped to avoid double accounting.
- The HTTP memory-limit middleware keeps its upfront Content-Length charge
but now also accounts the bytes actually streamed beyond it, so chunked
requests and understated Content-Length headers no longer bypass the
aggregate quota.
- Routes that decompress request bodies via RequestDecompressionLayer
(InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards)
now charge the decompressed bytes as handlers consume them: the global
middleware marks Content-Encoding requests, and a route-local
accounting layer inside the decompression layer charges the decoded
stream. Plain requests are skipped to avoid double-counting.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on memory admission guard lifetimes and 429 mapping
- Retain the gRPC pre-decode reservation for the whole request: the
extensions holding the guard were dropped when the request was consumed
(into_inner), releasing the reservation while per-message charges
stayed skipped. Both the unary and streaming handlers now clone and
hold the reservation marker for the duration of request handling.
- Retain the HTTP body permits across the handler: the AccountedBody
wrapper and the request extensions are dropped once the extractors
finish collecting the body, before the handler is done with the decoded
data. Both accounting middlewares now keep their own accounting handle
alive across next.run(req).await.
- Return a ChargedBuffer from the Loki snappy decompressor so the
reservation outlives the decompressed bytes through the caller's
protobuf decoding, matching the Prometheus path.
- Map mid-stream quota exhaustion to 429 instead of the generic body
error (400): the accounting flags exhaustion and the middlewares
rewrite the extractor rejection, so clients can distinguish
backpressure from malformed input.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: resolve the in-flight body acquisition before trying a new one
A parked acquisition in AccountedBody::charge always belongs to the
currently buffered frame, so it must be resolved before any new
acquisition is tried. Letting the fast-path try_acquire succeed while a
waiter is parked left the waiter alive, and its late completion would
then be credited with a later frame's byte count, under-reserving memory
relative to what was marked charged.
Adds a regression test that reproduces the misattribution: with the
buggy ordering the test delivers a body the quota cannot cover; with the
fix the over-quota frame waits for its own acquisition and times out.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(promql): resolve dotted column names as unqualified columns
col(), From<&str>/From<String> for Column and string join keys go through
Column::from_qualified_name, which splits `service.name` into relation
`service` and column `name` and lowercases unquoted identifiers. Build
PromQL column references with Column::from_name / ident() instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): resolve remote read matcher labels as unqualified columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover same-name columns differing in case and without()
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Previously the nightly release was scheduled only on Mondays, so a
failed run left users without nightly builds for a whole week.
Now the schedule triggers every weekday at 00:00 UTC, but the release
only proceeds when the latest published nightly release is older than
NIGHTLY_RELEASE_MAX_AGE_DAYS (5) days. The check runs in
allocate-runners before any EC2 runner is allocated, and gates all
schedule-driven jobs (including the Slack notification, so a skipped
nightly is silent). Tag pushes and manual dispatches are unaffected.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(promql): apply offset to subquery evaluation window
`prom_subquery_expr_to_plan` destructured `SubqueryExpr` without reading
`offset`, so `<subquery>[range:step] offset <d>` planned exactly the same
window as the un-offset form and silently returned data for the wrong time
range. The plain vector/matrix-selector paths already threaded the offset
through `selector_to_series_normalize_plan` and `RangeManipulate`.
Shift the inner evaluation window back by the offset and pass the offset to
the subquery's `RangeManipulate`, which maps the inner samples forward onto
the evaluation timeline before bucketing them into ranges. This matches
Prometheus, whose `evaluator.subqueryTimeRange` evaluates the inner
expression over `(start - offset - range, end - offset]` and whose
`evalSubquery` then hands the samples to the outer range-vector function as
a `MatrixSelector` that still carries the subquery offset. An offset on the
inner selector composes additively, as `subqueryTimes` documents.
`RangeManipulate`'s protobuf message has no offset field and recovers it on
decode from an immediately underlying `SeriesNormalize`. Since
`RangeManipulate` is commutative in `dist_plan` and can be pushed below a
`MergeScan`, insert that carrier node so the offset survives distributed
planning instead of decoding as zero.
Known divergence, unchanged by this commit: Prometheus anchors subquery step
points on absolute epoch multiples of the step, while GreptimeDB anchors them
on the evaluation start. The two agree whenever the offset is a multiple of
the subquery step; the added sqlness cases stay within that range.
Closes#9330
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* docs(promql): correct the subquery-offset rationale and pin the histogram shift
Follow-up to e132c267, which stated two things inaccurately and left one
path untested. No behaviour change beyond a conditional clone.
The `SeriesNormalize` carrier was described as existing "so the offset
survives distributed planning", implying standalone is unaffected. That is
wrong. `local_offset` is read on the substrait decode path
(`range_manipulate.rs`, `instant_manipulate.rs`), and GreptimeDB routes
PromQL plans through `MergeScan`/substrait in standalone too -- the
standalone results `promql/encode_substrait.result` and `precisions.result`
both show `MergeScan [is_placeholder=false, remote_input=[...]]`. Deleting
the carrier therefore empties subquery-with-offset results in standalone as
well, while `count_over_time_subquery_with_offset` keeps passing because it
asserts the pre-serialization plan. Say so at the node, so the next reader
does not remove it believing standalone is safe.
The carrier is also not semantically inert: `SeriesNormalizeStream::normalize`
shifts native histogram `start_timestamp` payloads by `offset`. On the
subquery path that runs on a computed inner result, on top of any offset the
inner selector already applied, and nothing pinned it. It is consistent --
`RangeManipulate` moves the millisecond time index forward by the same
`offset_ms` -- so the distance from a histogram's start timestamp to the
sample carrying it, which reset/rate detection reads, is invariant. Add
`subquery_offset_shifts_histogram_start_and_time_index_together`, which
executes the subquery's node stack over native histogram samples for a
zero and a non-zero inner offset. It cannot be a sqlness case: native
histograms are a struct column with no SQL type or literal
(`sql_data_type_to_concrete_data_type` rejects structs) and the sqlness
runner speaks only MySQL/Postgres, so such rows only arrive over
gRPC/remote-write v2.
The divergence note blamed sub-step offsets. Measured, that condition is too
narrow: an unaligned evaluation timestamp alone diverges, with no offset at
all -- `sum_over_time(fine[20s:10s])` at t=57 samples 47s and 57s here
against Prometheus's 40s and 50s. The real condition is that
`(start - offset)` is not a multiple of the subquery step, and it is
pre-existing: the committed `tql eval (359, 359, '1s')
sum_over_time(metric_total[60s:10s])` case is already an instance of it
(`359 - 60 + 10 = 309`). Reword the comment and the sqlness header
accordingly. A sub-step offset stays accepted: `foo[20s:10s] offset 5s` is
valid PromQL, erroring on it would be a regression and would not close the
gap anyway. This repo has no PromQL-compatibility documentation page, so
there is nowhere else to record it.
Also stop cloning the series key columns when there is no offset, and stop
the `offset 0s` sqlness comment implying its parser error is evidence about
this fix -- the same message comes from a `u32` overflow in
`promql-parser`'s shared duration check (`offset 9999999999d`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* refactor(promql): trim subquery-offset comments and tests per review
Shorten the planner and sqlness comments to the semantic contract, build
`SeriesDivide` with a clone instead of an `Option`, and drop the
normalize.rs histogram test and the `offset 0s` parser case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* fix(promql): record the subquery offset as the range fold offset
The subquery's `RangeManipulate` now shifts the payload timestamps by the
offset, so `predict_linear` over an offset subquery must recover the
evaluation time with that offset instead of 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>
---------
Signed-off-by: Aarav <aaravsjadav@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* revert(ci): pin the query-regression runner toolchain to nightly-2026-03-21
The query-regression benchmark compiles both the candidate and the
BASE checkout (the previous nightly build). Base refs can predate the
stable-toolchain migration and still use #![feature] gates, so the
runner toolchain must stay a nightly that can build historical
revisions; deriving it from the workspace rust-toolchain.toml (now
stable 1.96.1) breaks base builds.
Revert the toolchain-toml coupling introduced in #9369 and keep the
parts that were correct:
- query-regression.yml: RUSTUP_TOOLCHAIN hard-pinned to
nightly-2026-03-21 again (with a comment explaining why), Verify
assertions back to the exact nightly versions, and the
test-tooling pin-derivation machinery removed
- runner Dockerfile: ARG RUST_TOOLCHAIN=nightly-2026-03-21 + baked
ENV restored; the COPY rust-toolchain.toml parsing removed
- build-ecs-image.py: the toml staging in the builder user-data
removed; base-image auto-resolution kept but retargeted to Ubuntu
26.04 to match the Verify tool pins (python3 3.14 etc.)
- the rebuild job keeps the epoch-bump lockstep, now as a PR from a
timestamped ci/ branch mirroring update-dev-builder-version.sh
(direct pushes to main fail GH006 under branch protection)
Complements #9376 (actions-runner bump), which requires a rebuilt
image with the non-deprecated runner.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: skip qreg for rust-toolchain change
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta-srv): use NoTls for disabled and Unix socket Postgres KV backends
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* fix(meta-srv): treat Postgres config as unix socket only when every host is a socket
tokio-postgres dials hostaddr over TCP even when host is a socket path,
and a mixed host list with Require/VerifyFull TLS would otherwise end up
sending plaintext over TCP. is_unix_socket_url now parses via
tokio_postgres::Config and requires no hostaddr and all-Unix hosts.
Adds regression cases for the libpq keyword form, the user@ percent
encoded socket URL, and the mixed host / hostaddr negatives.
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* chore: remove dead code left behind by removed features
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: remove unused RouteInfoCorrupted error
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: allow customized time index unit for metric engine table
* test: provide query tests
* refactor: revert unnecessary change
* refactor: share timestamp unit conversions in api helper
Address review feedback on the time index unit changeset:
- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
(the only crate that sees both proto ColumnDataType and TimeUnit due
to layering; common-time and datatypes have no greptime-proto dep).
This removes the ColumnDataType -> TimeUnit match duplicated between
operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
convert_timestamp_value_data by reusing api::helper::to_grpc_value
for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
values, so an overflow mid-batch leaves the request half-converted;
harmless because the error aborts the whole insert request.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: align time units per destination table and floor remote-read timestamps
Address review feedback on PR #9236:
- Align each metric insert request to the unit of the table it actually
targets: an existing logical table keeps its own unit (it may be bound
to a different physical table than the one selected by the request),
and only new tables use the selected physical table's unit. The
previous blanket conversion rewrote valid millisecond samples to the
selected physical table's unit and the engine rejected them.
Regression test: writing an existing millisecond logical table and a
new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
negative infinity (div_euclid), consistent with
Timestamp::convert_to on the ingestion path; arrow's cast truncates
towards zero and returned -1ms for a stored -1001us. Widening
(second -> millisecond) keeps the exact arrow cast. Regression test:
a negative, non-aligned timestamp round-trips as -2ms.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: fold time unit alignment into existing table lookups
Address review feedback on PR #9236:
- The per-destination unit alignment no longer runs its own pass of
table lookups: create_or_alter_tables_on_demand gains an
align_time_index_unit parameter (metric engine path only) and
converts each request inside the lookups it already performs —
existing tables to their own unit, new tables to the selected
physical table's. Default ingest paths now issue zero additional
catalog lookups compared to main; the separate alignment pass remains
only in the opt-in logical batcher pre-gate, next to the eligibility
check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
(validate_column_count_match guarantees row widths) instead of
Optional get_mut; the gate-side alignment validates widths itself.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: resolve the batcher time index guard once per write target
All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: support non-millisecond time index units in the logical batcher
Make the logical table batcher's bulk encode path unit-aware so physical
metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can
use logical batching instead of falling back to the ordinary insert path.
- rows_to_aligned_record_batch builds the time index column in the
TARGET schema's unit, converting any timestamp encoding via
Timestamp::convert_to (flooring on narrowing, consistent with the
ordinary insert path).
- New tables created by the batcher use the selected physical table's
time index unit (resolved once per submit; a missing physical table
keeps the millisecond auto-create default).
- columns_taxonomy and the can_batch_metric_rows schema whitelist accept
any timestamp unit; the prometheus remote write v1/v2 batcher gates
and the OTLP pre-gate alignment are removed together with
Inserter::align_metric_row_inserts_time_unit, as the batcher now
converts internally.
Closes#9342
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
* fix: address review issue
* refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path
The main merge of #9236 (squash) resurrected the OTLP pre-gate
alignment and Inserter::align_metric_row_inserts_time_unit, which this
branch had removed. Drop them again:
- The pre-gate existed because the #9236-era bulk eligibility gate only
accepted millisecond schemas, so nanosecond-encoded OTLP requests had
to be converted before the check. This branch makes the bulk path
unit-aware (the gate accepts all time index units and batch alignment
converts each request to its destination's unit), so the pre-gate is
redundant and only added N+1 catalog lookups per batched request —
the very lookup-count overhead raised in the #9236 review.
- The per-destination unit semantics it implemented remain enforced in
the two paths that need them: the ordinary insert path
(create_or_alter_tables_on_demand converts inside its existing table
lookups) and the batched path (batch alignment resolves each
destination schema and converts to it).
test_otlp_logical_batcher_alignment (the test the pre-gate originally
fixed) and the mixed-physical-table regression both pass without it.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cover OTLP batcher cross-physical fallback and nanosecond physical
Extend the logical batcher integration coverage for the cases
previously guarded by the removed OTLP pre-gate alignment:
- test_otlp_logical_batcher_fallback_for_cross_physical_destination:
with the batcher enabled, an OTLP request targeting an existing
logical table bound to another physical table must NOT enter the
batcher (the bulk eligibility check rejects the destination binding)
and the ordinary insert path must convert it to the destination's
unit (60s -> 60_000_000us).
- test_otlp_logical_batcher_non_millisecond_physical_table now covers
both microsecond and nanosecond physical tables (parameterized),
asserting batcher submissions and unit-precise stored values.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: validate per-batch physical bindings and avoid intermediate timestamp buffers
Address review feedback on PR #9346:
- accepts_bulk_destinations dedupes on (schema, table, selected physical)
instead of (schema, table): one request can select different physical
tables per batch (per-series x_greptime_physical_table labels), and the
old key let a second selection skip validation and flush rows through
the wrong physical's regions. Missing tables additionally reject
conflicting physical selections within the same request. Regression
test covers an existing destination, a missing destination, and a
consistent selection (which must still batch).
- The timestamp column builder appends each value directly into the
target-unit Arrow builder; values already in the target unit (the
unchanged millisecond fast path) are appended without conversion, so
the default millisecond physical pays no Timestamp construction or
intermediate Vec allocation.
- The non-millisecond batching tests assert the submit_build_and_align
counter, which increments on every batcher submission in both
acknowledgement modes, so a silent fallback to ordinary insertion
fails the tests instead of passing on stored values alone.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(query): implement PromQL @ modifier on vector and matrix selectors
The planner previously dropped the `at` field of selectors (`at: _`),
so `some_metric @ 300` silently returned step-following values instead
of the samples anchored at the fixed timestamp.
Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch`
semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply
`offset` to the anchor, rewrite the selector offset to
`eval_start - anchor`, scan only the anchored window, then report the
same window at every evaluation step via a grid-wide-replay
`InstantManipulate`. `@ start()` / `@ end()` resolve against the
statement's evaluation range (`stmt_start` / `stmt_end`), which subquery
planning does not rewrite.
Timestamps before the Unix epoch are accepted; unrepresentable ones are
rejected with `AtModifierTimestampOutOfRange` instead of wrapping.
Selectors without `@` are planned exactly as before.
Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(promql): center predict_linear on the evaluation timestamp
predict_linear_impl used the window's last sample time as the evaluation
timestamp, and the UDF took only (ts_range, value_range, t) with no channel
for the current instant. After a range selector is folded for @ (or with an
offset) the same window is replayed at every step, so the prediction stayed
constant at the anchor's answer; even a plain window ended before the step
produced a stale value.
Give prom_predict_linear a 4th argument carrying the step's evaluation
instant (ms timestamp), derived in the planner from the row's time index
plus the offset the window was folded with (at_offset for @, offset_ms
otherwise, recorded on PromPlannerContext). The regression is centered on
that instant, matching Prometheus' use of enh.Ts. The mixed
float/native-histogram path forwards the same argument.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(query): tighten @ modifier planner and predict_linear
- range_fold_offset is always a concrete millisecond offset, not an
Option; the single-step replay helper no longer wraps an infallible
plan in Result.
- predict_linear's eval timestamp is always cast to Timestamp(ms) at the
call site, so the UDF drops its dead Int64 branch and the extra
func_name argument.
- Drop two redundant comparison cases from the at_modifier sqlness test
and fix the lookback window comment to the half-open (0s, 300s].
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): accept parse-time rejection of @ on Windows
`@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime
holds it and the planner rejects the anchor it cannot represent, but a
Windows SystemTime tops out near 1.8e12 seconds, so the parser's
checked_add fails first and the same literal is rejected while parsing.
Accept either rejection path so the test passes on both platforms.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): avoid replaying label_join over rewritten series keys
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): narrow anchored range call promotion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): address @ modifier review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): satisfy super import format check
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(query): address at modifier review follow-ups
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL
Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog
thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the
object store WAL config, validated by the datanode and the store.
In the `enqueued` mode `append_batch` returns on admission with the
entry ids assigned and the object is created in the background. At a
backlog threshold the next append is held back until an upload
completes. A transient create failure is repeated under the same
sequence with the same bytes; any conflicting object poisons the
store. `stop` uploads the backlog, or returns the error that dropped
it once stop began. `obsolete` clamps the watermark to the durable
entry id, and an id the store handed out needs no sequence floor.
Add `LogStore::wait_durable` with a default that returns at once. The
object store WAL answers it once the region is durable and indexed
through the entry id, and fails it after a backlog was lost.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): poison an enqueued store on permanent create failures
Repeat a failed create in the enqueued mode only when the storage error
is retryable; any other storage error poisons the store. A conflicting
object poisons an enqueued store without reading its epoch, so a failed
header read cannot turn the conflict into a retry.
A durability wait for an id above the highest id the store handed out
now waits for the handed-out ids of the region below it instead of
returning at once.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): poison on permanent create failures after stop begins
A create that fails with a storage error that is not retryable poisons
an enqueued store even after stop began; only a transient failure drops
the backlog without poisoning. Durability waiters whose callers stopped
waiting are pruned before a new waiter is queued. The backlog age test
no longer depends on a follow-up append finishing within the threshold.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): answer a durability wait once no earlier entry is pending
A durability wait now returns once the region holds no entry at or
below the target that is handed out but not durable, instead of waiting
for the largest id handed out to the region. A later object of the
region that is still being created no longer holds back a wait whose
target it does not cover.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): order the pending durability wait check after the actor
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(store-api): state that the default wait_durable keeps each store's guarantee
The default `LogStore::wait_durable` returns at once, which keeps each
log store's own acknowledgement guarantee; Raft Engine with
`sync_write = false` acknowledges before its periodic sync, so the
documentation no longer claims that every entry id a caller holds is
durable. The object store WAL configuration test now also serializes
the new options and reads them back.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): limit the acknowledgement guarantees to the durable mode
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): repeat enqueued creates that a retry layer marks persistent
An object store wrapped in the OpenDAL retry layer reports a temporary
error that outlasted its retries as persistent rather than temporary.
The enqueued mode now repeats a create after any storage error that is
not permanent, so a transient outage behind the retry layer no longer
poisons the store and drops the acknowledged backlog; after stop began
such a failure still drops the backlog without poisoning.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): cover a persistent create failure after stop begins
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Lei, HUANG
Weny Xu
github-actions[bot]andgreptimedb-ci
fys
Ning Sun
dennis zhuang
Aether
jeremyhi
shuiyisong
yihong
Yingwen
Dragon RoarandDennis Zhuang
discord9
DeviousCardiandClaude Opus 5.5
LFCandgreptimedb-ci
Mohd Quamar Tyagi
