* test: cover windowed sort cast unit mismatch
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit- or timezone-changing time casts
A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.
The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit-changing timestamp wrappers
to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.
The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: regen windowed sort sqlness after wrapper guard
The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): simplify windowed sort cast regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(partition): support iceberg_hash in partition rules
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): reject partition column type changes and quote rule identifiers
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(table): keep time index widening for partition columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(partition): replace PartitionFunction with IcebergHash
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(partition): describe iceberg_hash in the function module
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: quote partition rule identifiers in SHOW CREATE TABLE expectations
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(partition): check hash rules in information_schema and repartition target columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): coerce hash partition rules against the evaluation schema
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): keep quotes in partition column names when building filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: Support procedure_state() using the last procedure ID
Signed-off-by: hahaha <a310608318@gmail.com>
* feat: Fix and add tests according to review comments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: the zero-argument signature in ADMIN swallows explicit arguments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: fix the CI failures caused by SQL version issues
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: apply the changes based on the review
Signed-off-by: hahaha <a310608318@gmail.com>
---------
Signed-off-by: hahaha <a310608318@gmail.com>
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito): return from manual index builds only after the index is published
A manual build_index could report success while the index was missing:
the request was rejected or coalesced behind another build, the manifest
commit failed, the indexer dropped an error, or the region was staging.
Manual builds now wait for active builds of the same SST, keep their
waiters across coalescing and schema-change retries, are notified after
the worker applies the edit, and verify the files against the manifest
before returning.
The consistency check also honors inverted_index.ignore_column_ids and a
zero segment row count, so such SSTs are no longer rebuilt on every call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): cover manual index build waiting for a schema-change retry
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): select manual index build files by the applied version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): log why index builds are skipped or aborted
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito): split IndexBuildTask::index_build into phases
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): stop scheduling index builds when the region state changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): document region file id and silent aborts in index builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): log why a manual index build is incomplete
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): expose incomplete index builds and treat compacted files as done
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): pin index build concurrency in the compaction test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mysql): cap timestamp output at microsecond precision
Truncate MySQL text timestamp results to microseconds while preserving the existing second and millisecond formatting. Keep stored nanoseconds and explicit integer/string representations unchanged.
Fixes#9828
Signed-off-by: luofucong <luofc@foxmail.com>
* chore: ignore local output directory
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
* fix(partition): coerce partition exprs against the batch schema before routing
Batch partition routing (MultiDimPartitionRule::split_record_batch) builds
physical expressions via PartitionExpr::try_as_physical_expr, which did no
type coercion. After widening the time index unit (e.g. ms -> ns via
ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition
bounds keep their original unit, so evaluating the bounds against the batch
failed with:
Invalid comparison operation: Timestamp(ns) < Timestamp(ms)
Row inserts were unaffected because find_region compares Values and
Timestamp::cmp converts units; scans were unaffected because
PredicateGroup::new coerces the region partition expr against the region
schema first.
Coerce the logical partition expression against the evaluation schema in
try_as_physical_expr before creating the physical expression, the same way
PredicateGroup does. This also covers the region partition filter built for
SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this
conversion.
Fixes#9747
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: regenerate change_col_type result after comment update
The ts_widen_part comment in change_col_type.sql was reworded after the
result file was last generated; sqlness echoes SQL comments into the
result, so the stale comment made the case fail. Regenerate it.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(query): forward scalar subquery filters to region scans
Since DataFusion 55, uncorrelated scalar subqueries run in ScalarSubqueryExec
instead of being rewritten into joins, so filters such as
ts = (SELECT ts ... ORDER BY ts DESC LIMIT 1) no longer reach region scans
through a join dynamic filter. MergeScanExec now captures such filters as
remote dynamic filters, keeps them in the parent FilterExec, and replaces the
subquery with its result before querying regions.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): check the remote dyn filter budget across pushdown phases
Filters captured in the Pre and Post phases are sent in one request, but each
phase validated the initial registration budget alone. When their sum exceeded
the budget, serialization dropped every registration, including join and TopK
filters. Validate the combined set and keep filters accepted by the other phase.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): keep IN-bearing scalar subquery filters at the parent
Oversized remote dyn filter payloads replace IN lists with true. That only
widens a conjunction of bounds, but inside an arbitrary predicate such as
(host IN (...)) = (SELECT flag ...) it can drop rows the parent filter would
keep. Do not forward scalar subquery filters that contain IN lists.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(query): note the IN fallback restriction for scalar subquery filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(query): document the scalar subquery dyn filter bridge removal condition
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): keep only dyn_filters of region scans in the scalar subquery plan case
#9565 added scan diagnostics to EXPLAIN ANALYZE VERBOSE and partition filter
rendering changed; the case only checks the forwarded dyn filter.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): keep equality two-valued for NULL rows and accept unsigned -0
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): treat NULL masks as false in batch routing and keep comparisons bare
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(compat): update partition filter in analyze verbose expected output
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(meta): validate persisted partition bounds before altering regions
A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.
Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.
Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.
Fixes#9454
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix: access ColumnSchema fields directly and use plain format args
ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: access ColumnSchema data_type field in alter table tests
Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop borrows now that data_type is read as a field
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop useless format! in nested expression test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: cover the partition tail in change_col_type sqlness case
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop spurious blank line in change_col_type result
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): compare partition bounds with the scan comparison coercion
The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.
Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: update the sqlness fixture for the new incompatibility message
The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency
common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock
file was not refreshed, so cargo check --locked refused to run on CI.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): reject every non-widening type change of a partitioned column
The region routes rows on write through `MultiDimPartitionRule::find_region`,
which compares the persisted bound against row values directly: `Value::cmp`
panics on mismatched variants and equality misroutes silently. After a column
partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT
compares `Value::Int32` rows against the persisted `Value::Int64` bound and
panics in the frontend, even though the scan-side coercion this check mirrors
still works.
Reject any type change of a column referenced by a persisted partition
expression unless it is a timestamp unit widening, which keeps the bound and
the rows comparable because `Timestamp` comparisons convert units first; the
widening keeps the scan cast check unchanged.
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(promql): honor @ modifier on subqueries
`prom_subquery_expr_to_plan` never read the `at` field of a subquery, so
`sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and
followed the outer evaluation grid.
Handle `@` the way Prometheus does: fold the anchor into the subquery
offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier`
does), evaluate the subquery once at the start of the evaluation, and
replay the folded window over the outer grid, reusing the anchored
range selector machinery. A call whose range argument is an anchored
subquery is promoted as step-invariant, like one over an anchored range
selector. `@ start()`/`@ end()` and `offset` combine as for selectors,
and an unrepresentable anchor is rejected with the existing error.
Closes#9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* test(promql): add sqlness cases for @ on subqueries
Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`,
`@` combined with `offset` in either order, an instant query away from
the anchor, `*_over_time` functions and `rate` over an anchored
subquery, a computed inner expression, and an aggregation above the
anchored call. Two series per table so a dropped or merged series
shows up as a wrong row.
Refs #9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
---------
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark
Extend the object store WAL S3 benchmark beyond one load per configuration:
- Inputs for a full image reference, the configurations to run (R0 adds Raft
Engine with the standalone default sync_write = false), extra environment
for every server container, a restart after each load, and a JSON list of
named cell groups that override those inputs.
- Restart cells SIGKILL the loaded server, restart it on the same prefix and
data home, and time /health, the first correct count and the log store
recovery; min_objects fails a cell that left too few objects indexed.
- Crash groups write striped 1000-row batches from several writers, kill the
server at a random point, restart it and check every acknowledged batch
against the table (durable: present exactly once; enqueued: no duplicate,
no partial batch, losses counted). A pinned variant keeps one row of a
second table unflushed and samples the indexed objects while it runs.
- Idle windows scrape the server before and after a quiet period.
- A load group may split the loaded table into equal event_time ranges, so
every batch writes to every region, and checks that each region took rows.
- Drop groups write a table, drop it, kill and restart the server, then
write and flush another table and wait for the dropped table's objects to
be collected.
- The summary adds WAL traffic, objects per second, mean object size,
entries and appends per object, S3 requests per operation, server and host
CPU, the loader's own request latency, rows per region, crash cycles, and
each restart's replay per region from the server log.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: report the object sequences a replay spans instead of objects replayed
The width of a region's replayed entry range counts every object sequence in it, including objects that hold no entry of the region and are skipped, so the summary labels it as a span.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a crash cycle that loses a batch an earlier restart recovered
The oracle accepted every missing acknowledged batch in the enqueued mode,
so a batch that one restart recovered and a later restart no longer found
passed. Every cycle keeps what its restart found, and a cycle now fails in
either mode unless every batch an earlier restart recovered is still
present, so only the current cycle's backlog that was not durable at its
kill may be missing.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: keep the pinned WAL pinned and label OpenDAL operations as such
The pin table inherited the engine's 10-minute auto flush, so a longer
pinned cycle could flush its row and release the WAL before the kill. It now
sets its own auto_flush_interval of one day, longer than any crash group can
run, and a pinned cycle fails unless its indexed objects never fall while
it runs.
The summary's request counts come from opendal_operation_duration_seconds,
which counts OpenDAL calls rather than HTTP requests (a paginated list or a
multipart write is one operation), so the fields and columns now say S3
operations.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check a pinned cycle's retention by the collected-object counter
Net growth of the indexed objects can hide collection, and short cycles take no periodic sample, so a pinned cycle now fails unless deleted_objects_total is the same in the scrape before its writers start and in the one before the kill, both of the same server process. The operation columns name OpenDAL.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let a pinned cell collect only the objects below its pin
The first write after a start lets the store collect the empty start object asynchronously, so a deletion after a cycle's baseline does not mean the pin let go. A pinned cell now scrapes the server before it writes the pin row and fails only once its servers together have collected more objects than were indexed then, counting the first server from that scrape and every restarted one from its start.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: test that a pinned cell's allowance spans its server restarts
Two servers that each collect fewer objects than lie below the pin but more together now fail the cell in the second cycle.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat: add column encoding option and set it for trace span ids
COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'.
Either disables the dictionary for the column; delta_binary_packed is
limited to integer and timestamp columns.
OTLP trace v1/v2 tables now create span_id and parent_span_id with plain
encoding and 64KiB pages, and timestamp_end with delta_binary_packed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add compatibility cases for column encoding options
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: move column encoding downgrade checks into downgrade_compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): keep state column names unique when splitting aggregates
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): probe allocated names when renaming colliding state columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
RegionScanExec reported null_count = 0 and distinct_count = total_rows for
every column of an append-only table. DataFusion answers count(col) and
count(DISTINCT col) from these statistics, so both returned the row count.
Keep the exact row count and an exact zero null count for non-nullable
columns, so count(*) is still answered from statistics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
On append-mode tables a region reports an exact row count. When it is
zero, DataFusion's AggregateStatistics replaces the region-side
__min_state/__max_state aggregate with a non-null struct literal, so the
remote stream advertises the state column as non-nullable. The MergeScan
schema check required equal nullability and failed the query.
Accept a remote field that is non-nullable where the expected field is
nullable, matching DataFusion's own physical-plan schema invariant.
Batches are already rebuilt with the expected schema. Widening is still
rejected, and nested field nullability is still compared exactly as part
of the data type.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(log-store): persist object store WAL region releases
A region's release lived only in memory: a restart after DROP TABLE and
before the next object never re-established the dropped region's
watermark, so its segments counted as live and pinned garbage collection
of the whole prefix.
A release now writes a release record (region id and the entry id the
region is released through) into an object sealed at once, and returns
once that object is durable and indexed, in both acknowledgement modes.
Recovery seeds the obsolete watermarks from the release records of the
indexed objects before it computes any reclaim boundary. obsolete_all
releases in full; the new LogStore::release_region, a no-op by default,
releases a dropped region through the largest entry id the store handed
out for it, and Mito calls it after obsolete when it drops a region.
Objects with release records use format version 2, the version 1 layout
with the records at the end of the footer; every other object is still
written as version 1. A counter of release records made durable is added.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe persisted object store WAL region releases
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): carry a failed region release into the next object
A release whose batch failed left its record only in an orphan object,
so a drop that failed there could still leave the region unreleased
after a restart. The store now keeps the release records of failed
batches and seals them with its next batch, on a timer tick when no
entry is admitted, so the release becomes durable once a create
succeeds again.
The version 1 and version 2 object fixtures are frozen under
tests/compatibility/wal/object_store and loaded by the format tests,
and the RFC no longer says that state is persisted only in headers.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): complete a region drop whose WAL release fails
A drop request is not sent again once the region was removed, so a drop
whose WAL release failed used to leave the region half dropped. The
drop now completes, cleaning up the region's runtime state and
scheduling its files for deletion, and reports the release failure.
In the durable mode the object store WAL now creates one more object at
stop with the release records of failed and dropped batches, so a
graceful stop does not lose a release that failed. The release record
counter is described as counting the records of indexed objects.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): describe the stop-time release create of the WAL actor
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): state the accepted window of unreleased dropped regions
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): describe release waiters in the WAL actor's outcome table
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): fail region releases while the object store WAL is failing
A failing store keeps its backlog and rejects appends until a create
succeeds, so a release would otherwise wait for an upload that is not
coming. A release that arrives in that state now fails at once with the
retryable error, and one whose batch is still pending when the store
enters it fails as well. Neither poisons the store: the record of the
first waits for the next sealed batch, like that of a batch that failed,
and the record of the second stays in the backlog, so both become durable
once a create succeeds.
Rejected releases count in rejected_appends_total.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe region releases while the object store WAL is failing
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): place the drop crash window in the drop procedure
The crash window between the deletion of a table's metadata and a
durable release record is a gap of the drop procedure, not of the WAL.
Describe it as far as the code shows, name the remedy that belongs in
the procedure, cleaning up offline a region a resumed region drop finds
not open, and list it under future work.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): keep rejected release records coalesced while failing
A release rejected while the store is failing kept its record for the
next sealed batch, but every timer tick then sealed that record alone
behind the refused backlog, so a release repeated across ticks grew the
backlog without bound. A failing store now keeps such records coalesced
and seals them on the first tick after a create succeeds.
The fault-matrix case of a release while the backlog is refused now runs
with a 10 ms flush interval, repeats the release, and checks that the
backlog does not grow and that recovery reads the record back.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): keep rejected release records unsealed while failing
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): cite the drop procedure issue for the crash window
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): let a stop seal rejected release records of a failing store
Keeping the records of rejected releases out of sealed batches while the
store is failing also kept them out of the seal a stop requests, so a
stop whose last attempt of the backlog succeeded dropped them and the
released region pinned collection after the restart. A stop now seals
them with the backlog.
A fault-matrix case rejects a release while the store is failing, stops,
lets both creates succeed, and checks the record after a reopen.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe the stop of a failing store with rejected releases
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): keep release records readable by older object store WAL readers
Release records were written as a version 2 footer, which a build without
it refuses to read, so rolling a node back after a drop failed to open
the whole WAL prefix. They are now the single entry of a segment of the
reserved region u64::MAX at the first position of a version 1 object:
an older reader indexes that segment as a region it never opens, and
recovery reads the segment back with one range read per object that
holds one. The segment never holds the reclaim boundary and names no
namespace.
A durable stop whose last create of release records fails now reports
the error instead of returning Ok and losing the records silently.
The tests are trimmed: the release unit tests and fault-matrix cases
that repeated each other are merged, and the Mito drop tests share one
parameterized case.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe release records as a segment of a reserved region
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): report release records a poisoning stop drops
A durable stop whose last create of release records met a conflict
poisoned the store and returned Ok, losing the records silently. Stop
now reports the error whenever release records are dropped after it
began, by a rollback or by poisoning. The test of a failed release
covers that conflict, and seals its record on a tick after a one-shot
create failure, so the next tick cannot fail too.
The documentation now says that recovery reads the segments of release
records, and that one that does not decode fails recovery.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe how recovery reads segments of release records
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): say that stop reports release records a conflict drops
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): restore the transiently failing release create in enqueued mode
The fault matrix again covers a release whose create fails transiently in
the enqueued mode: the release waits for the repeat under the same
sequence and returns once it succeeds. The encode_batch doc now says the
release segment is the last segment of the object.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): place the release segment and count the restored case
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): do not fail a region drop on a failed WAL release
Returning the release error from the drop left a metric physical region
half dropped: the metric engine stops after the data region and never
drops the metadata region or forgets the physical region, and a retried
drop finds the data region gone and counts it as done. The release error
gave the caller nothing to retry, since the region is already removed
and the log store keeps the record for its next object. The drop now
logs the error and succeeds.
A metric engine test drops a physical region while the WAL releases of
both mito regions fail, and checks that both regions and the physical
region state are gone.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): a failed release no longer fails the drop
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): do not fail a region drop on a failed WAL obsolete
A store that the release of another region poisoned fails obsolete as
well, which left the metric engine with its metadata region undropped.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(metric-engine): drop a physical region whose WAL obsolete or release fails
Cover a failed data release, two failed releases and a conflicting
release that poisons the store, after the regions wrote to the WAL, and
check that a store opened later keeps no segment of either region.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): a failed obsolete does not fail the drop either
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): fail the drop on an obsolete error of other WAL providers
Only the object store WAL is shared by the regions of a store that the
release of one can poison; Raft Engine and Kafka keep failing the drop
when obsolete fails.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): a stopped or poisoned store keeps no failed release
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: align fulltext bloom tokens with matches_term and use them for LIKE
The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.
Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop unused lazy_static from index and avoid typo-flagged test tokens
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name probe extraction with verbs and document CharClass
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: rebuild legacy fulltext bloom indexes and probe lower(col)
The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.
Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover legacy bloom index rewrite by compaction in the compat case
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: log outdated fulltext bloom indexes in the consistency check
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: explain the unrecorded fulltext bloom version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add per-column COMPRESSION option for SST columns
CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N,
page_size = '<size>') on a column. The options are stored in the column
metadata, carried through gRPC column options, shown by SHOW CREATE
TABLE, and applied by the mito SST writer and bulk part encoder as
parquet column properties. Unset fields keep the SST defaults. Nested
types are rejected.
A page size below the dictionary page limit also lowers the column's
dictionary page limit: parquet sizes write batches by the dictionary
budget and keeps that size after falling back to plain encoding, which
would otherwise ignore the smaller data page limit.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: validate column compression options before persistence
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject compression overrides on metric logical tables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: benchmark the object store WAL against S3 from an EC2 runner
Add a workflow_dispatch workflow that loads the o11ybench agent observability
dataset into a fresh GreptimeDB container per cell and compares three WAL
configurations that differ only in [wal]: Raft Engine with sync_write on the
instance store, and the object store WAL in durable and enqueued mode, with
the data files on S3 in the runner's region for all three. Rounds alternate
the configuration order. A summary reports rows per second, D/R and E/R as
the geometric mean over rounds, append acknowledgement p50 and p99, and the
object store WAL counters. A smoke mode runs the same chain on ubuntu-latest
against the MinIO fixture.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scrape WAL metrics on failed loads and keep logic out of the benchmark YAML
Scrape the object store WAL metrics before stopping a cell's server whether
the load succeeded or failed, so failed cells keep their stall, poison and
create failure counters. Build the summary from the plan so that cells which
never ran are listed and leave their ratios undefined. Move budget, region,
runtime image, manifest and credential scan logic from the workflow into the
helper scripts, and drive the cell lifecycle in the tests through fake docker
and curl binaries.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: drop redundant required flags from benchmark inputs
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a benchmark cell whose metrics scrape fails and zero untouched WAL counters
A load that succeeds without its metrics scrape now fails the cell with exit
code 4 and records metrics_scraped in cell.json. The summary reports an
event counter that never registered as zero when the cell is an object store
WAL cell whose scrape succeeded and created objects, and as unknown otherwise.
Planned cells that never ran are read through the same path as recorded ones.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: register the S3 benchmark workflow for dispatch from its branch
A workflow_dispatch workflow can only be dispatched once GitHub has
registered it, which happens on the default branch or when one of its
triggers fires. A push trigger limited to this branch registers it; validate
runs only for workflow_dispatch, so a push run skips every job.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: delete the benchmark S3 prefix one object at a time
boto3 now sends a CRC checksum instead of Content-MD5 with DeleteObjects,
which the MinIO fixture of the smoke run rejects. DeleteObject needs neither
and the prefix of one run holds at most a few thousand objects.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: default the S3 benchmark to c6id.2xlarge
c7id is not offered in the runner region ap-southeast-1, where DescribeInstanceTypes
rejects it. c6id is the compute optimized x86 family with a local NVMe instance
store that the region offers, with the same 8 vCPU and 16 GiB at 2xlarge.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: drop the registration trigger of the S3 benchmark workflow
The workflow is registered now, so dispatches no longer need a push trigger
on its branch.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail an S3 benchmark run on EC2 without a local instance store
Without an instance store the Raft Engine WAL of the R cells silently stayed
on the root EBS volume, so the run measured a different baseline. Only a smoke
run may keep it on the root volume now.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let the instance store check fail its workflow step
The step piped wal-disk into tee without pipefail, so a rejected host still
passed. wal-disk now writes its record itself and the step runs it alone.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(promql): follow IEEE 754 float semantics in operators and aggregations
- Comparisons: Arrow orders floats totally (NaN == NaN, NaN > +Inf); PromQL
comparisons with NaN are false except !=.
- min/max: skip NaN unless the whole group is NaN. DataFusion's grouped
min/max let NaN replace and be replaced by neighbours, so the result
depended on input order.
- topk/bottomk rank NaN last.
- round() rounds ties up and returns NaN for a zero step, like Prometheus.
- sqrt() of a negative number is NaN instead of an error.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round() without a step rounds to integers
The planner passed a zero step for single-argument round(), which now
yields NaN like Prometheus; pass 1 instead. Found by the compatibility harness.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): apply the sqrt guard to builtin functions and register min/max for plan decoding
sqrt resolves through the session's builtin functions, not the UDF branch.
Distributed plans are decoded with the serializer's session, which did not
know prom_min/prom_max.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover NaN and rounding semantics
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep the nested count rewrite for PromQL min/max
count(min(x) by (l)) was rewritten by count_nest_aggr only for the builtin
min/max names. prom_min/prom_max return NULL for the same groups, so the
rewrite stays valid. Update plan snapshots for the IEEE comparison, sqrt,
topk and min/max changes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): register prom_min/prom_max in the function registry
Distributed plans split an aggregation into state and merge steps only for
functions whose state function exists in FUNCTION_REGISTRY; prom_min and
prom_max lived in the promql crate, so min/max stopped being pushed down to
datanodes. Move them to common-function and register them before the state
functions are derived.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: update sqlness results for PromQL float semantics
Plans and column names now show prom_min/prom_max, the NaN guards of
comparisons and sqrt, and the NaN sort key of topk. round() results follow
Prometheus' formula. Distributed min/max keep the state/merge split.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep column names and NULLs in IEEE comparisons and sqrt
Rewriting comparisons as `a op b AND NOT (isnan(a) OR isnan(b))` changed
TQL output column names and turned NULL operands into true for `!=` and
false for the bool forms. Implement comparisons and sqrt as PromQL UDFs
that propagate NULL and report the column name of the expression they
replace; prom_min/prom_max report min/max column names the same way.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: restore column names in sqlness results for PromQL float semantics
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): share the prom_min/prom_max groups accumulator with tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep distinct names for prom_min/prom_max
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): default a stepless subquery to a one-minute step
`x[5m:]` evaluated the inner expression at the query step, which is 1s
for instant queries. Prometheus uses the evaluation interval, one minute
by default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step in a range query
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): return range-vector results for instant queries
An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): keep encode_substrait on a valid range query
A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): plan wrapped range-vector roots and reject a zero subquery step
TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover wrapped range-vector roots and a zero subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): default subquery step and reject range vectors in range queries
A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover root subquery default step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): align root subqueries with subquery_child_window
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): return millisecond timestamps for range-vector results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): keep the time index timezone and name its table in errors
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
discord9
dennis zhuang
Chenglong Zheng
shuiyisong
Weny Xu
jeremyhi
B. Naut
fys
LFC
Ning Sun
Yingwen
fengyue-xve
DeviousCardi
Lei, HUANG
NightG1owZandNightG1owZ