Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.
Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(datatypes): escape object keys when rendering jsonb as text
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): skip unicode normalization for escaped jsonb keys
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* refactor(datatypes): serialize jsonb to text directly with serde_json
Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that
both object keys and string values are escaped by serde_json. This drops
fix_unicode_point, its regex and the hand-written key escaping. Errors are
propagated instead of falling back to lossy raw bytes.
Characters such as U+2028 are now emitted as-is, which is still valid
JSON, so the affected sqlness result lines are regenerated.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): accept JSON text bytes in jsonb_to_string
Some write paths (COPY FROM csv/json/parquet, JSON bound as text via
MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode
such bytes as jsonb before serializing with serde_json, so they still
render (properly escaped) instead of failing with InvalidJsonb.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
---------
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(promql): collapse duplicate samples at one timestamp before range functions
A read path that does not deduplicate, such as an append-mode region, can
hand PromQL two rows for one series at one timestamp. Every range function
reads its samples from RangeManipulate, so the duplicate entries were
treated as independent samples: count_over_time counted both, idelta and
irate returned NULL because the last two entries shared a timestamp, and
delta/increase/rate deflated because the average sampling interval divides
the observed span by sample_count - 1.
Prometheus sample identity is one sample per series per timestamp, so its
range functions can never see this input. Restore that identity once, at
the operator every range function shares, instead of special-casing each
function's arithmetic.
The collapse compares native ticks, never the millisecond payload, so
distinct sub-millisecond samples that share a millisecond stay distinct as
calculate_range documents and tests. Batches without duplicate timestamps
take a single adjacency scan and are returned untouched; only when
duplicates exist is a take built, keeping tag and value columns aligned.
Keeping the last row of a group matches MergeMode::LastRow.
The normalize test that fed three rows at one timestamp now uses distinct
timestamps: it exercises histogram start-timestamp offsetting, and its
repeated timestamp was incidental, not asserted behavior.
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover duplicate samples end to end and keep the newer value
The operator now keeps the first row of a tied timestamp group, which is the row a
deduplicating read keeps: storage orders equal primary key and timestamp by sequence
descending, and FlatDedupReader takes the first row of each group. The end-to-end case
pins that choice by writing 100 and then 101 at the same timestamp: the window reports
increase 6, while keeping the stale 100 would report 8.
The sqlness case writes duplicate rows into an append_mode region, which is the read
path that does not deduplicate, and asserts every range function plus the unaffected
ordinary table in both the standalone and distributed environments.
Signed-off-by: discord9 <discord9@163.com>
* docs(promql): qualify duplicate-timestamp conflict selection
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): avoid copying tags when collapsing duplicate samples
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* perf(mito2): merge compaction inputs on a separate task
Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.
The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.
One output of 4.8M rows took 1.28s instead of 1.65s.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): parallelize decoding and encoding of compaction outputs
Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.
The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.
The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): cancel compaction producers before consumer polling
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): support per-table compaction output parallelism
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoding tasks before returning failures
Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.
Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoders on SST write errors
Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.
Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain compaction outputs on cancellation
Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.
Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
fix(query): pin DataFusion cast constraint propagation fix
Update the pinned GreptimeTeam/datafusion revision from e3221b8 to
cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger
regression test).
CastExpr::propagate_constraints used to cast a filter's output interval
straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton
output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so
FilterExec advertised k as constant, the equivalence classes dropped the
ordering required by ORDER BY k, and rows returned in scan order.
The sqlness case keeps every #9600 assertion and the explicit ORDER BY,
and adds ascending and descending checks for the partitioned table, the
control table and a table whose matching rows are stored in neither
ascending nor descending k order.
Cargo.lock only rewrites the 35 datafusion source lines.
Signed-off-by: discord9 <discord9@163.com>
* test: cover windowed sort cast unit mismatch
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit- or timezone-changing time casts
A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.
The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit-changing timestamp wrappers
to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.
The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: regen windowed sort sqlness after wrapper guard
The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): simplify windowed sort cast regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(partition): support iceberg_hash in partition rules
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): reject partition column type changes and quote rule identifiers
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(table): keep time index widening for partition columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(partition): replace PartitionFunction with IcebergHash
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(partition): describe iceberg_hash in the function module
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: quote partition rule identifiers in SHOW CREATE TABLE expectations
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(partition): check hash rules in information_schema and repartition target columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): coerce hash partition rules against the evaluation schema
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): keep quotes in partition column names when building filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: Support procedure_state() using the last procedure ID
Signed-off-by: hahaha <a310608318@gmail.com>
* feat: Fix and add tests according to review comments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: the zero-argument signature in ADMIN swallows explicit arguments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: fix the CI failures caused by SQL version issues
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: apply the changes based on the review
Signed-off-by: hahaha <a310608318@gmail.com>
---------
Signed-off-by: hahaha <a310608318@gmail.com>
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito): return from manual index builds only after the index is published
A manual build_index could report success while the index was missing:
the request was rejected or coalesced behind another build, the manifest
commit failed, the indexer dropped an error, or the region was staging.
Manual builds now wait for active builds of the same SST, keep their
waiters across coalescing and schema-change retries, are notified after
the worker applies the edit, and verify the files against the manifest
before returning.
The consistency check also honors inverted_index.ignore_column_ids and a
zero segment row count, so such SSTs are no longer rebuilt on every call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): cover manual index build waiting for a schema-change retry
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): select manual index build files by the applied version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): log why index builds are skipped or aborted
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito): split IndexBuildTask::index_build into phases
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): stop scheduling index builds when the region state changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): document region file id and silent aborts in index builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): log why a manual index build is incomplete
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): expose incomplete index builds and treat compacted files as done
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): pin index build concurrency in the compaction test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mysql): cap timestamp output at microsecond precision
Truncate MySQL text timestamp results to microseconds while preserving the existing second and millisecond formatting. Keep stored nanoseconds and explicit integer/string representations unchanged.
Fixes#9828
Signed-off-by: luofucong <luofc@foxmail.com>
* chore: ignore local output directory
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
* fix(partition): coerce partition exprs against the batch schema before routing
Batch partition routing (MultiDimPartitionRule::split_record_batch) builds
physical expressions via PartitionExpr::try_as_physical_expr, which did no
type coercion. After widening the time index unit (e.g. ms -> ns via
ALTER TABLE ... MODIFY COLUMN ts TIMESTAMP_NS), the persisted partition
bounds keep their original unit, so evaluating the bounds against the batch
failed with:
Invalid comparison operation: Timestamp(ns) < Timestamp(ms)
Row inserts were unaffected because find_region compares Values and
Timestamp::cmp converts units; scans were unaffected because
PredicateGroup::new coerces the region partition expr against the region
schema first.
Coerce the logical partition expression against the evaluation schema in
try_as_physical_expr before creating the physical expression, the same way
PredicateGroup does. This also covers the region partition filter built for
SST pruning (src/mito2/src/sst/parquet/reader.rs), which shares this
conversion.
Fixes#9747
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: regenerate change_col_type result after comment update
The ts_widen_part comment in change_col_type.sql was reworded after the
result file was last generated; sqlness echoes SQL comments into the
result, so the stale comment made the case fail. Regenerate it.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(query): forward scalar subquery filters to region scans
Since DataFusion 55, uncorrelated scalar subqueries run in ScalarSubqueryExec
instead of being rewritten into joins, so filters such as
ts = (SELECT ts ... ORDER BY ts DESC LIMIT 1) no longer reach region scans
through a join dynamic filter. MergeScanExec now captures such filters as
remote dynamic filters, keeps them in the parent FilterExec, and replaces the
subquery with its result before querying regions.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): check the remote dyn filter budget across pushdown phases
Filters captured in the Pre and Post phases are sent in one request, but each
phase validated the initial registration budget alone. When their sum exceeded
the budget, serialization dropped every registration, including join and TopK
filters. Validate the combined set and keep filters accepted by the other phase.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): keep IN-bearing scalar subquery filters at the parent
Oversized remote dyn filter payloads replace IN lists with true. That only
widens a conjunction of bounds, but inside an arbitrary predicate such as
(host IN (...)) = (SELECT flag ...) it can drop rows the parent filter would
keep. Do not forward scalar subquery filters that contain IN lists.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(query): note the IN fallback restriction for scalar subquery filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(query): document the scalar subquery dyn filter bridge removal condition
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): keep only dyn_filters of region scans in the scalar subquery plan case
#9565 added scan diagnostics to EXPLAIN ANALYZE VERBOSE and partition filter
rendering changed; the case only checks the forwarded dyn filter.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): keep equality two-valued for NULL rows and accept unsigned -0
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): treat NULL masks as false in batch routing and keep comparisons bare
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(compat): update partition filter in analyze verbose expected output
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(meta): validate persisted partition bounds before altering regions
A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.
Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.
Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.
Fixes#9454
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix: access ColumnSchema fields directly and use plain format args
ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: access ColumnSchema data_type field in alter table tests
Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop borrows now that data_type is read as a field
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop useless format! in nested expression test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: cover the partition tail in change_col_type sqlness case
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop spurious blank line in change_col_type result
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): compare partition bounds with the scan comparison coercion
The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.
Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: update the sqlness fixture for the new incompatibility message
The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency
common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock
file was not refreshed, so cargo check --locked refused to run on CI.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): reject every non-widening type change of a partitioned column
The region routes rows on write through `MultiDimPartitionRule::find_region`,
which compares the persisted bound against row values directly: `Value::cmp`
panics on mismatched variants and equality misroutes silently. After a column
partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT
compares `Value::Int32` rows against the persisted `Value::Int64` bound and
panics in the frontend, even though the scan-side coercion this check mirrors
still works.
Reject any type change of a column referenced by a persisted partition
expression unless it is a timestamp unit widening, which keeps the bound and
the rows comparable because `Timestamp` comparisons convert units first; the
widening keeps the scan cast check unchanged.
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(promql): honor @ modifier on subqueries
`prom_subquery_expr_to_plan` never read the `at` field of a subquery, so
`sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and
followed the outer evaluation grid.
Handle `@` the way Prometheus does: fold the anchor into the subquery
offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier`
does), evaluate the subquery once at the start of the evaluation, and
replay the folded window over the outer grid, reusing the anchored
range selector machinery. A call whose range argument is an anchored
subquery is promoted as step-invariant, like one over an anchored range
selector. `@ start()`/`@ end()` and `offset` combine as for selectors,
and an unrepresentable anchor is rejected with the existing error.
Closes#9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* test(promql): add sqlness cases for @ on subqueries
Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`,
`@` combined with `offset` in either order, an instant query away from
the anchor, `*_over_time` functions and `rate` over an anchored
subquery, a computed inner expression, and an aggregation above the
anchored call. Two series per table so a dropped or merged series
shows up as a wrong row.
Refs #9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
---------
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark
Extend the object store WAL S3 benchmark beyond one load per configuration:
- Inputs for a full image reference, the configurations to run (R0 adds Raft
Engine with the standalone default sync_write = false), extra environment
for every server container, a restart after each load, and a JSON list of
named cell groups that override those inputs.
- Restart cells SIGKILL the loaded server, restart it on the same prefix and
data home, and time /health, the first correct count and the log store
recovery; min_objects fails a cell that left too few objects indexed.
- Crash groups write striped 1000-row batches from several writers, kill the
server at a random point, restart it and check every acknowledged batch
against the table (durable: present exactly once; enqueued: no duplicate,
no partial batch, losses counted). A pinned variant keeps one row of a
second table unflushed and samples the indexed objects while it runs.
- Idle windows scrape the server before and after a quiet period.
- A load group may split the loaded table into equal event_time ranges, so
every batch writes to every region, and checks that each region took rows.
- Drop groups write a table, drop it, kill and restart the server, then
write and flush another table and wait for the dropped table's objects to
be collected.
- The summary adds WAL traffic, objects per second, mean object size,
entries and appends per object, S3 requests per operation, server and host
CPU, the loader's own request latency, rows per region, crash cycles, and
each restart's replay per region from the server log.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: report the object sequences a replay spans instead of objects replayed
The width of a region's replayed entry range counts every object sequence in it, including objects that hold no entry of the region and are skipped, so the summary labels it as a span.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a crash cycle that loses a batch an earlier restart recovered
The oracle accepted every missing acknowledged batch in the enqueued mode,
so a batch that one restart recovered and a later restart no longer found
passed. Every cycle keeps what its restart found, and a cycle now fails in
either mode unless every batch an earlier restart recovered is still
present, so only the current cycle's backlog that was not durable at its
kill may be missing.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: keep the pinned WAL pinned and label OpenDAL operations as such
The pin table inherited the engine's 10-minute auto flush, so a longer
pinned cycle could flush its row and release the WAL before the kill. It now
sets its own auto_flush_interval of one day, longer than any crash group can
run, and a pinned cycle fails unless its indexed objects never fall while
it runs.
The summary's request counts come from opendal_operation_duration_seconds,
which counts OpenDAL calls rather than HTTP requests (a paginated list or a
multipart write is one operation), so the fields and columns now say S3
operations.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check a pinned cycle's retention by the collected-object counter
Net growth of the indexed objects can hide collection, and short cycles take no periodic sample, so a pinned cycle now fails unless deleted_objects_total is the same in the scrape before its writers start and in the one before the kill, both of the same server process. The operation columns name OpenDAL.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let a pinned cell collect only the objects below its pin
The first write after a start lets the store collect the empty start object asynchronously, so a deletion after a cycle's baseline does not mean the pin let go. A pinned cell now scrapes the server before it writes the pin row and fails only once its servers together have collected more objects than were indexed then, counting the first server from that scrape and every restarted one from its start.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: test that a pinned cell's allowance spans its server restarts
Two servers that each collect fewer objects than lie below the pin but more together now fail the cell in the second cycle.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat: add column encoding option and set it for trace span ids
COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'.
Either disables the dictionary for the column; delta_binary_packed is
limited to integer and timestamp columns.
OTLP trace v1/v2 tables now create span_id and parent_span_id with plain
encoding and 64KiB pages, and timestamp_end with delta_binary_packed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add compatibility cases for column encoding options
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: move column encoding downgrade checks into downgrade_compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): keep state column names unique when splitting aggregates
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): probe allocated names when renaming colliding state columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
WenyXu
Yvan Wang
shuiyisong
discord9
fys
Dragon Roar
Lei, HUANG
dennis zhuang
Chenglong Zheng
jeremyhi
B. Naut
Ning Sun
Yingwen
fengyue-xve
DeviousCardi