* test(recordbatch): compare float zero predicates with DataFusion
`SimpleFilterEvaluator` filters SST rows exactly, so it has to agree
with the SQL predicate that DataFusion evaluates above the scan.
DataFusion compares -0.0 and +0.0 as equal. Evaluate comparisons
against both zeros, NaN and NULL in Float64 and Float32 columns, and
check the selected rows against DataFusion's filter.
On main this fails: `v = Float64(0)` selects only the +0.0 row, while
DataFusion selects both zeros.
Signed-off-by: dayan02dev <pawarudayan92@gmail.com>
* fix(recordbatch): compare -0.0 and +0.0 as equal in simple filters
DataFusion 55 normalizes -0.0 to +0.0 before Arrow's comparison
kernels, so SQL compares the two zeros as equal. SimpleFilterEvaluator
called the kernels directly, and for floats they compare bit patterns:
-0.0 != +0.0 and -0.0 < +0.0. Mito applies the evaluator to SST rows as
an exact filter, so after a flush `v = 0.0`, `v >= 0` and `v IN (0.0)`
dropped the rows holding -0.0, and `v = -0.0` dropped the rows holding
+0.0. Rows still in a memtable matched.
Normalize the literal when the predicate is built and the input before
it is evaluated, with the helpers DataFusion uses for the same purpose.
Signed-off-by: dayan02dev <pawarudayan92@gmail.com>
* test(sqlness): cover flushed float rows holding -0.0
Flush rows holding -0.0, 0.0 and 1.0, then filter with `= 0.0`,
`>= 0`, `= -0.0` and `IN (0.0)`. Both zeros have to match every one of
them, as they do while the rows are still in the memtable.
Signed-off-by: dayan02dev <pawarudayan92@gmail.com>
---------
Signed-off-by: dayan02dev <pawarudayan92@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions
`iceberg_hash` partition rules could not be evaluated from SQL, so users had no
way to measure a hash distribution or check which bucket a value falls into.
Expose the same UDF that partition rules plan, plus `iceberg_hash_bounds(n)`
which prints the equal-width breakpoints `bi = i32::MIN + floor(i * 2^32 / n)`
as text for copy-paste into `PARTITION ON COLUMNS`.
Both functions are immutable and read-only, and no persisted format changes.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(function): handle untyped NULL and small integer counts
`iceberg_hash(NULL)` arrived as ScalarValue::Null, which becomes a NullArray
that the shared array form had no arm for, so it failed with an unsupported-type
error instead of hashing to NULL. Add a DataType::Null arm in the shared
implementation, so both the SQL function and any future rule evaluation keep the
advertised NULL-to-NULL behavior.
`iceberg_hash_bounds` rejected Int8/Int16/UInt8/UInt16 constants even when the
value was a valid bucket count, because Signature::any leaves the argument width
unchanged. Accept every integer width like the existing Int32/Int64/UInt32/UInt64
arms, and cover them plus the untyped NULL case in unit and SQL tests.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(function): add iceberg_hash_breakpoints SQL function
Equal-width `iceberg_hash_bounds(n)` buckets cannot follow a key distribution:
a handful of hot keys can still land in one bucket, and a uniform split keeps
hashing them together. Add `iceberg_hash_breakpoints(v1, v2, ...)`, which takes
any number of constant values in any order (one shared hashable type, exactly
the values a partition rule compares the hashed column against), hashes them
through the shared implementation, and prints them sorted and deduplicated:
SELECT iceberg_hash_breakpoints('iceberg', 34);
-- 1210000089, 2017239379
The output drops straight into the breakpoint list of a `PARTITION ON COLUMNS`
rule, so buckets can be placed around known key values instead of equal widths.
Columns and NULLs are rejected with explicit errors: a column cannot be hashed
at planning time, and a NULL hash never matches a comparison bound, so a NULL
value cannot be a breakpoint (NULL rows always fall in the lowest bucket).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(function): add iceberg_hash_ranges_sql SQL function
The helpers so far print bare breakpoint numbers, so a user still had to expand
them into comparison rules by hand: n buckets mean n rules with up to
2n - 1 `iceberg_hash` call sites. `iceberg_hash_ranges_sql(column, n)` renders
the whole list from the same equal-width breakpoints:
SELECT iceberg_hash_ranges_sql('host', 2);
-- iceberg_hash(host) < 0,
-- iceberg_hash(host) >= 0
The output drops straight into the expression list of `PARTITION ON COLUMNS`
(rule per line). It only renders text; it does not create or change partitions.
The breakpoints come from the shared `IcebergHashBoundsFunction` logic, so the
rendered rules and `iceberg_hash_bounds(n)` always agree. NULL name or bucket
count propagates as NULL; columns, non-string names, empty names, out-of-range
counts and wrong arity are rejected with explicit errors.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(function): include function identity in ScalarUdf equality
The ScalarUdf wrapper compared and hashed only the signature, so any two
functions sharing a signature were the same ScalarUDF: isnull, iceberg_hash
and iceberg_hash_bounds all have Signature::any(1, Immutable) and compared
equal. DataFusion compares expressions structurally, and its SQL planner
rewrites SELECT expressions against GROUP BY keys with base_exprs.contains
(rebase_expr); the collision made SELECT isnull(x) FROM t GROUP BY x,
iceberg_hash(x) fail planning, because isnull(x) matched the grouped
iceberg_hash(x) and was rewritten into an aggregate column that does not
exist.
Compare and hash the function name alongside the signature, so functions are
distinguishable while two instances of the same function still compare (and
hash) equal.
Add regressions: same-signature functions stay distinct ScalarUDFs and stay
separable in hash-based containers, and grouping by one while selecting the
other plans and executes in both directions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(function): quote rendered column names and reject mixed hash domains
`iceberg_hash_ranges_sql` interpolated the column argument verbatim, so a
legal name such as `host-name` produced `iceberg_hash(host-name)`, which parses
as a subtraction; names with spaces, dots, quotes or upper case could not be used
either. Quote the name as a SQL identifier and double embedded quotes, so the
rendered rules stay one identifier and survive parsing. Cover the round trip by
feeding the output back into `CREATE TABLE ... PARTITION ON COLUMNS` for plain,
dashed, upper-case, quoted and spaced names.
`iceberg_hash_breakpoints` accepted values of different types, but strings and
integers hash through different encodings (UTF-8 bytes versus Iceberg longs), so
mixing them yields breakpoints that describe no real column. Require one hash
domain (all strings or all integers, integer widths may mix) and name the
offending argument otherwise. The existing SQL case that mixed
`('iceberg', 34)` now uses two strings.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(function): register the raw hash UDF and tidy the helpers
Registering `iceberg_hash` through a `Function` wrapper built a different
`ScalarUDFImpl` type from the one partition rules plan, so DataFusion compared
the SQL expression and the rule expression as unequal; the wrapper also rebuilt
a `ScalarUDF` per batch. `ScalarFunctionFactory` already has `From<ScalarUDF>`,
so register `iceberg_hash_udf()` directly and add a test that the registered
function and `IcebergHash::to_logical_expr` produce equal expressions.
Also: collapse the arity check and slice destructuring into one `let ... else`;
make the bucket-count helpers associated functions instead of constructing a
`default()` instance; pass the caller name through so an error raised from
`iceberg_hash_ranges_sql` names that function rather than
`iceberg_hash_bounds`; and fetch the SQL test's UDFs from `FUNCTION_REGISTRY`
instead of rebuilding and wrapping each one.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(function): accept every string column name and cover the rules end to end
`iceberg_hash_ranges_sql` matched only `Utf8` and `LargeUtf8` for its
column-name argument, so a valid constant string that arrives as
`Utf8View` (for example `arrow_cast('host', 'Utf8View')`) was rejected as
non-string. `Signature::any` leaves argument types unchanged, so the function
has to accept all string representations itself; use `ScalarValue::try_as_str`
and keep the NULL and empty-name checks. A unit case and a SQL-level case pin
that `Utf8View` renders the same rules as a plain literal.
Add a sqlness case that renders the four buckets for the quoted column
`"host-name"`, creates the table from that exact text, inserts one value per
bucket plus NULL, and checks the row counts implied by the rendered predicates
against the rows each region stored. This closes the gap between rendering the
rules and routing rows with them.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(partition): hash constants through IcebergHash::evaluate_array
`iceberg_hash_values` duplicated `IcebergHash::evaluate_array` line for
line. Call the existing method from `into_hashed_scalar` and drop the extra
public entry point, so the SQL helpers still hash through the code a partition
rule evaluates without a second identical API.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
information_schema.tables.create_options (and SHOW TABLE STATUS) printed
object store credentials of external tables verbatim, and SHOW CREATE TABLE
masked only access_key_id and secret_access_key. Mask all four credential
keys in both paths, and sort extra options in create_options by key.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito2): record pending WAL tails in the region manifest
Adds `pending_tails` and `handoff_floors` to the region manifest and its
checkpoint, and an optional `pending_tails` / `handoff_floor` pair to
`RegionEdit`. An edit without them leaves the records unchanged. A floor is
dropped once the flushed entry id reaches it; a tail stays until a commit
removes it. Neither is written when empty, so manifests without them encode
as before.
Opening a region replays its WAL after this node's tails and floor, and WAL
releases at open and after a flush stay below this node's tails. Nothing
writes these records yet, so every region behaves as before.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(mito2): hold back reads and region changes while WAL tails are pending
A region with pending WAL tails is missing data that only those tails hold.
Until they are recovered, scans and memtable and SST stats fail with the
retryable `RegionPendingTails` error. Truncate, entering staging, a
repartition flush, a close-time flush on request, and ALTERs other than
adding nullable columns without a default or changing options are rejected
before they change any region state. Compaction pauses. A full-listing GC
pass skips the region while it has pending tails, or while a newer manifest
may exist or the manifest listing fails.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(mito2): name the WAL releases that keep pending tails
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): check that reopening restores the pending-tail read gate
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): pin what the pending-tail fixtures and rejections promise
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(mito2): name the replay start in the open log and share the pending-tail check
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(mito2): state the per-node order of pending tails and who keeps later writes out of them
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending
When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): drop the pending compaction task state when WAL tails skip it
Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(mito2): add shared SST storage and reclamation foundations
Add shared-level isolation, receipt replay and recovery, migration support, and fail-closed file protection. Keep coordinator submission and verified publication in the next stack layer.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): isolate table compaction manifest semantics
Keep atomic manifest extension envelopes and lifecycle safety in the common engine. Gate table-specific receipts and remap admission with enterprise, preserving their wire format and rejecting unsupported state before replay or GC.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): remove manifest-backed compaction task state
Keep file-only edits, shared-reference admission and GC safety. Task retries no longer require persisted receipts or task epochs; reject unpublished prototype task-state records rather than silently dropping them.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): guard staging manifests and reclaim shared range indexes
Reject prototype task state when decoding staging blobs, matching checkpoint and delta decoding. Delete the region-scoped range index when a shared SST reference is retired, while leaving the shared object to global GC. Document the level-2 merge-split rationale and bind the delta guard's action tag with a test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): simplify GC deletion and lingering rules
Require unknown files in a directory without a manifest to pass the lingering window before deletion, so an unpublished table-compaction output cannot be reclaimed out from under an active attempt. Replace the per-batch deletion loop with a single publication recheck taken before remote deletion, and drop the batching constant, the deferred-deletion accounting, and their test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): keep regions open after manifest validation rejection
Classify InvalidRequest from pre-write applicability checks as definitely not persisted. Cover both shared-removal and unsupported-level rejections without stopping the manifest manager, while keeping ambiguous storage errors conservative.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): reject ambiguous shared SST reclassification
Reject edits that both add a shared SST and remove the same file ID before admission and under the manifest lock. Keep ordinary edit semantics and verify add-only L0 reclassification survives reopen without diverging from the live Version.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): define truncate semantics for shared SST references
Keep explicit truncation applicable to shared references and document the distinction from physical reclamation. Exercise full and time-range truncation with real shared SSTs, an old query, another referencing region, and reopen.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): define shared copy results and reject metadata imports
Document copied_file_ids as installed logical references, including no-op retries. Reject shared SSTs in metadata-region copies before copying any objects or changing the target, and cover shared, mixed and repeated copy behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(mito2): unify shared SST level references and bounds docs
Use module-qualified shared-level constants consistently and document MAX_LEVEL as an exclusive bound for SST version operations. No runtime behavior changes.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): move the shared SST level to a max sentinel
Use the u8 maximum as the shared classification so future ordinary levels can never collide with it, and map wire levels to fixed storage slots instead of indexing by value. Shared metadata, admission and filters keep their semantics; unallocated levels stay rejected.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): keep dropped-region tombstones while files are deferred
Young unknown objects and hook-protected candidates on a dropped region must report needs_retry so the meta tombstone survives until they are actually reclaimed; reporting success would let the procedure drop the tombstone and orphan the retained objects. Cover both deferral paths plus the eventual clear.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): reject adopting an existing L1 SST as shared
Adopting a shared reference is L0-only: the manifest would replace an existing L1 entry while the live version keeps both handles, duplicating rows until reopen. Reject the edit at admission before any state change and cover the rejection with a real SST.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.
Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(datatypes): escape object keys when rendering jsonb as text
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): skip unicode normalization for escaped jsonb keys
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* refactor(datatypes): serialize jsonb to text directly with serde_json
Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that
both object keys and string values are escaped by serde_json. This drops
fix_unicode_point, its regex and the hand-written key escaping. Errors are
propagated instead of falling back to lossy raw bytes.
Characters such as U+2028 are now emitted as-is, which is still valid
JSON, so the affected sqlness result lines are regenerated.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): accept JSON text bytes in jsonb_to_string
Some write paths (COPY FROM csv/json/parquet, JSON bound as text via
MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode
such bytes as jsonb before serializing with serde_json, so they still
render (properly escaped) instead of failing with InvalidJsonb.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
---------
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(promql): collapse duplicate samples at one timestamp before range functions
A read path that does not deduplicate, such as an append-mode region, can
hand PromQL two rows for one series at one timestamp. Every range function
reads its samples from RangeManipulate, so the duplicate entries were
treated as independent samples: count_over_time counted both, idelta and
irate returned NULL because the last two entries shared a timestamp, and
delta/increase/rate deflated because the average sampling interval divides
the observed span by sample_count - 1.
Prometheus sample identity is one sample per series per timestamp, so its
range functions can never see this input. Restore that identity once, at
the operator every range function shares, instead of special-casing each
function's arithmetic.
The collapse compares native ticks, never the millisecond payload, so
distinct sub-millisecond samples that share a millisecond stay distinct as
calculate_range documents and tests. Batches without duplicate timestamps
take a single adjacency scan and are returned untouched; only when
duplicates exist is a take built, keeping tag and value columns aligned.
Keeping the last row of a group matches MergeMode::LastRow.
The normalize test that fed three rows at one timestamp now uses distinct
timestamps: it exercises histogram start-timestamp offsetting, and its
repeated timestamp was incidental, not asserted behavior.
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover duplicate samples end to end and keep the newer value
The operator now keeps the first row of a tied timestamp group, which is the row a
deduplicating read keeps: storage orders equal primary key and timestamp by sequence
descending, and FlatDedupReader takes the first row of each group. The end-to-end case
pins that choice by writing 100 and then 101 at the same timestamp: the window reports
increase 6, while keeping the stale 100 would report 8.
The sqlness case writes duplicate rows into an append_mode region, which is the read
path that does not deduplicate, and asserts every range function plus the unaffected
ordinary table in both the standalone and distributed environments.
Signed-off-by: discord9 <discord9@163.com>
* docs(promql): qualify duplicate-timestamp conflict selection
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): avoid copying tags when collapsing duplicate samples
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* perf(mito2): merge compaction inputs on a separate task
Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.
The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.
One output of 4.8M rows took 1.28s instead of 1.65s.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): parallelize decoding and encoding of compaction outputs
Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.
The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.
The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): cancel compaction producers before consumer polling
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): support per-table compaction output parallelism
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoding tasks before returning failures
Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.
Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain encoders on SST write errors
Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.
Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): drain compaction outputs on cancellation
Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.
Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
fix(query): pin DataFusion cast constraint propagation fix
Update the pinned GreptimeTeam/datafusion revision from e3221b8 to
cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger
regression test).
CastExpr::propagate_constraints used to cast a filter's output interval
straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton
output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so
FilterExec advertised k as constant, the equivalence classes dropped the
ordering required by ORDER BY k, and rows returned in scan order.
The sqlness case keeps every #9600 assertion and the explicit ORDER BY,
and adds ascending and descending checks for the partitioned table, the
control table and a table whose matching rows are stored in neither
ascending nor descending k order.
Cargo.lock only rewrites the 35 datafusion source lines.
Signed-off-by: discord9 <discord9@163.com>
* test: cover windowed sort cast unit mismatch
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit- or timezone-changing time casts
A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.
The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): stop windowed sort on unit-changing timestamp wrappers
to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.
The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: regen windowed sort sqlness after wrapper guard
The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): simplify windowed sort cast regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(partition): support iceberg_hash in partition rules
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): reject partition column type changes and quote rule identifiers
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(table): keep time index widening for partition columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(partition): replace PartitionFunction with IcebergHash
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(partition): describe iceberg_hash in the function module
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: quote partition rule identifiers in SHOW CREATE TABLE expectations
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(partition): check hash rules in information_schema and repartition target columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): coerce hash partition rules against the evaluation schema
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): keep quotes in partition column names when building filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: Support procedure_state() using the last procedure ID
Signed-off-by: hahaha <a310608318@gmail.com>
* feat: Fix and add tests according to review comments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: the zero-argument signature in ADMIN swallows explicit arguments
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: fix the CI failures caused by SQL version issues
Signed-off-by: hahaha <a310608318@gmail.com>
* fix: apply the changes based on the review
Signed-off-by: hahaha <a310608318@gmail.com>
---------
Signed-off-by: hahaha <a310608318@gmail.com>
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito): return from manual index builds only after the index is published
A manual build_index could report success while the index was missing:
the request was rejected or coalesced behind another build, the manifest
commit failed, the indexer dropped an error, or the region was staging.
Manual builds now wait for active builds of the same SST, keep their
waiters across coalescing and schema-change retries, are notified after
the worker applies the edit, and verify the files against the manifest
before returning.
The consistency check also honors inverted_index.ignore_column_ids and a
zero segment row count, so such SSTs are no longer rebuilt on every call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): cover manual index build waiting for a schema-change retry
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): select manual index build files by the applied version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): log why index builds are skipped or aborted
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito): split IndexBuildTask::index_build into phases
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): stop scheduling index builds when the region state changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito): document region file id and silent aborts in index builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): log why a manual index build is incomplete
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito): expose incomplete index builds and treat compacted files as done
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito): pin index build concurrency in the compaction test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mysql): cap timestamp output at microsecond precision
Truncate MySQL text timestamp results to microseconds while preserving the existing second and millisecond formatting. Keep stored nanoseconds and explicit integer/string representations unchanged.
Fixes#9828
Signed-off-by: luofucong <luofc@foxmail.com>
* chore: ignore local output directory
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>