mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
Lei, HUANG
ed62f8ff54
* feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions `iceberg_hash` partition rules could not be evaluated from SQL, so users had no way to measure a hash distribution or check which bucket a value falls into. Expose the same UDF that partition rules plan, plus `iceberg_hash_bounds(n)` which prints the equal-width breakpoints `bi = i32::MIN + floor(i * 2^32 / n)` as text for copy-paste into `PARTITION ON COLUMNS`. Both functions are immutable and read-only, and no persisted format changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): handle untyped NULL and small integer counts `iceberg_hash(NULL)` arrived as ScalarValue::Null, which becomes a NullArray that the shared array form had no arm for, so it failed with an unsupported-type error instead of hashing to NULL. Add a DataType::Null arm in the shared implementation, so both the SQL function and any future rule evaluation keep the advertised NULL-to-NULL behavior. `iceberg_hash_bounds` rejected Int8/Int16/UInt8/UInt16 constants even when the value was a valid bucket count, because Signature::any leaves the argument width unchanged. Accept every integer width like the existing Int32/Int64/UInt32/UInt64 arms, and cover them plus the untyped NULL case in unit and SQL tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(function): add iceberg_hash_breakpoints SQL function Equal-width `iceberg_hash_bounds(n)` buckets cannot follow a key distribution: a handful of hot keys can still land in one bucket, and a uniform split keeps hashing them together. Add `iceberg_hash_breakpoints(v1, v2, ...)`, which takes any number of constant values in any order (one shared hashable type, exactly the values a partition rule compares the hashed column against), hashes them through the shared implementation, and prints them sorted and deduplicated: SELECT iceberg_hash_breakpoints('iceberg', 34); -- 1210000089, 2017239379 The output drops straight into the breakpoint list of a `PARTITION ON COLUMNS` rule, so buckets can be placed around known key values instead of equal widths. Columns and NULLs are rejected with explicit errors: a column cannot be hashed at planning time, and a NULL hash never matches a comparison bound, so a NULL value cannot be a breakpoint (NULL rows always fall in the lowest bucket). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(function): add iceberg_hash_ranges_sql SQL function The helpers so far print bare breakpoint numbers, so a user still had to expand them into comparison rules by hand: n buckets mean n rules with up to 2n - 1 `iceberg_hash` call sites. `iceberg_hash_ranges_sql(column, n)` renders the whole list from the same equal-width breakpoints: SELECT iceberg_hash_ranges_sql('host', 2); -- iceberg_hash(host) < 0, -- iceberg_hash(host) >= 0 The output drops straight into the expression list of `PARTITION ON COLUMNS` (rule per line). It only renders text; it does not create or change partitions. The breakpoints come from the shared `IcebergHashBoundsFunction` logic, so the rendered rules and `iceberg_hash_bounds(n)` always agree. NULL name or bucket count propagates as NULL; columns, non-string names, empty names, out-of-range counts and wrong arity are rejected with explicit errors. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): include function identity in ScalarUdf equality The ScalarUdf wrapper compared and hashed only the signature, so any two functions sharing a signature were the same ScalarUDF: isnull, iceberg_hash and iceberg_hash_bounds all have Signature::any(1, Immutable) and compared equal. DataFusion compares expressions structurally, and its SQL planner rewrites SELECT expressions against GROUP BY keys with base_exprs.contains (rebase_expr); the collision made SELECT isnull(x) FROM t GROUP BY x, iceberg_hash(x) fail planning, because isnull(x) matched the grouped iceberg_hash(x) and was rewritten into an aggregate column that does not exist. Compare and hash the function name alongside the signature, so functions are distinguishable while two instances of the same function still compare (and hash) equal. Add regressions: same-signature functions stay distinct ScalarUDFs and stay separable in hash-based containers, and grouping by one while selecting the other plans and executes in both directions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): quote rendered column names and reject mixed hash domains `iceberg_hash_ranges_sql` interpolated the column argument verbatim, so a legal name such as `host-name` produced `iceberg_hash(host-name)`, which parses as a subtraction; names with spaces, dots, quotes or upper case could not be used either. Quote the name as a SQL identifier and double embedded quotes, so the rendered rules stay one identifier and survive parsing. Cover the round trip by feeding the output back into `CREATE TABLE ... PARTITION ON COLUMNS` for plain, dashed, upper-case, quoted and spaced names. `iceberg_hash_breakpoints` accepted values of different types, but strings and integers hash through different encodings (UTF-8 bytes versus Iceberg longs), so mixing them yields breakpoints that describe no real column. Require one hash domain (all strings or all integers, integer widths may mix) and name the offending argument otherwise. The existing SQL case that mixed `('iceberg', 34)` now uses two strings. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(function): register the raw hash UDF and tidy the helpers Registering `iceberg_hash` through a `Function` wrapper built a different `ScalarUDFImpl` type from the one partition rules plan, so DataFusion compared the SQL expression and the rule expression as unequal; the wrapper also rebuilt a `ScalarUDF` per batch. `ScalarFunctionFactory` already has `From<ScalarUDF>`, so register `iceberg_hash_udf()` directly and add a test that the registered function and `IcebergHash::to_logical_expr` produce equal expressions. Also: collapse the arity check and slice destructuring into one `let ... else`; make the bucket-count helpers associated functions instead of constructing a `default()` instance; pass the caller name through so an error raised from `iceberg_hash_ranges_sql` names that function rather than `iceberg_hash_bounds`; and fetch the SQL test's UDFs from `FUNCTION_REGISTRY` instead of rebuilding and wrapping each one. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): accept every string column name and cover the rules end to end `iceberg_hash_ranges_sql` matched only `Utf8` and `LargeUtf8` for its column-name argument, so a valid constant string that arrives as `Utf8View` (for example `arrow_cast('host', 'Utf8View')`) was rejected as non-string. `Signature::any` leaves argument types unchanged, so the function has to accept all string representations itself; use `ScalarValue::try_as_str` and keep the NULL and empty-name checks. A unit case and a SQL-level case pin that `Utf8View` renders the same rules as a plain literal. Add a sqlness case that renders the four buckets for the quoted column `"host-name"`, creates the table from that exact text, inserts one value per bucket plus NULL, and checks the row counts implied by the rendered predicates against the rows each region stored. This closes the gap between rendering the rules and routing rows with them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(partition): hash constants through IcebergHash::evaluate_array `iceberg_hash_values` duplicated `IcebergHash::evaluate_array` line for line. Call the existing method from `into_hashed_scalar` and drop the extra public entry point, so the SQL helpers still hash through the code a partition rule evaluates without a second identical API. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Sqlness Test
Sqlness manual
Case file
Sqlness has two types of file:
.sql: test input, SQL only.result: expected test output, SQL and its results
.result is the output (execution result) file. If you see .result files is changed,
it means this test gets a different result and indicates it fails. You should
check change logs to solve the problem.
You only need to write test SQL in .sql file, and run the test.
Case organization
The root dir of input cases is tests/cases. It contains several subdirectories stand for different test
modes. E.g., standalone/ contains all the tests to run under greptimedb standalone start mode.
Under the first level of subdirectory (e.g. the cases/standalone), you can organize your cases as you like.
Sqlness walks through every file recursively and runs them.
Kafka WAL
Sqlness supports Kafka WAL. You can either provide a Kafka cluster or let sqlness to start one for you.
To run test with kafka, you need to pass the option -w kafka. If no other options are provided, sqlness will use conf/kafka-cluster.yml to start a Kafka cluster. This requires docker and docker-compose commands in your environment.
Otherwise, you can additionally pass the your existing kafka environment to sqlness with -k option. E.g.:
cargo sqlness bare -w kafka -k localhost:9092
In this case, sqlness will not start its own kafka cluster and the one you provided instead.
Run the test
Unlike other tests, this harness is in a binary target form. You can run it with:
cargo sqlness bare
It automatically finishes the following procedures: compile GreptimeDB, start it, grab tests and feed it to
the server, then collect and compare the results. You only need to check if the .result files are changed.
If not, congratulations, the test is passed 🥳!