mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-17 01:12:27 +00:00
stack/pr9412-capability
93
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
d59725b04a |
test(sqlness): run environments concurrently and drop redundant restarts and sleeps (#9333)
* test(sqlness): run environments concurrently with external backends The runner forced both environment and instance parallelism to 1 whenever etcd/PG/MySQL or an external Kafka was set up, so the standalone and distributed environments ran one after the other. Only distributed uses the kv backend, and the two environments use different Kafka topic prefixes, so they can run at the same time. Keep one instance per environment, since instances would share the metadata table and Kafka topics. Runs with a runner-managed Kafka, an external server, a test filter, or `-j 1` stay fully serial. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): drop redundant restarts and sleeps - flow_rebuild: drop the R2 block, which repeats R1 from the same state, and the restart in R5, whose result R1 already asserts. - flow_advance_ttl, flow_view, ttl_instant: drop sleeps that no assertion depends on. - flow_basic, flow_null, flow_call_df_func: drop the bytes_log section (covered by flow_insert), a duplicate state_size query, an unchecked insert, and flushes that run with no new data. - alter_table_options, skip_wal: share restarts between independent tables. - session_skip_wal, copy_skip_wal: check every case after a single restart. Tables copied or inserted with skip_wal = true first get a WAL row that is truncated, so the check that truncated WAL entries are not replayed stays. - region_statistics: wait once for the statistics of all three tables. - build_index_table: fold its index_size checks into build_index_table_restart, which runs the same fixture and waits. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore session_skip_wal and copy_skip_wal Checking every case after a single restart dropped two state transitions the original cases cover: recovering a region whose memtable only held skipped rows, and writing to the recovered region before restarting again. Restore the original cases. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore the one-restart check in flow_auto_sink_table #5112 checked the auto-created sink before a restart and the flow and sink after it. #5987 moved the restart in front of the first SHOW and added a second one. Flow recovery only creates the sink when it is missing, so the second restart recovers from the same persisted state as the first. Restore the original before/after check with one restart. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): keep one instance per environment with external store addresses Instances of the distributed environment share the etcd behind --store-addrs, so run one instance per environment when it is set, same as --setup-etcd. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore the false-branch batch in flow_basic and assert it The kept batch (20, 22) is all above the threshold. The second batch (10, 23) is the only input that hits the false branch of the CASE and makes the flow compute again. Restore it and check the result, which the original case never did. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
  discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
 Lei, HUANG
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
6f1dd0cb76 |
feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN Previously MODIFY COLUMN rejected the time index column outright. Now the time index unit can be widened (Second -> Milli -> Micro -> Nano), which is lossless for data that fits the target unit: historical data in old SSTs is cast to the new unit on read by the existing schema-compat layer, and compaction rewrites it lazily. Narrowing and non-timestamp targets remain rejected; tag columns keep being rejected. Widening is rejected if any SST's time range would overflow the target unit's i64 range (e.g. millisecond -> nanosecond beyond year 2262), since the cast would silently null those values. Read-path correctness for old-unit SSTs (verified by new engine e2e tests and sqlness WHERE queries): - row-group min/max pruning: parquet statistics of a timestamp column are raw integers in the file's unit; when the region metadata's type differs (also the case for altered field columns), stats are now interpreted in the file's type and converted to the expected type before pruning. Without this, a new-unit predicate silently pruned whole row groups of old-unit files (wrong results, rows missing). - SST-level simple filters are skipped for columns whose file type differs from the expected type; the predicate is applied by the query layer's residual filter above the region scan. Also drop the stale "timestamp columns cannot change type" debug_assert. - retry idempotency: a same-type ModifyColumnType on the time index validates as a no-op and `need_alter` returns false, so a retried alter procedure (region already altered before the previous attempt failed) converges instead of aborting forever. - add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to - relax ModifyColumnType validation in store-api and table metadata - tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests (flushed SST + new writes + reopen + retry + overflow + predicate scans, cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl. partitioned table and WHERE filters over old-unit data Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve parquet filter issue * test: provide sqlness tests * refactor: drop trivial test cases and shorten comments Review pass over the branch's additions: - datatypes: keep a representative subset of the widening-matrix asserts - table: collapse the three single-branch rejection blocks into one loop - mito2: drop the boundary gt_eq and post-compaction predicate asserts (covered by the exact-filter regression test and sqlness); drop the engine-level gt_eq/lt_eq casts (full operator matrix stays in the cast_timestamp_unit unit tests) - sqlness: drop a bare full scan already covered by the filter above it - shorten function doc comments across datatypes/store-api/table/mito2/ recordbatch to the essential semantics Signed-off-by: Ning Sun <sunning@greptime.com> * fix: drop physical prefilter for columns whose file type differs Follow-up to the review feedback on CompatBatch/prune reader/filter handling for widened time index units. Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext, which builds its physical expression against the FILE's schema while the predicate literals are in the expected (post-alter) unit. Evaluating them against an old-unit SST raised a cross-unit comparison error (Timestamp(ms) >= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates are best-effort pruning hints (the query layer re-applies them above the scan), so drop the prefilter when the column's file type differs from the expected type, mirroring the simple-filter strategy. Verified: Between and a non-rewritten (large) InList on old-unit data no longer error and filter exactly end-to-end (sqlness), and no matching rows are lost at the engine level (engine test). Signed-off-by: Ning Sun <sunning@greptime.com> * test: add direct unit tests for stats cast and prefilter drop The two-step stats cast (reinterpret raw Int64 stats in the file's timestamp type, then rescale to the expected type) and the physical prefilter drop on file/expected type mismatch were only covered end-to-end; add localized unit tests so a regression fails at the exact site: - stats.rs (previously no tests): RowGroupPruningStats min/max over a hand-built RowGroupMetaData — passthrough with no expected metadata, passthrough on same type, and rescale (1000ms -> 1_000_000us, not 1000us) on a widened expected unit - reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter when file and expected types match and drops it on unit mismatch Signed-off-by: Ning Sun <sunning@greptime.com> * fix: tolerate mixed time units range cache key coverage check * docs: flag mixed-unit hazard in the (unwired) series index The series index stores per-series min/max ts as raw Int64 in the unit of the region metadata at write time, and the searcher builds its range predicates from a single per-region metadata. After a time index unit widen, files of one region would carry mixed units, so a per-file unit (or an index rebuild on such alters) is required before this index is wired into scans. Leave notes at both sites. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover mixed-unit compaction for sparse encoding and strict windows Compaction-path audit follow-up. The compat cast and window math were already covered for dense regions; add the two remaining e2e scenarios: - sparse primary key encoding (used by metric-engine physical regions): widening then compacting mixed-unit files rewrites the old-unit time index correctly through the sparse compaction compat path - strict-window manual compaction: each window output trims rows with a predicate built in the region's new unit against an old-unit file; every instant must survive exactly once (no loss, no cross-window duplication), rescaled Also documents the audit finding that Regular ranged (manual) compaction never trims rows: TwcsPicker sets output_time_range to None and the request time range only selects candidate windows. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover time index unit change in FlatCompatBatch directly The compat layer's rescaling of a widened time index was only verified end-to-end; add direct unit tests for both paths: - dense: identical units skip compat entirely; a widened unit rescales the time index column (1000ms -> 1_000_000us, not reinterpreted) while other columns pass through and the output schema matches the expected metadata - compact sparse (the metric-engine compaction path): same rescaling Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: move timestamp unit division into common-time The exact unit division (UnitQuotient + div_mod_units) is time semantics, not filter logic; move it next to TimeUnit in common-time with a compact test covering representable/non-representable values, negative (floor) instants, and quotient overflow. The ScalarValue helpers stay in filter.rs since common-time has no datafusion dependency. Signed-off-by: Ning Sun <sunning@greptime.com> * test: compat rescales a widened time index and fills an added column together The realistic multi-alter sequence (widen at T1, add column at T2, read a T0 SST) exercises cast and default-fill in the same compute_index_and_fields pass; assert both in one output batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preflight time index widening overflow before any region alters Address review feedback on the overflow guard: - preflight: when the frontend operator receives a widening alter on the time index, run an existence scan (ts outside the target unit's i64 range, LIMIT 1, via the query engine so it covers every region of the table in both standalone and distributed modes) BEFORE any DDL task is submitted. A region that fits can no longer commit the new schema while another region rejects the alter with a non-retryable error. File and row-group pruning keep the scan cheap when nothing overflows. The per-region check in mito2 stays as the final guard for data written after the preflight (the remaining race window); without a validate-only wire field (region.proto lives in the external greptime-proto repo) a fully atomic two-phase validate/commit is out of scope here. - fast path: cast_timestamp_unit returns the filter unchanged when the literal is already in the target unit, skipping the div-mod rebuild. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address auto review comments * fix: remove time index widening overflow preflight Overflow needs timestamps beyond the target unit's i64 range (~year 2262 for nanoseconds), which real workloads never write, so the two existence scans before every widening alter are not worth the cost. Region validation already rejects the alter when an SST's time range overflows the target unit; it now logs the rejection (with the offending file) and returns a deterministic client-facing message. Signed-off-by: Ning Sun <sunning@greptime.com> * test: make sqlness test stable * fix: log instead of rejecting time index widening overflow Overflowing values cast to NULL on read but do not otherwise affect reads or writes, so the alter is allowed; the region-level check now only logs (with the offending file) when an SST's time range exceeds the target unit's i64 range. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 Yingwen
|
0a002f4d65 |
feat: support per-region write buffer limits (#8473)
* feat(mito): add per-region write buffer limit Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): add default region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * docs: expand configuration change checklist Signed-off-by: evenyag <realevenyag@gmail.com> * test: cover table write buffer size option Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): refine region write buffer checks Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): keep region-stalled writes queued Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): allow zero region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): validate region state before write stall Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): reject writes beyond region hard limit Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update example Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 sribatsha dash
|
8f5d681e4b |
feat: support ALTER TABLE SET auto_flush_interval (#8403)
* feat: support ALTER TABLE SET auto_flush_interval Closes #8394. Add a new SetRegionOption::AutoFlushInterval variant so that the per-table auto flush interval can be changed on existing tables via 'ALTER TABLE t SET ...', following up the CREATE TABLE path from #8357. - region_request.rs: parse 'auto_flush_interval' with humantime and map it to the new variant. - metadata.rs: persist the value (or remove it, if None) in TableOptions.extra_options using the same humantime string format the engine already expects. - handle_alter.rs: apply the new interval in handle_alter_region_options_fast (no memtable flush needed, same pattern as Ttl) and group the variant with Ttl/Twsc in new_region_options_on_empty_memtable. Tests: - Two unit tests in metadata.rs covering set and unset-to-None. - A new sqlness case alter_auto_flush_interval.sql covering create-then-alter, alter-then-alter, invalid duration, and alter on a table that already had auto_flush_interval at create time. Signed-off-by: srivtx <crypticcc101@gmail.com> * fix: validate auto_flush_interval > 0 in ALTER SET path Gemini code assist flagged that the request parser accepted a zero duration, leaving the rejection to the downstream RegionOptions validation which only fires on next flush. Reject it at parse time so users get the error immediately at the ALTER TABLE statement. Also add a '0s' error case to the sqlness test. Signed-off-by: srivtx <crypticcc101@gmail.com> * fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result Address the rest of fengjiachun's review on #8403: 1. Empty value in ALTER SET clears the override (parallels Ttl). 'ALTER TABLE t SET ... = NULL' comes through as value = ''; we now return AutoFlushInterval(None) so the override is removed from TableOptions.extra_options, matching the Ttl pattern. 2. Add a unit test in region_request.rs covering the four cases (valid, empty-clears, zero-rejected, garbage-rejected). 3. Generate and check in alter_auto_flush_interval.result via 'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both the standalone and distributed sqlness jobs now pass locally, and the test extension covers the NULL-clears path end to end. Signed-off-by: srivtx <crypticcc101@gmail.com> --------- Signed-off-by: srivtx <crypticcc101@gmail.com> |
||
|
dennis zhuang
|
ca21c9f048 |
fix!: correct information_schema index metadata (#8275)
fix: correct information_schema index metadata Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
e403133eb2 |
feat: add information_schema statistics table (#8253)
* feat: add information_schema statistics table Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: use index-local sequence in statistics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: ordinal_position for pk Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: statistics.nullable uses empty string for non-nullable columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 fys
|
8d3ebde652 |
fix(mito2): schema-safe skipping index pruning (#8122)
* fix: schema-safe skipping index pruning * fix: cargo clippy * fix: sqlness test * remove default plan in BloomFilterIndexApplier * fix comment of plan_for_sst * add fast path for default_plan * minor refactor * some rename * fix: cr by ai |
||
|
fys
|
7d330cc4e6 |
fix(mito2): schema-safe inverted index pruning (#8089)
* fix(mito2): skip inverted index on per-SST type mismatch to avoid false negatives * restore INDEX_APPLY_MEMORY_USAGE * fix: cr * fix: cr |
||
 
|
d1873ca31d |
fix(metric-engine): validate column types and require time index in verify_rows (#8018)
* fix(metric-engine): validate column types and require time index in verify_rows The remote-write path into the metric engine previously bypassed schema validation. When a row's time index column carried a non-timestamp datatype (e.g. a string), the request reached mito's ValueBuilder::push for the timestamp builder and panicked instead of surfacing a typed error. Cache the (column_id, data_type, semantic_type) tuple for each physical column on PhysicalRegionState and use it in verify_rows to: - reject columns whose datatype or semantic type disagrees with the physical region's schema (mirrors mito's WriteRequest::check_schema) - reject requests that omit the time index column entirely Field columns stay optional; tag completeness needs per-logical-region metadata that verify_rows doesn't have and is left to a follow-up. Fixes #7990. Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> * refactor(metric-engine): simplify PhysicalColumnInfo construction - Add From<ColumnMetadata> and From<&ColumnMetadata> for PhysicalColumnInfo so call sites can use metadata.into() instead of repeating the field list. - Replace the four struct-literal constructions in create.rs, open.rs and alter.rs with the conversion. - In verify_rows, pass &col.column_name to ColumnNotFoundSnafu instead of cloning it explicitly (snafu's context handles the conversion). Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> * perf(metric-engine): cache time index column name in PhysicalRegionState verify_rows previously scanned every physical column on each row batch to find the timestamp column. Since the time index is fixed at region creation and never changes, stash its name on PhysicalRegionState when the region is first registered and read it directly from there. add_physical_columns carries a debug_assert to document the invariant that alter never introduces a new time index. Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> * perf(metric-engine): borrow physical column names when building name_to_id On the row-write path we built a HashMap<String, ColumnId> by cloning every column name out of the physical region's cached state. The map is scoped to the block that holds the state's read guard, so there's no need to own the keys. Switch the map to HashMap<&str, ColumnId> and widen RowsIter::new / IterIndex::new to accept any key type that borrows as str. Existing test helpers that pass HashMap<String, ColumnId> keep working through the Borrow<str> bound. Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> * fix: validate metric rows against physical schema Cache physical column metadata in the metric engine state so row validation and row modification can use the same source of truth for column IDs, data types, and semantic types. Validate incoming metric rows against the physical schema before writes. Put requests now require the time index and the expected field column, while delete requests keep accepting primary-key-plus-timestamp payloads by skipping the field completeness check. Pass physical column metadata directly into RowsIter instead of rebuilding a name-to-column-id map at each call site, and cover the new validation paths with tests for missing time indexes, missing fields, and duplicate field columns. Signed-off-by: evenyag <realevenyag@gmail.com> * fix: do not allow adding a new field Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fill default value for fields Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fill default for nullable fields Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Co-authored-by: BootstrapperSBL <yvanwww01@gmail.com> Co-authored-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
60fc455149 |
fix: always skip field pruning when using merge mode (#7957)
* test: add prefilter regressions for last_row null filters Signed-off-by: evenyag <realevenyag@gmail.com> * fix: skip fields in all merge mode Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify pre-filter skip fields handling Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
233e35c0c9 |
feat!: switch default sst format to flat (#7909)
* feat: support alter from primary_key to flat Signed-off-by: evenyag <realevenyag@gmail.com> * chore: alter flat to primary_key Signed-off-by: evenyag <realevenyag@gmail.com> * feat: change default_experimental_flat_format to true Signed-off-by: evenyag <realevenyag@gmail.com> * feat: compute channel size from splitted batch size Signed-off-by: evenyag <realevenyag@gmail.com> * test: add tests for split and channel size Signed-off-by: evenyag <realevenyag@gmail.com> * fix: always set sst_format from manifest on region open sanitize_region_options did not set options.sst_format when the default (PrimaryKey) matched the manifest value, leaving it as None after reopen. This caused the alter format change to appear lost. Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix tests Signed-off-by: evenyag <realevenyag@gmail.com> * test: show create table after alteration Signed-off-by: evenyag <realevenyag@gmail.com> * refactor!: rename default_experimental_flat_format to default_flat_format The flat format is no longer experimental. Remove "experimental" from the config field name, doc comments, and all references. Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
223f6cfdf7 |
feat: supports sst_format for x-greptime-hints and database options (#7843)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
42ad842434 |
feat: support changing table's append_mode to true (#7669)
* feat: support alter append_mode to true Signed-off-by: evenyag <realevenyag@gmail.com> * test: add sqlness test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove comment Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * fix: clear merge mode in mito when setting append mode Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sanitize open request and options with both append/merge mode Signed-off-by: evenyag <realevenyag@gmail.com> * feat: clear merge mode when append mode is true Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Weny Xu
|
567d3e66e9 |
feat: integrate repartition procedure into DdlManager (#7548)
* feat: add repartition procedure factory support to DdlManager - Introduce RepartitionProcedureFactory trait for creating and registering repartition procedures - Implement DefaultRepartitionProcedureFactory for metasrv with full support - Implement StandaloneRepartitionProcedureFactory for standalone (unsupported) - Add procedure loader registration for RepartitionProcedure and RepartitionGroupProcedure - Add helper methods to TableMetadataAllocator for allocator access - Add error types for repartition procedure operations - Update DdlManager to accept and use RepartitionProcedureFactoryRef Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: integrate repartition procedure into DdlManager - Add submit_repartition_task() to handle repartition from alter table - Route Repartition operations in submit_alter_table_task() to repartition factory - Refactor: rename submit_procedure() to execute_procedure_and_wait() - Make all DDL operations wait for completion by default - Add submit_procedure() for fire-and-forget submissions - Add CreateRepartitionProcedure error type - Add placeholder Repartition handling in grpc-expr (unsupported) - Update greptime-proto dependency Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: implement ALTER TABLE REPARTITION procedure submission Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(repartition): handle central region in apply staging manifest - Introduce ApplyStagingManifestInstructions struct to organize instructions - Add special handling for central region when applying staging manifests - Transition state from UpdateMetadata to RepartitionEnd after applying staging manifests - Remove next_state() method in RepartitionStart and inline state transitions - Improve logging and expression serialization in DDL statement executor - Move repartition tests from standalone to distributed test suite Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
aadfcd7821 |
feat(repartition): implement validation logic for repartition table (#7538)
* feat(repartition): implement validation logic for repartition_table Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minor refactor Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Ruihang Xia
|
5162c1de4d |
feat: repartition grammar candy (#7518)
* feat: repartition grammar candy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * align keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
 dennis zhuangandCopilot
|
1f91422bae |
feat!: improve mysql/pg compatibility (#7315)
* feat(mysql): add SHOW WARNINGS support and return warnings for unsupported SET variables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(function): add MySQL IF() function and PostgreSQL description functions for connector compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: show tables for mysql Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: partitions table in information_schema and add starrocks external catalog compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: async udf Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: set warnings Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: impl pg_my_temp_schema and make description functions simple Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add test for issue 7313 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: apply suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: partition_expression and partition_description Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: test Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: unit tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: saerch_path only works for pg Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: improve warnings processing Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: warnings while writing affected rows and refactor Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: improve ShobjDescriptionFunction signature Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: array_to_boolean Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
 LFC
|
fdab75ce27 |
feat: simple read write new json type values (#7175)
feat: basic json read and write Signed-off-by: luofucong <luofc@foxmail.com> |
||
 WaterWhisperer
|
7f1da17150 |
feat: support alter database compaction options (#7251)
Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com> |
||
|
Yingwen
|
bb6a3a2ff3 |
feat: support altering sst format for a table (#7206)
* refactor: remove memtable_builder from MitoRegion Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add alter format Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support changing the format and memtable Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support changing sst format via table options Signed-off-by: evenyag <realevenyag@gmail.com> * fix: set scanner and memtable builder with correct format Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix incorrect metadata in version after alter Signed-off-by: evenyag <realevenyag@gmail.com> * test: add sqlness test Signed-off-by: evenyag <realevenyag@gmail.com> * test: replace region_id in sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * fix: create correct memtable when setting sst_format explicitly Signed-off-by: evenyag <realevenyag@gmail.com> * test: sqlness alter_format test set sst_format to primary_key Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove verbose log Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 LFCandNing Sun
|
21532abf94 |
feat: new create table syntax for new json datatype (#7103)
* feat: new create table syntax for new json datatype Signed-off-by: luofucong <luofc@foxmail.com> * refactor: extract consts * refactor: remove unused error variant * fix tests Signed-off-by: luofucong <luofc@foxmail.com> * fix sqlness Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: Ning Sun <sunning@greptime.com> |
||
|
Ruihang Xia
|
aa98033e85 |
feat(parser): ALTER TABLE ... REPARTITION ... (#7082)
* initial impl Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * sqlness tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * tidy up Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
 Lin Yihai
|
b6cef77a5c |
feat: add SET DEFAULT syntax (#6421)
* feat: add `SET DEFAULT` syntax Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * test: add `CURRENT_TIMESTAMP()` as default value for `SET DEFAULT` syntax Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * refactor: Make the error types more precise. Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * chore: a minor error display enchancement for `SET DEFAULT` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * refactor: Using `MODIFY COLUMN` for `DROP/SET DEFUALT` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * chore: update `greptime-proto` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> --------- Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> |
||
|
Weny Xu
|
ffe0da0405 |
fix: correctly update partition key indices during alter table operations (#6494)
* fix: correctly update partition key indices in alter table operations Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add sqlness tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Zhenchi
|
599f289f59 |
feat: add granularity and false_positive_rate options for indexes (#6416)
* feat: add `granularity` and `false_positive_rate` options for indexes Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * upgrade proto Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |
||
|
Lin Yihai
|
438791b3e4 |
feat: Add DROP DEFAULT (#6290)
* feat: Add `DROP DEFAULT` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * chore: use `next_token` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> --------- Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> |
||
|
Lei, HUANG
|
69870e2762 |
fix(mito): use 1day as default time partition duration (#6202)
* fix unit tests * fix: sqlness * fix/default-time-window: ## Add Helper Functions and Enhance Compaction Tests - **Refactor Compaction Logic**: Introduced helper functions `flush` and `compact` in `compaction_test.rs` to streamline compaction operations. - **Enhance Compaction Tests**: Added a new test `test_infer_compaction_time_window` in `compaction_test.rs` to verify compaction time window inference. - **Testing Improvements**: Added `#[cfg(test)]` attribute to `new_multi_partitions` in `time_partition.rs` to ensure it's only included in test builds. * fix/default-time-window: - **Refactor `TimePartition` Struct**: Removed unnecessary comments regarding `time_range` in `time_partition.rs`. - **Enhance `TimePartitions` Functionality**: Added a method `part_duration_or_default` to provide a default partition duration in `time_partition.rs`. - **Update SQL Test Cases**: Modified SQL operations and expected results in `scan_big_varchar.result` and `scan_big_varchar.sql` to reflect changes in data manipulation logic. * fix/default-time-window: ### Update Time Partition Default Duration - **Refactor Default Duration**: Introduced `INITIAL_TIME_WINDOW` constant to define the default time window duration as `Duration::from_days(1)`. This change replaces multiple instances of the hardcoded default duration across the `time_partition.rs` file. - **Files Affected**: `time_partition.rs` * fix/default-time-window: ## Update Partition Duration Handling - **`time_partition.rs`**: Refactored `part_duration` to be non-optional, removing `Option` wrapper. Updated logic to use `unwrap_or` with `INITIAL_TIME_WINDOW` where necessary. Adjusted related methods and tests to accommodate this change. - **`version.rs` (memtable and region)**: Updated handling of `part_duration` to align with changes in `time_partition.rs`, ensuring consistent use of non-optional `Duration`. * fix/default-time-window: ### Improve Error Context in `time_partition.rs` - Enhanced error context message in `time_partition.rs` to provide clearer information on partition time range issues, including bucket size details. Signed-off-by: Lei, HUANG <lhuang@greptime.com> --------- Signed-off-by: Lei, HUANG <lhuang@greptime.com> |
||
 discord9
|
53752e4f6c |
fix: alter table update table column default (#6155)
* fix: alter table update table column default * fix: fuzz test also cast default value * chore: more testcase * test: non-zero value * refactor: per review * tests: unexpected alter result(WIP on fix) * ub * ub more * test: update sqlness |
||
|
dennis zhuang
|
b85429c0f1 | fix: set column index can't work in physical table (#6179) | ||
|
Lei, HUANG
|
4b71e493f7 |
feat!: revise compaction picker (#6121)
* - **Refactor `RegionFilePathFactory` to `RegionFilePathProvider`:** Updated references and implementations in `access_layer.rs`, `write_cache.rs`, and related test files to use the new struct name. - **Add `max_file_size` support in compaction:** Introduced `max_file_size` option in `PickerOutput`, `SerializedPickerOutput`, and `WriteOptions` in `compactor.rs`, `picker.rs`, `twcs.rs`, and `window.rs`. - **Enhance Parquet writing logic:** Modified `parquet.rs` and `parquet/writer.rs` to support optional `max_file_size` and added a test case `test_write_multiple_files` to verify writing multiple files based on size constraints. **Refactor Parquet Writer Initialization and File Handling** - Updated `ParquetWriter` in `writer.rs` to handle `current_indexer` as an `Option`, allowing for more flexible initialization and management. - Introduced `finish_current_file` method to encapsulate logic for completing and transitioning between SST files, improving code clarity and maintainability. - Enhanced error handling and logging with `debug` statements for better traceability during file operations. - **Removed Output Size Enforcement in `twcs.rs`:** - Deleted the `enforce_max_output_size` function and related logic to simplify compaction input handling. - **Added Max File Size Option in `parquet.rs`:** - Introduced `max_file_size` in `WriteOptions` to control the maximum size of output files. - **Refactored Indexer Management in `parquet/writer.rs`:** - Changed `current_indexer` from an `Option` to a direct `Indexer` type. - Implemented `roll_to_next_file` to handle file transitions when exceeding `max_file_size`. - Simplified indexer initialization and management logic. - **Refactored SST File Handling**: - Introduced `FilePathProvider` trait and its implementations (`WriteCachePathProvider`, `RegionFilePathFactory`) to manage SST and index file paths. - Updated `AccessLayer`, `WriteCache`, and `ParquetWriter` to use `FilePathProvider` for path management. - Modified `SstWriteRequest` and `SstUploadRequest` to use path providers instead of direct paths. - Files affected: `access_layer.rs`, `write_cache.rs`, `parquet.rs`, `writer.rs`. - **Enhanced Indexer Management**: - Replaced `IndexerBuilder` with `IndexerBuilderImpl` and made it async to support dynamic indexer creation. - Updated `ParquetWriter` to handle multiple indexers and file IDs. - Files affected: `index.rs`, `parquet.rs`, `writer.rs`. - **Removed Redundant File ID Handling**: - Removed `file_id` from `SstWriteRequest` and `CompactionOutput`. - Updated related logic to dynamically generate file IDs where necessary. - Files affected: `compaction.rs`, `flush.rs`, `picker.rs`, `twcs.rs`, `window.rs`. - **Test Adjustments**: - Updated tests to align with new path and indexer management. - Introduced `FixedPathProvider` and `NoopIndexBuilder` for testing purposes. - Files affected: `sst_util.rs`, `version_util.rs`, `parquet.rs`. * chore: rebase main * feat/multiple-compaction-output: ### Add Benchmarking and Refactor Compaction Logic - **Benchmarking**: Added a new benchmark `run_bench` in `Cargo.toml` and implemented benchmarks in `benches/run_bench.rs` using Criterion for `find_sorted_runs` and `reduce_runs` functions. - **Compaction Module Enhancements**: - Made `run.rs` public and refactored the `Ranged` and `Item` traits to be public. - Simplified the logic in `find_sorted_runs` and `reduce_runs` by removing `MergeItems` and related functions. - Introduced `find_overlapping_items` for identifying overlapping items. - **Code Cleanup**: Removed redundant code and tests related to `MergeItems` in `run.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Add Benchmarks - **Compaction Logic Improvements**: - Updated `reduce_runs` function in `src/mito2/src/compaction/run.rs` to remove the target parameter and improve the logic for selecting files to merge based on minimum penalty. - Enhanced `find_overlapping_items` to handle unsorted inputs and improve overlap detection efficiency. - **Benchmark Enhancements**: - Added `bench_find_overlapping_items` in `src/mito2/benches/run_bench.rs` to benchmark the new `find_overlapping_items` function. - Extended existing benchmarks to include larger data sizes. - **Testing Enhancements**: - Updated tests in `src/mito2/src/compaction/run.rs` to reflect changes in `reduce_runs` and added new tests for `find_overlapping_items`. - **Logging and Debugging**: - Improved logging in `src/mito2/src/compaction/twcs.rs` to provide more detailed information about compaction decisions. * feat/multiple-compaction-output: ### Refactor and Enhance Compaction Logic - **Refactor `find_overlapping_items` Function**: Changed the function signature to accept slices instead of mutable vectors in `run.rs`. - **Rename and Update Struct Fields**: Renamed `penalty` to `size` in `SortedRun` struct and updated related logic in `run.rs`. - **Enhance `reduce_runs` Function**: Improved logic to sort runs by size and limit probe runs to 100 in `run.rs`. - **Add `merge_seq_files` Function**: Introduced a new function `merge_seq_files` in `run.rs` for merging sequential files. - **Modify `TwcsPicker` Logic**: Updated the compaction logic to use `merge_seq_files` when only one run is found in `twcs.rs`. - **Remove `enforce_file_num` Function**: Deleted the `enforce_file_num` function and its related test cases in `twcs.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Testing - **Add `merge_seq_files` Functionality**: Implemented the `merge_seq_files` function in `run.rs` to optimize file merging based on scoring systems. Updated benchmarks in `run_bench.rs` to include `bench_merge_seq_files`. - **Improve Compaction Strategy in `twcs.rs`**: Modified the compaction logic to handle file merging more effectively, considering file size and overlap. - **Update Tests**: Enhanced test coverage in `compaction_test.rs` and `append_mode_test.rs` to validate new compaction logic and file merging strategies. - **Remove Unused Function**: Deleted `new_file_handles` from `test_util.rs` as it was no longer needed. * feat/multiple-compaction-output: ### Refactor TWCS Compaction Options - **Refactor Compaction Logic**: Simplified the TWCS compaction logic by replacing multiple parameters (`max_active_window_runs`, `max_active_window_files`, `max_inactive_window_runs`, `max_inactive_window_files`) with a single `trigger_file_num` parameter in `picker.rs`, `twcs.rs`, and `options.rs`. - **Update Tests**: Adjusted test cases to reflect the new compaction logic in `append_mode_test.rs`, `compaction_test.rs`, `filter_deleted_test.rs`, `merge_mode_test.rs`, and various test files under `tests/cases`. - **Modify Engine Options**: Updated engine option keys to use `trigger_file_num` in `mito_engine_options.rs` and `region_request.rs`. - **Fuzz Testing**: Updated fuzz test generators and translators to accommodate the new compaction parameter in `alter_expr.rs` and related files. This refactor aims to streamline the compaction configuration by reducing the number of parameters and simplifying the codebase. * chore: add trailing space * fix license header * feat/revise-compaction-picker: **Limit File Processing and Optimize Merge Logic in `run.rs`** - Introduced a limit to process a maximum of 100 files in `merge_seq_files` to control time complexity. - Adjusted logic to calculate `target_size` and iterate over files using the limited set of files. - Updated scoring calculations to use the limited file set, ensuring efficient file merging. * feat/revise-compaction-picker: ### Add Compaction Metrics and Remove Debug Logging - **Compaction Metrics**: Introduced new histograms `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` to track compaction input and output file sizes in `metrics.rs`. Updated `compactor.rs` to observe these metrics during the compaction process. - **Logging Cleanup**: Removed debug logging of file ranges during the merge process in `twcs.rs`. * feat/revise-compaction-picker: ## Enhance Compaction Logic and Metrics - **Compaction Logic Improvements**: - Added methods `input_file_size` and `output_file_size` to `MergeOutput` in `compactor.rs` to streamline file size calculations. - Updated `Compactor` implementation to use these methods for metrics tracking. - Modified `Ranged` trait logic in `run.rs` to improve range comparison. - Enhanced test cases in `run.rs` to reflect changes in compaction logic. - **Metrics Enhancements**: - Changed `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` from histograms to counters in `metrics.rs` for better performance tracking. - **Debugging and Logging**: - Added detailed logging for compaction pick results in `twcs.rs`. - Implemented custom `Debug` trait for `FileMeta` in `file.rs` to improve debugging output. - **Testing Enhancements**: - Added new test `test_compaction_overlapping_files` in `compaction_test.rs` to verify compaction behavior with overlapping files. - Updated `merge_mode_test.rs` to reflect changes in file handling during scans. * feat/revise-compaction-picker: ### Update `FileHandle` Debug Implementation - **Refactor Debug Output**: Simplified the `fmt::Debug` implementation for `FileHandle` in `src/mito2/src/sst/file.rs` by consolidating multiple fields into a single `meta` field using `meta_ref()`. - **Atomic Operations**: Updated the `deleted` field to use atomic loading with `Ordering::Relaxed`. * Trigger CI * feat/revise-compaction-picker: **Update compaction logic and default options** - **`twcs.rs`**: Enhanced logging for compaction pick results by improving the formatting for better readability. - **`options.rs`**: Modified the default `max_output_file_size` in `TwcsOptions` from 2GB to 512MB to optimize file handling and performance. * feat/revise-compaction-picker: Refactor `find_overlapping_items` to use an external result vector - Updated `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to accept a mutable result vector instead of returning a new vector, improving memory efficiency. - Modified benchmarks in `src/mito2/benches/bench_compaction_picker.rs` to accommodate the new function signature. - Adjusted tests in `src/mito2/src/compaction/run.rs` to use the updated function signature, ensuring correct functionality with the new approach. * feat/revise-compaction-picker: Improve file merging logic in `run.rs` - Refactor the loop logic in `merge_seq_files` to simplify the iteration over file groups. - Adjust the range for `end_idx` to include the endpoint, allowing for more flexible group selection. - Remove the condition that skips groups with only one file, enabling more comprehensive processing of file sequences. * feat/revise-compaction-picker: Enhance `find_overlapping_items` with `SortedRun` and Update Tests - Refactor `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to utilize the `SortedRun` struct for improved efficiency and clarity. - Introduce a `sorted` flag in `SortedRun` to optimize sorting operations. - Update test cases in `src/mito2/benches/bench_compaction_picker.rs` to accommodate changes in `find_overlapping_items` by using `SortedRun`. - Add `From<Vec<T>>` implementation for `SortedRun` to facilitate easy conversion from vectors. * feat/revise-compaction-picker: **Enhancements in `compaction/run.rs`:** - Added `ReadableSize` import to handle size calculations. - Modified the logic in `merge_seq_files` to clamp the calculated target size to a maximum of 2GB when `max_file_size` is not provided. * feat/revise-compaction-picker: Add Default Max Output Size Constant for Compaction Introduce DEFAULT_MAX_OUTPUT_SIZE constant to define the default maximum compaction output file size as 2GB. Refactor the merge_seq_files function to utilize this constant, ensuring consistent and maintainable code for handling file size limits during compaction. |
||
|
discord9
|
6ab0f0cc5c |
fix: alter table modify type should also modify default value (#6049)
* fix: select after alter * fix: insert a proper row&catch a bug * fix: alter table modify type modify default value type too * refactor: per review * chore: per review * refactor: per review * refactor: per review |
||
|
Yingwen
|
148d96fc38 |
fix: ensures logical and physical region have the same timestamp unit (#6041)
* fix: check time unit of logical region * test: enlarge ttl for alter test to avoid data expired during test * chore: fix unused |
||
|
Yingwen
|
56f319a707 |
fix: filter doesn't consider default values after schema change (#5912)
* test: sqlness test case * feat: use correct default while pruning row groups * fix: consider default in SimpleFilterContext * test: update sqlness test * test: add order by |
||
|
Zhenchi
|
8d485e9be0 |
feat: support altering fulltext backend (#5896)
* feat: add `greptime_index_type` to `information_schema.key_column_usage` Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix: show create Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |
||
|
Weny Xu
|
981d51785b |
fix: throw errors instead of ignoring (#5792)
* fix: throw errors instead of ignoring * fix: fix unit tests * refactor: remove schema version check * fix: fix clippy * chore: remove unused error * refactor: remove schema version check * feat: handle mutliple results * feat: introduce consistency guard * fix: release consistency guard on datanode operation completion * test: add tests * chore: remove schema version * refactor: rename * test: add more tests * chore: print all error * tests: query table after alteration * log ignored request * refine fuzz test * chore: fix clippy and log mailbox message * chore: close prepared statement after execution * chore: add comment * chore: remove log * chore: rename to `ConsistencyPoison` * chore: remove unused error * fix: fix unit tests * chore: apply suggestions from CR |
||
|
Lei, HUANGandYingwen
|
e375a18011 |
fix: conversion from TableMeta to TableMetaBuilder (#5693)
* refactor: use proc macro to generate conversion between TableMeta and TableMetaBuilder
* chore: format
* fix/partition-key-index:
### Update `TableMeta` and Add Partition and Alter Table Tests
- **`metadata.rs`**: Modified `new_meta_builder` method in `TableMeta` to manually remove `value_indices` by setting it to `None` in the `TableMetaBuilder`.
- **`partition_and_alter.result` & `partition_and_alter.sql`**: Added new test cases for creating, inserting, selecting, altering, and dropping a partitioned table `molestiAe`. These tests verify partitioning on the `sImiLiQUE` column and altering the table with a TTL
setting.
fix/partition-key-index:
### Remove Obsolete TODO Comment in `metadata.rs`
- Removed an outdated TODO comment regarding the `new_meta_builder` function in `src/table/src/metadata.rs`.
chore: check struct name in derive_meta_builder
refactor: Simplify TableMeta struct name check in macro
refactor: Improve ToMetaBuilder derive macro validation and error handling
refactor: Enforce ToMetaBuilder macro for table::metadata::TableMeta struct
* fix/partition-key-index:
Update `partition_and_alter.sql` to modify TTL setting
- Modified the TTL setting for the `molestiAe` table to '1d' in `partition_and_alter.sql`.
* fix: sqlness
* fix/partition-key-index:
### Update `TableMeta` and Test File Structure
- **Enhancement**: Added a note in `metadata.rs` to always use `new_meta_builder` for creating `TableMetaBuilder`.
- **Refactor**: Renamed test result and SQL files for better organization:
- `partition_and_alter.result` to `alter/partition_and_alter.result`
- `partition_and_alter.sql` to `alter/partition_and_alter.sql`
* refactor: Simplify `derive_meta_builder` by initializing fields with `Default::default()`
* fix/partition-key-index:
### Commit Summary
- **Refactor `TableMetaBuilder` Initialization**:
- Replaced `TableMetaBuilder::default()` with `TableMetaBuilder::empty()` across multiple files for initializing `TableMetaBuilder` instances.
- Affected files include:
- `src/catalog/src/system_schema.rs`
- `src/common/meta/src/key/test_utils.rs`
- `src/operator/src/req_convert/insert/fill_impure_default.rs`
- `src/query/src/log_query/planner.rs`
- `src/query/src/promql/planner.rs`
- `src/query/src/range_select/plan_rewrite.rs`
- `src/query/src/sql/show_create_table.rs`
- `src/table/src/test_util/memtable.rs`
- `src/table/src/test_util/table_info.rs`
- **Enhance `TableMetaBuilder`**:
- Added `custom_constructor` to `TableMeta` and implemented an `empty` method for `TableMetaBuilder`.
- Modified `TableMetaBuilder` to include a `new_external_table` method with default values.
- Updated `src/table/src/metadata.rs` to reflect these changes.
- **Add Testing Feature**:
- Introduced a conditional compilation for `test_util` in `src/table/src/lib.rs` to include testing utilities when the `testing` feature is enabled.
- **Update `Cargo.toml`**:
- Enabled the `testing` feature for the `table` module in `src/common/meta/Cargo.toml`.
- **Modify `NumbersTable` Initialization**:
- Replaced `TableMetaBuilder` with direct `TableMeta` struct initialization in `src/table/src/table/numbers.rs`.
- **Test Result Update**:
- Updated test results in `tests/cases/standalone/common/alter/partition_and_alter.result` to reflect changes in table meta handling.
* fix: rename default to empty
* docs: add doc for TableMetaBuilder::empty
* chore: Update src/table/src/metadata.rs
---------
Co-authored-by: Yingwen <realevenyag@gmail.com>
|
||
|
Weny Xu
|
1d637cad51 |
fix(metric-engine): group DDL requests (#5628)
* fix(metric-engine): group DDL requests * test: add sqlness tests * chore: apply suggestions from CR * chore: apply suggestions from CR |
||
|
Yingwen
|
3f6a41eac5 |
fix: update show create table output for fulltext index (#5591)
* fix: update full index syntax in show create table * test: update fulltext sqlness result |
||
|
Ruihang Xia
|
7fc935c61c |
feat!: support alter skipping index (#5538)
* feat: support alter skipping index Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update test results Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * cargo fmt Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * finalize Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
 jeremyhi
|
02af9dd21a |
refactor!: remove datetime type (#5506)
* feat remove datetime type * chore: fix unit test * chore: add column test * refactor: move create and alter validation to one place * chore: minor refactor ut * refactor: rename expr_factory to expr_helper * chore: remove unnecessary args |
||
|
Ruihang Xiaandjeremyhi
|
8026b1d72c |
feat!: unify all index creation grammars (#5486)
* column options Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle table constrain Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update test assertions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * change inverted index table constrain usage Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * don't create inverted index for pk on alter table Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove remaining pk-as-inverted-index Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * more inverted index magic Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * Update src/sql/src/statements.rs Co-authored-by: jeremyhi <jiachun_feng@proton.me> * drop support for index def in table constrain Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me> |
||
 yihong
|
adb5c3743c |
fix: flush table panic when table has interval column close #3235 (#5422)
* fix: flash table panic when table has interval column close #3235 Signed-off-by: yihong0618 <zouzou0208@gmail.com> * Revert "fix: flash table panic when table has interval column close #3235" This reverts commit ffc63efda39cd6ef525313b60ede061c5ec24b12. * fix: create table do not support interval type for now close #3235 Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: sqlness Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: address comments Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: address comments fix conflict and more tests Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: address final comments drop useless sqlness tests Signed-off-by: yihong0618 <zouzou0208@gmail.com> --------- Signed-off-by: yihong0618 <zouzou0208@gmail.com> |
||
 
|
35b635f639 |
feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)
* change dep Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: adapt to arrow's interval array * chore: fix compile errors in datatypes crate * chore: fix api crate compiler errors * chore: fix compiler errors in common-grpc * chore: fix common-datasource errors * chore: fix deprecated code in common-datasource * fix promql and physical plan related Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * wip: upgrading network deps Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * block on updating `sqlparser` * upgrade sqlparser Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * adapt new df's trait requirements Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix compiler errors in mito2 * chore: fix common-function crate errors * chore: fix catalog errors * change import path Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix some errors in query crate * chore: fix some errors in query crate * aggr expr and some other tiny fixes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix expr related errors in query crate * chore: fix query serializer and admin command * chore: fix grpc services * feat: axum serve * chore: fix http server * remove handle_error handler * refactor timeout layer * serve axum * chore: fix flow aggr functions * chore: fix flow * feat: fix errors in meta-srv * boxed() * use TokioIo * feat!: Remove script crate and python feature (#5321) * feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning * chore: fix servers tests compiler errors * feat: fix tests-integration errors * chore: fix unused * test: fix catalog test * chore: fix compiler errors for crates using common-meta testing feature is enabled when check with --workspace * test: use display for logical plan test * test: implement rewrite for ScanHintRule * fix: http server build panic * test: fix mito test * fix: sql parser type alias error * test: fix TestClient not listen * test: some flow tests * test(flow): more fix * fix: test_otlp_logs * test: fix promql test that using deprecated method fun() * fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64 * test: fix infer schema test case * test: fix tests related to plan display * chore: fix last flow test * test: fix function format related assertion * test: use larger port range for tests * fix: test_otlp_traces * fix: test_otlp_metrics * fix range query and dist plan Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: flow handle distinct use deprecated field * fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs * test: fix deserialize test * test: reduce split key case num * tests: lower case aggr func name * test: fix some sqlness tests * tests: more sqlness fix * tests: fixed sqlness test * commit non-bug changes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: make our udf correct * fix: implement empty methods of ContextProvider for DfContextProviderAdapter * test: update sqlness test result * chore: remove unused * fix: provide alias name for AggregateExprBuilder in range plan * test: update range query result * fix: implement missing ContextProvider methods for DfContextProviderAdapter * test: update timestamps, cte result * fix: supports empty projection in mito * test: update comment for cte test * fix: support projection for numbers * test: update test cases after projection fix * fix: fix range select first_value/last_value * fix: handle CAST and time index conflict * fix: handle order by correctly in range first_value/last_value * test: update sqlness result * test: update view test result * test: update decimal test wait for https://github.com/apache/datafusion/pull/14126 to fix this * feat: remove redundant physical optimization todo(ruihang): Check if we can remove this. * test: update sqlness test result * chore: range select default sort use nulls_first = false * test: update filter push down test result * test: comment deciaml test to avoid different panic message * test: update some distributed test result * test: update test for distributed count and filter push down * test: update subqueries test * fix: SessionState may overwrite our UDFs * chore: fix compiler errors after merging main * fix: fix elasticsearch and dashboard router panic * chore: fix common-functions tests * chore: update sqlness result * test: fix id keyword and update sqlness result * test: fix flow_null test * fix: enlarge thread size in debug mode to avoid overflow * chore: fix warnings in common-function * chore: fix warning in flow * chore: fix warnings in query crate * chore: remove unused warnings * chore: fix deprecated warnings for parquet * chore: fix deprecated warning in servers crate * style: fix clippy * test: enlarge mito cache tttl test ttl time * chore: fix typo * style: fmt toml * refactor: reimplement PartialOrd for RangeSelect * chore: remove script crate files introduced by merge * fix: return error if sql option is not kv * chore: do not use ..default::default() * chore: per review * chore: update error message in BuildAdminFunctionArgsSnafu Co-authored-by: jeremyhi <jiachun_feng@proton.me> * refactor: typed precision * update sqlness view case Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: flow per review * chore: add example in comment * chore: warn if parquet stats of timestamp is not INT64 * style: add a newline before derive to make the comment more clear * test: update sqlness result * fix: flow from substrait * chore: change update_range_context log to debug level * chore: move axum-extra axum-macros to workspace --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: luofucong <luofc@foxmail.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me> |
||
 Niwaka
|
0163ce8df9 |
feat: add column if not exists (#5393)
* feat: add column if not exists * chore: address reviews |
||
 Lanqing Yang
|
a4761d6245 |
feat: Alter inverted index (#5131)
feat: support alter inverted index. |
||
|
Niwaka
|
50583815de |
feat: support alter add multiple columns (#5262)
* feat: support alter add multiple columns * fix: address review * chore: add column format |
||
|
Zhenchi
|
c305b2b406 |
fix: show index to display skipping index (#5297)
* fix: show index to display skipping index Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix sqlness Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix sqlness Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |