* feat: table semantic layer information_schema view (Phase 3)
Add `information_schema.table_semantics`, a queryable view over the table
semantic layer. One row per table that carries at least one
`greptime.semantic.*` option: the signal-agnostic keys
(signal_type/source/pipeline/metadata_quality) are promoted to columns and
the remaining signal-specific keys are folded into a `semantic_options`
JSON string. Tables with no semantic key are excluded.
Stacked on Phase 2.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: address PR review on table_semantics
- fold JSON serialization failure into None instead of unwrap/panic
- drop per-row Vec allocation in predicate eval; use a fixed array
- align RFC view name with the shipped `table_semantics`
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: support defer_on_missing_source for pending flow creation
Add `defer_on_missing_source` flow option that allows creating flows
even when source tables do not yet exist. The flow enters a pending
state and is automatically activated when source tables become available.
Key changes:
- New `FlowStatus::PendingSources` and fields in `FlowInfoValue` for
unresolved source table names and last activation error
- `defer_on_missing_source` create-time-only option: stripped from
runtime/flownode `CreateRequest` but preserved in metadata for
SQL round-trip (`SHOW CREATE FLOW`, `information_schema.flows`)
- `CreateFlowProcedure` creates pending metadata when sources are
missing and `defer_on_missing_source=true`; falls back to
`FlowType::Batching` for missing-source flows
- `PendingFlowReconcileManager` in meta-srv periodically checks
pending flows and activates them when source tables resolve
- `ActivatePendingFlowProcedure` handles activation: allocates peers,
creates flows on flownodes, updates metadata, invalidates cache
- `OR REPLACE` properly handles pending<->active transitions,
including peer allocation and flownode flow teardown
- `FlowMetadataAllocator::alloc_peers` for peer allocation at
activation time
- Validated flow options: only `defer_on_missing_source` allowed;
unknown options rejected
- Known issue: standalone mode does not support flownodes, so
pending flow flush/sink behavior covered only in distributed
sqlness; operator and meta unit tests cover activation logic
Tests:
- operator `determine_flow_type_for_source_state` (3 passed)
- common-meta `create_flow` (19 passed) including replacement
- common-meta `activate_flow` (4 passed)
- meta-srv `flow` (11 passed)
- sqlness: `flow_pending` covers create/replace/round-trip
Signed-off-by: discord9 <discord9@163.com>
* chore: simplify pending flow PR scope
Reduce PR #8124 to the metadata-only MVP after complexity review.
Changes:
- Remove automatic activation procedure and meta-srv reconcile wiring
- Remove activation tests and activation-only metadata fields
- Reject cross-state pending<->active `OR REPLACE` transitions for now
- Keep pending metadata creation and SQL round-trip behavior
- Allow `DROP FLOW` for pending flows without routes
- Reduce flow_pending sqlness to metadata/round-trip/drop coverage only
Deferred follow-ups are documented locally in `.tmp/tasks/pending-defer-semantics/deferred-followups.md` and intentionally not committed.
Tests:
- `cargo test -p operator determine_flow_type_for_source_state`
- `cargo test -p common-meta create_flow`
- `cargo test -p common-meta drop_flow`
- `cargo sqlness bare --test-filter flow_pending --bins-dir /mnt/nvme_rust/rust-targets/pending_defer/debug`
Signed-off-by: discord9 <discord9@163.com>
* test: cover pending flow metadata edge cases
Signed-off-by: discord9 <discord9@163.com>
* test: fix pending flow metadata test lint
Signed-off-by: discord9 <discord9@163.com>
* docs: document pending flow metadata fields
Signed-off-by: discord9 <discord9@163.com>
* chore: more sleep when test
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* fix: schema-safe skipping index pruning
* fix: cargo clippy
* fix: sqlness test
* remove default plan in BloomFilterIndexApplier
* fix comment of plan_for_sst
* add fast path for default_plan
* minor refactor
* some rename
* fix: cr by ai
* feat: switch partition tree to bulk
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: keep partition tree memtable for migration test
Restore PartitionTreeMemtable construction when memtable.type=partition_tree
is explicit, and move the sparse-encoding bulk override into the default
(no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on
reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a
metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse,
so the test actually exercises a PartitionTreeMemtable in phase 1 and
verifies WAL replay into the new BulkMemtable on reopen without flushing.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: drop partition tree memtable from runtime
Re-apply the unconditional sparse-encoding override in
`MemtableBuilderProvider::builder_for_options` and route the
`MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation
warning. After this change, `PartitionTreeMemtableBuilder` is no longer
reachable from the engine runtime; benchmarks still reference the type.
Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the
`put_sparse_rows` helper added in the previous commit — that test only
existed to validate the PartitionTree -> Bulk reopen migration and is
unnecessary now that the override is in place.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): move timestamp_array_to_i64_slice into read module
Relocate the timestamp_array_to_i64_slice helper from
memtable/partition_tree/data.rs to the read module so that the read
path no longer depends on the partition_tree internals. All call sites
(both inside and outside the partition_tree module) now import from
crate::read.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests
The time_partition tests use the memtable builder purely as a generic
backend for the TimePartitions write/scan paths; nothing in them is
specific to the partition-tree memtable. Switch the seven affected
tests to TimeSeriesMemtableBuilder so the tests no longer depend on
PartitionTreeMemtableBuilder.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): delete PartitionTreeMemtable implementation
The runtime already falls back to BulkMemtable for the PartitionTree
variant. Drop the now-unreachable implementation, its metrics, the
partition_tree benchmarks, the metric-engine Unsupported fallback in
bulk_insert.rs, and the test helpers that only existed for the deleted
module.
MemtableOptions::PartitionTree, its parsing, the runtime fallback, the
store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures
remain so existing region options keep round-tripping.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito-codec): drop skip_partition_column parameter
PartitionTreeMemtable was the only caller passing
skip_partition_column=true; every other caller passes false. Now that
the partition_tree module is gone, the parameter is uniformly false
and the guard branch is dead. Drop the parameter from the trait method
and both impls, remove the guard and the is_partition_column helper,
and update the four remaining call sites in mito2 plus the bench.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): remove unused MemtableConfig enum
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fmt code
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: remove unused variant
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update test_config_api
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: remove unused memtable test helpers
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: address review comment
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: support bulk memtable options
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: sanitize config
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: remove partition tree options from region options
Move primary_key_encoding to the top level
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: make ssts test datetime replaced text stable
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update sqlness result
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: validate_enum_options consider bulk memtable
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: pass region id when parsing region options
Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with
`try_from_options(region_id, options_map)` so the legacy partition_tree
fallback can log the affected region. The fallback now also overrides
the SST format to flat in addition to clearing the memtable type.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: align sst_format with bulk memtable on parse and open
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(metric-engine): validate column types and require time index in verify_rows
The remote-write path into the metric engine previously bypassed schema
validation. When a row's time index column carried a non-timestamp
datatype (e.g. a string), the request reached mito's ValueBuilder::push
for the timestamp builder and panicked instead of surfacing a typed
error.
Cache the (column_id, data_type, semantic_type) tuple for each physical
column on PhysicalRegionState and use it in verify_rows to:
- reject columns whose datatype or semantic type disagrees with the
physical region's schema (mirrors mito's WriteRequest::check_schema)
- reject requests that omit the time index column entirely
Field columns stay optional; tag completeness needs per-logical-region
metadata that verify_rows doesn't have and is left to a follow-up.
Fixes#7990.
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* refactor(metric-engine): simplify PhysicalColumnInfo construction
- Add From<ColumnMetadata> and From<&ColumnMetadata> for PhysicalColumnInfo
so call sites can use metadata.into() instead of repeating the field list.
- Replace the four struct-literal constructions in create.rs, open.rs and
alter.rs with the conversion.
- In verify_rows, pass &col.column_name to ColumnNotFoundSnafu instead of
cloning it explicitly (snafu's context handles the conversion).
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* perf(metric-engine): cache time index column name in PhysicalRegionState
verify_rows previously scanned every physical column on each row batch to
find the timestamp column. Since the time index is fixed at region
creation and never changes, stash its name on PhysicalRegionState when
the region is first registered and read it directly from there.
add_physical_columns carries a debug_assert to document the invariant
that alter never introduces a new time index.
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* perf(metric-engine): borrow physical column names when building name_to_id
On the row-write path we built a HashMap<String, ColumnId> by cloning
every column name out of the physical region's cached state. The map is
scoped to the block that holds the state's read guard, so there's no
need to own the keys.
Switch the map to HashMap<&str, ColumnId> and widen RowsIter::new /
IterIndex::new to accept any key type that borrows as str. Existing
test helpers that pass HashMap<String, ColumnId> keep working through
the Borrow<str> bound.
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* fix: validate metric rows against physical schema
Cache physical column metadata in the metric engine state so row validation and row modification can use the same source of truth for column IDs, data types, and semantic types.
Validate incoming metric rows against the physical schema before writes. Put requests now require the time index and the expected field column, while delete requests keep accepting primary-key-plus-timestamp payloads by skipping the field completeness check.
Pass physical column metadata directly into RowsIter instead of rebuilding a name-to-column-id map at each call site, and cover the new validation paths with tests for missing time indexes, missing fields, and duplicate field columns.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: do not allow adding a new field
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: fill default value for fields
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: fill default for nullable fields
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: BootstrapperSBL <yvanwww01@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
* feat(operator): allow last_row merge_mode when append_mode is enabled
- Update RegionOptions::validate to allow last_row merge_mode with append_mode.
- Update fill_table_options_for_create to automatically set merge_mode to last_row when append_mode is enabled for LastNonNull table type.
- Add unit tests in mito2 and operator to verify options validation and table creation.
- Add integration test for InfluxDB write with append mode hint.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix(operator): simplify append mode options
Group `LastNonNull` auto-create options in a single append-mode branch.
Files:
- `src/operator/src/insert.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sqlness
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: allow ipv4_num_to_string to accept valid integers
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
* test: update sqlness result file
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
* fix: use coercible integer signature for ipv4_num_to_string
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
---------
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
