* feat(flow): handle time_ranges in DirtyWindowRequest
Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): fix doc comment spacing in align_time_window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time_ranges in handle_mark_dirty_time_window
Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): union merged dirty windows with the larger end
Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): clip bounded dirty ranges at the expire bound
merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): fall back to full dirty on dirty-window alignment failure
An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): apply rustfmt to new dirty-window merge tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time index units in dirty window marking
Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to 8127f179
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): return dirty-window alignment errors to callers
Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: update proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(catalog): expose soft drops in recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(sql): add UNDROP TABLE
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sql): cover UNDROP TABLE execution
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): keep successful UNDROP result
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject stale undrop by name-based lookup after tombstone consumed
When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.
Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.
- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
rejection case
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(log-store): keep Kafka obsolete_all as no-op
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): hide purging tables from recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): scope recycle bin scans by catalog
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sqlness): update recycle bin expectations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: avoid redundant recycle bin allocations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(json2): type hint
* test(datatypes): add JsonSettings serde round-trip test
* minor refactor
This reverts commit 7ff5a5249a09be5396536284fe822b5761ef4e6a.
* fix: code review
* chore: declare GreptimeDB Enterprise License for enterprise-gated sources
The `enterprise`-feature-gated sources (triggers, mito2 extension) were
excluded from the Apache-2.0 header check but carried no license of their
own. Declare a separate GreptimeDB Enterprise License and enforce it.
- Add LICENSE-ENTERPRISE (open-core split; core stays Apache-2.0).
- Add an Enterprise License header to each enterprise source file.
- Add licenserc-enterprise.toml and a second hawkeye step in CI to enforce
the Enterprise header on exactly those files.
- Cross-reference the two complementary file lists; document the layout in
licenses/README.md and the README License section.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: reference per-customer Enterprise Agreement instead of a terms URL
There is no public enterprise-terms page; each customer signs an individually
negotiated agreement. Point the license at a "separate written commercial
agreement with GrepTime Inc." and direct readers to the existing contact page.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer identity (Phase 1)
Attach a thin layer of semantic metadata to ingested tables via the existing
`table_options` slot, so machine consumers (LLM agents, alert/dashboard builders,
MCP servers, ETL) can align a table with the observability concept it stands for
without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md.
Phase 1 (identity) only:
- New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary
(signal/source/source_version/pipeline + trace/metric/log/resource-scope keys,
defined now, populated by later phases), value constants, the internal
`greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2,
deliberately outside the public namespace), and `is_semantic_option_key`.
- `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are
valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`.
- `fill_table_options_for_create` copies every semantic ctx extension into the new
table's options (prefix passthrough alongside the fixed allowlist).
- Frontend stamps identity on the context at each ingest entry: OTLP metrics
(metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for
the v1 model), logs (log/opentelemetry), and Prometheus remote write
(metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is
left for Phase 2 (declared).
- Trace identity is stamped only on the main span table; the derived
`_services` / `_operations` lookup tables keep the unstamped context and carry no
semantic identity (cross-table relationships are out of scope).
Semantic options appear in SHOW CREATE TABLE (like table_data_model /
otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its
semantics directly.
Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and
integration assertions that the common keys land for OTLP metrics (metric-engine
logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens
updated.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: prom batcher not cover and white list for semantic keys/values
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: typo
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: global switch for creating table automatically
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: make auto_create_table as comment by default
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: respect gloabl switch for metric engine
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: support defer_on_missing_source for pending flow creation
Add `defer_on_missing_source` flow option that allows creating flows
even when source tables do not yet exist. The flow enters a pending
state and is automatically activated when source tables become available.
Key changes:
- New `FlowStatus::PendingSources` and fields in `FlowInfoValue` for
unresolved source table names and last activation error
- `defer_on_missing_source` create-time-only option: stripped from
runtime/flownode `CreateRequest` but preserved in metadata for
SQL round-trip (`SHOW CREATE FLOW`, `information_schema.flows`)
- `CreateFlowProcedure` creates pending metadata when sources are
missing and `defer_on_missing_source=true`; falls back to
`FlowType::Batching` for missing-source flows
- `PendingFlowReconcileManager` in meta-srv periodically checks
pending flows and activates them when source tables resolve
- `ActivatePendingFlowProcedure` handles activation: allocates peers,
creates flows on flownodes, updates metadata, invalidates cache
- `OR REPLACE` properly handles pending<->active transitions,
including peer allocation and flownode flow teardown
- `FlowMetadataAllocator::alloc_peers` for peer allocation at
activation time
- Validated flow options: only `defer_on_missing_source` allowed;
unknown options rejected
- Known issue: standalone mode does not support flownodes, so
pending flow flush/sink behavior covered only in distributed
sqlness; operator and meta unit tests cover activation logic
Tests:
- operator `determine_flow_type_for_source_state` (3 passed)
- common-meta `create_flow` (19 passed) including replacement
- common-meta `activate_flow` (4 passed)
- meta-srv `flow` (11 passed)
- sqlness: `flow_pending` covers create/replace/round-trip
Signed-off-by: discord9 <discord9@163.com>
* chore: simplify pending flow PR scope
Reduce PR #8124 to the metadata-only MVP after complexity review.
Changes:
- Remove automatic activation procedure and meta-srv reconcile wiring
- Remove activation tests and activation-only metadata fields
- Reject cross-state pending<->active `OR REPLACE` transitions for now
- Keep pending metadata creation and SQL round-trip behavior
- Allow `DROP FLOW` for pending flows without routes
- Reduce flow_pending sqlness to metadata/round-trip/drop coverage only
Deferred follow-ups are documented locally in `.tmp/tasks/pending-defer-semantics/deferred-followups.md` and intentionally not committed.
Tests:
- `cargo test -p operator determine_flow_type_for_source_state`
- `cargo test -p common-meta create_flow`
- `cargo test -p common-meta drop_flow`
- `cargo sqlness bare --test-filter flow_pending --bins-dir /mnt/nvme_rust/rust-targets/pending_defer/debug`
Signed-off-by: discord9 <discord9@163.com>
* test: cover pending flow metadata edge cases
Signed-off-by: discord9 <discord9@163.com>
* test: fix pending flow metadata test lint
Signed-off-by: discord9 <discord9@163.com>
* docs: document pending flow metadata fields
Signed-off-by: discord9 <discord9@163.com>
* chore: more sleep when test
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* feat: expose node info for placement selectors
Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`.
Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation.
Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: skip datanode node info without stats
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: avoid unnecessary workload clones
Skip workload cloning for inactive nodes and for active node-info lookups without workload filters.
Files: `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: require frontend origin address
Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts.
Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: reuse resolved frontend address
Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins.
Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve datanode lease liveness
Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars.
Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Remove stale datanode lease helper
- `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs`
- `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/env-based-region-selector-oss: simplify lease discovery
- `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(operator): allow last_row merge_mode when append_mode is enabled
- Update RegionOptions::validate to allow last_row merge_mode with append_mode.
- Update fill_table_options_for_create to automatically set merge_mode to last_row when append_mode is enabled for LastNonNull table type.
- Add unit tests in mito2 and operator to verify options validation and table creation.
- Add integration test for InfluxDB write with append mode hint.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix(operator): simplify append mode options
Group `LastNonNull` auto-create options in a single append-mode branch.
Files:
- `src/operator/src/insert.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sqlness
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Error Handling Improvements**:
- Removed `CatalogSnafu` context from various `.await` calls in `dashboard.rs`, `influxdb.rs`, `jaeger.rs`, `prometheus.rs`, `event.rs`, and `pipeline.rs` to streamline error handling.
- **Prometheus Store Enhancements**:
- Added support for auto-creating tables and adding missing Prometheus tag columns in `prom_store.rs` and `pending_rows_batcher.rs`.
- Introduced `PendingRowsSchemaAlterer` trait for schema alterations in `pending_rows_batcher.rs`.
- **Test Additions**:
- Added tests for new Prometheus store functionalities in `prom_store.rs` and `pending_rows_batcher.rs`.
- **Error Message Improvements**:
- Enhanced error messages for catalog access in `error.rs`.
- **Server Configuration Updates**:
- Updated server configuration to include Prometheus store options in `server.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* reformat
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add DataTypes Error Handling and Column Renaming Logic
- **`error.rs`**: Introduced a new `DataTypes` error variant to handle errors from `datatypes::error::Error`. Updated `ErrorExt` implementation to include `DataTypes`.
- **`pending_rows_batcher.rs`**: Added functions `find_prom_special_column_names` and `rename_prom_special_columns_for_existing_schema` to handle renaming of special Prometheus columns. Updated `build_prom_create_table_schema` to simplify error handling with
`ConcreteDataType`.
- **Tests**: Added a test case `test_rename_prom_special_columns_for_existing_schema` to verify the renaming logic for Prometheus special columns.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- Refactored `PendingRowsBatcher` to accommodate Prometheus record batches:
- Introduced `accommodate_record_batch_for_target_schema` to normalize incoming record batches against existing table schemas.
- Removed `collect_missing_prom_tag_columns` and `rename_prom_special_columns_for_existing_schema` in favor of the new function.
- Added `unzip_logical_region_schema` to extract schema components.
- Updated tests in `pending_rows_batcher.rs`:
- Added tests for `accommodate_record_batch_for_target_schema` to verify handling of missing tag columns and renaming of special columns.
- Ensured error handling for missing timestamp and field columns in target schema.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Summary
- **Enhancement in Table Creation Logic**: Updated `prom_store.rs` to modify the handling of `table_options` during table creation. Specifically, `table_options` are now extended differently based on the `AutoCreateTableType`. For `Physical` tables, enforced
`sst_format=flat` to optimize pending-rows writes by leveraging bulk memtables.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Enhance Performance Monitoring in `pending_rows_batcher.rs`
- Added performance monitoring timers to various stages of the `PendingRowsBatcher` process, including schema cache checks, table resolution, schema creation, and record batch alignment.
- Improved schema handling by adding timers around schema alteration and missing column addition processes.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Enhance Concurrent Write Handling**: Introduced `FlushRegionWrite` and `FlushWriteResult` structs to manage region writes and their results. Added `flush_region_writes_concurrently` function to handle concurrent flushing of region writes based on
`should_dispatch_concurrently` logic in `pending_rows_batcher.rs`.
- **Testing Enhancements**: Added tests for concurrent dispatching of region writes and the logic for determining concurrent dispatch in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add Histogram for Flush Stage Elapsed Time
- **`metrics.rs`**: Introduced a new `HistogramVec` named `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to track the elapsed time of pending rows batch flush stages.
- **`pending_rows_batcher.rs`**: Replaced instances of `PENDING_ROWS_BATCH_INGEST_STAGE_ELAPSED` with `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to measure the elapsed time for various flush stages, including `flush_write_region`, `flush_concat_table_batches`,
`flush_resolve_table`, `flush_fetch_partition_rule`, `flush_split_record_batch`, `flush_filter_record_batch`, `flush_resolve_region_leader`, and `flush_encode_ipc`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Add design doc for physical table batching in PendingRowsBatcher
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Add implementation plan for physical table batching in PendingRowsBatcher
* feat/auto-schema-align:
### Commit Message
**Enhance Metric Engine with Physical Batch Processing**
- **Add `metric-engine` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `metric-engine` as a workspace dependency.
- **Expose Batch Modifier Functions**: Changed visibility of `TagColumnInfo`, `compute_tsid_array`, and `modify_batch_sparse` in `batch_modifier.rs` to public, and made `batch_modifier` a public module in `lib.rs`.
- **Implement Physical Batch Processing**:
- Added functions `bulk_insert_physical_region` and `bulk_insert_logical_region` in `bulk_insert.rs` to handle physical and logical batch insertions.
- Updated `pending_rows_batcher.rs` to attempt physical batch processing before falling back to logical processing, including new functions `flush_batch_physical` and `flush_batch_per_logical_table`.
- **Enhance Testing**:
- Added tests for physical region passthrough and empty batch handling in `bulk_insert.rs`.
- Introduced `with_mito_config` in `test_util.rs` for customized test environments.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Batch Processing for Table Creation and Alteration
- **`prom_store.rs`**:
- Added `create_tables_if_missing_batch` and
`add_missing_prom_tag_columns_batch` methods to handle batch creation of tables
and batch alteration to add missing tag columns.
- Implemented logic to determine missing tables and columns, and perform batch
operations accordingly.
- **`pending_rows_batcher.rs`**:
- Updated `PendingRowsBatcher` to utilize batch methods for creating tables an
adding missing columns.
- Enhanced logic to resolve table schemas and accommodate record batches after
batch operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf: concurrent catalog lookups and eliminate redundant concat_batches on ingest path
Replace sequential catalog_manager.table() calls with concurrent
futures::future::join_all in align_table_batches_to_region_schema.
This affects all three lookup loops: initial table resolution,
post-create resolution, and post-alter schema refresh. Reduces
O(N) sequential RPC latency to O(1) wall-clock time for requests
with many distinct logical tables (e.g. Prometheus remote_write).
Remove the per-logical-table concat_batches in flush_batch_physical.
Instead of merging all chunks of a table into one RecordBatch before
calling modify_batch_sparse, apply modify_batch_sparse directly to
each chunk and collect all modified chunks for a single final concat.
This eliminates one full data copy per logical table on the flush path.
* refactor: extract Prometheus schema alignment helpers into prom_row_builder module
Move six functions and their eight unit tests from pending_rows_batcher.rs
(~2386 lines) into a new prom_row_builder.rs module (~776 lines), leaving
the batcher at ~1665 lines focused on flush/worker machinery.
Extracted functions:
- accommodate_record_batch_for_target_schema (normalize incoming batch
against existing table schema)
- unzip_logical_region_schema (extract ts/field/tag columns)
- build_prom_create_table_schema (build ColumnSchema vec for table creation)
- align_record_batch_to_schema (reorder/fill/cast columns to target schema)
- rows_to_record_batch (convert proto Rows to Arrow RecordBatch)
- build_arrow_array (build Arrow arrays from proto values)
Cleaned up 12 now-unused imports from pending_rows_batcher.rs.
* feat/auto-schema-align:
### Enhance `PendingRowsBatcher` and `prom_row_builder` for Efficient Schema Handling
- **`pending_rows_batcher.rs`:**
- Refactored `submit` method to integrate table batch building and alignment into a single method `build_and_align_table_batches`.
- Removed intermediate `RecordBatch` creation, optimizing the process by directly converting proto `RowInsertRequests` into aligned `RecordBatch`es.
- Enhanced schema handling by identifying missing columns directly from proto schemas.
- **`prom_row_builder.rs`:**
- Introduced `rows_to_aligned_record_batch` for direct conversion of proto `Rows` into aligned `RecordBatch`es.
- Added `identify_missing_columns_from_proto` to detect absent tag columns without intermediate `RecordBatch`.
- Implemented `build_prom_create_table_schema_from_proto` to construct table schemas directly from proto schemas.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Add elapsed time metrics for bulk insert operations
- Updated `bulk_insert` method in `bulk_insert.rs` to record elapsed time metrics using `MITO_OPERATION_ELAPSED` for both physical and logical regions.
- Added a new test `test_bulk_insert_records_elapsed_metric` to verify that the elapsed time metric is recorded correctly during bulk insert operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* remove flush per logical region
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `flush_batch` and `flush_batch_physical` functions**
- Removed unused `catalog` and `schema` variables from `flush_batch` in `pending_rows_batcher.rs`.
- Updated `flush_batch_physical` to directly use `ctx.current_catalog()` and `ctx.current_schema()` for resolving table names.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Remove Unused Function and Associated Test
- **File:** `src/servers/src/prom_row_builder.rs`
- Removed the unused function `build_prom_create_table_schema` which was responsible for building a `Vec<ColumnSchema>` from an Arrow schema.
- Deleted the associated test `test_build_prom_create_table_schema_from_request_schema` that validated the removed function.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Remove Test**: Deleted the `test_bulk_insert_records_elapsed_metric` test from `bulk_insert.rs`.
- **Refactor Table Resolution**: Introduced `TableResolutionPlan` struct and refactored table resolution logic in `pending_rows_batcher.rs`.
- **Enhance Table Handling**: Added functions for collecting non-empty table rows, unique table schemas, and handling table creation and alteration in `pending_rows_batcher.rs`.
- **Add Tests**: Implemented tests for `collect_non_empty_table_rows` and `collect_unique_table_schemas` in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor Error Handling**: Updated error handling in `pending_rows_batcher.rs` and `prom_row_builder.rs` to use `Snafu` error context for more descriptive error messages.
- **Remove Unused Functionality**: Eliminated the `rows_to_record_batch` function and related test in `prom_row_builder.rs` as it was redundant.
- **Simplify Function Return Types**: Modified `rows_to_aligned_record_batch` in `prom_row_builder.rs` to return only `RecordBatch` without missing columns, simplifying the function's interface and related tests.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add Helper Function for Table Options in `prom_store.rs`
- Introduced `fill_metric_physical_table_options` function to encapsulate logic for setting table options, ensuring the use of flat SST format and physical table metadata.
- Updated `Instance` implementation to utilize the new helper function for setting table options.
- Added a unit test `test_metric_physical_table_options_forces_flat_sst_format` to verify the correct application of table options.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor `PendingRowsBatcher`**: Simplified worker retrieval logic in `get_or_spawn_worker` method by using a more concise conditional check.
- **Metrics Update**: Added `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` metric in `pending_rows_batcher.rs`.
- **Remove Unused Code**: Deleted multiple test functions related to record batch alignment and schema preparation in `pending_rows_batcher.rs` and `prom_row_builder.rs`.
- **Function Visibility Change**: Made `build_prom_create_table_schema_from_proto` public in `prom_row_builder.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: remove plan
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Refactor and Simplify Schema Alteration Logic
- **Removed Unused Methods**: Deleted `create_table_if_missing` and `add_missing_prom_tag_columns` methods from `PendingRowsSchemaAlterer` trait in `prom_store.rs` and `pending_rows_batcher.rs`.
- **Error Handling Improvement**: Enhanced error handling in `create_tables_if_missing_batch` method to return a specific error message for unsupported `AutoCreateTableType` in `prom_store.rs`.
- **Visibility Change**: Made `as_str` method public in `AutoCreateTableType` enum in `insert.rs` to support external access.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Message
Improve safety in `prom_row_builder.rs`
- Updated `unzip_logical_region_schema` to use `saturating_sub` for safer capacity calculation of `tag_columns`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Add TODO comments for future improvements in `pending_rows_batcher.rs`
- Added a TODO comment to consider bounding the `flush_region_writes_concurrently` function.
- Added a TODO comment to potentially limit the maximum rows to concatenate in the `flush_batch_physical` function.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Message
Enhance error handling in `pending_rows_batcher.rs`
- Updated `collect_unique_table_schemas` to return a `Result` type, enabling error handling for duplicate table names.
- Modified the function to return an error when duplicate table names are found in `table_rows`.
- Adjusted test cases to handle the new `Result` return type in `collect_unique_table_schemas`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor `partition_columns` Method**: Updated the `partition_columns` method in `multi_dim.rs`, `partition.rs`, and `splitter.rs` to return a slice reference instead of a cloned vector, improving performance by avoiding unnecessary cloning.
- **Enhance Partition Handling**: Added functions `collect_tag_columns_and_non_tag_indices` and `strip_partition_columns_from_batch` in `pending_rows_batcher.rs` to manage partition columns more efficiently, including stripping partition columns from record batches.
- **Update Tests**: Modified existing tests and added new ones in `pending_rows_batcher.rs` to verify the functionality of partition column handling, ensuring correct behavior of the new methods.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Schema Handling and Validation in `pending_rows_batcher.rs`
- **Schema Validation Enhancements**:
- Added checks for essential columns (`timestamp`, `value`) in `collect_tag_columns_and_non_tag_indices`.
- Introduced `PHYSICAL_REGION_ESSENTIAL_COLUMN_COUNT` to ensure minimum column count in `strip_partition_columns_from_batch`.
- Improved error handling for unexpected data types and duplicated columns.
- **Function Modifications**:
- Updated `strip_partition_columns_from_batch` to project essential columns without lookup.
- Modified `flush_batch_physical` to use `essential_col_indices` instead of `non_tag_indices`.
- **Test Enhancements**:
- Added tests for schema validation, including checks for unexpected data types and duplicated columns.
- Verified correct projection of essential columns in `strip_partition_columns_from_batch`.
Files affected: `pending_rows_batcher.rs`, `tests`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Add `smallvec` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `smallvec` as a workspace dependency.
- **Refactor Function**: Renamed `collect_tag_columns_and_non_tag_indices` to `columns_taxonomy` in `pending_rows_batcher.rs` and updated its return type to use `SmallVec`.
- **Update Tests**: Modified test cases in `pending_rows_batcher.rs` to reflect changes in function name and return type.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `pending_rows_batcher.rs` to Simplify Table ID Handling**
- Updated `TableBatch` struct to use `TableId` directly instead of `Option<u32>` for `table_id`.
- Simplified logic in `flush_batch_physical` by removing the check for `None` in `table_id`.
- Adjusted related logic in `start_worker` to accommodate the change in `table_id` handling.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Batch Processing Logic
- **`pending_rows_batcher.rs`**:
- Moved column taxonomy resolution inside the loop to handle schema variations across batches.
- Added checks to skip processing if both tag columns and essential column indices are empty.
- **Tests**:
- Added `test_modify_batch_sparse_with_taxonomy_per_batch` to verify batch modification logic with varying schemas.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Remove Primary Key Column Check in `pending_rows_batcher.rs`
- Removed the check for the primary key column and other essential column names in the function `strip_partition_columns_from_batch` within `pending_rows_batcher.rs`.
- Simplified the logic by eliminating the validation of column order against expected essential names.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Refactor error handling and iteration in `otlp.rs` and `pending_rows_batcher.rs`
- **`otlp.rs`**: Simplified error handling by removing `CatalogSnafu` context when awaiting table retrieval.
- **`pending_rows_batcher.rs`**: Streamlined iteration over tables by removing unnecessary `into_iter()` calls, improving code readability and efficiency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/metrics-for-bulk:
Add timing metrics for batch processing in `pending_rows_batcher.rs`
- Introduced `modify_elapsed` and `columns_taxonomy_elapsed` to measure time spent in `modify_batch_sparse` and `columns_taxonomy` functions.
- Updated `flush_batch_physical` to record these metrics using `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Summary
- **Remove Unused Code**: Eliminated the `#[allow(dead_code)]` attribute from the `compute_tsid_array` function in `batch_modifier.rs`.
- **Error Handling Improvement**: Enhanced error handling in `flush_batch_physical` function by adjusting the `match` block in `pending_rows_batcher.rs`.
- **Simplify Logic**: Streamlined the logic in `rows_to_aligned_record_batch` by removing unnecessary type casting in `prom_row_builder.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `flush_batch_physical` in `pending_rows_batcher.rs`:**
- Moved partition column stripping logic to a single location before processing region batches.
- Updated the use of `combined_batch` to `stripped_batch` for consistency in batch processing.
- Removed redundant partition column stripping logic within the region batch loop.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Update `batch_modifier.rs` Documentation and Parameter Naming
- Enhanced documentation for `compute_tsid_array` and `modify_batch_sparse` functions to clarify their logic and parameters.
- Renamed parameter `non_tag_column_indices` to `extra_column_indices` in `modify_batch_sparse` for better clarity.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
shuiyisong
yimeng
discord9
Lei, HUANG
Ruihang Xia
QuakeWang
fys
Weny Xu
dennis zhuang
Yash Agrawal
discord9
LFC
yxrxy
Ning Sun