mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-06 21:48:58 +00:00
dennis zhuang
09c0b23a23
* fix(meta): actually acquire logical table locks in alter-logical-tables procedure The procedure listed its logical table locks from table_info_values, which is only filled during Prepare, while procedure lock keys are fixed at submission — so the logical locks were never acquired. Today every writer of a logical table's info is serialized by the physical table lock, which hides the problem; a metadata-only alter procedure targeting a single logical table would race it. Resolve the logical table ids at submission, persist them in the procedure state (serde(default): state dumped by older versions keeps the previous behavior), lock physical + logical tables, and re-check the resolved ids against the locked set at Prepare so a table dropped and recreated after submission cannot be mutated without a lock. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: manage semantic table options via ALTER TABLE SET/UNSET CREATE TABLE accepts greptime.semantic.* options, but ALTER TABLE SET routed every option through SetRegionOption, whose closed match rejects them — tables auto-created by ingestion could never receive semantic declarations after the fact. Semantic options are pure metadata markers no region consumes, so they now take a metadata-only alter, following the repartition-hint precedent: - New AlterKind::SetAnnotations/UnsetAnnotations carrying an AnnotationFamily (currently only Semantic), so future marker-style option families reuse the same machinery. The converter classifies a SET/UNSET batch by key prefix and rejects batches that mix annotation keys with regular options. - The procedure reuses the MetadataOnly flow: no region dispatch, table-info update plus cache invalidation only. - Validation lives in the table-meta mutation layer, so it runs at frontend verification and again in the procedure's prepare step under the table lock: SET is strict (known key, value domain, entity columns exist and render as strings); UNSET is lenient inside the namespace so stale keys can be cleaned up. ModifyColumnTypes re-checks columns referenced by entity declarations at the same layer, closing a verify-then-execute race. - Logical metric tables are supported: an annotation alter submits a regular alter-table task locking only the logical table, and the DDL manager's physical-route guard admits it. - create_table_info re-checks semantic value domains for gRPC-built expressions that bypass the SQL parser. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(table): centralize annotation option classification and validation Address review feedback on the AnnotationFamily abstraction: with only one variant that every consumer immediately destructured, the generality was fake. Make it real and exhaustive instead: - AnnotationFamily gains RepartitionHint: repartition.column.hint is the same kind of marker option (pure metadata, no region consumes it) and previously had a hand-rolled special case in the converter, the metadata-only classifier, and a dedicated AlterKind pair — all deleted, one classification API remains. Per-family logical-table eligibility (allows_logical_tables) replaces the hard-coded Semantic check in the DDL manager guard. - One validation core in the table crate (check_annotation) serves both DDL entry points. CREATE and ALTER previously duplicated the rules; each keeps its existing error variants, status codes and messages via thin adapters over a typed error (ALTER missing column stays 4002 TableColumnNotFound, CREATE stays InvalidArguments). - The batch classifier returns Result instead of swallowing the mixed-batch error: a mixed SET on a logical table now reports the actual problem instead of UnexpectedLogicalRouteTable, and the flow classifiers propagate instead of guessing. The converter also moves its owned payloads instead of cloning them. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(meta): cover logical-table annotation alter routing The route-guard branch admitting metadata-only annotation alters on logical tables was only exercised end to end by sqlness. Pin it at the DDL manager level: a semantic SET on a logical table succeeds, updates only the logical table's metadata and dispatches nothing to datanodes; a mixed batch reports its own error instead of the route guard's; the repartition hint stays rejected on logical routes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(table): keep entity guard on ADD COLUMN and report missing columns first Review follow-ups: the old verify_alter loop scanned the post-alter schema, so it also caught DROP COLUMN followed by re-adding the declared column with a non-string type — the mutation-layer move only kept the MODIFY path. Guard add_columns the same way (this also covers ingestion auto-alter). And run the MODIFY drift check after the existence lookup, so altering a dropped-but-still-declared column reports ColumnNotExists (4002) like every other MODIFY on a missing column. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style(grpc-expr): drop a test comment restating the classifier doc Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(table): rename annotation validation helpers per review check_annotation* validated and normalized; align the names with the validate_and_normalize_* convention nearby, and spell out AnnotationContext (Cx is not used in this repo). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
207 lines
7.9 KiB
Plaintext
207 lines
7.9 KiB
Plaintext
DESC TABLE information_schema.table_semantics;
|
|
|
|
+------------------+--------+-----+------+---------+---------------+
|
|
| Column | Type | Key | Null | Default | Semantic Type |
|
|
+------------------+--------+-----+------+---------+---------------+
|
|
| table_catalog | String | | NO | | FIELD |
|
|
| table_schema | String | | NO | | FIELD |
|
|
| table_name | String | | NO | | FIELD |
|
|
| table_id | UInt32 | | NO | | FIELD |
|
|
| signal_type | String | | YES | | FIELD |
|
|
| source | String | | YES | | FIELD |
|
|
| source_version | String | | YES | | FIELD |
|
|
| pipeline | String | | YES | | FIELD |
|
|
| metadata_quality | String | | YES | | FIELD |
|
|
| semantic_options | String | | YES | | FIELD |
|
|
+------------------+--------+-----+------+---------+---------------+
|
|
|
|
CREATE TABLE metrics_tagged (
|
|
ts TIMESTAMP TIME INDEX,
|
|
val DOUBLE,
|
|
)
|
|
WITH (
|
|
'greptime.semantic.signal_type' = 'metric',
|
|
'greptime.semantic.source' = 'opentelemetry',
|
|
'greptime.semantic.source_version' = '2.0',
|
|
'greptime.semantic.pipeline' = 'greptime_metric_v1',
|
|
'greptime.semantic.metric.metadata_quality' = 'declared',
|
|
'greptime.semantic.metric.type' = 'counter',
|
|
'greptime.semantic.metric.unit' = 'By'
|
|
);
|
|
|
|
Affected Rows: 0
|
|
|
|
CREATE TABLE traces_tagged (
|
|
ts TIMESTAMP TIME INDEX,
|
|
span_name STRING,
|
|
)
|
|
WITH (
|
|
'greptime.semantic.signal_type' = 'trace',
|
|
'greptime.semantic.source' = 'opentelemetry',
|
|
'greptime.semantic.trace.conventions' = 'https://opentelemetry.io/schemas/1.27.0'
|
|
);
|
|
|
|
Affected Rows: 0
|
|
|
|
-- A table with no semantic options must not appear in the view.
|
|
CREATE TABLE plain_table (
|
|
ts TIMESTAMP TIME INDEX,
|
|
val DOUBLE,
|
|
);
|
|
|
|
Affected Rows: 0
|
|
|
|
SELECT table_schema, table_name, signal_type, source, source_version, pipeline, metadata_quality, semantic_options
|
|
FROM information_schema.table_semantics
|
|
ORDER BY table_name;
|
|
|
|
+--------------+----------------+-------------+---------------+----------------+--------------------+------------------+-----------------------------------------------------------------+
|
|
| table_schema | table_name | signal_type | source | source_version | pipeline | metadata_quality | semantic_options |
|
|
+--------------+----------------+-------------+---------------+----------------+--------------------+------------------+-----------------------------------------------------------------+
|
|
| public | metrics_tagged | metric | opentelemetry | 2.0 | greptime_metric_v1 | declared | {"metric.type":"counter","metric.unit":"By"} |
|
|
| public | traces_tagged | trace | opentelemetry | | | | {"trace.conventions":"https://opentelemetry.io/schemas/1.27.0"} |
|
|
+--------------+----------------+-------------+---------------+----------------+--------------------+------------------+-----------------------------------------------------------------+
|
|
|
|
-- Predicate pushdown on a promoted column.
|
|
SELECT table_name, signal_type
|
|
FROM information_schema.table_semantics
|
|
WHERE signal_type = 'metric'
|
|
ORDER BY table_name;
|
|
|
|
+----------------+-------------+
|
|
| table_name | signal_type |
|
|
+----------------+-------------+
|
|
| metrics_tagged | metric |
|
|
+----------------+-------------+
|
|
|
|
DROP TABLE metrics_tagged;
|
|
|
|
Affected Rows: 0
|
|
|
|
DROP TABLE traces_tagged;
|
|
|
|
Affected Rows: 0
|
|
|
|
DROP TABLE plain_table;
|
|
|
|
Affected Rows: 0
|
|
|
|
-- ALTER TABLE manages semantic declarations on existing tables: SET appears in
|
|
-- the view, UNSET disappears from it.
|
|
CREATE TABLE altered_semantics (
|
|
ts TIMESTAMP TIME INDEX,
|
|
svc STRING,
|
|
payload BINARY,
|
|
val DOUBLE,
|
|
);
|
|
|
|
Affected Rows: 0
|
|
|
|
ALTER TABLE altered_semantics SET 'greptime.semantic.signal_type' = 'metric', 'greptime.semantic.entity.service.id' = 'svc';
|
|
|
|
Affected Rows: 0
|
|
|
|
SELECT table_name, signal_type, semantic_options
|
|
FROM information_schema.table_semantics
|
|
WHERE table_name = 'altered_semantics';
|
|
|
|
+-------------------+-------------+-----------------------------+
|
|
| table_name | signal_type | semantic_options |
|
|
+-------------------+-------------+-----------------------------+
|
|
| altered_semantics | metric | {"entity.service.id":"svc"} |
|
|
+-------------------+-------------+-----------------------------+
|
|
|
|
-- Semantic options never share a statement with regular table options.
|
|
ALTER TABLE altered_semantics SET 'greptime.semantic.source' = 'prometheus', 'ttl' = '7d';
|
|
|
|
Error: 1004(InvalidArguments), Invalid table option request: `greptime.semantic.*` options must be altered separately from other table options
|
|
|
|
-- Unknown semantic keys are rejected on SET.
|
|
ALTER TABLE altered_semantics SET 'greptime.semantic.nonsense' = 'x';
|
|
|
|
Error: 1004(InvalidArguments), Invalid alter table(altered_semantics) request: unknown semantic option `greptime.semantic.nonsense`
|
|
|
|
-- Values outside the key's domain are rejected.
|
|
ALTER TABLE altered_semantics SET 'greptime.semantic.signal_type' = 'garbage';
|
|
|
|
Error: 1004(InvalidArguments), Invalid alter table(altered_semantics) request: invalid value `garbage` for semantic option `greptime.semantic.signal_type`
|
|
|
|
-- Entity columns must exist.
|
|
ALTER TABLE altered_semantics SET 'greptime.semantic.entity.host.id' = 'no_such_column';
|
|
|
|
Error: 4002(TableColumnNotFound), Column no_such_column not exists in table altered_semantics
|
|
|
|
-- Entity columns must render as strings.
|
|
ALTER TABLE altered_semantics SET 'greptime.semantic.entity.host.id' = 'payload';
|
|
|
|
Error: 1004(InvalidArguments), Invalid alter table(altered_semantics) request: entity column `payload` (option `greptime.semantic.entity.host.id`) has type `Binary`, which cannot render as a string
|
|
|
|
ALTER TABLE altered_semantics UNSET 'greptime.semantic.entity.service.id';
|
|
|
|
Affected Rows: 0
|
|
|
|
SELECT table_name, signal_type, semantic_options
|
|
FROM information_schema.table_semantics
|
|
WHERE table_name = 'altered_semantics';
|
|
|
|
+-------------------+-------------+------------------+
|
|
| table_name | signal_type | semantic_options |
|
|
+-------------------+-------------+------------------+
|
|
| altered_semantics | metric | |
|
|
+-------------------+-------------+------------------+
|
|
|
|
DROP TABLE altered_semantics;
|
|
|
|
Affected Rows: 0
|
|
|
|
-- Logical metric tables take the same metadata-only path.
|
|
CREATE TABLE phy_sem (ts TIMESTAMP TIME INDEX, val DOUBLE) engine=metric with ("physical_metric_table" = "");
|
|
|
|
Affected Rows: 0
|
|
|
|
CREATE TABLE logical_sem (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) engine=metric with ("on_physical_table" = "phy_sem");
|
|
|
|
Affected Rows: 0
|
|
|
|
ALTER TABLE logical_sem SET 'greptime.semantic.signal_type' = 'metric', 'greptime.semantic.entity.host.id' = 'host';
|
|
|
|
Affected Rows: 0
|
|
|
|
SELECT table_name, signal_type, semantic_options
|
|
FROM information_schema.table_semantics
|
|
WHERE table_name = 'logical_sem';
|
|
|
|
+-------------+-------------+---------------------------+
|
|
| table_name | signal_type | semantic_options |
|
|
+-------------+-------------+---------------------------+
|
|
| logical_sem | metric | {"entity.host.id":"host"} |
|
|
+-------------+-------------+---------------------------+
|
|
|
|
-- Regular options still cannot be altered on logical tables.
|
|
ALTER TABLE logical_sem SET 'ttl' = '7d';
|
|
|
|
Error: 1004(InvalidArguments), Alter logical tables invalid arguments: Only support add columns operation
|
|
|
|
ALTER TABLE logical_sem UNSET 'greptime.semantic.entity.host.id';
|
|
|
|
Affected Rows: 0
|
|
|
|
SELECT table_name, signal_type, semantic_options
|
|
FROM information_schema.table_semantics
|
|
WHERE table_name = 'logical_sem';
|
|
|
|
+-------------+-------------+------------------+
|
|
| table_name | signal_type | semantic_options |
|
|
+-------------+-------------+------------------+
|
|
| logical_sem | metric | |
|
|
+-------------+-------------+------------------+
|
|
|
|
DROP TABLE logical_sem;
|
|
|
|
Affected Rows: 0
|
|
|
|
DROP TABLE phy_sem;
|
|
|
|
Affected Rows: 0
|
|
|