mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-16 17:02:33 +00:00
cf16c7313bec7cbfc818c2250762e19b7a349e6e
119
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jeremyhi
|
60655a933b |
fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases A region's release lived only in memory: a restart after DROP TABLE and before the next object never re-established the dropped region's watermark, so its segments counted as live and pinned garbage collection of the whole prefix. A release now writes a release record (region id and the entry id the region is released through) into an object sealed at once, and returns once that object is durable and indexed, in both acknowledgement modes. Recovery seeds the obsolete watermarks from the release records of the indexed objects before it computes any reclaim boundary. obsolete_all releases in full; the new LogStore::release_region, a no-op by default, releases a dropped region through the largest entry id the store handed out for it, and Mito calls it after obsolete when it drops a region. Objects with release records use format version 2, the version 1 layout with the records at the end of the footer; every other object is still written as version 1. A counter of release records made durable is added. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe persisted object store WAL region releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): carry a failed region release into the next object A release whose batch failed left its record only in an orphan object, so a drop that failed there could still leave the region unreleased after a restart. The store now keeps the release records of failed batches and seals them with its next batch, on a timer tick when no entry is admitted, so the release becomes durable once a create succeeds again. The version 1 and version 2 object fixtures are frozen under tests/compatibility/wal/object_store and loaded by the format tests, and the RFC no longer says that state is persisted only in headers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): complete a region drop whose WAL release fails A drop request is not sent again once the region was removed, so a drop whose WAL release failed used to leave the region half dropped. The drop now completes, cleaning up the region's runtime state and scheduling its files for deletion, and reports the release failure. In the durable mode the object store WAL now creates one more object at stop with the release records of failed and dropped batches, so a graceful stop does not lose a release that failed. The release record counter is described as counting the records of indexed objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe the stop-time release create of the WAL actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): state the accepted window of unreleased dropped regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe release waiters in the WAL actor's outcome table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail region releases while the object store WAL is failing A failing store keeps its backlog and rejects appends until a create succeeds, so a release would otherwise wait for an upload that is not coming. A release that arrives in that state now fails at once with the retryable error, and one whose batch is still pending when the store enters it fails as well. Neither poisons the store: the record of the first waits for the next sealed batch, like that of a batch that failed, and the record of the second stays in the backlog, so both become durable once a create succeeds. Rejected releases count in rejected_appends_total. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe region releases while the object store WAL is failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the drop crash window in the drop procedure The crash window between the deletion of a table's metadata and a durable release record is a gap of the drop procedure, not of the WAL. Describe it as far as the code shows, name the remedy that belongs in the procedure, cleaning up offline a region a resumed region drop finds not open, and list it under future work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep rejected release records coalesced while failing A release rejected while the store is failing kept its record for the next sealed batch, but every timer tick then sealed that record alone behind the refused backlog, so a release repeated across ticks grew the backlog without bound. A failing store now keeps such records coalesced and seals them on the first tick after a create succeeds. The fault-matrix case of a release while the backlog is refused now runs with a 10 ms flush interval, repeats the release, and checks that the backlog does not grow and that recovery reads the record back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): keep rejected release records unsealed while failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): cite the drop procedure issue for the crash window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): let a stop seal rejected release records of a failing store Keeping the records of rejected releases out of sealed batches while the store is failing also kept them out of the seal a stop requests, so a stop whose last attempt of the backlog succeeded dropped them and the released region pinned collection after the restart. A stop now seals them with the backlog. A fault-matrix case rejects a release while the store is failing, stops, lets both creates succeed, and checks the record after a reopen. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the stop of a failing store with rejected releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep release records readable by older object store WAL readers Release records were written as a version 2 footer, which a build without it refuses to read, so rolling a node back after a drop failed to open the whole WAL prefix. They are now the single entry of a segment of the reserved region u64::MAX at the first position of a version 1 object: an older reader indexes that segment as a region it never opens, and recovery reads the segment back with one range read per object that holds one. The segment never holds the reclaim boundary and names no namespace. A durable stop whose last create of release records fails now reports the error instead of returning Ok and losing the records silently. The tests are trimmed: the release unit tests and fault-matrix cases that repeated each other are merged, and the Mito drop tests share one parameterized case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe release records as a segment of a reserved region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): report release records a poisoning stop drops A durable stop whose last create of release records met a conflict poisoned the store and returned Ok, losing the records silently. Stop now reports the error whenever release records are dropped after it began, by a rollback or by poisoning. The test of a failed release covers that conflict, and seals its record on a tick after a one-shot create failure, so the next tick cannot fail too. The documentation now says that recovery reads the segments of release records, and that one that does not decode fails recovery. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how recovery reads segments of release records Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): say that stop reports release records a conflict drops Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): restore the transiently failing release create in enqueued mode The fault matrix again covers a release whose create fails transiently in the enqueued mode: the release waits for the repeat under the same sequence and returns once it succeeds. The encode_batch doc now says the release segment is the last segment of the object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the release segment and count the restored case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL release Returning the release error from the drop left a metric physical region half dropped: the metric engine stops after the data region and never drops the metadata region or forgets the physical region, and a retried drop finds the data region gone and counts it as done. The release error gave the caller nothing to retry, since the region is already removed and the log store keeps the record for its next object. The drop now logs the error and succeeds. A metric engine test drops a physical region while the WAL releases of both mito regions fail, and checks that both regions and the physical region state are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed release no longer fails the drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL obsolete A store that the release of another region poisoned fails obsolete as well, which left the metric engine with its metadata region undropped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): drop a physical region whose WAL obsolete or release fails Cover a failed data release, two failed releases and a conflicting release that poisons the store, after the regions wrote to the WAL, and check that a store opened later keeps no segment of either region. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed obsolete does not fail the drop either Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fail the drop on an obsolete error of other WAL providers Only the object store WAL is shared by the regions of a store that the release of one can poison; Raft Engine and Kafka keep failing the drop when obsolete fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a stopped or poisoned store keeps no failed release Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
a08e5bcceb |
fix(log-store): keep the enqueued backlog on permanent object store errors (#9558)
* fix(log-store): keep the enqueued backlog on permanent object store errors On object storage the WAL's object store sits behind OpenDAL's retry layer, which marks every error it returns as persistent. The enqueued mode classified create failures by that status, so a 403 or a missing bucket was repeated every 100 ms on every create in flight, logged nothing, and made appends hang once the sealed batches were full. Object store errors are now classified by kind: denied access, an invalid configuration (how OpenDAL reports a missing S3 bucket) and a write to a missing location are non-retriable. In the enqueued mode such an error makes the store failing instead of poisoning it: the acknowledged backlog is kept under its sequences and bytes, its lowest pending batch is attempted alone with a delay that doubles from 100 ms to 30 s, appends are rejected at once with a retryable error that carries the last object store error, and the first create that succeeds ends the state. Stop uploads the backlog at once and reports a failure as before. The state is logged on entry, at most once a minute while it lasts and on exit, and exported as the `failing` gauge and the `rejected_appends_total` counter. Conflicts, encoding and catalog errors still poison the store; the durable mode is unchanged. An append held back because the sealed batches are full is now counted and timed by the stall metrics in both modes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how the enqueued object store WAL handles refused creates Describe the enqueued failure contract as the store now implements it: non-retriable object store errors are told apart by kind because OpenDAL's retry layer under the store marks every error persistent, a failing store keeps its backlog and attempts it with backoff while it rejects appends, and stop uploads the backlog at once. Add the failure matrix row, the failing gauge and the rejected appends counter, the sealed-batch admission limit and the stall metrics that now count it, and the fault matrix case counts. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): time the failing attempts where they start and fix two statements The test of the attempts of a failing store now records when each refused write starts and checks the gaps between them, instead of sampling a counter after a sleep of the test task. The delay comment and the RFC describe 30 seconds as the longest delay between two attempts, not as a bound on when appends resume, and the RFC states that the next append is held once seven sealed batches wait. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop the wait_until helper that main now defines Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): name errors that need an operator and trim the RFC contract Rename is_non_retriable to requires_intervention: such a create is still repeated, only slowly, until someone fixes the credentials, configuration or bucket. Name the backoff test after what it asserts, cut the test of the lowest pending batch to two failed rounds since the cadence is covered by the backoff test, and shorten the RFC failure contract to the contract and its rationale, leaving error mappings, request counts and logging cadence to the code comments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): tie the slow repeat to errors requiring intervention Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
83e62a7725 |
feat: remove the server-side Metric export gate (#9547)
* feat: remove the server-side Metric export gate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused export capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused CLI capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: import the packed export error selector Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
4ef312ecf2 |
docs: add the object store WAL RFC (#9195)
* docs: add the object store WAL RFC Propose an object store backed write-ahead log as a WAL provider next to Raft Engine and Kafka: a datanode batches the entries of its regions into immutable objects under one prefix, creates each object conditionally under a monotonically increasing sequence, acknowledges a write once its object is durable by default, and replays each region from its own segments after listing the prefix. A reference implementation exists and follows this RFC in a series of pull requests; the document describes it and marks what is proposed on top of it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: link the object store WAL RFC to its tracking issue Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: describe the predecessor chain and instance epoch of the object store WAL Recovery replays only the chain of objects that ends at the latest complete object, and every open starts a new epoch with an empty start object, so an object whose create was reported as failed is never replayed once a later append has been acknowledged. Update the object format, recovery, writing, the failure matrix, the garbage collection constraints, compatibility, evidence, drawbacks and the cluster mode alignment accordingly. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: identify object store WAL objects by sequence and epoch Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: derive the object store WAL epoch from the start object sequence Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: separate the retryable and the conflicting start object outcomes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: describe garbage collection of the object store WAL through the reclaim boundary Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: judge vanished objects after chain selection and fix the evidence references Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: record the stop-time conflict and the completed fault matrix in the object store WAL RFC Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: limit the stop-time conflict to objects of the same or a later epoch Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: keep the conflicting-content condition in the stop-time conflict sentence Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: report the S3 write path measurements of the object store WAL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: qualify the seal timer findings and link the S3 runs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: state the serial durable cycle behind the one-batch-per-interval observation Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
fc5831043f |
feat(cli): simplify packed metric exports with a boolean flag (#9531)
* feat(cli): select packed metric export with the layout option Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cli): inspect the packed capability error cause Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cli): follow the standard capability error source Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(cli): clarify the metric export gate helper name Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(cli): use a boolean flag for packed metric exports Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 shuiyisong
|
2df33380a7 |
feat(catalog): declare information-schema table scopes (#9504)
* feat(catalog): declare information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: rename cluster to global Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: update sqlness result Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(catalog): require information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
jeremyhi
|
18726c7602 |
fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(snapshot): bound writes and reduce logical export overhead Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: clarify metric snapshot acceptance criteria Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): split batches only for merged readers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(snapshot): cover supported types in both layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: consolidate metric snapshot experiment conclusions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): honor series statistics when splitting L0 scans Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito): explain memory risk of splitting short L0 series Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address snapshot export review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): accept source errors in lazy snapshot contexts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
c82da8308b |
feat: add bounded Metric logical-table batch DDL (#9427)
* feat: add bounded Metric logical-table batch DDL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: enforce logical batch DDL interceptor boundaries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: bound rewritten DDL size through shared SQL formatting Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve logical DDL error diagnostics Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: verify logical DDL error logging Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
shuiyisong
|
b5c0e24625 |
feat(otlp): add histogram rejection metrics and compact zero buckets (#9321)
* feat(otlp): add histogram rejection metrics and compact zero buckets Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test isolation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix minor naming Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
20dde2601f |
feat: enable native histogram ingestion by default (#9301)
* feat: enable native histogram ingestion by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Lei, HUANG
|
d0f8f4b80c |
feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
b5199bc59a |
feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: gate Jev filtering behind an opt-in Cargo feature Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: verify Jev registration with default features Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: clarify Jev concurrency scope and stabilization work Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: enable renamed ai-functions feature by default Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: remove extra AI feature-off checks Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: rename AI feature to ai_functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
4a980f350f |
docs: make packed snapshots part of the Metric export/import RFC (#9247)
* docs: revise Metric snapshots around packed Parquet objects Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: summarize experiment conclusions and bound multipart fallback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: specify packed snapshot persistence contract Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Weny Xu
|
6e2b514681 |
docs: refine Rust style guide (#9220)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
32bf865ffb |
docs: propose Metric export and import optimizations (#9121)
* docs: propose Metric export and import optimizations Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: colocate Metric RFC supporting materials Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: condense Metric PoC evidence into readable results Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: use the tracking issue as the implementation plan Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: clarify Metric export design and ordering acceptance Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Yingwen
|
9940de4917 |
feat: add scanbench query suites and structured results (#9050)
* perf: add scanbench partition diagnostics Port benchmark source and documentation from c5dd5e6356a8903f1dcf493eb46dc318612dc953. Exclude the Mito metrics changes. Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support scanbench query suites Signed-off-by: evenyag <realevenyag@gmail.com> (cherry picked from commit def9c22069b0d29c611695a68ad1b31110c845cb) Signed-off-by: evenyag <realevenyag@gmail.com> * docs: align scanbench port with existing engine metrics Remove documentation and fixture references to unported Mito metrics. Enable dev-tools in the build example and remove the obsolete force-flat-format option. Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
shuiyisong
|
b86da3d35f |
feat: support raw OTLP delta metrics (#8970)
* feat: support raw OTLP delta metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): update sqlness results for normalized label matching Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: derive temporality label from default column prefix Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): add analyze coverage for delta temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): scope label alignment to temporality marker Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: handle count-only histograms and vector broadcasts Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude temporality marker from entity descriptions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: use a fixed label for OTLP aggregation temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): preserve mixed-range semantics for raw delta Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
27165c2fdc |
feat(mito2): add SST range index writer (#8954)
* feat(mito2): add SST range index writer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): share parquet index writer Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): lazily construct index writer context Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 dennis zhuang
|
109257505e |
feat: report what the graph derives and fix two duplicate-node bugs (#8936)
* feat(semantic-graph): let the generic container yield to k8s.container A container inside a Kubernetes pod reaches the graph twice: as the k8s.container entity kube-state-metrics describes, identified by [pod uid, container name], and as the generic container the OTel resource attributes describe, identified by container.id. One physical container, two nodes. Kubernetes is the primary scenario, so k8s.container keeps its identity and the generic type stands down where it applies. Conventions gain a row-level condition for that: `suppressed_by` withdraws a declaration on rows where any of the named columns has a value. The test has to be per row, not per table — one descriptor table holds both pod rows and bare-runtime rows. Every branch that turns a declaration into rows now shares one guard (`declaration_predicate`), so the condition cannot apply to entities but not to the edges they carry. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report derived entity declarations in table_semantics information_schema.table_semantics only read table options, so the declarations the built-in conventions derive — for trace tables, for whitelisted Prometheus and OTel descriptor metrics — were invisible. "Why is my table not in the graph?" was answerable only from debug logs, which is not a self-service path. A new `entity_declarations` column reports the entities a table actually contributes: each one's identity, whether it came from an option or from the conventions, and any row-level condition attached to it. An expected entity missing from the list is the answer — the table name is not whitelisted, the source stamp is wrong, an id column is absent. The row filter widens to match: a table that declares nothing by option but derives entities by convention now appears, since it is in the graph and the view has to say so. The provider reaches the derivation through a new metadata-only trait method, keeping catalog below operator. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): never yield a container to an entity nothing derives The generic container withdrew on any row carrying a pod UID, on the assumption that k8s.container would cover it. Nothing guaranteed that: k8s.container came only from kube-state-metrics, so an OTLP-only deployment — or one whose KSM data had expired or fell outside the query window — lost the container node and its edges entirely instead of gaining a more specific one. The rule now names the superseding type rather than a trigger column, and withdraws only where that type's full identity is on the row. Both OTel sources declare k8s.container themselves, under the identity kube-state-metrics gives it ([pod uid, container name]), so the two sources name one node; `k8s.container.name` joins the descriptor's projected attributes to carry it. Resolution runs once every declaration for the table is known, so a superseding type that ends up undeclared — its columns are gone, or an explicit declaration of it was skipped — leaves the guard empty and the generic container standing. A container can change type; it cannot disappear. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(semantic-graph): build the declaration JSON by serializing a type The column was assembled entry by entry into a serde_json::Map, cloning every value and allocating a String per key. A Serialize struct that consumes the declaration moves the same data instead, and the field order now reads type, origin, identity, then description. The scan path around it was doing the same kind of avoidable work: declarations were derived before the predicate could discard the table, the supersession pass cloned every declaration's identity to look up one, and the option parse claimed the time index for tables that turn out to declare nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): keep the structured identity for single-column ids `entity_id_attrs` was NULL whenever the identity came from one column, so a consumer holding `host` = `a3f2...` had no way to tell which column produced it, and no way back to the source table. Most identities are single-column — host, k8s.pod, k8s.node, service — so the common case was the opaque one. Build the JSON object unconditionally. Entity equality still reads `entity_id` alone, so this changes no merging: it records which attributes the id was assembled from, beside an id that deliberately omits them. Also corrects the `entity_id` column doc, which still described the `k=v,k=v` rendering replaced in #8904 by values joined in declared order. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): keep what a container carried when it yields Yielding to k8s.container cost the row two things it had as a generic container. The runtime container id, which kube_pod_container_info keeps descriptive precisely because it is the handle back to runtime logs and metrics, went missing entirely: neither an id nor an attribute of any node. And the edge vocabulary knew only the generic type, so a row with the more complete labels ended up with fewer connections than one without — the container layer no longer reached its host. Both OTel sources now keep the runtime id and name descriptive on k8s.container, and the vocabulary gains the two edges that mirror the generic type's. Nothing checks a supersession against the edge vocabulary, so that requirement is written down where the rule is. Also: the conventions-failure path now reports the explicit half as its comment always claimed, entity_declarations reports scope columns, and identifies() is private again now that only declaration_predicate calls it. The two RFCs catch up with entity_id_attrs being unconditional, the view listing convention-derived tables, and supersession existing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report unmatched clients and the longest request `real_wins` pins a `calls` edge's RED metrics to the observed span pairs: when a window's edge key holds any pair, the unmatched client spans are suppressed so one (window, edge) yields one row. That leaves no way to tell a callee that stopped responding from traffic that stopped arriving — both show up as a lower request_count. Add two columns to `semantic_relationships`: - `unmatched_count` — client spans with no server span, counted outside `real_wins` on the same row, so the suppressed population stays visible without splitting the edge into two rows. NULL for agent calls, whose inner join leaves nothing unmatched, and for declared edges. - `duration_max` — the longest single request. It goes through `real_wins`: a pair is timed by the server span while an unmatched client is timed by its own (network wait included), so mixing them would make the max describe a different population than duration_sum and duration_count. Agent calls compute it from the child spans they already aggregate. The projection contract goes from 16 to 18 columns; every branch projects both. Explicit column queries are unaffected, `SELECT *` and ordinal-based readers see the new shape. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(semantic-graph): drop the duplicated source-gate case The whitelist gate on `source=opentelemetry` is already covered by `otel_implicit_declarations_are_gated` and by the wrong-source case in `table_semantics`; here it only paid for another table create, insert and full graph derivation. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style(semantic-graph): trim the comments back to what the code cannot say Several comments restated the code, repeated a rationale already stated at the type or in the RFC, or explained a test in more words than the test body. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
6d86e6ff06 |
feat: synthesize OTLP resource descriptor for the semantic entity graph (#8904)
* fix(servers): compose OTLP metrics job from service.namespace/service.name
The OTel Prometheus compatibility spec defines job as
"<service.namespace>/<service.name>" when the namespace is present.
The OTLP metrics path only used the bare service.name, so the job tag
diverged from target_info produced by Prometheus-side exporters for the
same resource. Compose the namespace form, and keep not fabricating a
job when service.name is absent.
Behavior change: resources carrying service.namespace now get
"namespace/name" as their job tag value.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion
Ordinary OTLP metrics scatter filtered resource attributes as tags over
every logical metric table, so metrics-only services contribute nothing
to the semantic entity graph. Each request now also projects its
distinct resources into one info-metric-shaped mito table,
otel_resource_info: a fixed allowlist of identity-relevant attributes
under their raw OTel keys (independent of the label translation
strategy and the promote/ignore headers) plus derived job/instance
compatibility columns, value 1.0, and the newest data-point timestamp.
The descriptor is written after the main insert is committed; a failure
there (conflicting pre-existing table, auto-create disabled) degrades
to an OTLP partial_success warning with rejected_data_points = 0
instead of failing the request and triggering client retries of
already-accepted data. A request writing a metric named
otel_resource_info suppresses synthesis. Legacy mode is unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): otel info-metric conventions with host/container entities
Whitelist the ingestion-synthesized otel_resource_info descriptor via a
new otel_info_metrics conventions map, gated on source=opentelemetry
(the existing gate hardcoded source=prometheus). Its declarations use
explicit descriptive lists instead of descriptive_rest so identifying
attributes of other entities do not leak into service.instance.
Conventions tightened per the Astronomy Shop findings: host identity is
host.id with host.name descriptive only (host.name is not stable across
SDKs and resource detectors), a generic container entity (new entity
type) is declared only when container.id is present, and trace-v1
tables now synthesize host/container from their flattened resource
attributes too. New co-declared edges: service.instance runs_on
container, container runs_on host.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): cover the resource descriptor in integration tests
Covers the descriptor's raw-key columns and info-metric options through
the HTTP path, the namespace/name job composition end-to-end, column
names being independent of the translation strategy, the allowlist
excluding unlisted resource attributes, auto-create after a drop, the
metric-name collision suppressing synthesis, and the partial-success
warning (rejected_data_points = 0) when a pre-existing incompatible
table fails the descriptor write while metric data is accepted.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cargo fmt
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(frontend): degrade descriptor permission denial to a warning
A table-level permission policy denying otel_resource_info would have
failed the whole OTLP metrics request because the descriptor's
permission check ran before the main insert. The descriptor is derived
enrichment: check its permission in the degrade path so a denial skips
the write and surfaces as the partial-success warning, like any other
descriptor write failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): guard descriptor writes with semantic ownership markers
A pre-existing schema-compatible table named otel_resource_info would
silently receive descriptor rows while its missing semantic stamps kept
it out of the entity graph. The descriptor write now requires the
auto-created table's ownership markers (mito engine + signal_type +
source + metric.type=info + metadata_quality=declared) and otherwise
degrades to the partial-success warning; the entity-graph gate for the
otel whitelist likewise requires metric.type=info, so a user table
stamped with only signal/source no longer picks up implicit
declarations.
Also fold the descriptor write cost into the response and surface the
degrade warning through the otel-arrow BatchStatus status_message.
Integration tests pin the full marker set on auto-create and that an
existing owned descriptor keeps accepting writes without degrading —
a missing marker would otherwise silently stop every descriptor write
after the first request.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): build descriptor rows without the per-resource BTreeMap
Projecting a resource allocated a BTreeMap and then collected it into the
row key, and every attribute was matched against the allowlist by linear
scan. Collect the tags into a Vec and sort once, and match the allowlist
instead of scanning it. Measured on the conversion path: descriptor work
drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape
(1000 resources with 4 data points each), where the cost tracks resource
count rather than data-point count.
Also trims the comments and tests added with the descriptor to what
carries information.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): pin the descriptor permission-denial degrade path
A policy denying the descriptor table must not fail the metrics request,
which the fix in
|
||
|
Yingwen
|
046862f067 |
feat(cmd): improve parquet rewrite fidelity and scanbench output (#8947)
* feat(cmd): preserve parquet rewrite metadata settings Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): print scanner metrics in verbose scanbench Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): preserve SST truncation settings in property dumps Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
0cc83c4570 |
feat(cmd): add parquet development tools (#8939)
* feat(cmd): add parquet metadata development tool Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): add parquet rewrite development tool Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): add SST replacement development tool Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): support local parquet files in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): clean up parquet development tools Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(cmd): share datanode tool utilities Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): satisfy parquet tool lints Signed-off-by: evenyag <realevenyag@gmail.com> * docs: document parquet development tools Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(cmd): rename SST replacement module Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): validate parquet rewrite options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
shuiyisong
|
1c5eabcbbf |
feat(otlp): support cumulative exponential histograms (#8900)
* feat: implement exponential histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove duplicate tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): enforce exponential histogram ingestion safety Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): remove protocol-coupled histogram checks Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): reuse native histogram schema across data points Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: merge repeated OTLP histogram fragments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): build rejection messages lazily Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: add doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
a28a0466b5 |
feat(promql): define native histogram semantics (#8758)
* feat(promql): define native histogram semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: change author of the rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update comments and tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add test Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
jeremyhi
|
448f973593 |
fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
8d9f24d1b6 |
docs: add entity relationships and graph query RFC (#8605)
* docs: add entity relationships and graph query RFC Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: revise entity-graph RFC after review - Zero-configuration declarations: the Prometheus-on-Kubernetes convention pack (job/instance per the OTel-Prometheus compatibility spec, k8s SD labels, *_info descriptors, target_info enrichment) alongside the OTLP trace auto-stamp, plus Remote Write 2.0 inline metadata. - Calls endpoints follow the service entity declaration; self-calls compare full endpoint ids; no silent identity fallback. - Strict time-window contract: the source window is never narrower than the query's observed_at range; unsafe-to-extract predicates error instead of silently defaulting. - scope removed from the relationship schema (kept on entities as a display property); entity row contract restated per projected observation; endpoint encoding documented as the v1 storage-level key with its known collision limitation. - Sampling caveats corrected (ratios are representative only under unbiased sampling); snapshot relation synthesizes endpoint-only vertices; shared attributes provide join keys while co-declaration provides relationship semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: align entity-graph RFC contracts and tighten prose - Metric/trace service unification is promised only when service.namespace is empty and job is not relabeled (the compatibility spec renders job as <namespace>/<name>); otherwise alignment needs pipeline normalization or explicit declarations. - Entity row contract stated once (per projected observation); the calls defining SQL is marked as the single-column simplification of the declaration-derived endpoint ids; Remote Write 2.0 metadata intake and the Prometheus implicit declarations are listed as M1 work. - Compress survey/example/reference prose. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: pin down the k8s convention pack rules and edge directions - kube_pod_owner implicitly declares k8s.workload with id (namespace, owner_kind, owner_name) and derives pod part_of workload; target_info's non-job/instance labels are implicit service descriptive columns — fixed rules, no new declaration syntax. - One direction for pod placement: k8s.pod runs_on k8s.node (pod added to runs_on sources; node->pod removed from contains). - Drop the remaining 'canonical' wording for the v1 storage-level id. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: make contains/part_of a true inverse pair part_of covers service.instance->service and k8s.pod->k8s.workload with contains as its inverse; has_instance is dropped. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: rewrite entity-graph RFC as a design narrative Restructure for human review: mainline first, cases illustrate the design instead of specifying it. - New Architecture and Benefits and Drawbacks sections; the calls derivation stays expanded as the flagship example while schema enumerations, window-rule listings, and executor edge-case handling move out of the document. - The cross-signal promise is stated honestly: neighbours and their source tables are discovered first, their telemetry is the next query — one engine, two statements; the worked example shows the full declaration -> entity -> edge -> telemetry flow. - Default materialisation added as the most direct alternative to read-time derivation, with its costs. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: final wording pass on the entity-graph RFC Mark the property-graph DDL as illustrative rather than settled M2 syntax, credit standards as foundations rather than claiming wholesale alignment, and clean up punctuation-heavy prose. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: keep the RFC at design altitude Demote the convention-pack rule details and snapshot property-merge semantics to the implementing changes; correct the single-trace-table assumption (traces can be routed to multiple tables); record attribute-key participation in entity equality as an open question. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: correct service graph terminology Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
59d40c39b1 |
feat: prw_v2 initial commit with sample ingestion (#8361)
* chore: add v2 entrance Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: decode request Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: implement remote write v2 for samples Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove hand-written proto Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: refactor Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: merge tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add source version field Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update proto Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update proto Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
d6c37778ae |
feat: table semantic layer information_schema view (Phase 3) (#8240)
* feat: table semantic layer information_schema view (Phase 3) Add `information_schema.table_semantics`, a queryable view over the table semantic layer. One row per table that carries at least one `greptime.semantic.*` option: the signal-agnostic keys (signal_type/source/pipeline/metadata_quality) are promoted to columns and the remaining signal-specific keys are folded into a `semantic_options` JSON string. Tables with no semantic key are excluded. Stacked on Phase 2. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: address PR review on table_semantics - fold JSON serialization failure into None instead of unwrap/panic - drop per-row Vec allocation in predicate eval; use a fixed array - align RFC view name with the shipped `table_semantics` Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: update results Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
31c2c1f6db |
feat: table semantic layer per-table enrichment (Phase 2) (#8218)
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
dennis zhuang
|
bce925b34d |
RFC: Table Semantic Layer (#8199)
* docs: RFC for Table Semantic Layer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: address comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: address comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: remove source_protocol and future work for sdk versions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
407d048136 |
feat: update project status and architecture (#8182)
* feat: update readme Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: image link Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: style Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: adds agent onboarding Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: address by CR Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: link Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 discord9
|
44b2d9b3a4 |
docs: rfc for remote dyn filter (#7931)
* docs: remote dyn filter rfc Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
jeremyhi
|
16fcbb2729 |
feat: export import v2 pr1 (#7785)
* feat: v2 schema handling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: impl m1.5 ddl export/import and schema tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: git ignore update Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: add license header Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: make fmt-check happy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: Run imported DDL against the intended schema Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: Canonicalize schema names after case-insensitive check Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: escape sql funcs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: Fixed by carrying explicit execution_schema in DdlStatement instead of parsing schema from SQL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: Fixed by encoding schema names as safe path segments in shared DDL path helpers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(cli): make export/import v2 schema recovery DDL-only Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by clippy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: follow our styling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): reject remote snapshot URIs with empty root Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): dedupe schema filters after canonicalization Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): schema-scoped detection to cover external tables Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 discord9andSisyphus
|
f2bccbd96a |
docs: flow inc query rfc (#7816)
* docs: flow inc query rfc Signed-off-by: discord9 <discord9@163.com> * chore: typo Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * docs: clarify flow incremental stale recovery Clarify that flush-boundary invalidation is part of IncrementalQueryStale, and document the in-memory checkpoint plus cold-start full snapshot recovery model. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> |
||
|
Yingwen
|
5c8ece27e0 |
feat: improve filter support for scanbench (#7736)
* feat: cast filters type for scanbench Signed-off-by: evenyag <realevenyag@gmail.com> * chore: pub file_range mod So we can use the pub struct FileRange in other places Signed-off-by: evenyag <realevenyag@gmail.com> * fix: add api as dev-dependency to cmd for clippy Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support profiling after warmup Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
0c30bf1a10 |
feat: add a subcommand to bench scan (#7722)
* feat: support scan bench Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support projection by name Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support force flat format Signed-off-by: evenyag <realevenyag@gmail.com> * feat: spawn tasks to poll streams Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support filter config Signed-off-by: evenyag <realevenyag@gmail.com> * feat: scan bench support wal Signed-off-by: evenyag <realevenyag@gmail.com> * chore: support not providing provider in wal Signed-off-by: evenyag <realevenyag@gmail.com> * fix: skip wal replay Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: wrap EngineComponents Signed-off-by: evenyag <realevenyag@gmail.com> * docs: add scanbench doc Signed-off-by: evenyag <realevenyag@gmail.com> * chore: change --skip-wal-replay to --enable-wal Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove limit from config Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Ruihang Xia
|
38b5df574c |
refactor: SQL and gRPC server handlers (#7637)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
discord9
|
8c5550bc62 |
feat: debug/mem/symbol (#7573)
Signed-off-by: discord9 <discord9@163.com> |
||
|
jeremyhi
|
0f7512a8c9 |
feat: export import v2 rfc (#7499)
* feat: export import v2 rfc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: csv format and --force arg Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: refine by comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuangandRuihang Xia
|
17e2b98132 |
docs: rfc for vector index (#7353)
* docs: rfc for vector index Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: explain why choose USearch and distributed query Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: row id mapping Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refine proposal Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * rename rfc file Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
discord9
|
29bbff3c90 |
feat: gc worker only local regions&test (#7203)
* feat: gc worker only on local region Signed-off-by: discord9 <discord9@163.com> * more check Signed-off-by: discord9 <discord9@163.com> * chore: stuff Signed-off-by: discord9 <discord9@163.com> * fix: ignore async index file for now Signed-off-by: discord9 <discord9@163.com> * fix: file removal rate calc Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Ruihang XiaandZhenchi
|
cc61af7c65 |
feat: dynamic enable or disable trace (#6609)
* wip Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * set `TRACE_RELOAD_HANDLE` Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * wrap http api Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update dependencies Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * type alias and unwrap_or_else Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * better error handling Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplify Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lazy initialize tracer Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * integration test Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Zhenchi <zhongzc_arch@outlook.com> |
||
|
Lei, HUANGandshuiyisong
|
e386a366d0 |
feat: add HTTP endpoint to control prof.gdump feature (#6999)
* feat/gdump: ### Add Support for Jemalloc Gdump Flag - **`jemalloc.rs`**: Introduced `PROF_GDUMP` constant and added functions `set_gdump_active` and `is_gdump_active` to manage the gdump flag. - **`error.rs`**: Added error handling for reading and updating the jemalloc gdump flag with `ReadGdump` and `UpdateGdump` errors. - **`lib.rs`**: Exposed `is_gdump_active` and `set_gdump_active` functions for non-Windows platforms. - **`http.rs`**: Added HTTP routes for checking and toggling the jemalloc gdump flag status. - **`mem_prof.rs`**: Implemented handlers `gdump_toggle_handler` and `gdump_status_handler` for managing gdump flag via HTTP requests. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * Update docs/how-to/how-to-profile-memory.md Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> * fix: typo in docs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> |
||
|
discord9
|
4507736528 |
docs: laminar flow rfc (#6928)
* docs: luminar flow Signed-off-by: discord9 <discord9@163.com> * chore: wording details Signed-off-by: discord9 <discord9@163.com> * more details Signed-off-by: discord9 <discord9@163.com> * rearrange phases Signed-off-by: discord9 <discord9@163.com> * refactor: use embed frontend per review Signed-off-by: discord9 <discord9@163.com> * todo per review Signed-off-by: discord9 <discord9@163.com> * docs: seq read impl for luminar flow Signed-off-by: discord9 <discord9@163.com> * rename Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Ning Sun
|
028effe952 |
docs: update memory profiling description doc (#6960)
doc: update memory profiling description doc |
||
 Sicong Hu
|
bc20b17bc5 |
docs(rfc): async index build (#6757)
* docs(rfc): async index build Signed-off-by: SNC123 <sinhco@outlook.com> * update rfc for retaining sync build Signed-off-by: SNC123 <sinhco@outlook.com> * fix bug and update rfc for index resource management Signed-off-by: SNC123 <sinhco@outlook.com> * update rfc for manual rebuild Signed-off-by: SNC123 <sinhco@outlook.com> --------- Signed-off-by: SNC123 <sinhco@outlook.com> |
||
  LFCandNing Sun
|
05529387d9 |
refactor: use DataFusion's UDAF implementation directly (#6776)
* refactor: use DataFusion's UDAF implementation directly Signed-off-by: luofucong <luofc@foxmail.com> * remove: delete how-to guide for writing aggregate functions Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * refactor: port json_encode_path to datafusion udaf Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> Signed-off-by: Ning Sun <sunning@greptime.com> Co-authored-by: Ning Sun <sunning@greptime.com> |
||
|
discord9
|
080b4b5d53 |
docs(rfc): rfc for gc worker (#6572)
* docs: rfc for gc worker Signed-off-by: discord9 <discord9@163.com> * revise: gc worker now run after compaction&long run queries check Signed-off-by: discord9 <discord9@163.com> * more drawback Signed-off-by: discord9 <discord9@163.com> * flowchart Signed-off-by: discord9 <discord9@163.com> * read consist optional&more detail Signed-off-by: discord9 <discord9@163.com> * chore: rephrase Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Ruihang Xia
|
869f8bf68a |
docs(rfc): compatibility test framework (#6460)
* docs(rfc): compatibility test framework Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * rename file Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix typo Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Yingwen
|
9527e0df2f |
feat: HTTP API to activate/deactive heap prof (activate by default) (#6593)
* feat: add HTTP API to activate/deactivate heap profiling Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add HTTP API to get profiling status Signed-off-by: evenyag <realevenyag@gmail.com> * feat: enable heap prof by default Signed-off-by: evenyag <realevenyag@gmail.com> * build: add "prof:true,prof_active:false" as default env to dockerfiles Signed-off-by: evenyag <realevenyag@gmail.com> * feat: activate heap profiling after log initialization Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add memory options to control whether to activate profiling Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update docs Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt toml Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix config test Signed-off-by: evenyag <realevenyag@gmail.com> * docs: usage of new api Signed-off-by: evenyag <realevenyag@gmail.com> * chore: log profile after version Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update how to docs Signed-off-by: evenyag <realevenyag@gmail.com> * docs: fix how to docs Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |