mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-18 08:44:07 +00:00
2d2e351dbd3414d760be20803f7e5fb03b62209b
6657
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2d2e351dbd |
fix(servers): report the dryrun row limit instead of "data is required" (#10491)
* test(servers): expect the dryrun row limit in the error message Signed-off-by: dayan02dev <pawarudayan92@gmail.com> * fix(servers): report the dryrun row limit instead of "data is required" Signed-off-by: dayan02dev <pawarudayan92@gmail.com> --------- Signed-off-by: dayan02dev <pawarudayan92@gmail.com> |
||
|
|
95809cd3da |
fix: return file logging initialization errors instead of panicking (#10478)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
ead519a8c1 |
feat(postgres): support Arrow IPC COPY via Flight bulk ingestion (#10470)
* feat(postgres): support Arrow IPC COPY via Flight bulk ingestion Signed-off-by: Ning Sun <sunning@greptime.com> * docs: remove Arrow COPY guide for separate documentation Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): bound Arrow COPY LZ4 decompression before ingestion Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
0794620b3c |
test(operator): allow memory admission before metering rejection (#10486)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
48aa0ed3ae |
feat(log-store): fence object store WAL I/O with node leases (#10438)
* feat(log-store): fence object store WAL I/O and recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(datanode): bind object store WAL I/O to node leases Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): scope lease fencing to pre-enablement support Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(datanode): wire node lease fencing behind a cluster WAL gate Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(wal): clarify fixed object-store directory generation Use a deployment-neutral name for the directory generation shared by standalone and distributed WAL stores. Keep the value and persisted prefixes unchanged. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): refine node lease expiry and retry handling Report stop failures only for discarded backlog, preserve retry counts for unissued creates, and request route refresh on lease expiry. Install I/O permits once, clarify shutdown semantics, and log permission transitions. Add deterministic regressions for stopped probes and lazy delete polling. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): observe WAL I/O permission availability Report unavailable and available permission transitions at info level without treating the first grant as an expired lease. Track unavailable actors with a gauge that is cleared on recovery and exit, and cover repeated checks, both ack modes, and stop/drop cleanup. Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(log-store): document WAL permission availability gauge Describe first-grant, idle, recovery and exit accounting in the RFC and align metric help terminology with the other object store logstore metrics. Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
ecee34fc5a |
feat(mito2): support index plugins in compaction (#10333)
* feat(mito2): build index plugins during compaction Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito2): support index plugins in multi-region compaction Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): expose primary key runs to index providers Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): reuse compaction index inputs with matching partition scopes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): compare interned partition scopes during index reuse Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): align compaction registration error context Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): implement explicit compaction handler validation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve compaction indexes across optional providers Allow compaction to continue with a warning when sketch providers are unavailable. Build active sketch and builtin creators together and avoid a later builtin-only replacement of the shared Puffin. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): cancel compaction index input probes Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
416f08de0f |
fix(mito2): release the manifest lock while a downgrade waits for the checkpoint (#10468)
* fix(mito2): release the manifest lock while a downgrade waits for the checkpoint A graceful change to follower or downgrading leader waited for the pending checkpoint under the manifest write lock. The region worker reads the manifest of that region inline when an index build finishes, so it stopped serving every region it owns until the checkpoint ended. The wait now uses `pending_checkpoint_completion` with the lock released, and takes the lock again afterwards. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): wait for the checkpoint when the region is already a follower A request that finds the region already a follower returned at once, while the change that made it a follower could still wait for the checkpoint with the lock released. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
453c35e9d3 |
feat: support trigger cache invalidation through frontend plugins (#10460)
Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
c6962802a9 |
feat(meta-srv): bind candidate region leases to the migration's registration (#10440)
* feat(meta-srv): bind candidate region leases to the migration's registration A region migration now gives its candidate region registration a random non-zero candidate lease id. Leases granted because of that registration carry it, but only while the table route exists, and the upgrade instruction carries the same id. The datanode passes the id of each lease to the engine, and Mito records the lease of the latest heartbeat per region outside the region instance, so it survives a reopen during catchup. The migration also drops the candidate registration before it restores the route on rollback, and before the procedure leaves the migration tracker when it ends without a rollback. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): record a lease only while the region stays open Record a granted lease under the same lock that removes it after the region closes, so a lease recorded during a close does not outlive the region. The metric engine records the data region's lease when the metadata region ignores it. Remove the roles-only extraction of operating regions, and cover a present candidate lease id in the upgrade JSON. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * build: point greptime-proto at the candidate lease id change Use the greptime-proto commit that adds GrantedRegion.candidate_lease_id (greptime-proto#357), which also brings in the RawSeriesSource Substrait payload message. Rely on the declaration order of RegionMigrationProcedure to drop the candidate registration before the tracker entry, and document it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drop lease records when workers stop Remove the lease records of the regions a stopping worker closes. Check under the removal lock deterministically in tests, cover a lease that arrives after a region is removed, assert the candidate lease id on the migration's real registration and its retry, and match ids per region in a batched upgrade. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): drop a redundant helper comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(store-api): pin the candidate lease id wire bytes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: document candidate registration and lease record lifecycles Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep candidate lease ids through customized lease renewers The heartbeat handler now sets the candidate lease id of each lease a customized renewer returns from the renewed region with the same id. The metric engine records a lease for the data region only when the metadata region ignored it, since the data region records its own leases. The countdown reset command carries a named struct instead of a tuple. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): cover a lease the data region ignores Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: tighten lease id lookups and lease test assertions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: align lease naming and share the txn-observing kv backend Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: tidy lease naming, docs and shared test helpers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: drop field comments that restate their names Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: drop method comments that restate their names Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: say which heartbeat a lease's send time refers to Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: drop the reset deadline field comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: state when an operating region's role and id reach a lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: pin greptime-proto to the merged candidate lease id change Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: add the candidate lease id argument to the direct demotion test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: order new keeper helpers after public items and test equal send times Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
c679ca5fdc |
feat(ingestion): configurable time index unit for auto-created physical metric tables (#10189)
* feat(ingestion): configurable time index unit for auto-created physical metric tables Auto-created physical metric tables hardcoded a millisecond time index. Add an `auto_create_physical_table_timestamp_unit` option (frontend and standalone configs, default "Millisecond") that selects the time index unit for physical metric tables created by auto table creation. New logical tables created under such a physical table inherit its unit, so Prometheus remote write, OTLP metrics, and other metric-engine ingestion paths write timestamps in the configured unit. The option threads through all auto-create sites: the operator inserter's `create_physical_table_on_demand`, the frontend instance's batched logical-table flush path (`create_metric_physical_table_if_missing`), the logical-table batcher's unit alignment fallback, and the OTLP metric preflight validation unit. Millisecond remains the default, so existing behavior is unchanged unless configured. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(otlp): match default physical schema to configured timestamp unit Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
7e76d28908 |
fix(postgres): account for buffered InfluxDB and OTLP COPY bodies (#10450)
Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
610794e447 |
perf: batch logical table cleanup when dropping databases (#10461)
* perf: batch logical table cleanup when dropping databases Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: refine drop database batching and recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify drop database traversal and batch sizing Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: tighten drop database batch size estimate Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: narrow logical table metadata cleanup API Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
0df6e25f3e |
feat(mito2): bound tail replay resources and recover from failed rounds (#10444)
* feat(mito2): bound tail replay resources and recover from failed rounds Tail replays now write SSTs through a MemtableSstWriter split out of RegionFlushTask, with their own WriteType and metrics, so replay chunks no longer count as flushes. A node runs at most 16 replays at once, gives their SST writes their own permits, and bounds their chunk memory by a budget of 1/8 of the global write buffer: each replay keeps a floor of budget/32 and the chunks share a pool of budget/2 above it. After a failed round, a writable region whose storage holds a newer manifest is fenced and reloaded once the replay ends, so tails that another writer already committed stop being retried. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): make the replay update check cancellable and keep flush logs The manifest check after a failed replay round listed storage under the manifest read lock and could not be cancelled, so closing a region whose listing stalled waited on it. The listing now runs under the replay's cancellation, and only the fence takes the write lock. Flush completion logs keep their reason, replay metrics record memtable compaction, and tests cover a stalled check on close, flushes running while replay writes are saturated, and the no-index case of async index builds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep the wait for the replay fence lock cancellable The replay now waits for the manifest write lock under its cancellation and only rechecks and fences once it holds the lock, so a demoted, closed or replaced region ends a replay queued behind another manifest operation. It logs when it waits for a replay permit, and the replay metrics are crate-private. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): wait for the replay to reach the fence before demoting Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): address review of the tail replay limits The update check after a failed round lists the manifest directory without holding the manifest lock; the write-lock recheck catches any version installed meanwhile. A replay snapshots its tails once it holds its permit, logs only when it waits for one, and logs when it fences or declines to. The writer takes the region hook instead of the manifest context, and names, docs and tests follow the review. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): cancel the tail replay on a direct demotion A direct role change that leaves a region unable to replay its tails, such as a demotion when its lease expires, now asks the worker to cancel the replay, as a graceful downgrade already does. A replay waiting on storage I/O then ends, and a later promotion starts another. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): cancel the tail replay when a lease demotes the region A lease that grants a non-writable role now cancels the region's tail replay like a direct demotion does; both share one helper, which sends nothing when the region has no pending tails. Docs, logs, metric observation and tests follow the second review. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): ignore demotion notices of replaced region instances A demotion notice now carries the region instance it applied to, and the worker cancels the replay only while that instance is still registered, so a late notice of a replaced instance leaves the new instance's replay running. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): narrow the tail replay byte counter doc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): align tail replay names and docs, test reservation bookkeeping Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): derive the replay log reason from its write type The SST writer now logs the flush reason for flushes and the write type otherwise, so the replay no longer passes a bare string. Drops an engine test that also passed without the replay memory pool; the reservation unit tests cover the pool's accounting. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): close the demotion and manifest version races of tail replay A role change now cancels the tail replay whenever it leaves the region unable to replay, instead of comparing with a role read before the change, which a concurrent promotion could make stale. The manifest check after a failed round lists from, and rechecks, the installed manifest version rather than the last version number, which a failed commit leaves ahead of it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin the demotion notice to the state after the role change Adds a case where the region is already a follower when the engine sets its role, and words the unwritable notice docs by the state after the change. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): rewrap the unwritable notice doc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): notify only when a role change makes the region unable to replay Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): assert every set_role return value Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): drop transition headings that repeat the code Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): restrict memtable writer construction and simplify write metrics Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): tie the flush reason to the writer kind and share replay test helpers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): correct the replay histogram range and listener doc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): keep tail replay doc links within the documented items' visibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
0a93810c23 |
fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(ingest): reserve memory for table batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): prefix the labels column Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): hide exemplar table from metadata Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): reconcile source tags before writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(exemplar): prune tags and retain source IDs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): record slow queries Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: account for held memory in batch admission Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: config comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: simplify exemplar and memory tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: deduplicate exemplar setup Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>v1.4.0-alpha.0-fcc1891ba-20261116-1794831195 |
||
|
|
fcc1891ba6 |
feat(mito2): support index plugins in region creation and rebuilds (#10298)
* feat(mito2): validate index plugin options before region creation Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito2): support index plugins in rebuilds Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): skip unnecessary SST reads during index rebuilds Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): adapt index rebuilds to shared file metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): keep async flush indexes independent of sketch providers Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): finish flush setup before seeding historical indexes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve index rebuild scope and published extensions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): remove speculative flush index guard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve existing indexes on incomplete rebuilds Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): separate background indexing from sketch maintenance Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
dd941a8e33 |
fix(mito2): wait for a stopped region's checkpoint before reopening it (#10376)
* fix(mito2): wait for a stopped region's checkpoint before reopening it A region's manifest checkpoint runs as a background task that writes the checkpoint, then `_last_checkpoint`, then deletes older manifest files: the deltas up to the previous checkpoint, or every older delta at the region's first checkpoint. Stopping the manifest manager does not wait for it, so a new instance of the region could read the manifest while the old task still wrote it. On a store that writes files in place the open could read an empty `_last_checkpoint`. On any store, an open that found no checkpoint could have deltas it had just listed deleted by the first checkpoint's cleanup. Wait for the stopped instance's checkpoint at engine stop, after the worker loop ends, and before a fenced region is reloaded. The wait does not hold the manifest lock, which the worker may take meanwhile: the checkpoint task owns the sender of a watch channel, which closes however the task ends. Fixes the flaky test_change_region_compaction_window (#7135). Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover the checkpoint wait through cleanup, panic and abort Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): align checkpoint wait names and log the wait Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover concurrent and late waits on the checkpoint Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): keep two checkpoint waiters through the end of the task Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that a wait after the checkpoint ended does not start waiting Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): address review on the stopped checkpoint wait Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): state the stop contract and assert it before the checkpoint wait Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the checkpoint completion future and keep its task state together Rename `pending_checkpoint_done` to `pending_checkpoint_completion` on `Checkpointer` and `RegionManifestManager`, and state how it differs from `wait_for_pending_checkpoint`. Keep the task handle and its completion receiver in one `CheckpointTask`, so reaping the handle also drops the receiver. Use plain backticks in the `stop` doc, which is public and cannot link to a crate-private method. Narrow the comment in `RegionWorkerLoop::clean`: a datanode or standalone shutdown closes every region before it stops the engines, so there the wait only covers regions whose close failed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
89e14ab936 |
fix(mito2): cancel compaction merges at batch boundaries (#10411)
* fix(mito2): cancel compaction merges at batch boundaries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve upload errors and expose merge cancellation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): normalize encoder cancellation at SST writer boundary Signed-off-by: WenyXu <wenymedia@gmail.com> * style(mito2): use imports for SST write types Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): share cooperative upload cancellation with tokens Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): classify nested upload cancellation errors Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
21ada9de09 |
chore: remove stray merge-main.log (#10445)
merge-main.log at the repository root holds the output of a `git merge`
that stopped on conflicts in src/flow, followed by an `exit=1` line. It
looks like it was committed by accident in
|
||
|
|
b30e002cef |
feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>]) ingests one Prometheus Remote Write request body over the postgres wire protocol — a GreptimeDB-specific, binary COPY format (wire format code 1). The payload is byte-for-byte the body a remote-write client POSTs to /v1/prometheus/write, compression included: buffered until CopyDone and completed through the exact ingestion core that endpoint runs. To share that core, the HTTP handlers' ingestion bodies are extracted into axum-free functions (ingest_remote_write_v1/v2 plus the context extension stamping), and the HTTP routes become thin wrappers — their behavior is unchanged, and the copy layer runs the identical decode, validation, conversion, batched/prepared write paths, per-batch write-progress accounting and context extensions, under the connection's regular postgres context. The WITH options carry what the HTTP endpoint takes as its content-type/-encoding and query parameters; session-provided values (database, user) are not repeated: - version 1|2 (default 1), matching the content-type's proto - zstd, matching Content-Encoding: zstd (snappy by default) - physical_table, matching the ?physical_table= parameter - pipeline_name/pipeline_version, matching the pipeline headers (v1 only; for v2 they are accepted and ignored, exactly like the HTTP route ignores them) Table-less like the other ingestion formats; one COPY statement is one bounded request (shared 64 MiB limit); the row count reports written rows (v1) or written samples plus histograms (v2). The flush enforces the frontend's shared request-memory quota (max_in_flight_write_bytes) — the same limiter the HTTP ingestion protocols use, so COPY cannot bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write carries the handler, metric-engine flag, validation mode and shared memory limiter. e2e: v1 body chunked across CopyData boundaries; v2 samples with readback; physical_table option routing; garbage-body decode failure with connection reuse; a tiny-quota regression where a compressed body whose decompressed size exceeds the budget fails like the HTTP route. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(servers): propagate COPY body reservations and test write accounting Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): share Prometheus logical batching with COPY ingestion Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): narrow Prometheus COPY scope and share write outcomes Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>v1.4.0-nightly-20261116 |
||
|
|
7e177b06fe |
fix(telemetry): avoid spurious log directory error on first start (#10446)
* fix(telemetry): avoid spurious log directory error on first start Since 0.2.4, tracing-appender prunes old log files when an appender with max_log_files is built, and it lists the log directory before creating it. When the directory does not exist yet, it prints "Error reading the log directory/files" to stderr. GreptimeDB sets max_log_files by default, so every start with a missing log directory printed this line. Create the log directory in build_file_appender before building the appender. Creation errors are left to the appender build, which reports them as before. Add a test that checks the stderr of a child process. Works around https://github.com/tokio-rs/tracing/issues/3612. Signed-off-by: shuxiachai <shuxiachai@163.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(telemetry): drop the first-start log directory test Review feedback: the test is not needed for such a minor change. Restore the unix-only test imports that it had merged. Signed-off-by: shuxiachai <shuxiachai@163.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: shuxiachai <shuxiachai@163.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dbd080b840 |
fix(recordbatch): treat -0.0 and +0.0 as equal in SimpleFilterEvaluator (#10443)
* test(recordbatch): compare float zero predicates with DataFusion `SimpleFilterEvaluator` filters SST rows exactly, so it has to agree with the SQL predicate that DataFusion evaluates above the scan. DataFusion compares -0.0 and +0.0 as equal. Evaluate comparisons against both zeros, NaN and NULL in Float64 and Float32 columns, and check the selected rows against DataFusion's filter. On main this fails: `v = Float64(0)` selects only the +0.0 row, while DataFusion selects both zeros. Signed-off-by: dayan02dev <pawarudayan92@gmail.com> * fix(recordbatch): compare -0.0 and +0.0 as equal in simple filters DataFusion 55 normalizes -0.0 to +0.0 before Arrow's comparison kernels, so SQL compares the two zeros as equal. SimpleFilterEvaluator called the kernels directly, and for floats they compare bit patterns: -0.0 != +0.0 and -0.0 < +0.0. Mito applies the evaluator to SST rows as an exact filter, so after a flush `v = 0.0`, `v >= 0` and `v IN (0.0)` dropped the rows holding -0.0, and `v = -0.0` dropped the rows holding +0.0. Rows still in a memtable matched. Normalize the literal when the predicate is built and the input before it is evaluated, with the helpers DataFusion uses for the same purpose. Signed-off-by: dayan02dev <pawarudayan92@gmail.com> * test(sqlness): cover flushed float rows holding -0.0 Flush rows holding -0.0, 0.0 and 1.0, then filter with `= 0.0`, `>= 0`, `= -0.0` and `IN (0.0)`. Both zeros have to match every one of them, as they do while the rows are still in the memtable. Signed-off-by: dayan02dev <pawarudayan92@gmail.com> --------- Signed-off-by: dayan02dev <pawarudayan92@gmail.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> |
||
|
|
1dbc0cd80f |
fix(mito2): avoid unrelated sparse tag expansion during compaction (#10414)
* fix(mito2): avoid unrelated sparse tag expansion during compaction Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): cover sparse compaction filter and compatibility boundaries Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle projected fields in sparse compaction mapper Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): remove redundant compaction projection mapper Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
1e39eab06b |
fix(ci): bound nightly kind deployment and restore inotify limits (#10423)
* fix(ci): bound nightly kind deployment and restore inotify limits Signed-off-by: evenyag <realevenyag@gmail.com> * fix(ci): use shell timeouts in composite actions Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
176bc07b53 |
docs(skills): focus release notes on user impact (#10442)
* docs(skills): focus release notes on user impact Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(skills): simplify compatibility guidance Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(skills): clarify release scope and highlight criteria Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(skills): align release targets and generated change groups Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7f7c003ff2 |
chore(ci): update compatibility versions (#10439)
Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com> Co-authored-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com> |
||
|
|
01d286dba8 |
docs(rfc): add object-store authority and leader election RFC (#10306)
* docs(rfc): add object-store authority and leader election RFC Signed-off-by: hahaha <a310608318@gmail.com> * docs(rfc): fix object-store authority and leader election RFC Signed-off-by: hahaha <a310608318@gmail.com> * docs(rfc): fix object-store authority and leader election RFC Signed-off-by: hahaha <a310608318@gmail.com> * docs(rfc): fix object-store authority and leader election RFC Signed-off-by: hahaha <a310608318@gmail.com> --------- Signed-off-by: hahaha <a310608318@gmail.com> |
||
|
|
beb1f88696 |
feat: persist bounded WAL releases on region close (#10329)
* feat(log-store): persist bounded region releases in shared WAL batches Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): release safe WAL history when closing regions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito): cover bounded WAL release across close and restart Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): clamp bounded releases to durable entries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): observe close release failures and cover recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: describe bounded WAL release lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): make bounded release fence sealing deterministic Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): bound queued releases and ignore shutdown hints Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover close release persistence and shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): wait for failed close flush cleanup Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: clarify bounded WAL release backpressure and shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(mito2): clarify staging test retry invariant Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): back off durable release retries Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): drive release retry after backoff Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): share and verify WAL retry backoff Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: clarify retained WAL release retry semantics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
68145fc727 |
fix(mito2): coordinate region edits with repartition lifecycle requests (#10354)
* fix(mito2): coordinate region edits with repartition requests Extend the pending ALTER queue to defer lifecycle requests blocked by region edits. Preserve scheduler handoffs and drain accepted edits before entering staging, rejecting new unbound imports at that boundary. Keep ApplyStagingManifest's asynchronous execution and close-time flush behavior. Cover queue bounds, failures, staging transitions, close/reload races, and flush concurrency. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): settle region waiters before completion returns Reject pending lifecycle requests when retiring a region. Give each completion one dispatch owner while preserving stalled-write ordering, edit-before-compaction dispatch, and fenced-region reload handling. Cover edit retirement with queued requests; the regression times out when removal only rejects edits. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve queued close flush requirements Defer later closes behind a pending close even when they do not request a flush themselves. Reuse the existing close-time flush and waiter handoff. Cover ordinary WAL writes with mixed flush_on_close values and verify committed SSTs before reopening. Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): clarify region request scheduling names Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): retain lifecycle integration after rebase Restore staging-session checks and bound test edits from the prior merge resolution. Keep StagingEnded dispatch and classify upstream tail replay notifications without adding lifecycle ownership. Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
643bdd72db |
fix(metric-engine): reuse existing column ids when adding columns is retried (#10372)
The metric engine derives the columns to add from the cached physical schema, and `DataRegion::assemble_alter_request` assigned every given column `max(non-reserved column id) + 1` without checking the existing names. That cache can lag behind: Mito reads its in-memory metadata without going through the region worker, and a caller future dropped on a client timeout does not withdraw the request that is already queued. A retry then treats an installed column as new and Mito rejects it, because the name exists with a different id. Recheck the real metadata in `assemble_alter_request`, validate an existing column with `validate_shared_column` and drop it from the payload, assign ids only to genuinely new columns, and skip the alter when nothing is new. Serialize the existence check, the id allocation and the physical alter per physical data region: the mutex lives in `PhysicalRegionState` and is held by the metric engine orchestration, so concurrent requests on logical regions sharing the region cannot allocate one id to different columns. `DataRegion` stays a stateless adapter. Build the logical mapping and the cached physical columns from the metadata read after the alter, and report a missing column as an error instead of panicking. Signed-off-by: Wenyu Xu <wenymedia@gmail.com> |
||
|
|
6e025eef61 |
fix(auth): secure MySQL and PostgreSQL database selection (#10382)
* fix(mysql): authorize the default database at login Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): validate MySQL handshake scope and SQL USE Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): authorize PostgreSQL schema selection Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): defer PostgreSQL database lookup until authorized Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(servers): propagate catalog lookup errors in protocol tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(postgres): skip unusable search path candidates Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): log database authorization and selection outcomes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): preserve search path responses and schema boundaries Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): clarify database selection diagnostics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mysql): format handshake username in structured logs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): secure database-less sessions and failed startup Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mysql): retain authorized default database selection Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(auth): separate database selection from request dispatch Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mysql): align schema validation with database-less login Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(auth): explain database-less session restrictions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
f88af153ec |
feat(query): serialize remote dynamic membership (#10341)
* feat(query): serialize remote dynamic membership Recover single-column UTF-8 build-key membership from InList and HashTableLookupExpr producers and send it to remote datanodes using the existing PhysicalInListNode. Non-monotone or unsupported subtrees widen to true; the receiver keeps the live filter authoritative. Signed-off-by: discord9 <discord9@163.com> * refactor(query): send lookup membership as a key list Serialize a HashTableLookupExpr as the existing PhysicalInListNode and preserve the rest of the expression tree. Remove per-subtree monotone and true rewrites; only an unrepresentable lookup or an over-budget payload falls back at the whole-payload level. Signed-off-by: discord9 <discord9@163.com> * docs: tighten narrow membership contract Clarify that the key-list transport is a positive pruning hint, not universal Boolean equivalence, and test that a NULL probe survives the IS NULL wrapper. Signed-off-by: discord9 <discord9@163.com> * fix(query): retain safe bounds on membership fallback When a key-list predicate cannot be shipped, keep top-level conjuncts that contain no membership and re-encode their conjunction. Drop membership-bearing conjuncts whole; fall back to true only when no safe bounds remain. Signed-off-by: discord9 <discord9@163.com> * fix(query): skip unrepresentable dynamic filter snapshots Do not synthesize a true predicate when a runtime hash-table lookup has no wire representation. Capture and fanout skip the affected snapshot, keep the query running, and report PushedDown::No so the parent filter is retained. Safe membership-free conjuncts still travel, and a producer-supplied true remains encodable. Cover restrictive-to-true updates, unsupported initial registrations with later supported updates, final unsupported cleanup, and NULL build keys. Signed-off-by: discord9 <discord9@163.com> * docs(query): clarify 642-key payload budget comment Signed-off-by: discord9 <discord9@163.com> * refactor(query): tighten dynamic membership encoding Preserve static IN conjuncts when only a runtime lookup is unrepresentable, apply the IN-list exclusion only on the over-budget path, and derive the membership count guard from the byte budget before materializing literals. Document that decoded dynamic filters are pruning-only, fix the NOT/CASE contract wording, deduplicate test helpers, and justify the schema-less InList proto encoding. Signed-off-by: discord9 <discord9@163.com> * fix(query): reset stale dynamic filters on unsupported updates Publish an explicit true reset when a producer generation cannot be carried, instead of leaving an older narrower predicate that could prune rows after widening. Keep the unsupported expression unrewritten; the reset is a transport-built invalidation. Type over-budget build-key memberships as DynFilterPayloadTooLarge, accept scalar-subquery IN lists under the conjunct-retention contract, and clarify the no-initial-predicate wording. Signed-off-by: discord9 <discord9@163.com> * docs(query): scope no-true rule to the encoder Clarify that the encoder never substitutes true for a producer node, while the long-lived fanout path publishes an explicit true reset to invalidate a stale region predicate. Point at the fanout caller instead of claiming producers always skip refused generations. Signed-off-by: discord9 <discord9@163.com> * docs(query): correct reset wording Fix the inverted conjunct-retention sentence and describe initial capture versus ongoing fanout resets consistently in the error docs. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
7eb766c9e4 |
feat(common-meta): make RDS metadata backends closable (#10417)
* feat(common-meta): make RDS metadata backends closable Standalone closes the metadata backend on shutdown, and `MetadataKvBackendCreator` now returns a `ClosableKvBackendRef`. Only the raft-engine backend implemented `ClosableKvBackend`, so external metadata backends could not be wired into the standalone creator. Implement `ClosableKvBackend` for `RdsStore` and return `ClosableKvBackendRef` from `with_mysql_pool`/`with_pg_pool` and the metasrv RDS builders. The stores only hold reference-counted connection pools that are released with the last backend handle, so `close()` has nothing to release explicitly. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(common-meta): close RDS connection pools on shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(common-meta): preserve closable RDS constructors and test shutdown semantics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
180dfc68f7 |
perf(mito): reduce SST metadata memory overhead (#10356)
* perf(mito): reduce SST metadata memory overhead Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(cache): add a sharded weak interner Share equal live allocations without exposing lock guards or retaining payloads. Sweep candidate buckets incrementally and cover concurrent publication and weak-reference churn. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): share SST metadata through the sharded interner Require shared metadata wiring for manifests, canonicalize checkpoint entries without cloning their payloads, and centralize runtime registration. Keep live content variants distinct and verify runtime and recovered allocation identity. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: compile SST metadata consumers across feature sets Wrap fixture manifest metadata in Arc explicitly and gate the test-only HashMap import with its consuming feature. Validate default mito tests and all cmd targets and features with Clippy warnings denied. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): reclaim retired metadata while idle Run bounded interner maintenance on the shared blocking runtime without retaining its owner between passes. Shrink sparse indexes and release empty backing storage, preserving live readers and reclaiming their metadata after delayed release. Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito): use intern-arc for shared SST metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): expose interner counts and decouple manifest setup Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(partition): support hashing partition expressions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): hash intern SST metadata without ordering values Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
6faa9ec862 |
feat(mito2): replay pending WAL tails in the background (#10342)
* feat(log-store): read the WAL of another node through the log store `LogStore::read_node` streams one region's entries in `(after, upper]` from the WAL of another node that shares the log store's storage, without writing to it. The default returns `None`. The object store WAL implements it with a registry of read-only readers over its own object store. The node prefix comes from the configured root and the store's own generation, and the returned stream holds the reader, so reads of the same node share its cached object indexes. `WalReadUpperBound` moves to store-api so the trait can name it; log-store re-exports it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(mito2): replay pending WAL tails in the background A leader that is writable, altering or editing and still has pending tails replays them in one background task per region and publishes the result in one manifest edit, which opens the read gate and resumes compaction once no tail remains. - The worker starts the task when a lease, a catchup or a role change makes the region writable, and cancels it on close, drop, downgrade and when it replaces the instance. The task always reports its end, so the worker can start another replay or reload a region a conflict fenced. - Each tail is read from this node's WAL, or through `LogStore::read_node` for another node, and replayed with the per-entry code of open-time replay into memtables outside the region's version. Chunks of about 64 MiB are written with the SST half of the flush task, never across two tails, since an SST gives its rows its largest sequence. - The edit adds the SSTs, clears the tails, and advances the flushed entry id to the last entry read and the flushed sequence to the largest chunk sequence. It commits outside the edit queue, under the manifest lock, only while the region is a writable leader recording exactly the replayed tails. A busy region or an unknown result retries the same bytes; a conflict keeps the chunks; a certain failure deletes them. - Another node's listed tail is only a bound. After a catchup made the region writable, the edit keeps its unread part until the region receives a Leader lease, then a second edit clears it. - After the commit the task releases the WAL up to the new replay start, kept below this node's remaining tails, and the worker raises the region's last entry id to the new flushed entry id, so a later truncate cannot pull it back. - WAL pressure no longer flushes a region whose WAL release one of this node's tails caps. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): harden tail replay across leases, cancellation and uncertain commits - A Leader lease counts as received after a catchup only if its heartbeat was sent after the catchup promotion: the writable fast path does not check the start of a downgrade, so a lease granted before an earlier downgrade could otherwise clear the mark. The mark is the promotion time, so the fast path still reads one atomic. - A direct promotion counts as a lease. - The tail commit also raises the committed sequence, so later writes of the region take sequences above the replayed rows. - Registering a writable region starts a replay, which covers a create that reopens an existing region. - Once an attempt may have persisted the edit, later certain failures retry the same content instead of building a new edit. - WAL reads stop when the task is cancelled, and a cancelled task does not start another commit attempt. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that closing a region cancels a stalled tail read Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): hold tail commit attempts by number instead of sleeping Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): cancel tail replays when the worker stops The worker's cleanup closes its remaining regions without the close handler, so it now cancels every running tail replay first; otherwise a replay stalled in a WAL read or waiting for a lease outlives the engine. The stalled-read test covers an engine stop as well as a close. The docs no longer claim that every SST gives all its rows its largest sequence. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): shorten two tail replay comments Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): stop tail reads at their upper bound A tail read over this node's WAL kept reading past the tail's upto until the log ended, so an unreadable later entry failed a replay that already had every entry it needed. Stop the read once it reaches upto. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): inline the tail replay notify helper Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that writes during a tail replay win over replayed rows Whatever records pending tails must raise the region's committed sequence to the tails' largest sequence, so writes accepted while the replay runs get larger sequences than the replayed rows. Record this in AGENTS.md and test it with a write that lands between two commit attempts of the replay. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): bound reads of this node's listed tails A listed bound may name no entry, so a read of this node's WAL never saw an entry equal to it and kept polling later objects; one that failed to read failed the tail. Read every listed tail through `read_node`, which stops at the bound, and keep reported tails on the node's own WAL. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that rows around a tail replay survive a reopen The tail commit raises the flushed entry id to the largest entry read, which can be another node's. Whatever records tails keeps this node's later entry ids above every tail and leaves none of its unflushed entries below one outside its own tails. Record this in AGENTS.md, and make the replay test meet it and reopen the region without a flush afterwards. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject graceful staging entry of a region with pending tails Entering staging through `set_region_role_state_gracefully` skipped the pending-tail check that `EnterStaging` runs, so a region with pending tails could enter staging, end its replay, and return to Leader with nothing replaying its tails. Run the same check before entering staging there. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): replay tails of a region altered right after promotion A promotion checked whether it made the region writable after setting the role. If the worker started an allowed ALTER or edit in between, the check saw Altering or Editing and asked for no replay, and nothing asked for one later. Check for a leader that a replay runs on instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover the WAL release and locked checks of tail commits Check that a tail commit releases the replayed WAL, that it is refused under the manifest lock once the region was demoted or its tails changed, that a commit retried while altering keeps its chunks, and that a replay waiting for a lease does not end and start again. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): note that read_node also reads this node's WAL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep no region version during a tail replay round A replay round held the region's version until it ended, and with it the memtables the region had when the round started. A flush removed them from newer versions, but their write buffer memory was only freed once the round dropped them, so a slow read, a commit retry or a lease wait could keep flushed memory charged and stall writes. Copy the tails and the flushed entry id instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): count a direct Leader role set as a lease only when it promotes A direct role set to Leader counted as a Leader lease even when the region already was a leader. On a region a catchup promoted, it cleared the mark that keeps another node's unread listed tail until a lease arrives, so the next commit could drop that tail. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): move the tail replay leader check onto MitoRegion The engine and the replay task both use it, next to the other role checks. Its doc now names the Writable, Altering and Editing states and why staging and downgrading regions replay nothing. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay waits for a Leader lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin the pressure skip, staging rejection and remote read error The WAL pressure test gets a case whose own tail starts above the flushed entry id and still flushes; the graceful staging case checks the rejection; and a remote tail on a log store that reads no other node's WAL fails as unsupported and keeps its tails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): list every tail replay trigger and the missing admission limit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): rename tail replay helpers and notifications The names now say what each one checks, records or announces: the replay predicate is a capability rather than a running replay, the lease hook records a lease, the writable notice is an event like its siblings, and the commit notice pairs with the end notice. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): ask whether a Leader lease came after a catchup `is_writable_by_catchup` was negated at both of its call sites. Its inverse, `leader_lease_received`, is what the replay reads. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): plan the read of a pending tail in the tail replay `Wal` no longer interprets `PendingTail` or `UptoKind`. The replay task decides whether a tail comes from this node's own WAL or up to its bound through `Wal::read_node`, which now takes plain arguments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay result goes stale Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): send the writable notice through notify_worker The tail replay already has one helper that sends a background notice to a region's worker; the writable notice now uses it too, and so logs a worker that has stopped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): note that tail replays skip staging regions and index like flushes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): say why each tail gets its own SSTs and what chunks are for Ordering between tails needs a tail per SST: an SST can give all its rows its largest sequence, so an earlier tail sharing an SST with a later one would take the later tail's sequence and outrank rows that a leader between the two tails flushed. Within a tail, the 64 MiB chunks only bound memory: the sequence floor already puts writes accepted during the replay above every tail. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the tail replay end guard like other drop guards Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): skip a tail replay edit that a sync already superseded A commit notice that the worker handles after a sync rebuilt the region's version from a later manifest no longer brings back tails or files that manifest removed. The notice carries the tails the replay read, and the worker applies the edit only while the version still records them. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay stops before publishing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover lock waits, busy edits and repeated triggers in tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log every exit that drops a tail replay Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): tell the event listener when a tail replay checks for a lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover uncertain commits, the SST hook and lease waits of tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log each tail replay exit once, where it happens Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): reuse the mock region hook and wait for role change tasks in tail replay tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover failed-round cleanup, instance replacement and leased first rounds of tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): make the replacement helper a TestRegion method and fix a lease comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
ed62f8ff54 |
feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions (#10361)
* feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions `iceberg_hash` partition rules could not be evaluated from SQL, so users had no way to measure a hash distribution or check which bucket a value falls into. Expose the same UDF that partition rules plan, plus `iceberg_hash_bounds(n)` which prints the equal-width breakpoints `bi = i32::MIN + floor(i * 2^32 / n)` as text for copy-paste into `PARTITION ON COLUMNS`. Both functions are immutable and read-only, and no persisted format changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): handle untyped NULL and small integer counts `iceberg_hash(NULL)` arrived as ScalarValue::Null, which becomes a NullArray that the shared array form had no arm for, so it failed with an unsupported-type error instead of hashing to NULL. Add a DataType::Null arm in the shared implementation, so both the SQL function and any future rule evaluation keep the advertised NULL-to-NULL behavior. `iceberg_hash_bounds` rejected Int8/Int16/UInt8/UInt16 constants even when the value was a valid bucket count, because Signature::any leaves the argument width unchanged. Accept every integer width like the existing Int32/Int64/UInt32/UInt64 arms, and cover them plus the untyped NULL case in unit and SQL tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(function): add iceberg_hash_breakpoints SQL function Equal-width `iceberg_hash_bounds(n)` buckets cannot follow a key distribution: a handful of hot keys can still land in one bucket, and a uniform split keeps hashing them together. Add `iceberg_hash_breakpoints(v1, v2, ...)`, which takes any number of constant values in any order (one shared hashable type, exactly the values a partition rule compares the hashed column against), hashes them through the shared implementation, and prints them sorted and deduplicated: SELECT iceberg_hash_breakpoints('iceberg', 34); -- 1210000089, 2017239379 The output drops straight into the breakpoint list of a `PARTITION ON COLUMNS` rule, so buckets can be placed around known key values instead of equal widths. Columns and NULLs are rejected with explicit errors: a column cannot be hashed at planning time, and a NULL hash never matches a comparison bound, so a NULL value cannot be a breakpoint (NULL rows always fall in the lowest bucket). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(function): add iceberg_hash_ranges_sql SQL function The helpers so far print bare breakpoint numbers, so a user still had to expand them into comparison rules by hand: n buckets mean n rules with up to 2n - 1 `iceberg_hash` call sites. `iceberg_hash_ranges_sql(column, n)` renders the whole list from the same equal-width breakpoints: SELECT iceberg_hash_ranges_sql('host', 2); -- iceberg_hash(host) < 0, -- iceberg_hash(host) >= 0 The output drops straight into the expression list of `PARTITION ON COLUMNS` (rule per line). It only renders text; it does not create or change partitions. The breakpoints come from the shared `IcebergHashBoundsFunction` logic, so the rendered rules and `iceberg_hash_bounds(n)` always agree. NULL name or bucket count propagates as NULL; columns, non-string names, empty names, out-of-range counts and wrong arity are rejected with explicit errors. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): include function identity in ScalarUdf equality The ScalarUdf wrapper compared and hashed only the signature, so any two functions sharing a signature were the same ScalarUDF: isnull, iceberg_hash and iceberg_hash_bounds all have Signature::any(1, Immutable) and compared equal. DataFusion compares expressions structurally, and its SQL planner rewrites SELECT expressions against GROUP BY keys with base_exprs.contains (rebase_expr); the collision made SELECT isnull(x) FROM t GROUP BY x, iceberg_hash(x) fail planning, because isnull(x) matched the grouped iceberg_hash(x) and was rewritten into an aggregate column that does not exist. Compare and hash the function name alongside the signature, so functions are distinguishable while two instances of the same function still compare (and hash) equal. Add regressions: same-signature functions stay distinct ScalarUDFs and stay separable in hash-based containers, and grouping by one while selecting the other plans and executes in both directions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): quote rendered column names and reject mixed hash domains `iceberg_hash_ranges_sql` interpolated the column argument verbatim, so a legal name such as `host-name` produced `iceberg_hash(host-name)`, which parses as a subtraction; names with spaces, dots, quotes or upper case could not be used either. Quote the name as a SQL identifier and double embedded quotes, so the rendered rules stay one identifier and survive parsing. Cover the round trip by feeding the output back into `CREATE TABLE ... PARTITION ON COLUMNS` for plain, dashed, upper-case, quoted and spaced names. `iceberg_hash_breakpoints` accepted values of different types, but strings and integers hash through different encodings (UTF-8 bytes versus Iceberg longs), so mixing them yields breakpoints that describe no real column. Require one hash domain (all strings or all integers, integer widths may mix) and name the offending argument otherwise. The existing SQL case that mixed `('iceberg', 34)` now uses two strings. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(function): register the raw hash UDF and tidy the helpers Registering `iceberg_hash` through a `Function` wrapper built a different `ScalarUDFImpl` type from the one partition rules plan, so DataFusion compared the SQL expression and the rule expression as unequal; the wrapper also rebuilt a `ScalarUDF` per batch. `ScalarFunctionFactory` already has `From<ScalarUDF>`, so register `iceberg_hash_udf()` directly and add a test that the registered function and `IcebergHash::to_logical_expr` produce equal expressions. Also: collapse the arity check and slice destructuring into one `let ... else`; make the bucket-count helpers associated functions instead of constructing a `default()` instance; pass the caller name through so an error raised from `iceberg_hash_ranges_sql` names that function rather than `iceberg_hash_bounds`; and fetch the SQL test's UDFs from `FUNCTION_REGISTRY` instead of rebuilding and wrapping each one. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): accept every string column name and cover the rules end to end `iceberg_hash_ranges_sql` matched only `Utf8` and `LargeUtf8` for its column-name argument, so a valid constant string that arrives as `Utf8View` (for example `arrow_cast('host', 'Utf8View')`) was rejected as non-string. `Signature::any` leaves argument types unchanged, so the function has to accept all string representations itself; use `ScalarValue::try_as_str` and keep the NULL and empty-name checks. A unit case and a SQL-level case pin that `Utf8View` renders the same rules as a plain literal. Add a sqlness case that renders the four buckets for the quoted column `"host-name"`, creates the table from that exact text, inserts one value per bucket plus NULL, and checks the row counts implied by the rendered predicates against the rows each region stored. This closes the gap between rendering the rules and routing rows with them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(partition): hash constants through IcebergHash::evaluate_array `iceberg_hash_values` duplicated `IcebergHash::evaluate_array` line for line. Call the existing method from `into_hashed_scalar` and drop the extra public entry point, so the SQL helpers still hash through the code a partition rule evaluates without a second identical API. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
ca1b8c7c98 |
fix(config)!: redact credentials from diagnostic TOML (#10381)
* fix(config): redact credentials from diagnostic TOML Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(config): redact credentials at their owning fields Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(config): cover skipped SAS tokens and export logs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cli): mask SAS tokens before overlapping account names Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cli): redact SAS tokens before overlapping account fields Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cli): mask containing credentials before shorter values Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7b455f2b81 |
docs: record authorization invariants in servers and frontend guides (#10384)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
e7f75a73dd |
fix(frontend): classify physical entity graph sources (#10383)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
56cb03f4fb |
fix(function): handle unaligned binary vector literals (#10380)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7d0186457b |
fix(mito2): notify flush waiters after failure cleanup (#10398)
* fix(mito2): notify flush waiters after failure cleanup Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): simplify flush failure replies and test notification ordering Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
4bda315fde |
fix(frontend): check table permissions for COPY FROM STDIN (#10374)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
5d88de5290 |
fix(servers): authorize HTTP database overrides (#10378)
* fix(servers): authorize HTTP database overrides Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): update the HTTP output benchmark handler call Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): authorize Prometheus database overrides Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): retain authorized label lookup scope Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
87af50bfdd |
fix(build): restrict dashboard tokens to the GitHub origin (#10379)
* fix(build): restrict dashboard tokens to the GitHub origin Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: remove standalone dashboard download test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
2e1ce9808a | feat: update dashboard to v0.14.6 (#10365) | ||
|
|
9f6c957e2b |
fix(mito2): commit staging edits once when staging ends (#10322)
* fix(mito2): commit staging edits once when staging ends Staging edits (the new partition expression, the imported files) were saved as deltas in the staging directory, each taking a manifest version. The end of staging listed that directory and merged every delta it found, including deltas a stale writer wrote late. Staged actions now stay in memory and take no version. The end of staging commits them once, as the next version, in a new EndingStaging state that stalls writes and holds role changes back until the commit returns. If the commit may have been written and the region stays staging, the encoded commit is kept: a retried request stages nothing new and commits the same content, and other staged edits are refused with a retryable error meanwhile. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): settle an uncertain end of staging before later commits Under overwrite publication a save that may have landed must not be overwritten, so every commit takes its version before the save and a failed save leaves that version unused. For the end of staging, the version is now taken only once the result is known: a failed save is read back at once, and if the read-back cannot tell, the commit stays unsettled. Every later commit first reads it back again, applies it to the manifest and the region version if it landed, and is refused with a retryable error while it still cannot tell. A follower sync past that version forgets it. Committed staging actions are applied to the region version through the manifest context, so the end of staging and a later commit that settles it share one path. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): settle before checking a commit and keep the lock into the end of staging - Settle an unknown end of staging when taking the manifest lock, before the write is checked against the manifest, and fire the hook of a landed one with the lock released, so a failing commit cannot drop it. Entering staging settles too; a staging region settles only when staging ends. - Pass the held manifest lock from the graceful Leader path into the end of staging, so a queued close cannot abort staging in between. - Wait for the worker to drain the stalled writes before clearing the staging directory. - Let an empty import retry a frozen end commit. - Count the size of an end delta found by reading it back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep settling out of staging and before a new staging - Do not settle while the region is staging or a staged edit is running; only the end of staging settles there, as it also leaves staging. - Settle before entering staging through a graceful role change, and fire the hook of a landed result even if entering staging then fails. - List the end-of-staging outcomes in the order they are decided. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): settle before the metadata backfill and fence the late staging clear - Settle an unknown end of staging before a graceful Leader on a writable region checks whether to backfill metadata; a landed result is merged into the backfill notification. - Skip the staging directory clear after the end of staging if the region was closed meanwhile, as it may be open and staging again. - Clear the staging directory after an end of staging with nothing to commit. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): take the overwrite version before the end of staging is saved Under overwrite publication the end of staging takes its version before the save again, as before this change, so a later commit never overwrites a save that may have landed. After an error that may have written it, the region goes back to staging with its staged actions and encoded bytes kept, and the retry writes the same bytes at the next version; a reopen replays both as one commit. This removes the immediate read-back of a failed end commit, the unsettled mark and the settling before later commits, the ManifestUnsettled error, and the region version handle on the manifest context. The end of staging applies the committed actions to the region version itself again. A graceful Leader from staging takes the manifest lock again after the end of staging and runs the metadata backfill, firing one notification for both, as before this change. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): check the lease and fence conflicts in the end of staging commit The end of staging commits its staged actions on its own path, so it skipped the checks other manifest commits make: in conditional publication it saved after the Leader deadline had passed, and a version conflict left the region unfenced, so the worker never reloaded it. Run both commits through one core that makes those checks. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): let the worker pass the drain to the end of staging The end of staging built and sent the worker's StagingEnded notification itself, so the region module depended on the worker's dispatch types. The worker now passes the drain as a future that the end of staging awaits after it drops the manifest lock. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep the write order across the end of staging The end of staging stores the region's next state under the manifest lock, and the worker handles the writes it stalled meanwhile later, on StagingEnded. A write that the worker took in between was applied before the earlier stalled ones. The worker now stalls a region's later writes behind those until StagingEnded drains them. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(mito2): read and encode the staged actions without copying them The end of staging copied the staged actions to read them, to encode them and to commit them. Read and encode them in place, and copy them for the region hook only when one is registered. The commit keeps its own copy, so the staged actions survive a failed commit for the retry. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover a flush while staging ends and the staging directory clear Check that a migration flush gets a retryable error while the region ends staging and succeeds after it. Plant a delta from an older binary in the staging directory before applying the staging manifest, so the check that the directory is cleared looks at the target region and fails if the end of staging leaves it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): tidy the end of staging wording and a test helper Nothing is merged at the end of staging any more, so its log, error reasons and bindings say staged or committed. Fix the region hook doc sentence and table cells for actions staged in memory, and move the import helper of the conditional manifest tests next to the other helpers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drop the staged files of a region that left staging An end of staging that leaves for Follower or Downgrading kept the staged actions in the manager, and the SST listing and all_manifest_files merged them into the files of a region that never committed them. The end of staging now drops them when it leaves staging, and both readers merge the staging manifest only while the region is staging. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(mito2): check a resent import against the staged files in linear time The check that a resent import only adds files that are already staged walked every staged edit for every file, quadratic in the file count. It now collects the staged file ids into a set once. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(mito2): apply the committed staged actions without another copy The end of staging cloned the staged actions to split them for the region version before the commit, on top of the copy the commit applies. The commit now hands the staged list back once it succeeds, and the end of staging checks the staged Change by reference before the commit and splits the returned list after it. One copy stays, because a successful commit uses the actions twice: a copy builds the new manifest, and the returned list updates the region version. The region hook gets its copy only when one is registered, as before. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): return the normal version from all_manifest_files and drop a dead fallback all_manifest_files preferred the staging manifest's version. It now returns the normal version, as its doc says. A test helper's unreachable `unwrap_or(path)` after `rsplit` becomes `unwrap_or_default()`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the staging helpers after what they do - Rename is_staging_or_staged_edit to in_staging_session. The Staging and EndingStaging pair lives in RegionRoleState::in_staging_session, which the compaction scheduler, region_role, should_abort_index and the region-level helper use. - Rename finish_ending_staging to leave_ending_staging, since it can also land on Follower, Downgrading or Staging, and tie end_staging, EndingStaging and StagingEnded together in the EndingStaging doc. - Rename contains_import to only_adds_staged_files and the staged list field to actions. - Move the staging paragraph to the public update(), state on end_staging that the drain is never awaited under the manifest lock, say when all_manifest_files merges staged files, say that the listings do not rely on the clear at the end of staging, and count the end of staging in the flush rule of the mito2 AGENTS.md, naming the predicates the flush paths test. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): keep the end-of-staging drain next to the stalled requests Move drain_ended_staging into worker.rs, and the set of regions whose writes wait for it into StalledRequests, which adds, checks and removes them, so the stall bookkeeping and its release live together. Rejecting all stalled writes now also forgets these regions. Their stalled writes are rejected with them, so a later write has nothing to keep order behind and is handled under the region's current state. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): keep the staged view with the staged actions The manifest with the staged actions applied was a separate field of the manager, set and cleared next to the actions by hand. It is now a field of StagedActions, so clearing the staged actions clears it as well, and its accessor says what it returns. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): split EndStagingPending out of RegionBusy A staged append that arrives while a possibly written end commit waits for its retry was refused with RegionBusy, which therefore had to be in the not-written arm of may_have_persisted_manifest_update for every producer. It now gets EndStagingPending (status RegionBusy, retryable), which alone takes that place, and RegionBusy keeps its earlier classification. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover regions that are ending staging Check that a region ending staging reports the StagingLeader role and skips compaction scheduling, and that its bulk writes stall, as do later ones behind them after staging ended, until the worker drains them. Share the stall wait and the StagingEnded drain between the tests. The tests wait for these stalls through a listener on their own engine, so the end-of-staging stalls notify the test listener as the write buffer stalls do, and CountingStallListener moves to the shared test listeners. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
6420082685 |
fix(meta): skip metric engine identity columns when merging partition columns (#10368)
CreateLogicalTables copies every physical partition column into the logical schema as a nullable String tag. After a physical metric table is repartitioned by the internal `__table_id` column, that column reached the region create request and the metric engine rejected it with "Internal column __table_id is reserved", so no logical table could be created. Filter engine-owned identity columns while merging, reusing `is_metric_engine_internal_column`. Filtering at the merge point also protects procedures that persisted an unfiltered `physical_partition_columns` list. User partition columns keep the previous behavior. Write routing for a physical table partitioned by `__table_id` is still tracked by #10352; this change only unblocks logical table creation. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
c937e3997b |
perf(mito2): prefilter single-column dynamic filters (#10340)
* perf(mito2): prefilter dynamic join membership Read the narrow dynamic-membership column before the full projection and use a stable per-row-group mask to select rows. The live filter above the scan stays authoritative; bounds-only and unsupported shapes keep all rows. Signed-off-by: discord9 <discord9@163.com> * test: cover dynamic prefilter cache and generations Prove the prefilter-result cache keeps the static mask but never reuses a dynamic mask across producer generations. Document why a superseded dynamic mask is discarded rather than assumed monotone. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b5cdcba82a |
fix(datanode): keep flush task error status code and retry hint (#10360)
The synchronous FlushRegions handler wrapped the error returned by the flush task into Unexpected before building the reply. InstructionError then only copied the Unexpected status code and the default NonRetryable hint, so metasrv could not tell a transient flush failure from a non-retryable one. Return the task error unchanged so its status code and retry hint reach FlushRegionReply, and rebuild the reply message with the same debug chain the previous wrapper produced to keep the diagnostics unchanged. Add handler-level tests covering RegionBusy, RegionNotFound and Unexpected task errors under both FailFast and TryAll. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
6a202adbe7 |
fix(log-store): skip objects collected below the boundary in WAL reads (#10343)
* fix(log-store): skip objects collected below the boundary in WAL reads The read-only WAL reader lists the source prefix and then reads the header and footer of every listed object. While the store writing the prefix is still collecting objects, a listed object can be deleted before its footer is read. The NotFound failed the whole read, even when the object lay below the reclaim boundary and was off the chain the read replays, so reads of a large prefix under steady writes failed often. Apply the rule startup recovery already uses. An object gone below the reclaim boundary at which the chain of the read stops is skipped. Any other vanished object, on the chain or off it, still fails the read: it may be the object that completes a later chain or carries the largest epoch. Startup recovery and the reader now share one NotFound split and one boundary check. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover unreadable objects in listed WAL reads A NotFound below the boundary is skipped, but any other error reading a header or footer must still fail the read. Cover it with a corrupted header below the boundary in both modes. Describe the boundary in `ensure_vanished_below` and `read` in terms that also fit the `LastEntry` mode, where the chain can stop at the object of `after` instead of a reclaim boundary. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe where WAL reads stop following links In the `LastEntry` mode a later object can carry a reclaim boundary above the object of the reported entry. The read then keeps that object as its tip below the boundary and follows no link from it. Say that the boundary stops the links to predecessors, and that the tip is the one object below it a read can need. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): pin the outcomes of vanished-object reader tests The unreadable-object test now asserts the InvalidWalObject error for object 1 inside the index cache's wrapper, instead of any error other than NotFound. The vanished chain-object test states the outcome it guards against: without object 2 a listed read could select the chain ending at object 1 and succeed with no entries. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
62dad907d0 |
fix: redact credential table options in user-visible output (#10346)
information_schema.tables.create_options (and SHOW TABLE STATUS) printed object store credentials of external tables verbatim, and SHOW CREATE TABLE masked only access_key_id and secret_access_key. Mask all four credential keys in both paths, and sort extra options in create_options by key. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |