mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-06 19:06:58 +00:00
8ecf49dca017adf873d31f7d4e6f01ebd4cef1cb
5375
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 fys
|
d33c664183 |
feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * chore: remove unused code files Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): pass build type to bloom filter indexer Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): share bloom filter field and JSON update loop Signed-off-by: fys <fengys1996@gmail.com> * refactor: better name * refactor(mito2): simplify JSON path bloom index handling Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): expose index targets from creators Signed-off-by: fys <fengys1996@gmail.com> * docs(mito2): note JSON path index rebuild risk Signed-off-by: fys <fengys1996@gmail.com> * chore: update greptime-proto * chore: update greptime-proto --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
  Weny XuandLei, HUANG
|
ee55c0c2fa |
feat: share concurrent SST metadata loads (#9729)
Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> |
||
|
Lei, HUANG
|
c7ea80558f |
fix(mito2): preserve live index files across remote compactions (#10046)
fix(mito2): reuse executor-owned compaction index resources Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
694ca48517 |
feat(meta): add drain migration batch operations (#10035)
* feat(meta): add drain migration batch operations Build migrations through the shared manager and collect batch results with route reconciliation and failure strategies. Keep the existing fence-and-plan flow while preparing tested operations for drain orchestration. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): gate procedure receiver helpers to unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 shuiyisong
|
ec240ad7e6 |
feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions. Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): preserve reserved sparse key semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): clarify sparse errors and avoid panics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito): reduce sparse key validation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito-codec): reduce sparse validation cost Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(mito): remove ordinary sparse support Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
efed847ca6 |
fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce sparse primary key schema Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: fix Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Dragon Roar
|
b21cd16f1d |
feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup Signed-off-by: lczllx <2181719471@qq.com> * fix(datanode): harden the startup object storage self-test Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): cover the File store skip in the object store self-test Signed-off-by: lczllx <2181719471@qq.com> * refactor(datanode): rename the object storage self-test to probe Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): disable object store self-test in sqlness and integration tests Signed-off-by: lczllx <2181719471@qq.com> * docs(datanode): correct what the object storage probe claims to cover Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
|
Lei, HUANG
|
045cf74c3f |
perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task Compaction read, merged and encoded each output on a single task, so parquet encoding waited for the merge and the merge waited for encoding. The merge now runs on its own compaction runtime task and hands merged batches to the SST writer through a bounded channel. The channel also closes when that task panics, so the reading side awaits the task before reporting the end of the input. A panic fails the compaction instead of producing a truncated SST. One output of 4.8M rows took 1.28s instead of 1.65s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): parallelize decoding and encoding of compaction outputs Parquet encoding took about two thirds of a compaction output's time and ran on a single thread. When `experimental_compaction_output_parallelism` is above 1, the SST writer encodes the columns of each row group on up to that many blocking threads of the compaction runtime while the next row group is buffered, and the merge decodes up to that many input files ahead of it on compaction runtime tasks. The parallel writer feeds every column writer the same input as `ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group reach the object store writer one row group later, so `max_file_size` may split a file one row group later. The option defaults to 1, which keeps today's CPU usage. One output of 4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): cancel compaction producers before consumer polling Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): support per-table compaction output parallelism Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoding tasks before returning failures Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime. Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoders on SST write errors Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error. Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain compaction outputs on cancellation Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures. Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
b83ceb5e7b |
feat: add reusable paged byte cache (#9791)
* feat: add reusable paged byte cache Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): include common cache in scope rules Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Yvan Wang
|
4c37ed4a0e |
feat(pipeline): add logfmt processor (#9586)
Add a built-in pipeline processor that parses logfmt lines (https://brandur.org/logfmt) into structured objects, with support for quoted values, escapes, bare-key flags, and ignore_missing. Closes #5049 Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> |
||
 jeremyhi
|
8e16710425 |
fix(log-store): accept a retried first create of the conditional create probe (#10034)
* fix(log-store): accept a retried first create of the conditional create probe The startup probe of the object store WAL wrote its first create by hand and failed on any error. When that create is stored but its response is lost, the retry layer repeats it, the repeat meets the object, and the probe reported a failed write, so the WAL refused to open. Move the create of put_if_absent, which reads the object back and accepts identical content, into a path-based core and use it for the probe's first create too. The second create, which the store must refuse, is unchanged. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): reuse LosingWriter for the retried probe create test Move LosingWriter from the fault matrix into the test module and use it, armed with the lose_next pattern, in place of RetriedWriter. Say why the probe's second create bypasses put_path_if_absent: a read-back would find the first create's object and let an overwriting store pass. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
89a5df3141 |
fix: ignore stale leader grants after a region is downgraded (#9987)
* fix: ignore stale leader grants after a region is downgraded Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: compare stale leader grants with the longest lease seen Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: treat a lease too long to store as unknown A lease longer than a u64 of nanoseconds saturated to a shorter one, so the derived send time could be later than the real one. Now such a lease makes the send time unknown, which counts as earlier than any gate. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address review comments on stale leader grants Scope the early send time claim to after a lease has been seen, drop the unused role parameter of demote, clarify a test comment and check that a fresh grant still makes a reopened region writable. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: compare stale leader grants with the heartbeat send time Pass the send time of the heartbeat behind a lease to set_region_role_with_lease, and return whether the role was applied. Mito compares the send time with the grant gate directly, so it no longer derives it from the longest lease seen. The datanode finalizes leadership only when the grant was applied. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover stale grants through the datanode and repeated downgrades Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: keep leader grant rules in one region method Apply a role grant in MitoRegion::apply_role_grant, which renews the lease and decides whether the role changes; the engine only looks up the region. Rename the gate to downgrade_started_at and demote to begin_downgrade, keep the field private, and copy it on reopen through inherit_downgrade_state_from. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: drop a needless borrow Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: keep lease concepts in the engine and role ordering in the region ManifestContext, which owns the role, records when the role was last demoted and promotes a region only for a promotion issued after that. The time starts when the context is created, so a reopened region ignores grants for heartbeats sent before the reopen, without copying state. The engine translates leases: it renews the deadline and promotes with the heartbeat send time. MitoRegion keeps no grant state. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep the data region of a metric region off a grant its metadata region ignores The metric engine applies a grant to the metadata region first and stops if it is ignored, so the data region, whose role the engine reports, never becomes a leader that skips finalization. Also describe what promoting returns. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: describe when a lease grant is ignored and flatten the finalize check Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: move lease types out of the region and the Leader deadline into ManifestContext Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: name the promotion fence after the downgrade it records Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
05536cd7f5 |
fix(meta): register failure detectors for regions switched before a rollback (#9989)
* fix(meta): register failure detectors for regions switched before a rollback A region migration switches the routes of its tables one table at a time. If it aborts after some tables were switched, those regions are already served by the target, but the rollback deregistered the target's failure detectors for every region and registered only the source's. The target's next heartbeat registers them again, so if the target stopped before sending one, nothing detected its failure and the switched regions stayed unavailable until the metasrv leader changed or the target came back. The rollback now collects, from the routes it already reads, the regions whose leader is the target and resets the target's failure detectors for them at the end. It also skips a table whose route no longer exists, so a table dropped meanwhile no longer fails the rollback before the detectors are handled. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(meta): roll back a downgrading leader in the rollback detector test The table that was not switched yet keeps a downgrading leader in a real abort, so the test now builds it that way and checks that the rollback clears it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(meta): monitor regions switched before a rollback only on the target The rollback registered the source's failure detectors for every region of the migration, including the regions already switched to the target. Those regions are served by the target after the rollback, so their source detectors watched the wrong peer: when one fired, the failover found the leader had changed and only removed it. The rollback now registers the source's detectors only for the regions that were not switched, and keeps resetting the target's detectors for the others. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(meta): rename reset_failure_detectors_for_candidate_regions to reset_failure_detectors Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
df33737e9d |
feat: add shared asynchronous cache primitives (#9790)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Ning Sun
|
c818b6ac2a |
feat(postgres): influxdb line protocol ingest via COPY FROM STDIN (#9982)
* feat(postgres): influxdb line protocol ingest via COPY FROM STDIN COPY FROM STDIN WITH (FORMAT influxdb [, precision 'ns']) ingests InfluxDB line protocol over the postgres wire protocol, reusing the HTTP ingestion pipeline wholesale for correctness and performance parity: the whole payload is buffered and completed once at CopyDone through the frontend's InfluxdbLineProtocolHandler::exec — conversion, permissions, line-protocol interceptors, timestamp alignment to table time-index units, merge mode and inserts are the same path an HTTP write of the same bytes takes, under the connection's regular postgres query context. Formats are an explicit, closed enum (CopyFormat::Csv | CopyFormat::Influxdb(options)); each variant states its own rules in the copy lifecycle's match arms — csv requires a target table and parses incrementally, influxdb forbids one (each line names its own measurement) and buffers one bounded request body. The COPY layer owns only the buffering: record boundaries, including newlines inside quoted field values and after escapes, belong to the line-protocol parser alone — one COPY statement is one bounded ingestion request (64 MiB; larger imports are multiple COPY statements). - Wiring: PostgresServer::with_influxdb_handler attaches the shared line-protocol handler; unwired servers reject the format when a copy starts. - Grammar: COPY FROM STDIN without a table is scanned (an unquoted FROM cannot be a table name); table-bound formats reject it with guidance, influxdb rejects a named table. - Options: precision accepts the HTTP endpoint's values via the shared parse_time_precision (moved to servers::influxdb), defaulting to nanoseconds. Review findings (fengjiachun) are fixed at the root by removing the pre-splitter rather than repairing it: the retired naive line splitter broke quoted multiline records at chunk/flush boundaries (found by removing record splitting entirely), drained the buffer quadratically (found by append-only buffering), and capped nothing (found by an explicit payload limit). e2e: multi-measurement ingest with chunked feeds; a 32,768-physical-line flush-threshold regression with a multiline quoted record split across CopyData chunks; precision 'ms' epoch verification; invalid UTF-8 and trailing-record-without-newline handling; payload limit; table-named rejection; malformed-line abort with connection recovery. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: typo --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 discord9
|
bd77b599cc |
refactor(flow): remove tracked-write protocol (#9992)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
cdc0d3a055 |
feat(procedure): support detached subprocedure recovery (#10009)
Select Discard or Detach when submitting children via Status::Suspended. Persist the policy from the child's first normal or rollback checkpoint. Recover detached children as independent roots with their original id and context, using existing execution and cleanup without rebuilding the graph. Keep fresh root admission serialized while allowing accepted procedures to recover above the submission limit. Cover first persistence, repeated recovery, rollback, cleanup, defaults, and concurrent admission. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Chenglong Zheng
|
724072bb33 |
feat: Add last_sequence for max allocated sequence and add tests for … (#9951)
* feat: Add last_sequence for max allocated sequence and add tests for the reproduced issue cases Signed-off-by: hahaha <a310608318@gmail.com> * test: Add a regression test for Kafka remote catchup Signed-off-by: hahaha <a310608318@gmail.com> --------- Signed-off-by: hahaha <a310608318@gmail.com> |
||
|
Dragon Roar
|
5f0e22171e |
fix(log-store): skip records with unparsable keys instead of aborting the read (#9749)
* fix(log-store): skip records with unparsable keys instead of aborting the read Signed-off-by: lczllx <2181719471@qq.com> * fix(log-store): keep undecodable RecordMeta keys fatal Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
 B. Naut
|
3d2b73762a |
fix(catalog): propagate frontend process-list failures (#9999)
* fix(catalog): propagate frontend process-list failures Return the existing invocation error when a selected frontend cannot list processes, so a successful response cannot omit that peer. Signed-off-by: Byte-Naut <Byte-Naut@proton.me> * test(frontend): cover remote process-list errors and aggregation Exercise real metadata discovery and gRPC listeners for a missing management service and successful catalog-scoped aggregation. Signed-off-by: Byte-Naut <Byte-Naut@proton.me> --------- Signed-off-by: Byte-Naut <Byte-Naut@proton.me> |
||
 LFC
|
b369d35d8b |
feat: enable JSON2 columns on non-append-only tables (#9985)
feat(mito2): support JSON2 in non-append-only tables Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yvan Wang
|
180e2716fc |
fix(log-store): make raft-engine periodic WAL sync actually fsync (#9715)
* fix(log-store): make raft-engine periodic WAL sync actually fsync On raft-engine 0.4.2, Engine::sync() writes an empty LogBatch and write() returns before fsync (tikv/raft-engine#395). The periodic sync task therefore never durable-flushed acknowledged WAL data. Work around by writing a minimal non-empty marker with sync=true so the fsync path is reached until a crates.io release includes tikv/raft-engine#400. Fixes #9712 Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> * perf(log-store): reuse LogBatch across periodic WAL syncs Keep the periodic sync task's LogBatch across ticks instead of allocating a fresh one (4 KiB encode buffer plus item vec) every period. A successful Engine::write drains the batch but retains its buffers; after a failed write or a panicked blocking task the batch is discarded and recreated on the next tick, since it may be left partially encoded. Keep the marker put rather than deleting an empty key: the put overwrites a single key, whereas every delete would become a recovery tombstone that raft-engine keeps in memory while replaying append files. Also document that the marker value must not decode as a NamespaceImpl (the marker key lies inside the range scanned by list_namespaces), add a test covering batch reuse and namespace listing, and apply rustfmt. Refs #9712 Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> --------- Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> |
||
|
jeremyhi
|
62ebf71f90 |
fix(meta): exclude route followers from failover targets (#10001)
* fix(meta): exclude route followers from failover targets Failover target selection did not exclude the region's route followers. The selector could keep picking a follower, the submit rejected it as a peer conflict, and the region never failed over. Group the failed regions by their followers and exclude them per group. A group without a candidate is skipped instead of failing every region of the datanode. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: check failover continues after a group without candidates Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(meta): keep failover going when follower routes can't be read Fall back to selecting without follower exclusion when the routes can't be read, log skipped groups with their followers at warn level, and test a region without a route. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(meta): select failover targets per exclusion set in select_peers generate_failover_tasks now only reads the route followers of the regions. select_peers resolves the affinity and the unschedulable datanodes once, then selects per group of regions that exclude the same datanodes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(meta): drop a comment that repeats the code Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(meta): tighten failover target selection Name the route followers after the parameter they fill, read the routes with unwrap_or_else, and log a group without a candidate as an error, as the caller did before. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(meta): let the region-stat-aware selector balance the whole batch The region-stat-aware selector now receives each region's excluded peers along with the whole batch, so it can still balance load across the regions while keeping a region off its own followers. A region without a candidate is left out of its output instead of failing the others. The selectors driven by SelectorOptions keep selecting per group of regions that exclude the same datanodes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: name the batch type of the stat-aware recorder Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(meta): treat a selector without candidates the same in both arms Both selector arms now place what they can and treat NoEnoughAvailableNode as nothing placed; the stat-aware selector's contract says so. Rename its parameter to exclude_peer_ids_by_region, check for unplaced regions in linear time, and describe what the code does in the comments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: keep the stat-aware recorder's placements off excluded peers Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
44df6e5ded |
fix(mito2): coordinate range cache invalidation and singleflight reads (#9935)
* fix(mito2): invalidate retired SST range results and coalesce loads
Coordinate decoded range loads by their full scan key, bound shared work, and prevent late fills after SST retirement. Invalidate range results for local purge and object-store GC while preserving old snapshot reads.
Fixes #9934
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): preserve SST lifetime across index replacement
Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.
Fixes #9944
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): list-based SST purge and single FilePurger::remove_file
The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.
- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
is_delete) method; is_delete means the file was retired AND this is the
last live view.
- LocalFilePurger deletes the SST and all its index files via the new
delete_file_and_all_indexes helper (list before delete, so a listing
failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
callers in the entire repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): elect final SST view release atomically
Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.
Add a deterministic concurrent-drop regression covering retired and live files.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): restore Arc-owned SST cleanup and clarify reference release
Restore the lifetime design from
|
||
|
discord9
|
984f80e277 |
fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit- or timezone-changing time casts A cast on the time index column keeps the windowed-sort optimization eligible only when it preserves the underlying timestamp values. Casts that change the time unit (e.g. ms -> s) or that add a timezone to a naive timestamp re-interpret the values, so the working ranges derived from the time index no longer describe the sort column and the scan fails with a working-range error or silently wrong order. Reject both cast shapes so such queries fall back to the normal SortExec path. The integration fixture uses two rows 1000 seconds apart so the cast output stays distinct and the ordering assertions are not vacuous; the descending arm gets its own expectation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit-changing timestamp wrappers to_timestamp_seconds/millis/... cast their argument to their own unit inside the function, so the hidden cast bypassed the explicit-cast guard and kept the windowed-sort optimization eligible on a sort column whose values no longer match the working ranges. The wrapper branch now applies the same unit- and value-preserving predicate as the cast branch, with the predicate extracted into preserves_timestamp_values and shared by both. The integration regression gains to_timestamp_seconds ASC/DESC arms and a to_timestamp_millis value-preserving control; unit tests cover unit-changing, tz-adding, and non-timestamp wrapper inputs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regen windowed sort sqlness after wrapper guard The windowed_sort_nanos and windowed_sort_advance cases sort by to_timestamp_nanos over a millisecond index and to_timestamp_millis over a nanosecond index. Both wrappers change the time unit inside the function, so the windowed-sort rule no longer applies and the plans render a plain SortExec. Query results are unchanged; only the EXPLAIN plan line differs. Regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): simplify windowed sort cast regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 dennis zhuang
|
5b4efd908c |
perf(mito2): index encoded bulk parts from memory during flush (#9913)
* perf(mito2): build encoded bulk part indexes from in-memory bytes during flush Sync flush uploaded encoded bulk parts without indexes and rebuilt them after the flush, which read every encoded SST back from the object store and needed a second manifest commit. Build the indexes from the encoded bytes still in memory before the flush commits, the same way the regular SST writer does, and drop the post-flush rebuild for these files. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): keep flush going when indexing encoded bytes fails Match ParquetWriter: a failure while building indexes from the encoded bytes logs and leaves the SST unindexed instead of failing the flush after the SST was uploaded. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): own encoded SSTs for cleanup before uploading them Index creation now runs after an encoded SST is uploaded, so a truncate or drop could cancel the flush after the upload but before UncommittedSsts tracked the file, leaving the SST and its index behind. Track the file before the upload, and fill the index metadata on the caller's SstInfo instead of a partial one. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(mito2): name encoded SST index build params explicitly Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): drop index metadata when the puffin file fails and size index decode batches Indexer::do_finish kept the per-index metadata when finishing the puffin file failed, so callers could publish indexes that were not written. Return an empty output instead. Indexing encoded bulk parts now decodes with the batch size estimated from the largest row group, the same estimate bulk merges use, instead of a fixed 8192 rows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(config): document the mito index build_mode option Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): decode encoded SSTs for indexing on the blocking pool Indexing encoded bulk parts decoded Parquet batches inside the flush task on the global async runtime. Decode each batch with spawn_blocking_global and keep the indexer update on the async task, with one batch in flight. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
17ed988f49 |
fix(mito): preserve SST lifetime across index replacement (#9950)
* fix(mito): preserve SST lifetime across index replacement
Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.
Fixes #9944
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): list-based SST purge and single FilePurger::remove_file
The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.
- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
is_delete) method; is_delete means the file was retired AND this is the
last live view.
- LocalFilePurger deletes the SST and all its index files via the new
delete_file_and_all_indexes helper (list before delete, so a listing
failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
callers in the entire repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): elect final SST view release atomically
Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.
Add a deterministic concurrent-drop regression covering retired and live files.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): restore Arc-owned SST cleanup and clarify reference release
Restore the lifetime design from
|
||
|
dennis zhuang
|
2e84833ac8 |
feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): reject partition column type changes and quote rule identifiers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(table): keep time index widening for partition columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(partition): replace PartitionFunction with IcebergHash Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(partition): describe iceberg_hash in the function module Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: quote partition rule identifiers in SHOW CREATE TABLE expectations Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(partition): check hash rules in information_schema and repartition target columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): coerce hash partition rules against the evaluation schema Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep quotes in partition column names when building filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Chenglong Zheng
|
2117f8b059 |
feat: Support procedure_state() using the last procedure ID (#9876)
* feat: Support procedure_state() using the last procedure ID Signed-off-by: hahaha <a310608318@gmail.com> * feat: Fix and add tests according to review comments Signed-off-by: hahaha <a310608318@gmail.com> * fix: the zero-argument signature in ADMIN swallows explicit arguments Signed-off-by: hahaha <a310608318@gmail.com> * fix: fix the CI failures caused by SQL version issues Signed-off-by: hahaha <a310608318@gmail.com> * fix: apply the changes based on the review Signed-off-by: hahaha <a310608318@gmail.com> --------- Signed-off-by: hahaha <a310608318@gmail.com> |
||
|
dennis zhuang
|
e6b60644b3 |
fix(mito2): stop the series distributor from stalling on unread partitions (#9991)
When every partition channel is full, the distributor walked the senders round-robin with a 100us send timeout each. Tokio timers fire on a 1ms tick, so a consumer that leaves some partitions unpolled costs about one millisecond per unpolled partition for every series batch. SeriesScan::build_stream and scan_all_partitions drain partitions one after another, and SortPreservingMergeExec above a multi-partition SeriesScan only pulls the partition holding the current minimum key. With 16 partitions and one batch per series after a merge, a 40-series scan read partition by partition took ~480ms (measured on the native PromQL two-input route). Wait on all senders at once and send to the first one that frees a slot. The send-timeout metric goes away with the timeout. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
5e9bb3c689 |
feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: align native histogram ingestion with remote read validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Weny Xu
|
2d00ae25c8 |
feat(meta): fence a draining datanode and plan the drain (#9960)
Introduce the metasrv side procedure that fences a datanode before it is drained, and collect the regions placed on it: - mark the datanode unschedulable through the node scheduling manager - build the drain plan from the datanode table index and the authoritative table routes, splitting leader regions from follower regions - keep the plan in the volatile context: it is derived from the table routes, so a restarted procedure rebuilds it instead of reading a persisted copy The leader regions are not migrated yet: the migration state machine lands separately. The procedure has no caller yet, so it stays inert. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
9f0e58e2af |
fix(meta): share peer discovery with plugin hooks (#9977)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
0dab3aef21 |
fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601)
* fix(query): skip LastRow pushdown for IGNORE NULLS Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep IGNORE NULLS aggregates above remote scans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): record IGNORE NULLS results before and after flush Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): share IGNORE NULLS coverage across deployments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh last-value plans after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
df8f6848cc |
fix(query): avoid lossy casts in partition pruning (#9600)
* fix(query): keep lossy casts out of partition pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): align cast pruning expectations with conservative fallback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): record lossy cast partition pruning regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain safe pruning conjuncts beside unsupported casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh cast pruning plans after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
91a59b82fb |
fix: acknowledge writes only within the region's leader lease (#9904)
* fix: acknowledge writes only within the region's leader lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: drop cached logical columns after a failed logical drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep the prepared region for the lease check after the WAL append Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: serialize a logical drop with fills of the logical column cache Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: check the drop lock and grant long leases deterministically Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover a bulk insert that loses its lease during the WAL append Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
1e97a0e7c3 |
fix(log-store): refuse an object store that ignores conditional create (#9909)
* fix(log-store): refuse an object store that ignores conditional create The object store WAL checks only the write_with_if_not_exists capability before it opens, and OpenDAL declares that capability per service. A store behind the S3 service can accept If-None-Match and overwrite the object anyway, so the WAL could overwrite its own objects and lose acknowledged writes without an error. Before recovery, the store now creates a probe key, unique to the open, twice under <node prefix>/probes/, outside the objects it lists, and deletes it. The second create must be refused with a condition failure, otherwise the open fails with an error naming the object store. The probe costs two writes and one delete per open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop a comment that restates the keys helper Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): accept AlreadyExists from the second probe create A backend may report an existing conditional-create target as AlreadyExists. The first create stored the probe key, so that error can only refer to it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
6f5c064f54 |
fix(meta): fence datanode processes replaced by a newer process (#9878)
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
|
||
|
Weny Xu
|
c96cdd0026 |
perf: stream and prune SST manifest scans (#9931)
* refactor: add reusable metadata pruning and projected SST batches Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): stream and prune manifest SST metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(datanode): stream inspection queries through reusable providers Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(catalog): push manifest filters through inspection queries Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: align manifest column constants with module conventions Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: reduce manifest inspection allocations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: narrow manifest inspection API visibility Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: retain residual filters for manifest inspection Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
shuiyisong
|
513b9cee58 |
feat(catalog): add public-global information schema scope (#9957)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Yingwen
|
371c4e9004 |
feat: support query concurrency limits in standalone (#9958)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Dragon Roar
|
88cb2a986e |
fix: reclaim HDFS temporary files left behind by a datanode crash (#9616)
* fix: reclaim HDFS temporary files left behind by a datanode crash Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): stop the temporary file sweep from starving GC jobs Signed-off-by: lczllx <2181719471@qq.com> * feat(mito2): add metrics for the temporary file sweep Signed-off-by: lczllx <2181719471@qq.com> * refactor(object-store): share one definition of the temporary file name Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): skip the temporary file sweep when no GC job is allowed Signed-off-by: lczllx <2181719471@qq.com> * refactor: gate the temporary file sweep behind the hdfs-object-store feature Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): sweep temporary files when a create opens an existing region Signed-off-by: lczllx <2181719471@qq.com> * docs(config): document the temporary file sweep in the GC settings Signed-off-by: lczllx <2181719471@qq.com> * refactor: align the temporary file sweep code with the surrounding style Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
 Dhruv VaishnavandWeny Xu
|
2e37e19af4 |
feat(datanode): expose ingest task panic and cancellation metrics (#9830)
* feat(datanode): record ingest task join failures Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(datanode): remove extra ingest task metric tests Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
Ning Sun
|
80df1c397b |
feat: COPY FROM STDIN for postgres interface (#9297)
* feat: postgres copy-in support * feat: add postgres copy-in framework * feat(postgres): route COPY FROM STDIN flushes through the pending-rows batcher Follow-up to #9302: begin_copy_in now uses the batching-aware new_query_context, so copy-in batches are submitted to the pending-rows batcher like SQL INSERTs when PostgreSQL batching is enabled for the connection, and keep the direct write path when it is not. The batcher acknowledges synchronously, so the CommandComplete row count stays exact in both modes. Adds test_postgres_copy_in_with_batching asserting the batcher flush metric increments only when batching is enabled. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): open up the COPY FROM STDIN WITH clause for format-specific options The WITH (...) option list is now open instead of restricted to sqlparser's COPY grammar: - COPY ... FROM STDIN statements are parsed textually (new copy_in::parse module, tokenizer-based) so arbitrary KEY value entries reach the codecs; sqlparser would reject options it does not know before we ever see them. WITH is optional, matching PostgreSQL. - CopyOptionSet becomes an open ordered map of KEY -> String | Number | Ident | Boolean | Flag with typed take_* accessors (take_string, take_single_byte, take_flag, raw take_value) and case-insensitive keys; duplicate options are rejected. - Each format codec takes the options it understands and rejects the leftovers with one message naming them ('COPY option(s) X are not supported by format Y'), so a typo cannot be silently ignored. The text format now also rejects HEADER/QUOTE/ESCAPE instead of ignoring them, matching PostgreSQL. A future format (e.g. influxdb line protocol with WITH (FORMAT influxdb_line, precision 'ns')) only needs its codec and a registry entry; no parser changes. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): reuse arrow-csv decoder for COPY FROM STDIN csv parsing Replace the hand-rolled CSV record parser and per-value conversion with arrow-csv's incremental Decoder — the same engine backing COPY ... FROM 'file.csv' in common-datasource — so both COPY routes accept identical data. The decoder handles record splitting (quoting, escaped quotes, custom delimiters, CRLF, headers, chunk boundaries) and typed conversion against the targeted columns' schema in one pass; decoded batches become insert rows via the existing api::helper::vectors_to_rows and datatypes Helper::try_into_vectors, and the decode schema comes from the greptime Schema's arrow projection. The codec framework now has parsers emit complete insertable rows (api Row) instead of raw byte fields: create_parser takes the targeted columns and is fallible; text converts with the shared text-representation helpers, csv with the arrow decoder. Documented consequences of sharing the engine: JSON and Binary columns and bare epoch-integer timestamps are rejected for csv, and an empty field — quoted or not — is NULL; a custom NULL string is rejected accordingly. Signed-off-by: Ning Sun <sunning@greptime.com> * docs(postgres): make the copy-in framework explicitly format-open Reframe the codec framework docs so nothing assumes csv is the only other text format: the shared pieces (Field, convert_text_field, check_text_representable_type) are described as building blocks for any format whose records carry values as text, while formats with their own typed decoding (csv today) build rows themselves. State explicitly that formats are not limited to PostgreSQL's own — GreptimeDB-specific formats (e.g. a line protocol) register like any other and define their own options through the open WITH clause. No code changes beyond a test rename and a doc clarifier. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): driver-level coverage for CSV copy-in options Add an end-to-end test through tokio-postgres exercising HEADER, custom DELIMITER, quoted fields with embedded newlines, escaped quotes and CRLF record separators, with read-back verification through the real insert path. WITH is omitted, matching PostgreSQL. Signed-off-by: Ning Sun <sunning@greptime.com> * test(postgres): drop the raw-socket copy-in tests The simple-protocol raw client duplicated what tokio-postgres and the integration tests already cover; keep the driver-based tests only. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): address review: fix text-format O(n^2) drain, share type checks, hex::decode - text feed: take the longest complete line prefix in one drain and iterate lines over the slice, instead of draining per line (each drain memmoved the whole buffer; ~1300 drains for a 64KB chunk of short lines) - csv check_column_type: delegate to the shared check_text_representable_type and additionally reject Binary/Json - decode_bytea: use the hex crate instead of a hand-rolled decoder - semantic_type_of: use ColumnSchema::is_time_index() like the flow adapter's column_schemas_to_proto, instead of index lookups - parse: drop the unused expect_ident parameter; document the identifier-case decision - PostgresServer:🆕 document that query_handler and copy_in_handler are the same frontend instance Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove unrelated files * feat(postgres)!: drop the text codec; csv is the only COPY FROM STDIN format Remove the hand-rolled PostgreSQL text-format codec (backslash escapes, octal, \N nulls) that arrow-csv cannot express, keeping arrow-csv-based csv as the only implemented format. - COPY FROM STDIN without FORMAT now fails with guidance ('requires an explicit format in this version; use WITH (FORMAT csv)') rather than silently defaulting to the removed format - FORMAT text/txt and FORMAT binary report 'not supported yet' via registry stubs instead of unknown-format errors - codec.rs shrinks to the trait framework: the text-representation helpers (Field, convert_text_field, bytea hex, timestamp parsing) are removed with the codec; the module docs note where to find their shape should a future text-serializing format need them - csv implements check_column_type directly (no shared default) Breaking for clients relying on the PostgreSQL default text format; explicit WITH (FORMAT csv) is required. Signed-off-by: Ning Sun <sunning@greptime.com> * perf(postgres): raise COPY FROM STDIN flush batch to 32768 rows and harden flush diagnostics - COPY_IN_FLUSH_ROWS 8192 -> 32768: without the pending-rows batcher each flush pays the full insert pipeline (catalog lookup, schema check, partition split, region write round trip), so 4x larger batches cut round trips 4x at linearly-bounded memory; the constant also sizes the arrow decoder batch. Comment documents the trade-off and points high-rate users at the pending-rows batcher, which turns flushes into enqueues. - rows buffer pre-allocates COPY_IN_FLUSH_ROWS. - flush_copy_in warns on unexpected output variants instead of silently counting 0 rows. - batching wire test feeds 70000 rows to stay above the new threshold. Verified against a real psql 18.6 client: \copy ... WITH (FORMAT csv) sends FORMAT csv and succeeds; default \copy (text) gets the 'requires an explicit format' guidance; COPY TO STDOUT round-trips. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): reset copy-in state on successful CopyDone on_copy_done left the CopyInState in place (inert but holding buffers) until the next COPY overwrote it; take it like on_copy_fail does, so buffers are freed eagerly and a stray post-done CopyData gets the protocol-violation error instead of a finished parser. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass typo check * chore: format change * fix(postgres): multi-chunk COPY FROM STDIN corrupted at chunk boundaries The csv feed loop flushed the arrow-csv decoder at every CopyData chunk end — including partial batches at record boundaries — but arrow-csv's contract permits flush() only after decode() returned 0 (a full batch of complete records). Flushing outside that point corrupts subsequent decoding: fields mis-parse shifted by the size of the record prefix buffered so far ('000001,host-001,...' parses as '00001h,ost-001u,...'), either erroring (22P02/22P04) or, worse, silently writing wrong values. Which chunks trigger it depends on where record boundaries land, so it appears intermittent with random-length data (psql 8KB / pgx ~64KB chunking both hit it). Adopt the strict decoder protocol: feed decodes until input is exhausted and flushes only when decode returns 0 (batch full); trailing partial batches surface at the EOF signal in finish(). With batch_size == COPY_IN_FLUSH_ROWS the flush threshold is unchanged; sub-threshold streams now emit their rows at CopyDone instead of per chunk, which is semantically identical. Adds test_csv_chunked_feeds_are_alignment_safe: the same payload fed at chunk sizes 1..65531 (plus shifted starts) with every row's content verified — reproduces the corruption on the old loop at chunk sizes 7, 8, 13, 64 and 1000. Verified with the reporter's Go reproducer (pgx, content-checked): 10/10 clean at -chunk 8192, and 4096/1000/97/65531 plus the previously failing benchmark sizes 500/1000/2000/200/800 all round- trip exactly. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: bypass spell check * chore: bypass spell check * fix(postgres): address PR review — cross-catalog guard, identifier folding, tab delimiter Three review findings (shuiyisong, PR #9297): [P1] Enforce the catalog restriction on COPY targets. copy_in_table now applies validate_catalog_and_schema when QueryOptions:: disallow_cross_catalog_query is set, mirroring SQL INSERT's check_permission path, BEFORE begin_copy_in switches the query context to the target catalog (which previously made the restriction unenforceable downstream). Cross-catalog COPY is now rejected with 'Illegal access to catalog' like INSERT. [P2] Canonicalize unquoted identifiers like the SQL parser (ParserContext::canonicalize_identifier): unquoted table-name parts and columns fold to lowercase, quoted ones keep their case. COPY Metrics (Host, Ts) now addresses the table CREATE TABLE Metrics created, and no longer silently misroutes into a coexisting quoted "Metrics". [P2] Allow a tab as CSV delimiter: take_single_byte accepts a literal tab while still rejecting newlines and other control characters, and the option parser now also accepts E'...' escape-string constants (DELIMITER E'\t'), processing PostgreSQL's backslash escapes. Verified end-to-end with psql 18.6: unquoted/quoted mixed-case round- trip into the right tables, cross-catalog rejection asserted in a new frontend unit test (QueryOptions plugin + second catalog), and both DELIMITER '<tab>' and DELIMITER E'\t' import tab-separated data. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): accept PG boolean spellings in COPY IN; drop copy state on failure Two review findings (shuiyisong, PR #9297): [P2] Boolean columns round-trip between the COPY directions. arrow-csv only parses true/false while PostgreSQL COPY OUT emits t/f, so boolean tables failed to re-import their own export (22P02). Boolean columns now decode as Utf8 in the arrow schema and flush_batch converts them with PostgreSQL's boolean input grammar (true/false, t/f, yes/no, y/n, on/off, 1/0, case-insensitive, whitespace trimmed). Round-trip covered end-to-end: COPY ... TO STDOUT's exact bytes are fed back via COPY FROM STDIN and verified. [P2] Drop the COPY state on every server-side failure. on_copy_data and on_copy_done now take ownership of the state up front, so parse, conversion or insert errors drop the buffered data instead of retaining it on long-lived or pooled connections until the next COPY or connection close (the reviewer measured ~17 MiB retained after a failed 8 MiB row); on_copy_data success puts the state back. An e2e test drives a failed COPY followed by a successful one on the same connection. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): interpret offset-free COPY timestamps in the session timezone Review finding (shuiyisong, PR #9297): the arrow decode schema carried Timestamp fields with no timezone, so the CSV decoder parsed offset-free timestamp strings as UTC while SQL INSERT parses them in the session timezone — the same input string silently stored a different instant depending on the ingestion method (8h apart in Asia/Shanghai). CopyInCodec::create_parser now receives the session timezone and the csv decode schema attaches it to Timestamp fields, so naive strings parse like INSERT; explicit offsets in the input (Z, +08:00) still take precedence, and UTC sessions are unchanged. Covered by test_csv_timestamp_session_timezone (naive vs +08:00 vs Z) and the end-to-end test_postgres_copy_in_session_timezone_parity, which SETs TIMEZONE TO 'Asia/Shanghai', inserts and COPY-loads the same string, and asserts both CAST(ts AS BIGINT) readbacks agree on 1704038400000. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): don't unescape E-string COPY options twice Review finding (shuiyisong, PR #9297): sqlparser's tokenizer already decodes PostgreSQL escape-string constants before returning Token::EscapedStringLiteral, so the second unescape_pg_string pass altered the decoded value — DELIMITER E'\\t' (a literal backslash followed by t, two characters) was turned into a tab and wrongly accepted as a single-byte delimiter. Use the EscapedStringLiteral payload as-is, like SingleQuotedString, and delete unescape_pg_string. Regression coverage distinguishes the three cases: E'\t' -> tab, E'\\' -> single backslash, and E'\\t' -> rejected as a multi-character delimiter. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): COPY OUT timestamps carry session-tz offset; quoted empty strings survive COPY IN Two review findings (shuiyisong, PR #9297): [P2] Preserve timestamp instants across COPY OUT/IN in non-UTC sessions. COPY OUT rendered naive UTC text while COPY IN (since the session-tz fix) interprets offset-free strings as local time, so re-importing an export shifted instants by the zone offset. COPY OUT now renders timestamps with an explicit session-timezone offset (2024-01-01 08:00:00.000000+08): the copy-out arrow schema carries the session tz (arrow-pg converts to a tz-aware DateTime) and the pg field type becomes TIMESTAMPTZ so the offset-bearing text format applies. Embedded offsets win on re-import regardless of session, keeping the OUT -> IN round trip exact; UTC sessions are unchanged. [P2] Preserve quoted empty strings. The shared arrow-csv decoder nulls every empty field, so a quoted empty string ("" — an empty STRING in PostgreSQL) imported as NULL. A shadow quote-mask scanner runs over the same bytes with the decoder's own splitting rules and records per field whether it was quoted; flush_batch restores quoted empties to empty strings for string columns, while unquoted empty fields stay NULL. Masks track records across chunk boundaries and align with decoder batches. Covered end-to-end: test_postgres_copy_out_in_timestamp_round_trip (Asia/Shanghai export asserted to carry +08, exact-epoch re-import comparison) and test_postgres_copy_in_quoted_empty_string (NULL, quoted empty and non-empty in one import), plus unit tests including a chunk-boundary sweep for the quoted-empty mask alignment. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): keep COPY IN quote masks aligned after doubled quotes; allow quoted empties in NOT NULL columns Two review findings (fengjiachun, PR #9297): [P2] Doubled quotes desynchronized the shadow quote-mask scanner. In QuoteInQuoted the doubled-quote branch skipped the second quote AND the following byte, so a closing quote or delimiter directly after an escaped quote was swallowed and later fields lost their masks -- import semantics depended on CopyData chunk boundaries. The branch now consumes only the escaped quote and returns to the in-quoted state, mirroring the decoder. Regression: "a""b","" ... verified across chunk sizes 1-64. [P2] A NOT NULL string column rejected quoted empty strings before restoration could run: the decode schema preserved the target nullability, and arrow-csv's batch validation rejects the temporary NULL a quoted empty decodes to. String fields now decode as nullable (safe: quoted-vs-unquoted is exactly the restoration's job), and the target NOT NULL constraint is enforced in flush_batch after restoration, so "" imports as an empty string while an unquoted empty field fails with a not-null-constraint error. Regressions at unit and e2e level: NOT NULL column accepting "" and rejecting unquoted NULL. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): align COPY IN quote-mask scanner with the CSV decoder on blank lines and CR terminators Two review findings (fengjiachun, PR #9297): [P2] Blank lines emitted a spurious empty mask. csv-core skips blank lines, but the scanner queued one mask per blank record, so every subsequent decoded row consumed the wrong mask (a leading blank line turned a quoted empty into NULL and an unquoted empty into an empty string, and tripped NOT NULL on the wrong row). A blank line before the header also shifted the header expectation. end_record now returns no mask for a completely blank record without consuming the header expectation. [P2] Standalone CR was not a record terminator. The decoder accepts CR, CRLF and LF alike; the scanner only recognized CR before LF or at EOF, so CR-separated input desynchronized masks and flush_batch failed with 'quote masks out of sync'. is_record_end now treats CR alone as a terminator, consuming one byte (two for CRLF), while a CR at a chunk boundary still defers via carry so CRLF split across CopyData chunks stays one terminator. Regressions: blank lines before/inside records and before a header (NULL/empty preservation plus NOT NULL behavior), and mixed CR/CRLF/LF terminators across chunk sizes 1-64 with a trailing CR-only record. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres)!: drop the quote-mask scanner; COPY IN CSV behaves exactly like file COPY The shadow quote-mask scanner — kept to preserve PostgreSQL's quoted-empty-string vs unquoted-NULL distinction — mirrored the decoder's record-splitting grammar by hand and required a fix for every rule it drifted on (doubled quotes, blank lines, CR terminators, NOT NULL interplay). Per maintainers' decision, that complexity is not worth the divergence: COPY FROM STDIN now behaves EXACTLY like the file-based COPY ... FROM 'file.csv' — an empty field, quoted or not, imports as NULL (arrow-csv's engine semantics, identical in both routes). Removed: QuoteMaskScanner and helpers, the always-nullable string decode schema, the restoration + NOT NULL enforcement in flush_batch, set_string_at, and the scanner-era unit tests. The decode schema again preserves target nullability, so the decoder enforces NOT NULL itself, same as the file route. Kept (small, schema-level, no second grammar): boolean spellings (t/f) and session-timezone timestamp decoding, both addressing separate review findings. E2E: test_postgres_copy_in_empty_field_is_null pins the parity — quoted empty, unquoted empty and non-empty in one import. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: remove empty file * chore: lint issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
jeremyhi
|
1c5d15fd36 |
perf(log-store): read object store WAL footers while the recovery listing goes on (#9889)
* perf(log-store): read object store WAL footers while the recovery listing goes on Recovery listed the whole prefix before it read any footer, so its LIST pages, about 80-120 ms each on S3 and one per 1000 objects, added to the footer reads. The listing now runs in a task of its own and feeds the footer reads as its pages arrive; a listing that fails part-way still fails recovery, and a failed footer read stops the listing. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): fail object store WAL recovery as soon as its listing fails A listing that failed part-way was reported only after every object it had already listed was read, which on S3 can be tens of thousands of reads. The stream now settles a finished listing before it yields another object, so its failure goes first. Tests cover a large listed backlog and a listing task that panics; the counting and parking test object accesses forward list_ahead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): hold the listing until the first footer reads wait in the backlog test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): check a failed listing settles before buffered objects with a finished task Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): bound the objects the recovery listing holds ahead of the reads The listing fed the footer reads through an unbounded channel, so a listing much faster than the reads could hold most of the prefix in memory. It now holds at most 4096 objects, four LIST pages of S3, and waits for the reads while that many are queued. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): poll the recovery listing directly to show it waits on a full backlog Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): check the bound with the channel the recovery listing uses Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
f2be11a69e |
fix: upload staging manifests in chunks (#9919)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhiandYingwen
|
1cb01d0d44 |
fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898)
* fix(mito2): keep SSTs when a manifest save ends with RateLimited The object store's retry layer retries temporary errors and returns the last attempt's error. A manifest delta PUT can be stored by the server while the client sees a timeout, reset or 5xx, and the retries that follow can be throttled, so the save ends with RateLimited even though the delta exists. may_have_persisted_manifest_update treated RateLimited as proof that nothing was written, so flush and compaction deleted the SSTs the stored delta references. A later open or follower sync replays the delta and reads fail on the missing files. Treat RateLimited like other ambiguous errors: the SSTs are kept and become orphans when the save really failed. Fixes #9897 Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep SSTs when a manifest save ends with PermissionDenied The retry layer returns the last attempt's error, so a 403 from a revoked key, a changed bucket policy or clock skew can follow a delta PUT that the server already stored. Like RateLimited, PermissionDenied does not prove that nothing was written, so it no longer lets flush or compaction delete the SSTs; they become orphans when the save really failed. The compaction test that rejects a manifest write before persistence now injects Unsupported, which is raised before any request is sent. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
shuiyisong
|
afbe77e31f |
perf: avoid duplicate OTLP histogram conversion (#9880)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
815da8a6ef |
feat(mito): expose scan concurrency metrics and dashboard (#9758)
* feat(mito): expose planned scan concurrency histogram Signed-off-by: evenyag <realevenyag@gmail.com> * feat(grafana): visualize planned scan concurrency Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): distinguish query and compaction scan metrics Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
5523cf3874 |
feat(meta): track per-node scheduling state and fence region placement (#9884)
feat: track per-node scheduling state and fence region placement
Add a cluster-wide, persistent "can this node receive new regions" flag and
make every metasrv placement path respect it. This is the fence half of
`ADMIN drain_node`; the drain procedure, gRPC entry and region evacuation land
separately.
State layout
- New namespace `__node_scheduling/{datanode,flownode}/{node_id}` holding
`{"schedulable": false}`. A missing key means schedulable, so undrain is a
delete and no stale `schedulable: true` records are left behind.
- `NodeSchedulingKey`/`NodeSchedulingValue` mirror the existing KV conventions
(`NodeInfoKey`'s `{role, node_id}` shape, `LeaseValue`'s `*Value` suffix).
The role is part of the prefix, so extending to flownodes needs no key change.
- `NodeSchedulingManager` writes with `put_if_not_exists`/`compare_and_put`, so
concurrent marks change the state exactly once, and reads a single prefix
range per placement decision (DDL/failover only, not per query).
Placement
- `NodeInfo` gains `#[serde(skip)] schedulable: Option<bool>` plus
`is_schedulable()`; `None` (unresolved) counts as schedulable, and the field
never reaches heartbeat bytes, so the wire/in-memory format is unchanged.
- `SchedulingAwarePeerDiscovery` stamps the flag on `active_datanodes`. It is
additive (never hides nodes) because failover still needs the full active set
to compute its exclusion list.
- All three built-in selectors now fetch candidates through
`selector::common::active_schedulable_datanodes`; `RegionSupervisor` adds
unschedulable datanodes to `exclude_peer_ids` for both selector variants.
Enforcing the single fence
- `MetaPeerClient` no longer implements `PeerDiscovery`; its discovery methods
are inherent, and `DatabaseOperator` (frontends only) takes the concrete
client. `SchedulingAwarePeerDiscovery`, built once in `MetasrvBuilder`, is
therefore the only way to obtain a `PeerDiscoveryRef`, so bypassing the fence
is a compile error rather than a silent misplacement.
Compatibility
- New key namespace only; `NodeInfo` serialization is byte-identical, so old
metasrv nodes and persisted heartbeats keep working. The flag itself is stored
in the durable KV, so a follower metasrv sees marks written by the leader.
Verification
- `cargo nextest run -p meta-srv node_scheduling` (13), `selector` (12),
`supervisor` (14), `-p common-meta cluster` (9), `-p meta-client` (36),
`-p standalone` (compiles, no tests).
- `cargo clippy -p meta-srv -p common-meta -p meta-client -p standalone
--all-targets -- -D warnings`, `cargo fmt --all -- --check`.
Signed-off-by: WenyXu <wenymedia@gmail.com>
|