mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-13 23:42:32 +00:00
stack/pr9412-capability
1494
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jeremyhi
|
18726c7602 |
fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(snapshot): bound writes and reduce logical export overhead Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: clarify metric snapshot acceptance criteria Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): split batches only for merged readers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(snapshot): cover supported types in both layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: consolidate metric snapshot experiment conclusions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): honor series statistics when splitting L0 scans Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito): explain memory risk of splitting short L0 series Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address snapshot export review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): accept source errors in lazy snapshot contexts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Weny Xu
|
e6f3ca10a5 |
fix(deps): bump OpenDAL to 0.59.4 for OSS RRSA credentials (#9520)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 fys
|
9a57e1195d |
feat(json2): json_get respect type hints (#9349)
Signed-off-by: fys <fengys1996@gmail.com> |
||
 discord9
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 shuiyisong
|
a907cbfbe4 |
perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues for tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: bound asynchronous trace auxiliary writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf: bound trace auxiliary cache by estimated memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: size trace auxiliary cache from host or pod memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add metrics for cache hit and miss Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: unify frontend cache metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: merge and reduce Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add init for the metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 zhaiAohan
|
21536a092b |
build(deps): update opensrv-mysql to fix MariaDB 12.3 USE information_schema hang (#9357)
Update opensrv-mysql from f3135e5 to 1f3d88c, which replies a proper EOF packet to COM_FIELD_LIST instead of a malformed OK packet that MariaDB 12.3 misparses and hangs on USE information_schema. Fixes #9158 Signed-off-by: tian1220A <1573612521@qq.com> |
||
 dependabot[bot]
|
27c1aa76d3 |
chore(deps): bump xxhash-rust from 0.8.15 to 0.8.19 (#9465)
Bumps [xxhash-rust](https://github.com/DoumanAsh/xxhash-rust) from 0.8.15 to 0.8.19. - [Commits](https://github.com/DoumanAsh/xxhash-rust/commits) --- updated-dependencies: - dependency-name: xxhash-rust dependency-version: 0.8.19 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
fys
|
c46fbde9ac |
fix(json2): preserve metadata when adding columns via alter table (#8696)
* fix(json2): support adding columns via alter table Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): scope missing-column null handling to JSON2 --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
 yihong
|
beed56c305 |
chore: update jieba-rs (#9453)
Signed-off-by: yihong0618 <zouzou0208@gmail.com> |
||
 Ning Sun
|
3e86422ef8 |
chore: bump version and start new 1.4 release cycle (#9403)
chore: start new 1.4 release cycle |
||
|
Ning Sun
|
3131cdbcb6 |
fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests The request-memory accounting only bounds and charges the encoded bytes on the wire, but a compressed body can expand far beyond that during decompression, so tiny requests could allocate disproportionate frontend memory before any protobuf validation or quota charge. - Handler-level decompression (Prometheus remote read/write v1+v2, Loki) now enforces a hard 512 MiB decoded-size cap, checked before any output buffer is allocated, and charges the decoded bytes to the shared ServerMemoryLimiter, holding the permits for the lifetime of the decompressed buffer. - gRPC requests with transport compression reserve the configured max_recv_message_size before tonic decompresses, so the decoding phase is admitted against max_in_flight_write_bytes; the later per-message charge is skipped to avoid double accounting. - The HTTP memory-limit middleware keeps its upfront Content-Length charge but now also accounts the bytes actually streamed beyond it, so chunked requests and understated Content-Length headers no longer bypass the aggregate quota. - Routes that decompress request bodies via RequestDecompressionLayer (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards) now charge the decompressed bytes as handlers consume them: the global middleware marks Content-Encoding requests, and a route-local accounting layer inside the decompression layer charges the decoded stream. Plain requests are skipped to avoid double-counting. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on memory admission guard lifetimes and 429 mapping - Retain the gRPC pre-decode reservation for the whole request: the extensions holding the guard were dropped when the request was consumed (into_inner), releasing the reservation while per-message charges stayed skipped. Both the unary and streaming handlers now clone and hold the reservation marker for the duration of request handling. - Retain the HTTP body permits across the handler: the AccountedBody wrapper and the request extensions are dropped once the extractors finish collecting the body, before the handler is done with the decoded data. Both accounting middlewares now keep their own accounting handle alive across next.run(req).await. - Return a ChargedBuffer from the Loki snappy decompressor so the reservation outlives the decompressed bytes through the caller's protobuf decoding, matching the Prometheus path. - Map mid-stream quota exhaustion to 429 instead of the generic body error (400): the accounting flags exhaustion and the middlewares rewrite the extractor rejection, so clients can distinguish backpressure from malformed input. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve the in-flight body acquisition before trying a new one A parked acquisition in AccountedBody::charge always belongs to the currently buffered frame, so it must be resolved before any new acquisition is tried. Letting the fast-path try_acquire succeed while a waiter is parked left the waiter alive, and its late completion would then be credited with a later frame's byte count, under-reserving memory relative to what was marked charged. Adds a regression test that reproduces the misattribution: with the buggy ordering the test delivers a body the quota cannot cover; with the fix the over-quota frame waits for its own acquisition and times out. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Weny Xu
|
133296f298 |
chore: bump version to 1.3.0-beta.1 (#9392)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
3c0e2a8d55 |
feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject empty packed export time ranges Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover cancellation during packed export I/O Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
  
|
75bd8e9ce6 |
feat: add HDFS object storage backend (#8701)
* feat: add HDFS object storage backend Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * fix: make HDFS storage operations durable Gate the native HDFS backend behind an explicit feature. Publish writes through same-directory temporary files and atomic HDFS Rename2 replacement, and provide streaming copy fallback for COPY_REGION. Add regression coverage for interrupted writes and the region-copy path. Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * ci: run HDFS object store tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: note HDFS temporary file cleanup follow-up Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: enable HDFS object storage by default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: remove redundant HDFS build feature notes Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: Minghan2005 <cambrianocean@gmail.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Minghan2005 <cambrianocean@gmail.com> Co-authored-by: jeremyhi <fengjiachun@gmail.com> |
||
 dennis zhuang
|
ffdd6d09a6 |
perf(index): cut allocations when building bloom and inverted indexes (#9359)
* perf(index): build bloom filters from element hashes without per-token allocation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): speed up inverted index building with hashed buffers and sync pushes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): use BuildHasher::hash_one for element hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): require callers to act on spill requests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): insert bloom hashes one by one to keep segment set capacity bounded Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): add index build and bloom search benchmarks Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): keep applier setup out of the bloom search benchmark Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): skip empty spills and keep the old inverted sort memory estimate Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): stream fulltext token hashes and test spill dispatch Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): note non-ASCII tokens still allocate in analyze_text_hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): narrow the allocation note to case-insensitive non-ASCII tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
4ffd93d8dc |
fix(query): preserve global limits with DataFusion optimizer fixes (#9071)
* fix(query): preserve global fetch through physical-optimizer repartitioning Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch) below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then dropped or rewrote that fetched coalesce, losing the global limit. Root-caused and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo pins to that fix commit and adds regression coverage. - pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix) - global_limit.rs: adapt to new DF API (input_distribution_requirements, child_distribution, replace_children+Recompute); accept HashPartitioned and KeyPartitioned as partitioning to restore; add regression unit tests - sqlness: extend standalone common ssts and refresh distributed ssts_limit result for the preserved fetch plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: drop dead HashPartitioned compat arms and test wrapper - remove HashPartitioned compatibility matches (pinned DF only emits KeyPartitioned); drop their #[expect(deprecated)] attributes - remove single-use agg_with_limit helper whose seed limit is always overwritten by the soft-limit transform - restore Cargo.lock dependency edges unrelated to the DataFusion pin bump (cargo update --precise had downgraded 26 unrelated packages) Addresses oracle ablation review. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(deps): pin DataFusion to merged fetch preservation fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Yingwen
|
3c2aac0a55 |
feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(storage): route series index build requests Signed-off-by: evenyag <realevenyag@gmail.com> * feat(admin): add BUILD_SERIES_INDEX Signed-off-by: evenyag <realevenyag@gmail.com> * test: specify compaction type in series index fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * test: correct series index SQL fixtures and error assertions Signed-off-by: evenyag <realevenyag@gmail.com> * test(compat): preserve legacy index rebuild across upgrades Signed-off-by: evenyag <realevenyag@gmail.com> * test(sql): cover series index admin validation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): bound series index maintenance queue Signed-off-by: evenyag <realevenyag@gmail.com> * docs: remove series index how-to guide Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove index build upgrade compatibility case Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address series index reconciliation review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * fix: bound manual series index reconciliation admission Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto to merged index build options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
b9f991502c |
refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
d910183cc4 |
feat(flow): freeze recovery windows and retention bounds for incremental flows (#9312)
* feat(flow): discover recovery windows from exact sequence reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(flow): freeze explicit recovery windows and retention bounds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): validate retained source windows before recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): qualify recovery source tables before serialization Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
  
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
fys
|
045441e3cc |
feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Lei, HUANG
|
b5199bc59a |
feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: gate Jev filtering behind an opt-in Cargo feature Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: verify Jev registration with default features Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: clarify Jev concurrency scope and stabilization work Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: enable renamed ai-functions feature by default Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: remove extra AI feature-off checks Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: rename AI feature to ai_functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
19c127d9c3 |
feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: restore packed metric snapshots Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: stream large packed parquet entries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: allow default S3 endpoint in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: skip unconfigured S3 in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use portable file URLs in packed restore fixtures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate packed snapshot structure before restore Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve strict manifest decoding and verify fixture Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject packed layout on both database export paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: implement file size in coordinator test storage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use expect_err for rejected export layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
263e229103 |
feat: add experimental Metric export to V2 snapshots (#9233)
* feat: add experimental Metric export to V2 snapshots Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate the complete Metric export capability response Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: construct portable file URLs for Metric export fixtures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address Metric export review nits Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
6a084d9a4f |
fix: release completed SST write buffers during compaction (#9243)
* fix: upgrade OpenDAL and honor SST write buffer size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: track removal of the OpenDAL bridge fork Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): honor SST write buffers across upload paths Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 shuiyisongandLei, HUANG
|
66d38e8e1c |
feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add `InsertLimitInterceptor` hook to `Inserter` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: attribute insert limit checks to the target table's database Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: unify write admission through metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce write admission for pending row batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: distinguish internal requests for ingestion metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: admit split ingestion requests once per database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: OpenTSDB throws error reason Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use meter crate main rev Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude database ingest rate limit from table options Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: reserve channel 255 for internal requests Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
b332bb67d6 |
perf(telemetry): remove the shared lock from TraceLayer (#9269)
* perf(telemetry): remove the shared lock from TraceLayer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(telemetry): stop collecting span data while tracing is disabled Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
6647dbc437 |
feat!: upgrade DataFusion fork to 55.1.0 (#9177)
Rebase the GreptimeDB DataFusion fork from official 55.0.0 to 55.1.0. DataFusion 55.1.0 is a patch release on branch-55 containing eleven cherry-picked fixes (schema-adaptation struct filters, cast/projection metadata propagation, nested-nullability aggregation adaptation, UnnestExec batch_size, RightMark join ordering panic, empty-struct ScalarValue, and FFI codec fixes). All twenty GreptimeDB fork patches rebase onto it with no textual or semantic overlap; none of them is absorbed upstream, so all are retained. Fork pin moves to discord9/datafusion branch greptimedb-55.1.0, commit 2aa87d52cdce7006af492330064738f33ed294c1 (55.1.0 + 20 patches). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
055bbbc93d |
feat: update pgwire to 0.41 (#9025)
* feat: update pgwire to 0.41 * feat: update datafusion-pg-catalog and arrow-pg |
||
 LFC
|
697cc5fa29 |
fix(json): fix JSONPath panic with jsonb 0.5.6 (follow-up to #9192) (#9228)
* fix(json): upgrade jsonb to fix unterminated JSONPath panic Signed-off-by: luofucong <luofc@foxmail.com> * fix(json): align integer extraction with JSON2 conversions Signed-off-by: luofucong <luofc@foxmail.com> * style: format jsonb dependency declaration Signed-off-by: luofucong <luofc@foxmail.com> * fix(otlp): report concrete unsupported JSONB type names Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
jeremyhi
|
dca01654bc |
feat: prepare and execute database Metric exports (#9180)
* feat: prepare and authorize captured database exports Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: bound database export jobs and drain failures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: validate database export identity and restore equivalence Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate database export directory URLs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve Windows export directory paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject local database export filename aliases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: consolidate database export planning policies Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: restore escaped database export filenames Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: align database restore assertions with shared policies Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: clarify database export boundaries and names Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
ab21f684bf |
perf(mito2): lazily materialize sparse primary key tags (#9199)
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations The new revision (v0y4g3r/memcomparable main, PR #1) includes: - Deserializer::read_bytes_into for reusable-buffer decoding - read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain) - bulk put_slice in both normal and reverse modes - Error::Eof instead of panic on truncated input Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): extract sparse tag values lazily without full PK decode Replace the full primary-key decode in two hot consumers of SortField::deserialize with column-lazy extraction over SparsePrimaryKeyView, eliminating the per-label owned values (Vec<Value>, one heap allocation per string) for every distinct key: - flat_format: DecodedPrimaryKeys keeps sparse keys encoded and extracts each tag column directly from the raw dictionary bytes into a reusable buffer. FlatConvertFormat (flush/compaction write path) builds all projected tag columns in one pass per key, sharing offset discovery between columns; the read path (file_range filters) keeps per-column lazy extraction. - series_index writer: builds rows from the reserved prefix and only the indexed tags via a shared offsets cache and buffer, instead of decoding all labels and copying out a few strings. - mito-codec: add SparsePrimaryKeyView::reserved_value for typed table_id/tsid extraction. Persisted formats are unchanged; values pushed to column builders are identical (NULL vs empty string, reserved column types, duplicate dictionary entries and consecutive-run order are preserved). Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion 30 samples): old = full decode + per-column builds, new = lazy extraction (+ one-pass for multi-column projection). | Workload | Before | After | Change | | --- | ---: | ---: | ---: | | 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% | | 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% | | 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% | | 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% | | 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% | Also adds a decode_sparse benchmark group to mito-codec's bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): share sparse tag scans across filters Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics. Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches. Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark sparse tag predicate count scaling Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
LFC
|
aacf04cf6e |
feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Weny Xu
|
be15c88e92 |
feat: batch ordinary table writes across HTTP protocols (#9115)
* feat: integrate table batching across HTTP protocols Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: skip empty prepared writes before batch admission Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: load batching protocols from environment and document frontend wiring Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: remove experimental prefix from pending rows batcher config Signed-off-by: WenyXu <wenymedia@gmail.com> * style: sort frontend test dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: count batched ingestion once and update config snapshot Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
d45f5d6eaf |
feat(log-store): add the object store WAL object format (#9200)
* feat(log-store): add the object store WAL object format Add the byte format of a single object store WAL object: a header with the GTWALOBJ magic, format version 1, the object sequence and the writer instance id; one segment per region ordered by region id with entries ordered by entry id; a footer that records each segment's region id, entry id range, entry count, byte range and CRC32; and a fixed trailer with the GTWALTRL magic, the footer location, the footer CRC32 and the whole-object CRC32. The module encodes objects deterministically and decodes the header, trailer, footer and segments separately, with structural checks on footer ranges and segment tiling. It has no callers yet; the store that writes and reads objects follows in later changes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): pin the object store WAL format with a byte fixture Add a fixed version 1 object with two regions and five entries as a hex literal. The test decodes it and checks the exact header, trailer, footer entries and records, and checks that encoding the same records, in either input order, reproduces the fixture byte for byte. Round-trip tests alone pass when a refactor changes field order, endianness or checksum coverage in both the encoder and the decoder. The fixture bytes were derived from the documented layout rather than from the encoder, so such a change now fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): reject an empty footer when decoding a WAL object The encoder never writes an object without records, but decode_footer accepted a footer that declares zero segments, and verify_segment_ranges accepts an empty footer too. Only the test-only decode_object rejected it, so a checksum-valid empty object would pass the header, trailer and footer checks that recovery runs. Reject a zero entry count in decode_footer and drop the now unreachable check in decode_object. Add a test that builds a checksum-valid object with an empty footer and checks that decode_footer and decode_object reject it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): use pub(crate) for the WAL object format API Other log-store modules use pub(crate) for items shared across module boundaries. Switch the format module from pub(super) to pub(crate) to follow that convention. No behavior change. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
7c7132ea65 |
refactor(flow): execute streaming flows with DataFusion (#8976)
* test(mito2): cover regex inverted index pruning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(flow): execute streaming flows with DataFusion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(flow): remove legacy streaming runtime
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): avoid retrying stateless sink inserts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): align stateless writes with sink schema
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): reject stale stateless source schemas
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): validate stateless flow routing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "test(mito2): cover regex inverted index pruning"
This reverts commit
|
||
|
Weny Xu
|
e0915a8380 |
refactor: reuse common batching components in Prom ingestion (#9114)
* refactor: compose Prom ingestion with common batcher components Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: share bulk insert IPC encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: centralize batcher worker channel creation Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: share timestamp extraction for batched writes Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: preserve sharded batcher worker lookup Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker lookup and configuration validation Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: validate batcher limits inside fallible constructors Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
d7ada1761d |
feat: export logical tables from Metric physical scans (#9159)
* feat: add physical Metric table exporter Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: drain Metric export writes before cancellation cleanup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: construct Metric export error context lazily Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: name the logical table export entry point Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: share Parquet writer for logical table exports Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve Parquet destinations and cancellation boundaries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: clarify logical table export field names Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: clarify logical table export helper responsibilities Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate logical export membership by table ID Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify logical table export coverage and strengthen assertions Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
Weny Xu
|
737025760e |
feat: support request-level WAL skipping for bulk inserts (#9110)
* feat: support request-level WAL skipping for bulk inserts Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover bulk insert WAL skipping across protocols Signed-off-by: WenyXu <wenymedia@gmail.com> * test: align WAL snapshot naming with sequence watermarks Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: bump proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
0059508e29 |
perf: avoid directory listing for explicit COPY input files (#9126)
* perf: avoid directory listing for explicit COPY input files Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve COPY input symlink filtering Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
21aed0371f |
feat(pprof): switch CPU profiler to framehop unwinder (#9125)
* feat(pprof): switch CPU profiler to framehop unwinder Replace the default libgcc-based unwinder in pprof-rs with the framehop unwinder, which is designed to be async-signal-safe: - framehop performs no heap allocation during unwinding (MustNotAllocateDuringUnwind) - It handles prologue/epilogue interruption correctly - It falls back to frame-pointer unwinding when CFI is unavailable - It does not depend on libgcc's unwind implementation, which is documented as not signal-safe (see tikv/pprof-rs#36) Bump pprof from 0.14 to 0.15 in all three consumers (common-pprof, cmd, servers) to unify on a single version. pprof 0.15 also replaces parking_lot with spin-rs to avoid a potential profiler deadlock (#268). This addresses the libgcc_s.so.1 #GP crash observed in production when CPU profiling is active, by eliminating the libgcc unwinder from the signal handler path entirely. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(pprof): gate framehop-unwinder to supported targets framehop-unwinder is only available on x86_64/aarch64 Linux/macOS. Enabling it unconditionally for all Unix targets breaks the build on riscv64 and other platforms: pprof disables its backtrace-rs fallback when framehop-unwinder is set, but the framehop module is not compiled on unsupported targets, leaving no TraceImpl implementation. Split the pprof dependency: the base target.'cfg(unix)' block carries the common features (flamegraph, prost-codec, protobuf), and a separate target block adds framehop-unwinder only on supported targets. Cargo unions features from both blocks on matching targets, so x86_64 and aarch64 Linux/macOS get the full feature set while other Unix targets fall back to the default backtrace-rs implementation. Addresses review comment discussion_r3987313632. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
9619085310 |
chore: bump tikv-jemalloc-sys patch to jemalloc dev (ff80bf2d) (#9119)
* chore: bump tikv-jemalloc-sys patch to jemalloc dev (ff80bf2d) Update the [patch.crates-io] rev for tikv-jemalloc-sys from e1846d8c (5.3.1 + 54f22c83 backport) to ff444d4 (upstream dev HEAD, 161 commits ahead of 5.3.1). The dev branch includes additional TSD/tcache fixes beyond the original backport: - fb5499aa9c: Handle jemalloc calls after TSD teardown - 61dc1da395: Fix possible tcache corruption on fiber migration - 1e92317014: Fix thread-exit TSD cleanup See GreptimeTeam/jemallocator branch bump-jemalloc-dev and tikv/jemallocator#182. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main GreptimeTeam/jemallocator#2 has been merged; reference the merge commit e254a7ea on main instead of the PR head branch. Jemalloc source content is unchanged (still upstream dev ff80bf2d). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
ab0b1f5cce |
fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix (#9103)
* fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix Upgrade tikv-jemallocator / tikv-jemalloc-ctl / tikv-jemalloc-sys from 0.6 to 0.7, which embeds jemalloc 5.3.1 (includes a056c20d 'Handle tcache init failures gracefully'). On top of that, patch tikv-jemalloc-sys to the GreptimeTeam fork that adds the remaining upstream fix 54f22c83 'Initialize TSD tcache before enabling it' (GreptimeTeam/jemalloc#1, GreptimeTeam/jemallocator#1). Without the ordering fix, a reentrant allocation during TSD bootstrap (e.g. heap-profiling prof_tdata init / sampled backtrace when prof:true is active) can observe an enabled-but-uninitialized tcache, corrupting per-thread tcache metadata and crashing the process in arena_stats_merge, calloc, or the libgcc unwinder. The patch is pinned by rev and should be removed once tikv/jemallocator ships a jemalloc snapshot that includes 54f22c83. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump tikv-jemalloc-sys patch rev to merged release-5.3.1 GreptimeTeam/jemalloc#1 has been merged; point the patch at the jemallocator commit referencing the merge commit on release-5.3.1. Jemalloc source content is unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main GreptimeTeam/jemallocator#1 has been merged; reference the merge commit e1846d8c on main instead of the PR head branch. Jemalloc source content is unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(servers): bump tikv-jemallocator dev-dependency to 0.7 main added a target.'cfg(not(windows))'.dev-dependencies entry on tikv-jemallocator 0.6 for servers after this branch diverged. On the merge ref it pulled tikv-jemalloc-sys 0.6 from crates.io, which conflicts with the patched 0.7 (links = "jemalloc" may only appear once in the dependency graph), failing version selection in CI. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
5d5f0d6d70 |
fix(query): prevent incomplete aggregate dynamic filtering (#9102)
* fix(query): prevent incomplete aggregate dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover mixed expression and column maxima Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): pin upstream aggregate regression backport Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): assert aggregate filter behavior on datanode scans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): pin merged aggregate dynamic filter fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
60899d19ee |
perf(servers): defer Prometheus sample value formatting to serialization (#9091)
Building a matrix response allocated one `String` per sample while the record batches were scanned, then dropped it after the JSON body was written. Keep the `f64` in `PromSampleValue::Number` instead and format it with ryu while serializing, so no per-sample string is allocated. `PromSampleValue::Text` keeps values parsed from a JSON body, so deserializing and re-serializing a response is unchanged. Vector and scalar results still expose `String`, since they hold a single sample. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> |
||
|
Weny Xu
|
b46de8c828 |
feat(telemetry): add log directory size retention (#8997)
* feat(telemetry): add log directory size retention Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update config API logging fixture Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(telemetry): recover log retention state Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(telemetry): handle log retention cleanup errors Signed-off-by: WenyXu <wenymedia@gmail.com> * test(telemetry): cover log count retention on rotation Signed-off-by: WenyXu <wenymedia@gmail.com> * test(telemetry): cover log directory retention Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(telemetry): avoid log filename allocation Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Lei, HUANGandCopilot Autofix powered by AI
|
be482553b0 |
fix(mito2): harden flat merge and use winner_tree (#9064)
* fix(mito2): reject non-flat batches in flat merge instead of panicking - SortColumns::new becomes fallible try_new: batches missing the flat-format internal columns (time index, __primary_key, __sequence) at the fixed trailing positions now yield InvalidRecordBatch instead of a downcast panic, completing the generic-schema gate that only covered BatchBuilder output assembly. Document the flat-format input contract on FlatMergeIterator/FlatMergeReader. - Clarify why BatchBuilder's schema gate uses >= 3 columns when a real flat-format schema always has at least 4. - Add schema-structure tests: empty primary keys (tables without tags), dictionary-encoded string tag columns with per-source dictionaries, and graceful rejection of batches without internal columns. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test(mito2): scan tables with various schemas through flat merge Add an engine-level test that writes, flushes and scans regions without tags (empty primary key) and with multiple string tags (dictionary-encoded in the flat input schema), so the flat merge reader merges an SST with the memtable on real schemas instead of hand-built batches. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): use winner_tree dependency Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Update comments for FlatMergeIterator struct Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
shuiyisong
|
9a65561226 |
feat(auth): support bearer token authentication over SQL protocols (#8899)
* chore: add mysql and pg auth method choose hook Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(auth): support bearer token authentication over SQL protocols Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mysql): authenticate bearer tokens in the handshake catalog Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |