rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-19 10:22:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,367 Commits 327 Branches 279 Tags
stack/pr9412-capability
Commit Graph
11 Commits
Author SHA1 Message Date
jeremyhi 18726c7602 fix: bound metric snapshot memory and reduce export overhead (#9505) 
* fix(mito): bound retained scan memory

Compact unused tag dictionary values before merge retention and avoid splitting
a single scan source into redundant batches. Include regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(snapshot): bound writes and reduce logical export overhead

Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible
string dictionaries, bound conversion work, combine Parquet row groups and
filter single-table exports before routing. Include resource-boundary and
packed-restore regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: clarify metric snapshot acceptance criteria

Specify populated scale fixtures, controlled S3 request delay, native macOS
validation, paging exclusions and alternating comparison rounds.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): split batches only for merged readers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(snapshot): cover supported types in both layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: consolidate metric snapshot experiment conclusions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): honor series statistics when splitting L0 scans

Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics.

Cover short-series overlapping L0 files with an ordered latest-value regression.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito): explain memory risk of splitting short L0 series

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address snapshot export review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): accept source errors in lazy snapshot contexts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-10-12 06:17:03 +00:00
jeremyhi 6265c31e4a feat: batch logical table DDL during import-v2 (#9467) 
* feat: batch logical table DDL during import-v2

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve one-shot batch DDL transport and context

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: check schema-only import state at its own path

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve oversized import DDL compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: reuse import DDL encoding and metric constants

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-10-08 13:14:06 +00:00
jeremyhi 78eefbe4c4 perf: batch schema export requests (#9408) 
* perf: batch schema export requests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: propagate legacy schema export failures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep credentials out of SQL response errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use a typo-safe dotted catalog name

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(cli): address schema export review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-30 06:23:33 +00:00
jeremyhi 3c0e2a8d55 feat: export Metric snapshots with packed Parquet objects (#9382) 
* feat: export Metric snapshots with packed Parquet objects

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject empty packed export time ranges

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover cancellation during packed export I/O

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-28 15:22:02 +00:00
jeremyhi bf67246645 perf: bound concurrent Metric export writers (#9296) 
* fix: preserve objects after conditional Metric export collisions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: share Metric export writer and payload budgets

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover bounded Metric export conversion and mixed roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve export output after ambiguous close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align HTTP export timeout with managed cancellation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: trim redundant Metric export roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: bound pending Metric export writer handles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: protect export files on failed writes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve ordinary export cleanup and large rows

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: report Metric export collisions as invalid arguments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed conditional export closes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support view arrays in managed ordinary export

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: expect successful cleanup after failed close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up unsynced overwrite after close failure

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: scope export JSON expansion estimates to JSON columns

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: rechunk ordinary export batches within the write budget

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-28 03:10:21 +00:00
dennis zhuang f3eb8e6c72 test: cut integration test time and make the storage matrix meaningful (#9308) 
* test: cut integration test time and make the storage matrix meaningful

tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.

- Point the PR CI object-store matrix at the MinIO already started by
  tests-integration/fixtures. Three GT_S3_* consumers did not read
  GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
  they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
  release integration-test container. The release previously ran every
  remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
  /tmp/greptimedb_cache, which the datanode wipes on startup, so a
  starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
  non-trivial SST representation: JSON/JSON2 columns, native histograms,
  metric-engine logical tables, and tables carrying fulltext or skipping
  indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
  shorten it in the event tests, which otherwise wait a 5s window per DDL
  they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
  test_grpc_message_size_ok and is now rewritten to assert the negotiated
  grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
  prefixes of their copy_from siblings; two standalone/distributed event
  test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
  in a parallelism-dependent order, so its last digits are unstable; the
  test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
  ttl=7d with 2025 timestamps, so its rows were only readable as long as
  they stayed in the memtable.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: address review — wire nightly real-S3 job into check-status, keep the short event interval

The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.

In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-09-23 01:52:52 +00:00
jeremyhi 19c127d9c3 feat: restore packed metric snapshots (#9250) 
* test: cover snapshot parquet restore compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: restore packed metric snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: stream large packed parquet entries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: allow default S3 endpoint in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: skip unconfigured S3 in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use portable file URLs in packed restore fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate packed snapshot structure before restore

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve strict manifest decoding and verify fixture

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject packed layout on both database export paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: implement file size in coordinator test storage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use expect_err for rejected export layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-22 00:07:04 +00:00
jeremyhi 263e229103 feat: add experimental Metric export to V2 snapshots (#9233) 
* feat: add experimental Metric export to V2 snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate the complete Metric export capability response

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: construct portable file URLs for Metric export fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address Metric export review nits

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-21 09:10:01 +00:00
jeremyhi 33cfb72a43 fix: make database export assertions portable on Windows (#9256) 
fix: compare database export paths portably on Windows

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-20 04:23:37 +00:00
jeremyhi dca01654bc feat: prepare and execute database Metric exports (#9180) 
* feat: prepare and authorize captured database exports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: bound database export jobs and drain failures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: validate database export identity and restore equivalence

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate database export directory URLs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve Windows export directory paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject local database export filename aliases

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: consolidate database export planning policies

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: restore escaped database export filenames

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align database restore assertions with shared policies

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify database export boundaries and names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-17 12:02:39 +00:00
jeremyhi d7ada1761d feat: export logical tables from Metric physical scans (#9159) 
* feat: add physical Metric table exporter

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: drain Metric export writes before cancellation cleanup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: construct Metric export error context lazily

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: name the logical table export entry point

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: share Parquet writer for logical table exports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve Parquet destinations and cancellation boundaries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify logical table export field names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify logical table export helper responsibilities

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate logical export membership by table ID

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify logical table export coverage and strengthen assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-16 02:48:47 +00:00