* fix: reclaim HDFS temporary files left behind by a datanode crash
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): stop the temporary file sweep from starving GC jobs
Signed-off-by: lczllx <2181719471@qq.com>
* feat(mito2): add metrics for the temporary file sweep
Signed-off-by: lczllx <2181719471@qq.com>
* refactor(object-store): share one definition of the temporary file name
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): skip the temporary file sweep when no GC job is allowed
Signed-off-by: lczllx <2181719471@qq.com>
* refactor: gate the temporary file sweep behind the hdfs-object-store feature
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): sweep temporary files when a create opens an existing region
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): document the temporary file sweep in the GC settings
Signed-off-by: lczllx <2181719471@qq.com>
* refactor: align the temporary file sweep code with the surrounding style
Signed-off-by: lczllx <2181719471@qq.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
* refactor(mito2): pick TWCS L1 merges by closed units
TWCS could keep rewriting L1 files without new writes. Overlap-only picks
could split their inputs into one more file than they read, and L1 file
reduction then rewrote up to 16 files to remove that single file. The
outputs, split by key, created new overlaps and the cycle repeated.
Replace the three global pick phases with one pick per window, scheduled by
class across windows. Every pick now decreases a termination measure when the
window receives no writes.
- Active windows only take merges that remove files. L1 files are grouped by
conflicts and merged in key order, so size-split tails are merged together.
A reduction rewrites at most 4 output files' worth of bytes per removed file.
- Inactive windows absorb L0 files, then take closed merges: no other file
intrudes on the inputs' bounding box, so outputs cannot add overlaps. Each
merge resolves conflicts or removes files, decreasing (conflicts, files).
Files that do not take part in an overlap are not rewritten.
- Overlaps that only share a boundary key no longer count as conflicts, so a
size-split sorted run does not look overlapping.
- Closed merges may take 4x the input limit to resolve a whole window once.
- Cache primary key ranges per pick instead of resolving them per comparison.
- Log the pick branch of each compaction output.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): state the compactor assumptions behind TWCS termination
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): resolve active window overlaps on manual compaction
A window that stops receiving writes keeps being the active window, whose
picks only remove files, so its overlaps were never resolved. Manual
compactions now pick every window as inactive, so a closed merge can resolve
them. Automatic compactions are unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): defer overlap resolution in the window before the active one
Late data keeps landing in the window right before the active window. A
closed merge that resolves its overlaps early rewrites the whole window,
then the late data brings the overlaps back, and the balanced run check
keeps them there. Overlap-only merges now skip that window until it is
older. File-reducing merges and manual compactions are unchanged.
On the 12h/1GB out-of-order bench this reaches overlap depth 1.0 instead
of 2.08 at the same write amplification (3.01 vs 3.02).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): only check earlier-starting files when closing units
Closing units only checks files that start before a unit ends, and the
overlap checks compare key bounds without cloning them.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): keep settling windows from rewriting L1 files
The window before the active one still receives late data. Skipping only
its overlap-only merges was not enough: file-reducing merges of
conflicting files and mixed L0/L1 merges still rewrote its L1 files,
and a mixed merge could rewrite a nearly full output for each small late
file. A settling window now only merges L0 files among themselves and
conflict-free files; the window resolves its overlaps once it is older.
On the out-of-order bench, write amplification drops from 3.88 to 3.30
at 2GB/6h, 3.05 to 2.94 at 1GB/6h and 3.01 to 2.96 at 1GB/12h, with
overlap depth 1.0 in all three.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve late overlaps once a window is quiet
Late data that lands in a window older than the settling one is merged
into a small file covering the whole key range. Resolving it rewrites
the window's dominant sorted run, which the balanced run check refuses,
so the overlap stayed. Removing the check instead rewrites the window
for every late flush.
A window is quiet once the region has written at least as many rows as
the window holds since its newest file. Overlap merges in a quiet window
may rewrite a dominant run, so each quiet period pays for at most one
such rewrite. Sequences are per row and kept by compaction, so this
needs no state.
On the bench with late data up to 2 hours, mean overlap depth drops
from 1.67 to 1.17 at 2GB/6h and from 2.33 to 1.83 at 1GB/6h, at write
amplification 3.70 to 4.25 and 3.26 to 3.74; all inactive windows of
2GB/6h realtime reach depth 1 (write amplification 2.80 to 3.30). The
remaining depth is in the final active window.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): check closed merges of quiet and settling windows terminate
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): split TWCS closures over the input limit
A file spanning the keys of a large sorted run joins every run file into one
closure. Once a closure held more files than one compaction accepts, closed
merges skipped it, so the window never merged those files again and late
files kept piling up.
Split such closures into single files, like active-window picks do, and
replace the closed check with a conflict bound: outputs only conflict with
files intruding on the inputs' box, so a merge is taken if `outputs · I`
does not exceed the conflicts its inputs had with other files. Intervals of
whole closures have no such conflicts and still need to be closed, so other
windows pick as before, and `(conflicts, files)` still strictly decreases.
Small run files and late files over a large run can be merged again. A late
file over a run that cannot shrink stays: resolving it would rewrite the
whole run.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): address TWCS picker review findings
- Manual compactions may rewrite a dominant sorted run to resolve overlaps.
The active window is never quiet, so manual compactions could not resolve
its late-data overlaps before.
- Single-key intervals predict one output: the writer never splits a series,
so large files of one series were never merged.
- The active window falls back to single files when no group of conflicting
files qualifies, so a large file no longer keeps its small peers unmerged.
- Compute the settling window distance with checked_sub, which overflowed for
windows at the timestamp limits.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): count inner keys as TWCS conflicts and check balance per group
- A single key strictly inside another file's key range now conflicts, so
a late file of one series inside a wide file is resolved like other
overlaps. Files sharing only a boundary key still do not conflict.
Intrusion covers the new pairs, which a randomized test checks.
- Check the dominant-run guard per group of conflicting files. Runs of
separate groups never overlap, so pairing one group's run with another
group's peer made an interval of two dominant runs look balanced.
- A window with unknown row counts, written before they were recorded, is
not quiet: its threshold would drop to a single row.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): index TWCS files by key to avoid quadratic grouping
Grouping a window's files compared every file with all files sharing its
time range, and closing groups and checking intrusions scanned the whole
window. A sorted run split by key shares one time range, so a window of N
such files took O(N²) checks without any conflict. The 1024-file fallback
only covers the sorted-run path, not these groups.
Index the files by lower primary key in a segment tree that keeps each
subtree's largest upper key and its time bounds. Grouping, closing, and the
conflict bound only check the files whose key and time ranges meet, so a
file of a key-sliced run only meets its neighbors. Conflicts among split
files are precomputed per file and counted per interval with a binary
search instead of pairwise.
On a run of 20,000 key-sliced files with 5 late files spanning all keys,
a closed merge pick takes ~62 ms instead of ~3.9 s and an active pick
~19 ms instead of ~3.5 s.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(object-store): retry transient S3 transport failures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): bound TWCS closed-merge conflict graph work
Budget the candidate files visited while building conflict units and split-unit adjacency. When the budget is exceeded, treat every file as a split unit and pick a bounded file reduction instead of materializing a quadratic graph.
The fallback candidate is admitted only through adds_no_conflicts, so outputs do not add conflict pairs and the (P, N) termination invariant still holds. Dense picks are logged with their own branch.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): charge TWCS index nodes to the conflict-graph budget
The conflict-graph work budget only counted files that reached a callback, so a traversal with poor pruning could still visit a quadratic number of internal nodes without triggering the dense fallback. Charge every ExtentIndex node, including pruned nodes, and stop the traversal as soon as the shared budget is exhausted.
Add a regression test where the root straddles the query time but neither leaf matches, so a callback-only budget would never fire.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): retry smaller TWCS dense compaction candidates
If a dense fallback candidate fails admission, try one single-output subinterval within it before giving up. Reuse the reduction guards and check both candidates against the full window's external conflicts, preserving bounded work and termination.\n\nCover the chain regression through both fallback unit constructions and reject one-output retries that introduce conflicts across time gaps.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): compact shared primary-key boundaries
Count shared primary-key endpoints as conflicts and apply the same overlap rule to bounding-box intrusions. Existing compaction writers keep series intact, so fresh output ranges remain disjoint.
Align simulated output splits with the writer and cover boundary-only cleanup in inactive and manual windows, plus stable real output files after a subsequent TWCS request.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): compact small L1 files in inactive windows
Allow bounded small-L1 reductions in settling and older inactive windows when normal picks cannot proceed. Preserve full-window conflict checks, large-file protection, and LastNonNull closure semantics.\n\nCover selection limits, termination, and real SST compaction with tombstones and reopen checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): clarify TWCS unit grouping guarantees
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(hdfs): clean up temporary file when an atomic write fails
Signed-off-by: fengyue-xve <1759519137@qq.com>
* style: apply rustfmt to hdfs layer test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(hdfs): report earlier write failures and log cleanup failures
- Later `write`/`close` calls on a writer whose failed write was
cleaned up now mention the original failure instead of the bare
"unavailable after close or abort" message.
- `abort_and_delete` logs abort and deletion failures with the
temporary path instead of discarding them; the original write error
is still returned to the caller.
- The temporary-file cleanup test now lists the directory holding the
temporary file, so it fails if the cleanup is reverted.
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
* chore: remove dead code left behind by removed features
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: remove unused RouteInfoCorrupted error
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks
cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.
- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
misplaced deps to the correct sections, drop orphaned
[workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
for dependencies that are structurally required despite no textual
reference: sqlparser (required by sqlparser_derive expansions in
datatypes, common-query), common-error (required by common-macro's
stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
(feature-pinning for the transitive kube dependency in tests-fuzz),
tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
common-mem-prof), protobuf (required by build.rs-generated bindings in
log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections
Part of #9289
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test(meta): verify physical metric column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
---------
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* fix(postgres): return empty responses for comment-only SQL (#9295)
fix(postgres): handle parsed empty queries in both protocols
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
* ci: create docs follow-up issue on PR merge instead of on label (#9237)
* ci: create docs follow-up issue on PR merge instead of on label
The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.
Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
carries the docs-required label
This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on docs issue creation timing
- Only touch docs labels when the docs checkbox state actually changed
in an edit. Previously, editing any other part of the PR body while
the checkbox stayed checked removed the docs-required label, silently
dropping the docs follow-up now that issue creation happens at merge.
Unchanged checkbox now leaves labels untouched, which also preserves
manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
re-read the live PR via the API and create the docs issue if the
docs-required label is present OR the checkbox is ticked in the
current body.
- Make the workflow concurrency group action-aware so a merge run does
not cancel an in-flight label update from an edit run.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: make docs-required label the single source of truth at merge
The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.
At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): check matcher safety on the receiving operand
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): spell out the shapes a filter may cross
`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.
Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.
No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): assert the new pushdowns reach the scan
The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.
Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns
Signed-off-by: fys <fengys1996@gmail.com>
* fix(json2): preserve rows on type hint mismatch during compaction
* refactor(json2): simplify alter settings handling
* fix(json2): preserve coerced values during compaction
* chore: remove unnecessary clone
* chor: reduce memory allocations
* fix: cargo clippy
* chore: update greptime-proto to main branch
* refactor(datatypes): unify string handling with other JSON type hints
* fix: cr
---------
Signed-off-by: fys <fengys1996@gmail.com>
* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
* fix: run compaction pruner tasks on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep compaction metadata and index work on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev
Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add Jev choice and score functions
Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: use generic AI SQL function names
Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: share constant AI criteria within each batch
Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: preserve AI score uncertainty in JSONB results
Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: explain reuse of volatile AI evaluations
Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: unify pending rows batch acknowledgement policy
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align logical batcher example configuration expectations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align batcher worker channel defaults to 65536
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: sync lock
* fix: shear and check issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* test: cut integration test time and make the storage matrix meaningful
tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.
- Point the PR CI object-store matrix at the MinIO already started by
tests-integration/fixtures. Three GT_S3_* consumers did not read
GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
release integration-test container. The release previously ran every
remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
/tmp/greptimedb_cache, which the datanode wipes on startup, so a
starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
non-trivial SST representation: JSON/JSON2 columns, native histograms,
metric-engine logical tables, and tables carrying fulltext or skipping
indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
shorten it in the event tests, which otherwise wait a 5s window per DDL
they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
test_grpc_message_size_ok and is now rewritten to assert the negotiated
grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
prefixes of their copy_from siblings; two standalone/distributed event
test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
in a parallelism-dependent order, so its last digits are unstable; the
test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
ttl=7d with 2025 timestamps, so its rows were only readable as long as
they stayed in the memtable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: address review — wire nightly real-S3 job into check-status, keep the short event interval
The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.
In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
fix(object-store): make removed-entry lister test portable
On Windows, DirEntry is a FindFirstFileW snapshot: file_type() and
metadata() keep returning cached data after the file is removed, so
read_list_entry() may yield the stale entry instead of None. The strong
assertion is Unix-specific; keep the portable checks (call succeeds,
path is correct) on all platforms and gate only the is_none() assertion.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and
metadata() keep returning cached data after the file is removed, so
read_list_entry() cannot observe the deletion. The test asserts the
Unix behavior (lstat returns ENOENT) and fails deterministically on
Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))].
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
test_writer_abort_is_unsupported_without_atomic_write asserted the file
content immediately after abort() returned Unsupported. SecureFsWriter
writes through tokio::fs::File, whose write_all() only enqueues a blocking
write task (tokio's poll_write returns Ready before the write completes),
so the data may not be visible yet when the test reads the file. Drop the
race-prone content assertion and only verify the Unsupported contract.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
refactor/building-backend-in-object-store:
### Refactor Object Store Configuration
- **Centralize Object Store Configurations**: Moved object store configurations (`FileConfig`, `S3Config`, `OssConfig`, `AzblobConfig`, `GcsConfig`) to `object-store/src/config.rs`.
- **Error Handling Enhancements**: Introduced `object-store/src/error.rs` for improved error handling related to object store operations.
- **Factory Pattern for Object Store**: Implemented `object-store/src/factory.rs` to create object store instances, consolidating logic from `datanode/src/store.rs`.
- **Remove Redundant Store Implementations**: Deleted individual store files (`azblob.rs`, `fs.rs`, `gcs.rs`, `oss.rs`, `s3.rs`) from `datanode/src/store/`.
- **Update Usage of Object Store Config**: Updated references to `ObjectStoreConfig` in `datanode.rs`, `standalone.rs`, `config.rs`, and `error.rs` to use the new centralized configuration.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: change cache policy for file cache
* feat: file cache run pending task after put
* feat: run pending task in put_dir
* feat: run pending task after stager recovered
* feat: purge recycle bin periodically
* feat: use lru policy for read cache