* fix(pipeline): coalesce concurrent pipeline cache misses
The pipeline cache reads with a plain `moka::sync::Cache::get` and falls
through to a distributed query on a miss, so when the 10s TTL expires every
in-flight write request on a frontend issues its own scan of the single-region
`greptime_private.pipelines` table. Concurrent scans per expiry scale with
write QPS, and every frontend's burst lands on the same datanode. A user
running high-throughput ingestion through a pipeline saw that datanode
overloaded.
Switch to `moka::future::Cache::try_get_with` so concurrent misses on the same
key share one loader. This requires a single-key lookup, so cache entries are
now keyed by the requested schema rather than the schema the pipeline is stored
under; resolving a request to a stored schema stays in the loader, which is the
authoritative path and already handles the empty-schema and multi-schema cases.
A lookup for a schema not yet cached costs one extra read, now protected from
amplification by the coalescing it enables.
`remove_cache` previously only walked the compiled-pipeline cache, so an entry
populated by `get_pipeline_str` alone (the pipeline read API) survived deletion
until it expired. It now walks all three caches.
Also make the TTL configurable as `pipeline.cache_ttl`, default unchanged at
10s. The TTL is what propagates a pipeline change to other frontends, so
raising it trades staleness for fewer reads.
Refs #9021
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): restore cross-schema semantics broken by the new cache key
Keying cache entries by the requested schema dropped two behaviours that the
previous stored-schema key provided for free.
Creating a new version only wrote the creating request's schema, so another
schema on the same frontend kept serving its cached `latest` — an older
version — until the entry expired. Since the whole point of making the TTL
configurable is to let operators raise it, that window is not bounded by
anything useful. Creation now invalidates every schema's `latest` alias for
that name before priming the cache, leaving the version-pinned keys alone.
The failover cache lost its reach across schemas the same way: a global
pipeline (stored under the empty schema) loaded by schema A was cached under
`A`, so schema B using it for the first time while the pipeline table was down
missed and failed ingestion. The failover cache has no loader and so is not
subject to the single-key model of `try_get_with`; it keeps the stored-schema
key and the empty-schema-first resolution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(pipeline): drop cache priming on create and fold the sweep helpers
Priming the cache on create saved one read on a low-frequency operation and
cost a concept: entries were written under the creating request's schema while
`PipelineContent.schema` said empty, so the two schemas in play disagreed.
Invalidating the `latest` aliases is required regardless — that is what makes
a new version visible to other schemas — so dropping the priming loses only
the saved read, which coalescing now protects anyway. `insert_and_compile` no
longer needs the caller's schema.
`remove_cache` and the create-time invalidation collapse into one
`invalidate(name, version)`; `None` sweeps only the `latest` aliases, which is
exactly what creation wants. That leaves `invalidate_by_suffixes` and
`cache_keys` with a single caller each, so both are inlined.
Drop the `PipelineOptions` humantime test: `load_config_test` loads both
example TOMLs, which now carry `cache_ttl = "10s"`, and would fail the same
way if the serde attribute were lost. The `toml` dev-dependency goes with it.
The two invalidation tests are now checked to be orthogonal: removing the
version suffix fails only the delete test, and sweeping just the compiled
cache fails both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): keep failover populated across a create
The `latest` sweep on create clears the failover cache along with the loaded
ones, and after dropping the priming there was nothing writing it back. An
outage between the create and the first read-back left neither `latest` nor the
explicit version with anything to fall back on, failing ingestion — worse than
before, since the previous version's failover entry was swept too.
Creation now goes through `PipelineCache::on_pipeline_created`, which pairs the
sweep with a failover write of the new empty-schema definition. The two must
happen together, so they live behind one method rather than at the call site.
Also commit the Cargo.lock entry for the dropped `toml` dev-dependency, and
trim the comments added over the last few commits down to what the code does
not already say.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(json2): concretize JSON2 schemas at merge scan boundaries
Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec.
Add SQLness coverage for whole JSON2 columns in windows and joins.
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
feat(query): support JSON2 paths in SQL functions
Update the DataFusion fork to expose scalar function planning hooks.
Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs.
Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions.
Signed-off-by: luofucong <luofc@foxmail.com>
* feat(runtime): add weighted workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics
Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to dynamic-config revision
Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): runtime-adjustable workload scheduler parameters
Expose dynamic adjustment of the experimental workload scheduler at
runtime:
- common-runtime: set_workload_scheduler_weights and
set_workload_scheduler_max_concurrent_polls, which forward to the
catio scheduler's set_weight/set_max_concurrent_polls when the
scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
/debug/workload_scheduler/max_concurrent_polls POST handlers, so
operators can rebalance query/write shares or admission concurrency
without restarting the datanode.
Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): add GET /debug/workload_scheduler status endpoint
Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to time-accounting revision
Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to lock-free sampling revision
Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio to scheduler PR head
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): add scheduler bypass control
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: advance catio scheduler fixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: regenerate config docs for workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio scheduler test fix
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(http): satisfy scheduler lifecycle clippy
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: add distributed scheduler toggle coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat: finalize workload scheduler runtime controls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio atomic weights
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: preserve unrelated lockfile resolution
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(runtime): downsample scheduler time accounting
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(runtime): verify cross-runtime scheduler progress
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): configure scheduler poll sampling
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): clarify scheduler activation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): explain scheduler use case
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
* feat: Add built-in daemon mode for standalone service
Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`.
- Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built.
- stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender.
- On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground).
Closes#7314
Signed-off-by: tian1220A <1573612521@qq.com>
* test(cmd): add integration test for standalone daemon mode
Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking.
Signed-off-by: tian1220A <1573612521@qq.com>
* fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt
Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional.
Also fix the rustfmt formatting issues that caused fmt-check to fail.
Signed-off-by: tian1220A <1573612521@qq.com>
* chore(cmd): declare daemonize dependency directly in cmd
daemonize is only used by the cmd crate. Move it out of the workspace
dependencies and declare it directly in the unix-only dependencies of
src/cmd/Cargo.toml.
Signed-off-by: tian1220A <1573612521@qq.com>
---------
Signed-off-by: tian1220A <1573612521@qq.com>
* fix(perf): align direct-SST CREATE TABLE with baked index metadata
The offline fixture generator (query_perf_fixture::direct_sst::
build_region_metadata) bakes greptime:inverted_index /
greptime:skipping_index field metadata into the region manifest for
tag/field columns, but create_table_sql emitted a bare CREATE TABLE
without those declarations. MergeScan's remote-schema validation then
failed on any tag/field projection (HTTP 500 'advertised remote stream
schema field mismatch'), breaking direct_readable_sst perf cases.
CREATE TABLE now declares the matching SKIPPING INDEX WITH
(granularity='1') / INVERTED INDEX column options. A round-trip test
proves the emitted SQL is parser-valid and yields the exact catalog
metadata.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): speed up Prometheus JSON response building with ryu and per-series entry reuse
PrometheusJsonResponse::record_batches_to_data spends ~47% of its CPU
in f64::to_string() per sample and ~32% in IndexMap::entry() per row
(60s profile of concurrent query_range workloads, ~800k series).
- Replace f64::to_string() with ryu::Buffer::format_finite for finite
values (shortest round-trip, 3-5x faster); NaN/+Inf/-Inf keep the
previous std formatting so wire output is unchanged.
- Remember the previous row's label vector and entry index; query output
is clustered by series, so consecutive rows reuse the same IndexMap
entry via get_index_mut instead of rebuilding and hashing the label
vector (worst case adds one Vec comparison per series transition).
Also adds a query-regression case (prom_json_response) that measures the
real Prometheus HTTP range API path (/v1/prometheus/api/v1/query_range),
which is the only frontend path that builds the Prometheus JSON response
(TQL ANALYZE formats the SQL JSON shape instead), plus a prom_http query
kind in the regression runner.
Perf (aligned base d90cca4b75, 256 series x 481 points):
- prom_range_2h (JSON response path): 29.31ms -> 21.46ms (-26.8%)
- tql_range_2h_control (non-JSON path): +1.87% (noise)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(servers): keep Prometheus wire format for integral floats with ryu
ryu::Buffer::format_finite prints integral values as "1.0", but the
Prometheus JSON wire format (matching std f64::to_string) expects "1".
Strip the trailing ".0" that ryu only emits for integral values; extreme
values keep ryu scientific notation, and NaN/Inf keep std output. Adds
wire-format tests covering 1.0, 0.0, -0.0, 1.5, 0.1, 1e21, 1e30, 1e-7,
f64::MAX, NaN, ±Inf.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(servers): address Prometheus response review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers)!: use ryu for Prometheus sample values
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(cmd): skip Prometheus execution time extraction
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): restore columnar group-by for dictionary-encoded tags
Bump the DataFusion fork to be93ffd85 (feat/dict-group-column-53), which
backports apache/datafusion #23187: DictionaryGroupValuesColumn lets
dictionary-encoded group keys use the columnar GroupValuesColumn fast
path (hash distinct dictionary values once per batch, resolve rows by
key index) instead of falling back to row-based GroupValuesRows.
This fixes the TSBS double-groupby regression introduced by #8541
(preserve dictionary-encoded query labels): v1.2.0-beta.1 scan output
changed tag columns to Dictionary(UInt32, Utf8), which DataFusion 53.1.0
did not support in GroupValuesColumn's supported_type allow-list, so
GROUP BY queries silently dropped to the ~60% slower row path
(time_calculating_group_ids +57%, peak_mem +50%, end-to-end +38%).
Adds an end-to-end integration test (dict_groupby_sst) that flushes a
flat-format SST with dictionary-encoded hostname, runs the tsbs-style
double-groupby query, and asserts correct results with no CastExec
inserted before the aggregate.
Signed-off-by: discord9 <discord9@greptime.dev>
* chore(deps): pin merged dictionary group-by support
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@greptime.dev>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: discord9 <discord9@greptime.dev>
* feat: embed the derivation conventions as data and adopt gen_ai entity naming
Move the co-declared edge vocabulary, the agent-edge vocabulary and the
virtual-destination candidates from Rust consts into an embedded
conventions.yaml (include_str!), parsed once behind a LazyLock and
validated against the entity-type grammar and the closed rel_type set; a
broken file propagates as a plan error instead of panicking. The agent
vocabulary entity types follow the GenAI semantic-convention namespace
as written: gen_ai.agent / gen_ai.model / gen_ai.tool.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: drop the tag requirement for entity identity columns
Entity declarations no longer require id columns to be tag/primary-key
columns; only column existence is validated. Trace pipelines flatten the
identifying attributes (span_attributes.gen_ai.agent.id, ...) into field
columns, so the tag rule locked real trace tables out of declaring
entities while buying no correctness — the read-time derivation works on
any column.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: implicit declarations for well-known prometheus info metrics
Tables stamped signal_type=metric + source=prometheus whose name matches
the conventions.yaml whitelist (kube_pod_info, kube_node_info,
kube_pod_owner, target_info) get implicit entity declarations: k8s.pod /
k8s.node / k8s.workload with name-based identity and target_info's
service / service.instance with the remaining tags as the descriptive
snapshot. The existing co-declared vocabulary then derives runs_on and
part_of from the same rows, so no new edge branch is needed. Explicit
declarations of a type always suppress the implicit one, and the metric
engine's physical table is excluded (it aggregates every logical
table's columns).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover the prometheus conventions in sqlness and compact the graph cases
Add the whitelisted-info-metric scenario (kube_pod_info, kube_pod_owner,
target_info deriving runs_on / part_of, a non-whitelisted metric
contributing nothing), fold the single-table calls, cross-table pairing
and virtual-node cases into one trace scenario (they exercise the same
union-before-join path), merge the two declaring-metric-table cases, and
reuse one rename probe for both reserved names.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject entity id columns without a stable string form
Review follow-ups: the DDL check now validates against the schema and
rejects binary-backed and nested types for identity columns (the
derivation renders ids via CAST to Utf8, so the failure used to surface
only when the graph was scanned); the agent sqlness case keeps its
identity columns as fields to cover the relaxed tag rule end to end;
stale tag-rule comments and a dangling const reference are cleaned up.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: type-check every entity column role, not only ids
The registry renders scope and descriptive values through the same
CAST-to-string path as ids, so a binary-backed column in any role fails
at scan time; the DDL check is now role-independent (and simpler).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name the code-anchored vocabulary constants
Entity types and edge attributes the derivation code itself anchors on
(service, gen_ai.agent, calls, trace/attribute provenance) become
constants in the conventions module; the rest of the vocabulary stays
YAML-only data. ImplicitEntity is renamed PromImplicitEntity, and the
implicit-declaration path logs each skip of a whitelisted info metric
(wrong stamps, suppressed by an explicit declaration, missing id
column) so a missing graph entity is diagnosable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: single-source the graph constants
The graph tables' column names move to common-catalog (the schemas
catalog exposes and the plans operator builds must match column by
column), and the conventions module now carries the complete built-in
vocabulary — entity types, rel_types, provenances and connection types —
with the embedded YAML validated by membership against it, so an edit
drifting outside the vocabulary fails the conventions test instead of
deriving nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: treat empty identity components as absent
kube-state-metrics emits empty-string labels an entity id must not be
built from: an unscheduled pod's node and an owner-less pod's owner_kind
/ owner_name. Standard Prometheus drops empty labels (they arrive as
NULL and the existing predicate handles them), but other remote-write
agents may keep them, which produced ghost entities with empty ids and
false runs_on / part_of edges. Every identity predicate (registry,
co-declared edges, span endpoints) now requires non-NULL and non-empty
components through one shared helper.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: tighten the conventions DSL semantics
Rename the co-declaration rule lists to what they are (co_declared_edges
/ trace_co_declared_edges — derivation rules, not a relation
vocabulary), stop overstating the GenAI entity types (Greptime types
derived from GenAI attributes; OTel defines no model/tool entities),
move target_info's descriptive snapshot to service.instance (the
remaining labels are the target's resource attributes, and instances
would write conflicting snapshots onto the logical service), and extend
the descriptor whitelist with the stable KSM sources: container info
metrics (closing the k8s.pod contains k8s.container rule),
kube_service_info (new k8s.service entity type) and the fuller
descriptive label sets.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: guard entity column types on ALTER as well
ALTER MODIFY COLUMN could change a declared entity column to a type
without a stable string form, deferring the failure to graph scan time;
verify_alter now checks the post-alter schema. Dropping a declared
column stays allowed — the read-time derivation skips the stale
declaration, and semantic options cannot be altered off yet.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: bridge traces and kube-state-metrics on the pod UID
Trace-v1 tables now get implicit declarations from their flattened
resource attributes (otlp_trace_entities in conventions.yaml): the
service identity — replacing the hardcoded fallback — plus
service.instance and k8s.pod, each applied only when its columns exist.
A new co-declared rule derives service.instance runs_on k8s.pod, and
the whitelisted kube-state-metrics pod identity switches from
namespace+pod names to the UID, so the trace-side pod and every KSM
descriptor land on one entity while names stay descriptive. This also
removes pod identity from the multi-cluster same-name collision.
The conventions rejection tests were passing for the wrong reason (a
half-renamed fixture key failed deserialization before reaching any
validation rule); they now assert the specific error each case targets.
Sqlness covers the UID merge across descriptor tables, pod-contains-
container, the k8s.service node, and the empty-uid/empty-node rows
deriving nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover the OTLP-to-graph chain end to end
One real OTLP export must come out of semantic_relationships as the
zero-configuration chain: service calls service, instance part_of
service, instance runs_on pod (bridged by k8s.pod.uid). Resources
without service.instance.id or k8s.pod.uid derive nothing extra.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: identify k8s.service by UID
Same reasoning as pods: a recreated same-name service must not merge
into the old entity and same-named services across clusters must not
collide; kube_service_info carries a stable uid and nothing joins on the
service's name. Also drop a stale tag-rule mention from the option
validation docs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cut duplicated test coverage and redundant comments
The trace service-fallback test collapsed into the resource-entities
test (same synthesis path since the fallback moved to YAML; only the
invalid-explicit-no-fallback case was distinct), role-duplicate and
subsumed DDL cases are gone, the embedded-conventions test is just the
parse (its assertions were decorative), and the YAML section comments no
longer restate the struct docs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Bump the GreptimeTeam/datafusion fork rev from 6d6ae9a to 452cb4b,
which includes fix(substrait): support Dictionary literals in producer.
This fixes flow queries against dictionary-encoded PK string columns
(metric tables) failing with:
Failed to encode DataFusion plan:
NotImplemented("Unsupported literal: Dictionary(UInt32, Utf8(...))")
The substrait producer now encodes ScalarValue::Dictionary as its inner
value wrapped in a cast to the dictionary type, so the original SQL
works without CAST workarounds.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor: port query regression runner to Rust
Signed-off-by: discord9 <discord9@163.com>
* ci: remove optional OTLP report plotter
Signed-off-by: discord9 <discord9@163.com>
* refactor: split query regression runner into modules
Signed-off-by: discord9 <discord9@163.com>
* style: use crate-qualified imports in query regression runner
Signed-off-by: discord9 <discord9@163.com>
* refactor: simplify query regression runner internals
Signed-off-by: discord9 <discord9@163.com>
* feat: abstract inspect-footer storage access behind object store destination
Add an optional --destination <TOML> to inspect-footer (and
--base-destination/--candidate-destination to finalize-remote) so the
storage inspection reads DB data files through the opendal-backed
object_store abstraction instead of bare std::fs. Local paths keep
working unchanged via the --root shortcut (File backend); remote
backends (S3/GCS/...) are described by a DestinationConfig TOML
reusing the object-store crate's ObjectStoreConfig serde shape.
- inspect_footer: list via ObjectStore::list + ObjectMeta filtering
(parquet keys, non-zero size, metadata/ segment), read footers
async via ParquetObjectReader + ParquetMetaDataReader with known
file size (no extra HEAD); output JSON schema unchanged
- finalize-remote: --base-data-home/--candidate-data-home become
optional, mutually exclusive with the new --*-destination args
- cmd deps: add object_store_opendal + datafusion_object_store
- tests: fs-backend list+footer integration tests (metadata filtering,
destination TOML mode, root/destination exclusivity)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style: drop needless borrow in inspect footer test
Fix clippy::needless_borrows_for_generic_args in the inspect-footer test
(fs::create_dir_all(table.join("metadata"))). Missed by the earlier
focused clippy run because it only covered --bin targets.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Parse PostgreSQL DSNs (URL or libpq keyword) with tokio_postgres::Config —
the backend's own parser — and log its Debug, which redacts the password.
This matches the backend grammar exactly (multi-host URIs, backslash
escapes, any Unicode whitespace, percent-encoded query keys, and '&'/';'/
'://' inside values) rather than approximating it by hand. Other URLs are
redacted via the url crate; a best-effort keyword fallback covers inputs
neither parser accepts.
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
* feat(flow): handle time_ranges in DirtyWindowRequest
Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): fix doc comment spacing in align_time_window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time_ranges in handle_mark_dirty_time_window
Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): union merged dirty windows with the larger end
Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): clip bounded dirty ranges at the expire bound
merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): fall back to full dirty on dirty-window alignment failure
An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): apply rustfmt to new dirty-window merge tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time index units in dirty window marking
Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to 8127f179
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): return dirty-window alignment errors to callers
Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: update proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Weny Xu
dennis zhuang
LFC
discord9andRuihang Xia
zhaiAohan
Ning Sun
dependabot[bot]
Lei, HUANG
jeremyhi
Yingwen
shuiyisongandfys
raphaelroshan