* feat: add CORS support for gRPC-Web on the frontend gRPC server
A browser-based gRPC-Web client cannot read a cross-origin response. The
request carries a non-simple content type, so the browser sends an
`OPTIONS` preflight first, and it drops the call when the response has no
`Access-Control-Allow-Origin`.
Add `enable_cors` and `cors_allowed_origins` to `[grpc]`, threaded from
`GrpcOptions` to `GrpcServerConfig` the same way `tls` is.
The CORS layer sits outside `tonic_web::GrpcWebLayer`, so it answers the
preflight before the request reaches the gRPC routes. It exposes
`grpc-status`, `grpc-message` and `grpc-status-details-bin`, which a
browser cannot read otherwise. An empty `cors_allowed_origins` allows any
origin.
`enable_cors` defaults to false, unlike `[http]`. `GrpcOptions` is shared
by the public `[grpc]` section and the internal `[internal_grpc]` one, and
serde cannot tell them apart, so a true default would also turn CORS on
for the internal gRPC listeners: the frontend internal gRPC server, and
the datanode and flownode gRPC servers. None of them authenticate callers,
and a browser on the host or in the cluster network can reach all of them.
Set `enable_cors = true` to turn it on.
Tests start a real server on an ephemeral port and cover the preflight, a
custom origin list, and the disabled case.
Update the example TOMLs and regenerate config/config.md.
Signed-off-by: lczllx <2181719471@qq.com>
* fix: drop the redundant OPTIONS from the gRPC CORS allow_methods
Signed-off-by: lczllx <2181719471@qq.com>
* test: assert the allow-headers and allow-methods headers in the gRPC CORS preflight
The preflight answers with `access-control-allow-headers: *` from
`AllowHeaders::any()`, and with `access-control-allow-methods: post` from the
POST-only `allow_methods` list. Pin both, the way the HTTP CORS test pins its
own headers: a missing allow-headers header would let the preflight pass the
origin check and still have the browser block every gRPC-Web call, which a
non-browser client would never notice.
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): stop the example configs from setting the new gRPC CORS keys
Signed-off-by: lczllx <2181719471@qq.com>
* test(servers): cover the exposed gRPC CORS headers and pin the option plumbing
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): document the gRPC CORS origin example with `#+`
Signed-off-by: lczllx <2181719471@qq.com>
* fix(frontend): keep CORS off the internal gRPC server
Signed-off-by: lczllx <2181719471@qq.com>
* docs(grpc): document that the internal gRPC server never serves CORS
Signed-off-by: lczllx <2181719471@qq.com>
* Update src/servers/src/grpc.rs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim redundant comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: share CORS origin parsing and simplify gRPC-Web CORS tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: enable gRPC CORS only on the frontend public server
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(config): clarify the gRPC CORS options
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
* feat(datanode): wire the object store WAL into the datanode and standalone
A datanode configured with `provider = "experimental_object_store"` now
builds an `ObjectStoreLogStore` instead of failing with the placeholder
"not supported yet" error. The builder resolves `storage_provider`
against the object store manager (empty selects the default store, an
unknown name is rejected as an invalid `storage_provider`), opens the
store under the node id and the standalone generation, stops it if a
later build step fails, and hands it to `Datanode`, whose shutdown stops
it after the region server. Datanode shutdown is now best-effort: every
step runs and the first error is returned. A datanode with a metasrv
client still rejects the provider.
The standalone bootstrap builds `WalProvider::ObjectStore` with the same
derived node prefix before the metasrv WAL conversion, so the WAL
options it allocates match the prefix the store runs under.
`log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's
test utilities can build an engine on the object store WAL over an
in-memory object store, and new engine tests cover create, reopen,
replay isolation between regions, batch open, drop and offline cleanup,
the latest entry id after replay, and the rejection of object store WAL
options on a Raft Engine log store.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(datanode): stop every region engine before the object store WAL
The region server returned at the first engine whose stop failed, so the
engines after it in map order, Mito included, were never stopped before
the datanode stopped the object store WAL. It now attempts every engine,
returns the first error and logs the later ones.
`ObjectStoreLogStore` is also exported from the `log-store` crate root,
and its callers import it from there.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL
Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog
thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the
object store WAL config, validated by the datanode and the store.
In the `enqueued` mode `append_batch` returns on admission with the
entry ids assigned and the object is created in the background. At a
backlog threshold the next append is held back until an upload
completes. A transient create failure is repeated under the same
sequence with the same bytes; any conflicting object poisons the
store. `stop` uploads the backlog, or returns the error that dropped
it once stop began. `obsolete` clamps the watermark to the durable
entry id, and an id the store handed out needs no sequence floor.
Add `LogStore::wait_durable` with a default that returns at once. The
object store WAL answers it once the region is durable and indexed
through the entry id, and fails it after a backlog was lost.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): poison an enqueued store on permanent create failures
Repeat a failed create in the enqueued mode only when the storage error
is retryable; any other storage error poisons the store. A conflicting
object poisons an enqueued store without reading its epoch, so a failed
header read cannot turn the conflict into a retry.
A durability wait for an id above the highest id the store handed out
now waits for the handed-out ids of the region below it instead of
returning at once.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): poison on permanent create failures after stop begins
A create that fails with a storage error that is not retryable poisons
an enqueued store even after stop began; only a transient failure drops
the backlog without poisoning. Durability waiters whose callers stopped
waiting are pruned before a new waiter is queued. The backlog age test
no longer depends on a follow-up append finishing within the threshold.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): answer a durability wait once no earlier entry is pending
A durability wait now returns once the region holds no entry at or
below the target that is handed out but not durable, instead of waiting
for the largest id handed out to the region. A later object of the
region that is still being created no longer holds back a wait whose
target it does not cover.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): order the pending durability wait check after the actor
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(store-api): state that the default wait_durable keeps each store's guarantee
The default `LogStore::wait_durable` returns at once, which keeps each
log store's own acknowledgement guarantee; Raft Engine with
`sync_write = false` acknowledges before its periodic sync, so the
documentation no longer claims that every entry id a caller holds is
durable. The object store WAL configuration test now also serializes
the new options and reads them back.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): limit the acknowledgement guarantees to the durable mode
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): repeat enqueued creates that a retry layer marks persistent
An object store wrapped in the OpenDAL retry layer reports a temporary
error that outlasted its retries as persistent rather than temporary.
The enqueued mode now repeats a create after any storage error that is
not permanent, so a transient outage behind the retry layer no longer
poisons the store and drops the acknowledged backlog; after stop began
such a failure still drops the backlog without poisoning.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): cover a persistent create failure after stop begins
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(mito): limit series index disk usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): enforce series index quota during reconciliation
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): remove series index disk budget layer
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): simplify series index limit to estimated usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): trust index catalogs when loading snapshots
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): minimize series index disk limit changes
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): keep series index cleanup running at capacity
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): avoid no-op index clones and stabilize capacity tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: update config API expectation and stabilize index build tests
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: reject duplicate region engine configurations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: reuse canonical engine names in config validation
Replace duplicated engine-name literals with the existing common-catalog constants so duplicate-config validation uses the shared engine names. Keep the TOML regression inputs independent to verify the public configuration tags.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: clarify zero-based duplicate engine config indices
State explicitly that duplicate region engine configuration indices are zero-based so users can map them to the order of TOML entries. Preserve the existing index values and error classification.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add config flag to control range index reads
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: disable range index builds when configured and default to off
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat(wal): add the object store WAL provider identity and configuration
Add the identity and configuration of the experimental object store WAL
without the log store implementation:
- store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a
remote WAL.
- common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`)
with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and
`on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as
`object_store` with the key `wal.object_store.prefix`. The metasrv config
conversion rejects the new provider.
- common-meta, meta-srv, mito2: handle the new variants, map the region WAL
options to the provider and reject them on a Raft Engine or Kafka log store.
- datanode: validate the configuration and fail with a "not supported yet"
error until the log store lands.
- Example configs and the generated config docs.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(wal): accept object store WAL options when re-enabling WAL
Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(mito2): reconcile series indexes in background
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): clean up series indexes published during region drop
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(config): align series index examples with upstream enable flag
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): run series index tasks on compaction runtime
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(pipeline): coalesce concurrent pipeline cache misses
The pipeline cache reads with a plain `moka::sync::Cache::get` and falls
through to a distributed query on a miss, so when the 10s TTL expires every
in-flight write request on a frontend issues its own scan of the single-region
`greptime_private.pipelines` table. Concurrent scans per expiry scale with
write QPS, and every frontend's burst lands on the same datanode. A user
running high-throughput ingestion through a pipeline saw that datanode
overloaded.
Switch to `moka::future::Cache::try_get_with` so concurrent misses on the same
key share one loader. This requires a single-key lookup, so cache entries are
now keyed by the requested schema rather than the schema the pipeline is stored
under; resolving a request to a stored schema stays in the loader, which is the
authoritative path and already handles the empty-schema and multi-schema cases.
A lookup for a schema not yet cached costs one extra read, now protected from
amplification by the coalescing it enables.
`remove_cache` previously only walked the compiled-pipeline cache, so an entry
populated by `get_pipeline_str` alone (the pipeline read API) survived deletion
until it expired. It now walks all three caches.
Also make the TTL configurable as `pipeline.cache_ttl`, default unchanged at
10s. The TTL is what propagates a pipeline change to other frontends, so
raising it trades staleness for fewer reads.
Refs #9021
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): restore cross-schema semantics broken by the new cache key
Keying cache entries by the requested schema dropped two behaviours that the
previous stored-schema key provided for free.
Creating a new version only wrote the creating request's schema, so another
schema on the same frontend kept serving its cached `latest` — an older
version — until the entry expired. Since the whole point of making the TTL
configurable is to let operators raise it, that window is not bounded by
anything useful. Creation now invalidates every schema's `latest` alias for
that name before priming the cache, leaving the version-pinned keys alone.
The failover cache lost its reach across schemas the same way: a global
pipeline (stored under the empty schema) loaded by schema A was cached under
`A`, so schema B using it for the first time while the pipeline table was down
missed and failed ingestion. The failover cache has no loader and so is not
subject to the single-key model of `try_get_with`; it keeps the stored-schema
key and the empty-schema-first resolution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(pipeline): drop cache priming on create and fold the sweep helpers
Priming the cache on create saved one read on a low-frequency operation and
cost a concept: entries were written under the creating request's schema while
`PipelineContent.schema` said empty, so the two schemas in play disagreed.
Invalidating the `latest` aliases is required regardless — that is what makes
a new version visible to other schemas — so dropping the priming loses only
the saved read, which coalescing now protects anyway. `insert_and_compile` no
longer needs the caller's schema.
`remove_cache` and the create-time invalidation collapse into one
`invalidate(name, version)`; `None` sweeps only the `latest` aliases, which is
exactly what creation wants. That leaves `invalidate_by_suffixes` and
`cache_keys` with a single caller each, so both are inlined.
Drop the `PipelineOptions` humantime test: `load_config_test` loads both
example TOMLs, which now carry `cache_ttl = "10s"`, and would fail the same
way if the serde attribute were lost. The `toml` dev-dependency goes with it.
The two invalidation tests are now checked to be orthogonal: removing the
version suffix fails only the delete test, and sweeping just the compiled
cache fails both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): keep failover populated across a create
The `latest` sweep on create clears the failover cache along with the loaded
ones, and after dropping the priming there was nothing writing it back. An
outage between the create and the first read-back left neither `latest` nor the
explicit version with anything to fall back on, failing ingestion — worse than
before, since the previous version's failover entry was swept too.
Creation now goes through `PipelineCache::on_pipeline_created`, which pairs the
sweep with a failover write of the new empty-schema definition. The two must
happen together, so they live behind one method rather than at the call site.
Also commit the Cargo.lock entry for the dropped `toml` dev-dependency, and
trim the comments added over the last few commits down to what the code does
not already say.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(runtime): add weighted workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics
Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to dynamic-config revision
Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): runtime-adjustable workload scheduler parameters
Expose dynamic adjustment of the experimental workload scheduler at
runtime:
- common-runtime: set_workload_scheduler_weights and
set_workload_scheduler_max_concurrent_polls, which forward to the
catio scheduler's set_weight/set_max_concurrent_polls when the
scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
/debug/workload_scheduler/max_concurrent_polls POST handlers, so
operators can rebalance query/write shares or admission concurrency
without restarting the datanode.
Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): add GET /debug/workload_scheduler status endpoint
Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to time-accounting revision
Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to lock-free sampling revision
Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio to scheduler PR head
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): add scheduler bypass control
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: advance catio scheduler fixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: regenerate config docs for workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio scheduler test fix
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(http): satisfy scheduler lifecycle clippy
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: add distributed scheduler toggle coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat: finalize workload scheduler runtime controls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio atomic weights
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: preserve unrelated lockfile resolution
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(runtime): downsample scheduler time accounting
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(runtime): verify cross-runtime scheduler progress
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): configure scheduler poll sampling
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): clarify scheduler activation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): explain scheduler use case
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
* fix(servers): compose OTLP metrics job from service.namespace/service.name
The OTel Prometheus compatibility spec defines job as
"<service.namespace>/<service.name>" when the namespace is present.
The OTLP metrics path only used the bare service.name, so the job tag
diverged from target_info produced by Prometheus-side exporters for the
same resource. Compose the namespace form, and keep not fabricating a
job when service.name is absent.
Behavior change: resources carrying service.namespace now get
"namespace/name" as their job tag value.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion
Ordinary OTLP metrics scatter filtered resource attributes as tags over
every logical metric table, so metrics-only services contribute nothing
to the semantic entity graph. Each request now also projects its
distinct resources into one info-metric-shaped mito table,
otel_resource_info: a fixed allowlist of identity-relevant attributes
under their raw OTel keys (independent of the label translation
strategy and the promote/ignore headers) plus derived job/instance
compatibility columns, value 1.0, and the newest data-point timestamp.
The descriptor is written after the main insert is committed; a failure
there (conflicting pre-existing table, auto-create disabled) degrades
to an OTLP partial_success warning with rejected_data_points = 0
instead of failing the request and triggering client retries of
already-accepted data. A request writing a metric named
otel_resource_info suppresses synthesis. Legacy mode is unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): otel info-metric conventions with host/container entities
Whitelist the ingestion-synthesized otel_resource_info descriptor via a
new otel_info_metrics conventions map, gated on source=opentelemetry
(the existing gate hardcoded source=prometheus). Its declarations use
explicit descriptive lists instead of descriptive_rest so identifying
attributes of other entities do not leak into service.instance.
Conventions tightened per the Astronomy Shop findings: host identity is
host.id with host.name descriptive only (host.name is not stable across
SDKs and resource detectors), a generic container entity (new entity
type) is declared only when container.id is present, and trace-v1
tables now synthesize host/container from their flattened resource
attributes too. New co-declared edges: service.instance runs_on
container, container runs_on host.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): cover the resource descriptor in integration tests
Covers the descriptor's raw-key columns and info-metric options through
the HTTP path, the namespace/name job composition end-to-end, column
names being independent of the translation strategy, the allowlist
excluding unlisted resource attributes, auto-create after a drop, the
metric-name collision suppressing synthesis, and the partial-success
warning (rejected_data_points = 0) when a pre-existing incompatible
table fails the descriptor write while metric data is accepted.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cargo fmt
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(frontend): degrade descriptor permission denial to a warning
A table-level permission policy denying otel_resource_info would have
failed the whole OTLP metrics request because the descriptor's
permission check ran before the main insert. The descriptor is derived
enrichment: check its permission in the degrade path so a denial skips
the write and surfaces as the partial-success warning, like any other
descriptor write failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): guard descriptor writes with semantic ownership markers
A pre-existing schema-compatible table named otel_resource_info would
silently receive descriptor rows while its missing semantic stamps kept
it out of the entity graph. The descriptor write now requires the
auto-created table's ownership markers (mito engine + signal_type +
source + metric.type=info + metadata_quality=declared) and otherwise
degrades to the partial-success warning; the entity-graph gate for the
otel whitelist likewise requires metric.type=info, so a user table
stamped with only signal/source no longer picks up implicit
declarations.
Also fold the descriptor write cost into the response and surface the
degrade warning through the otel-arrow BatchStatus status_message.
Integration tests pin the full marker set on auto-create and that an
existing owned descriptor keeps accepting writes without degrading —
a missing marker would otherwise silently stop every descriptor write
after the first request.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): build descriptor rows without the per-resource BTreeMap
Projecting a resource allocated a BTreeMap and then collected it into the
row key, and every attribute was matched against the allowlist by linear
scan. Collect the tags into a Vec and sort once, and match the allowlist
instead of scanning it. Measured on the conversion path: descriptor work
drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape
(1000 resources with 4 data points each), where the cost tracks resource
count rather than data-point count.
Also trims the comments and tests added with the descriptor to what
carries information.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): pin the descriptor permission-denial degrade path
A policy denying the descriptor table must not fail the metrics request,
which the fix in 2401b3dd9c does but nothing covered. Verified as a
regression guard by mutation: moving the permission check back before
the main write makes this test fail.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): keep legacy mode covered after trimming the unit tests
Trimming the descriptor tests dropped the only assertion that legacy
mode skips the job/instance remap and the promote filter. Both alter
the columns of tables already in use, so fold the check into the legacy
conversion test rather than leaving it uncovered.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): stop encoding column names into composite entity ids
A composite entity id rendered the identifying columns as sorted
`col=value` pairs, so the same identity split into one entity per
signal: a trace table names its columns service_name and
resource_attributes.service.instance.id where a metric table names them
job and instance. One service instance became two nodes with two
parallel edge sets, breaking the walk from a trace to that instance's
metrics.
Render an id as its values in declared order instead, escaping the
separator so components stay distinguishable, which is what single-column
ids already did by keeping only the value. entity_id_attrs still carries
the structured form.
Values alone are not enough for a namespaced service: the metric side
folds service.namespace into job while traces keep the bare name. Add
qualified_by to the conventions so the trace declarations compose the
namespace the same way, per the OTel rule that job is
<service.namespace>/<service.name> or the bare name when the namespace
is empty. A table without the namespace column keeps the unqualified
identity rather than losing the declaration.
Conventions validation now rejects one entity type declared with a
different number of id columns by two sources, which would silently
produce ids that can never match.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(otlp): import the parent module by crate path
check-super-imports.py, part of the CI format gate, rejects a
file-level `use super::`.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): gate the resource descriptor, and fix what review found
Synthesizing greptime_otel_resource_info creates and writes a table the
user never sent, so it is now off unless
otlp.experimental_enable_resource_info says otherwise. With it off the
request costs exactly what it did before the descriptor existed: nothing
is projected, no table is created, no write and no permission check
happen. Tests run with it on. StandaloneOptions carried no otlp field,
so the whole [otlp] section was silently dropped in standalone mode; map
it through, or the new option (and trace_ingest_chunk_size before it)
would do nothing there.
Renamed from otel_resource_info: the greptime_ prefix marks the table as
engine-managed and makes a collision with a user metric unlikely, which
is what the pre-existing-table ownership check and its per-request
catalog lookup were defending against. Both are gone.
A request may carry data for several graph windows, but the descriptor
folded every data-point time into one row at the newest of them, leaving
the earlier windows with metric rows and no entities. Key the rows by
window as well, and take the times from the data points the encoder
actually writes: it drops exponential histograms, and a resource
carrying nothing else was being described as an entity with no
measurements.
Projecting a resource cloned its attributes once per data point. Nest
the windows under the attributes instead, so they are moved once per
resource, and walk the data-point times through a visitor rather than
collecting a Vec per metric.
Also documents what the two maps key and hold, and lifts the projected
attribute names to constants beside KEY_SERVICE_NAME.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): skip the descriptor's work entirely when it is disabled
The collision scan over the request's output tables ran even with the
feature off. Short-circuit on the option instead, and update the config
snapshot the new [otlp] section changed.
Also drops the doc comment orphaned by the deleted ownership check: it
had attached itself to the trait impl and described a check that no
longer exists.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): drop the expect and name the service identity
The CASE is built through Case directly rather than the fallible
when().otherwise() builder, so the non-test path no longer carries an
expect (architecture-invariants $4).
service_identity returned two same-typed Options that both call sites
destructured positionally; a named struct makes a swap fail to compile.
Also records that id-column order is part of the identity, where the
option docs and the conventions authors will read it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(semantic-graph): drop comments that narrate the code
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): cast duration_nano before the trace-table union
Trace tables written before the signed-integer ingest change hold
duration_nano as UInt64 and later ones as Int64. The calls derivation
unions the per-table selects, and the two have no common integer type,
so a deployment holding both shapes could not build the plan. The
cross-table test now spans both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): drop the redundant duration_nano casts
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): decide exponential histogram acceptance in one place
The resource descriptor mirrored only the experimental gate, so with both
experimental flags on a resource whose only metric is a delta exponential
histogram was described as an entity with no measurements. The encoder's
whole-metric rules move into exponential_histogram_gate, which both call,
and the descriptor takes its timestamps through exponential_histogram_value
so per-point rejections drop out too.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: cap default runtime sizes to a minimum of 2 threads
RuntimeOptions derived its default sizes directly from num_cpus. On
single-core machines every runtime (global, compact, query, ingest)
ended up with one worker thread, which can easily deadlock async code
(e.g. block_on combined with spawn).
Clamp all CPU-derived runtime sizes to at least 2 threads.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: init logging before runtimes so runtime options are logged
The global runtimes were initialized before the global logging
subscriber, so the "Creating runtime ..." info logs that carry the
runtime sizes were silently dropped. Initialize logging first in all
node start paths; common-telemetry has no dependency on
common-runtime, so the reorder is safe.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
The example TOMLs and generated config.md documented the default of
wal.sync_period as "10s", but since #5677 moved the WAL sync task to a
background RepeatedTask, an unset sync_period falls back to 5s in
RaftEngineLogStore. The two paths therefore had different fsync
periods: deployments based on the example configs used 10s while bare
configs used 5s.
Align the documentation with the actual code behavior (5s) instead of
changing the code fallback to 10s, so that no existing deployment
silently gets a larger data-loss window on host power loss.
- config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s
- config/config.md: regenerated via make config-docs
- src/cmd/tests/load_config_test.rs: update assertions accordingly
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: adjust the position of experimental_enable_prometheus_native_histogram
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: move prom_validation_mode as well
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add a dedicated http api server port
* fix: integration test
* refactor: make http-api-port opt-in
* refactor: rename attribute to http-api-server
* feat: use middleware to check different http server port
* refactor: rename config option
* fix(prometheus): make remote write timeout retryable
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(prometheus): enforce pending row timeout budget
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(prometheus): skip pending-row timeout fallback when batcher is disabled
PendingRowsBatcher::try_new returns None when max_batch_rows,
max_concurrent_flushes, worker_channel_capacity or max_inflight_requests
is zero, meaning remote writes bypass batching entirely. The timeout
fallback predicate now mirrors these enablement conditions so the HTTP
timeout is not raised when no request can wait for a pending-row flush.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(prometheus): skip pending-row timeout fallback in async batch mode
With PENDING_ROWS_BATCH_SYNC=false, pending-row submissions return right
after enqueue and no request waits for a flush, so raising the global
HTTP timeout only delays unrelated routes. Export the batch sync mode
predicate from the servers crate and consult it in the frontend's
effective_http_options so the fallback is skipped in asynchronous mode.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Yingwen
Dragon RoarandDennis Zhuang
jeremyhi
shuiyisong
Weny Xu
Lei, HUANG
Dhruv Vaishnav
discord9
Whis Liao
Ning Sun