From 3b8f55e490656d0d44ed0a52f521adef73691999 Mon Sep 17 00:00:00 2001 From: dennis zhuang Date: Wed, 24 Jun 2026 17:14:09 +0800 Subject: [PATCH] docs(agents): add per-crate guides, architecture invariants, and generated-files list (#8346) * docs(agents): add per-crate guides, architecture invariants, and generated-files list Add agent/contributor navigation docs modeled on the AGENTS.md convention: - Per-crate AGENTS.md for hot crates (mito2, metric-engine, flow, frontend, meta-srv): module map, read/write paths, change-coupling points, test commands, and gotchas. - .agents/architecture-invariants.md: repo-wide rules that clippy and the style guide do not cover (format compatibility, crate layering, async runtimes, error handling, experimental gating, the DataFusion fork). - .agents/generated-files.md: tool-generated artifacts that must not be hand-edited (sqlness .result, config.md, dashboards, build.rs output, proto). - Anchor the .gitignore CLAUDE.md/AGENTS.md rules to the repo root so per-crate AGENTS.md files are tracked while root-level personal config stays ignored. Signed-off-by: Dennis Zhuang * chore: update crate AGENTS.md and fix config.md path Signed-off-by: Dennis Zhuang * docs(agents): fix DataFusion patch layout and SQL query lifecycle order Address review feedback on #8346: - architecture-invariants: the DataFusion sub-crates pin an exact crates.io version in [workspace.dependencies] and are redirected to the fork rev in [patch.crates-io]; the two sections hold different forms, not the same rev. - frontend: the SQL query lifecycle runs the pre_parsing/post_parsing interceptors around parsing, before the per-statement permission check. Signed-off-by: Dennis Zhuang --------- Signed-off-by: Dennis Zhuang --- .agents/README.md | 28 +++++++- .agents/architecture-invariants.md | 111 +++++++++++++++++++++++++++++ .agents/generated-files.md | 60 ++++++++++++++++ .gitignore | 5 +- src/flow/AGENTS.md | 90 +++++++++++++++++++++++ src/frontend/AGENTS.md | 83 +++++++++++++++++++++ src/meta-srv/AGENTS.md | 90 +++++++++++++++++++++++ src/metric-engine/AGENTS.md | 85 ++++++++++++++++++++++ src/mito2/AGENTS.md | 88 +++++++++++++++++++++++ 9 files changed, 637 insertions(+), 3 deletions(-) create mode 100644 .agents/architecture-invariants.md create mode 100644 .agents/generated-files.md create mode 100644 src/flow/AGENTS.md create mode 100644 src/frontend/AGENTS.md create mode 100644 src/meta-srv/AGENTS.md create mode 100644 src/metric-engine/AGENTS.md create mode 100644 src/mito2/AGENTS.md diff --git a/.agents/README.md b/.agents/README.md index 605831f1b0..d55aac50f0 100644 --- a/.agents/README.md +++ b/.agents/README.md @@ -1,4 +1,8 @@ -# Agent Skills +# Agent Resources + +Shared resources for coding agents (Codex, Claude Code, ...) and contributors. + +## Skills Shared agent skills live in `.agents/skills`. @@ -6,3 +10,25 @@ Shared agent skills live in `.agents/skills`. - Claude Code reads the same skills through `.claude/skills`, which is a symlink to this directory. - Add or update shared skills under `.agents/skills//SKILL.md`. - If a new skill does not appear, restart the agent or start a new thread. + +## Per-crate guides + +Hot crates carry an `AGENTS.md` next to their code as a navigation aid (module +map, read/write paths, change-coupling points, test commands, gotchas): + +- [`src/mito2/AGENTS.md`](../src/mito2/AGENTS.md) — primary time-series storage engine +- [`src/metric-engine/AGENTS.md`](../src/metric-engine/AGENTS.md) — metrics engine (logical/physical regions) +- [`src/flow/AGENTS.md`](../src/flow/AGENTS.md) — stream processing / continuous aggregation +- [`src/frontend/AGENTS.md`](../src/frontend/AGENTS.md) — request entry point and orchestration +- [`src/meta-srv/AGENTS.md`](../src/meta-srv/AGENTS.md) — metadata and cluster coordination + +## Architecture invariants + +[`architecture-invariants.md`](architecture-invariants.md) lists repo-wide rules +that are easy to violate and expensive to get wrong (format compatibility, crate +layering, async runtimes, error handling, feature gating, the DataFusion fork). + +## Generated files + +[`generated-files.md`](generated-files.md) lists tool-generated artifacts that +must not be hand-edited (sqlness `.result`, `config.md`, dashboards, ...). diff --git a/.agents/architecture-invariants.md b/.agents/architecture-invariants.md new file mode 100644 index 0000000000..60c73a26e2 --- /dev/null +++ b/.agents/architecture-invariants.md @@ -0,0 +1,111 @@ +# Architecture Invariants + +Repo-wide rules that are easy to violate and expensive to get wrong. They are +**not** general best practices — each one is specific to GreptimeDB, has high +blast radius, and is not caught by `cargo clippy`. + +This complements the other docs rather than repeating them: + +- `docs/style-guide.md` — micro-level code style (formatting, modules, comments). +- `docs/rfcs/` — per-feature architecture decisions. +- `CONTRIBUTING.md` — how to build, test, and submit. + +Paths below are relative to the repo root. + +## 1. Persisted and wire formats must stay backward/forward compatible + +Anything written to disk or sent over the wire outlives the process that wrote +it: region manifests, WAL entries, SST/Parquet files and their metadata, +metadata KV values (`common-meta` keys, metric-engine metadata), and gRPC +messages. A node running an old version may read data written by a new one and +vice versa. + +- Add fields, don't repurpose or reorder them. For serde types use + `#[serde(default)]` / `#[serde(alias)]`; never change the meaning of an existing + field or the discriminant of an existing enum variant. +- Monotonic version counters (e.g. the mito2 manifest version) only ever move + forward — never reset or skip. +- When a change touches a persisted or wire format, add a case to the + compatibility test suite. See `docs/rfcs/2025-07-04-compatibility-test-framework.md` + — compatibility has been broken on releases before (v0.14.1, v0.15.1) precisely + because this step was skipped. +- Wire types are generated from the external `greptime-proto` crate; change the + format there first, then bump the dependency (see invariant 6's pattern). + +## 2. Respect crate layering and dependency direction + +The workspace is layered; dependencies point downward only. + +- `common-*` crates are the base. They must not depend on storage engines, + `frontend`, `datanode`, or `meta-srv`. +- `store-api` defines the engine contract (e.g. `RegionEngine` in + `src/store-api/src/region_engine.rs`). Engines (`mito2`, `metric-engine`, + `file-engine`) implement it; `datanode` drives engines **through the trait**, + not through engine internals. +- `frontend` reaches storage through `operator` / `query` / `catalog`, not by + depending on `datanode` internals. Standalone mode is the one bridge, via a + `RegionServer` wrapper (`src/frontend/src/instance/standalone.rs`). +- Do not introduce circular dependencies. New deps go through + `[workspace.dependencies]` in the root `Cargo.toml`, not per-crate version + literals. + +## 3. Use the shared async runtimes; never block them + +Runtimes are partitioned by workload so one workload can't starve another. They +live in `common-runtime` (`src/common/runtime/`). + +- Use the categorized spawns — `spawn_global`, `spawn_query`, `spawn_ingest`, + `spawn_compact`, `spawn_hb` — instead of constructing your own tokio runtime, + and pick the category that matches the work. +- Run CPU-bound or synchronous-blocking work via `spawn_blocking_*`; never do + heavy CPU or blocking syscalls directly inside an async task. +- Never call `block_on*` from inside an async context or an engine worker — it + deadlocks the runtime. + +## 4. Errors: snafu + `ErrorExt`, no panics in non-test code + +Each crate defines its own snafu `Error` enum and implements `ErrorExt` +(`src/common/error/src/ext.rs`). + +- Set a meaningful `status_code()`. It drives the client-visible result and + whether the message is masked (`Internal`/`Unknown` are masked from end users). +- Mark errors the caller may retry with the appropriate `retry_hint()` + (`Retryable`). Default is non-retryable. +- In non-test code, return errors instead of `unwrap()` / `expect()` / `panic!()`. + Use `unimplemented!()` (not `todo!()`) for paths that won't be implemented, per + `docs/style-guide.md`, which also covers `with_context` vs `context`. + +## 5. Gate unstable features behind `experimental_` config + +Features whose behavior or surface may still change ship behind config keys +prefixed `experimental_` (see existing examples in `config/datanode.example.toml`, +`config/flownode.example.toml`, `config/standalone.example.toml`). Some can be +overridden per-object (e.g. a flow's `WITH (experimental_... = '...')`). This lets +unfinished work merge without freezing it into the stable config surface. + +When you stabilize such a feature, drop the prefix and document the migration. + +## 6. DataFusion is a pinned fork — two sections, two forms + +GreptimeDB uses a fork at `GreptimeTeam/datafusion`, wired up in the root +`Cargo.toml` through **two sections that hold different things**: + +- `[workspace.dependencies]` pins each DataFusion sub-crate to an **exact + crates.io version** (e.g. `datafusion = "=53.1.0"`). +- `[patch.crates-io]` redirects those same crates to the **fork at a git rev** + (e.g. `datafusion = { git = ".../GreptimeTeam/datafusion.git", rev = "..." }`). + +So: + +- Adding a new DataFusion sub-crate dependency means adding it to **both** + sections — the `=` entry under `[workspace.dependencies]` and the + matching git-rev patch under `[patch.crates-io]`. +- Upgrading DataFusion means bumping the version in `[workspace.dependencies]` + **and** the rev in `[patch.crates-io]` together, for all of them. + +## Maintenance contract + +Update this file when a new repo-wide invariant emerges (a new persisted format, +a new runtime category, a layering rule), or when an existing one changes. Keep +each entry high-signal: if `clippy` or `docs/style-guide.md` already enforces it, +it does not belong here. diff --git a/.agents/generated-files.md b/.agents/generated-files.md new file mode 100644 index 0000000000..1a16fcb28c --- /dev/null +++ b/.agents/generated-files.md @@ -0,0 +1,60 @@ +# Generated Files — Do Not Hand-Edit + +These artifacts are produced by tooling. Editing them by hand is almost always +wrong: your change is either overwritten on regeneration or causes CI to fail. +When the underlying source changes, regenerate them with the listed command. + +## sqlness `.result` files — `tests/cases/**/*.result` + +Generated by the sqlness runner from the sibling `.sql` files. **Never edit a +`.result` by hand.** To update expectations, change the `.sql` (or the engine +behavior) and re-run the test; the runner rewrites the `.result`. + +```bash +cargo sqlness bare # run all local cases +cargo sqlness bare -t # run one case +``` + +Review the regenerated output with `git diff tests`. + +Note: `tests/cases/distributed/common` is a **symlink** to +`tests/cases/standalone/common`. They are the same files — editing one changes +both, and there is no need to diff standalone vs distributed. + +## Configuration docs — `config/config.md` + +Generated from the `config/*.example.toml` files and +`config/config-docs-template.md`. + +```bash +make config-docs +``` + +Edit the example TOMLs and/or the template, then regenerate. Do not edit +`config/config.md` directly. + +## Grafana dashboards + +Generated artifacts; regenerate rather than editing by hand. + +```bash +make dashboards +``` + +## `build.rs`-generated code + +Some crates generate code at build time into the Cargo `OUT_DIR` (e.g. +`common-version` for build/git info, plus `common-catalog`, `common-function`'s +system tables, `log-store`, `servers`). There is no checked-in file to edit — +change the corresponding `build.rs` or its inputs instead. + +## Protobuf / gRPC types + +Generated from the external `greptime-proto` crate, not from sources in this +repo. To change the wire format, change `greptime-proto` upstream and bump the +dependency. + +## License headers + +Managed by the license-header tooling and the pre-commit hooks; let the tooling +add/fix them rather than editing headers by hand. diff --git a/.gitignore b/.gitignore index 1c3dfd626a..4cd336b0e9 100644 --- a/.gitignore +++ b/.gitignore @@ -66,8 +66,9 @@ greptimedb_data !/.github # AI related -CLAUDE.md -AGENTS.md +# Root-level personal/local agent config only; per-crate AGENTS.md files are tracked. +/CLAUDE.md +/AGENTS.md .codex .gemini .opencode diff --git a/src/flow/AGENTS.md b/src/flow/AGENTS.md new file mode 100644 index 0000000000..aba019e02a --- /dev/null +++ b/src/flow/AGENTS.md @@ -0,0 +1,90 @@ +# flow — Agent & Contributor Guide + +Navigation aid for `src/flow`. Keep it short and point to code. Paths are +relative to the repo root. + +Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +## What this crate does + +Flownode is the stream-processing engine behind continuous aggregation / +materialized views. It runs in two modes: + +- **Batching mode** (the default and the actively developed path): splits data + into time windows and periodically runs aggregation SQL through the frontend, + writing results back to a sink table. +- **Streaming mode** (the older dataflow engine): an incremental DFIR/dataflow + compute graph that processes row-level diffs. + +A flow without an explicit `flow_type` is created as **batching**. + +## Module map + +| Module | Path | Purpose | +| --- | --- | --- | +| `engine` | `src/flow/src/engine.rs` | `FlowEngine` trait: create/remove/flush/insert lifecycle | +| `adapter` | `src/flow/src/adapter.rs`, `src/flow/src/adapter/` | `StreamingEngine`, worker pool, dual-engine dispatch, table sources/sinks | +| `batching_mode` | `src/flow/src/batching_mode.rs`, `src/flow/src/batching_mode/` | `BatchingEngine`, task scheduling, time windows, frontend client, checkpoints | +| `compute` | `src/flow/src/compute/` | Streaming dataflow render/state | +| `expr` | `src/flow/src/expr.rs`, `src/flow/src/expr/` | Scalar/aggregate expressions and Map-Filter-Project | +| `plan` | `src/flow/src/plan.rs` | `TypedPlan` (reduce/join/MFP) | +| `transform` | `src/flow/src/transform.rs` | Substrait → flow plan | +| `df_optimizer` | `src/flow/src/df_optimizer.rs` | SQL → DataFusion logical plan → optimized plan | +| `repr` | `src/flow/src/repr.rs` | `Row`, `DiffRow`, `Batch`, `RelationDesc` | +| `server` | `src/flow/src/server.rs` | gRPC `Flow` service, `FlownodeBuilder`/`FlownodeInstance` | +| `heartbeat` | `src/flow/src/heartbeat.rs` | Reports flownode state/stats to metasrv | + +Flow metadata lives in `common-meta`, not here: +`src/common/meta/src/key/flow/` and `src/common/meta/src/ddl/create_flow.rs`. + +## Data flow + +`Frontend → Flownode (gRPC)` → `FlowService` (`server.rs`) → +`FlowDualEngine` (`adapter/flownode_impl.rs`) routes by `FlowType`: + +- Batching: marks dirty windows; a task later runs aggregation SQL via the + frontend client and writes the sink table (`batching_mode/task.rs`). +- Streaming: worker threads apply incremental diffs and push to the sink + (`adapter/worker.rs`, `compute/render.rs`). + +## Public surface + +- gRPC `Flow` service in `src/flow/src/server.rs` + (`handle_create_remove`, `handle_mirror_request`, `handle_mark_dirty_time_window`). +- Sink writes for the streaming engine go through `FrontendInvoker` + (`row_inserts`, `row_deletes`). +- `FlowEngine` trait in `src/flow/src/engine.rs`. +- Started from the `cmd` crate via `FlownodeBuilder` / `FlownodeInstance`. + +## When you change X, also touch Y + +- **Flow definition / options**: validation in `common-meta`'s `ddl/create_flow.rs` + and the serialized `FlowInfoValue` in `common-meta`'s `key/flow/`. +- **New scalar/aggregate function** (`expr/`): also wire up evaluation in + `compute/render.rs` (streaming) and ensure batching SQL handles it. +- **Persisted flow metadata**: keep `FlowInfoValue` backward compatible + (`serde(default)` / `serde(alias)`). +- A change to one engine often needs the mirror change in the other. + +## Testing + +```bash +cargo nextest run -p flow +``` + +Helpers in `src/flow/src/test_utils.rs` (test context, test query engine). + +## Gotchas + +- Batching vs streaming differ a lot in latency, debuggability, and code path — + confirm which mode a flow uses before reasoning about it. Batching is the + default and the primary target. +- Streaming workers are `!Send`; cross-thread interaction goes through + `WorkerHandle`, not the worker directly. +- Internal flow timestamps (`repr::Timestamp`, ms) are not necessarily the + table's time column; window functions key off the diff timestamp. + +## Maintenance contract + +Update this file when the dual-engine routing, the gRPC surface, or the flow +metadata contract (shared with `common-meta`) changes. diff --git a/src/frontend/AGENTS.md b/src/frontend/AGENTS.md new file mode 100644 index 0000000000..2321b65732 --- /dev/null +++ b/src/frontend/AGENTS.md @@ -0,0 +1,83 @@ +# frontend — Agent & Contributor Guide + +Navigation aid for `src/frontend`. Keep it short and point to code. Paths are +relative to the repo root. + +Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +## What this crate does + +Frontend is the request entry point and orchestration layer. It accepts +multi-protocol requests (gRPC, HTTP, MySQL, PostgreSQL, InfluxDB, OTLP, Jaeger, +Prometheus, OpenTSDB), checks permissions, parses/plans SQL, and dispatches: +reads go to the query engine (`query` crate), writes go to the inserter/deleter +(`operator` crate). + +Boundary with `servers`: the `servers` crate implements the wire protocols and +network I/O; `frontend` provides the business logic by implementing handler +traits (`SqlQueryHandler`, `GrpcQueryHandler`, `InfluxdbLineProtocolHandler`, +...). In standalone mode the frontend embeds a datanode `RegionServer`; in +distributed mode it talks to remote datanodes via `operator`/`client`. + +## Module map + +| Module | Path | Purpose | +| --- | --- | --- | +| `instance` | `src/frontend/src/instance.rs` | `Instance`: the core handler; implements `SqlQueryHandler`, `PrometheusHandler`, etc. | +| `instance/builder` | `src/frontend/src/instance/builder.rs` | `FrontendBuilder` assembles `Instance` from its dependencies | +| `instance/grpc` | `src/frontend/src/instance/grpc.rs` | `GrpcQueryHandler`: insert/delete/query/promql over gRPC | +| `instance/standalone` | `src/frontend/src/instance/standalone.rs` | Calls the local `RegionServer` instead of RPC | +| `instance/region_query` | `src/frontend/src/instance/region_query.rs` | Routes distributed region reads to datanodes | +| `instance/*` | `src/frontend/src/instance/` | Per-protocol handlers (`influxdb.rs`, `promql.rs`, `otlp/`, `jaeger.rs`, `logs.rs`, `prom_store.rs`, ...) | +| `frontend` | `src/frontend/src/frontend.rs` | `Frontend` lifecycle wrapper (`FrontendOptions`, start/shutdown) | +| `server` | `src/frontend/src/server.rs` | `Services`: builds and wires the protocol servers | +| `heartbeat` | `src/frontend/src/heartbeat.rs` | Heartbeat to metasrv; handles suspend / cache invalidation | +| `service_config` | `src/frontend/src/service_config/` | Per-protocol option structs | + +## Request lifecycles + +- **SQL query** (`instance.rs`): `do_query` → `pre_parsing` interceptor → parse → + `post_parsing` interceptor → then per statement: `check_permission` → + `statement_executor.plan` (logical plan) → `query_engine.execute` → for + distributed reads, `region_query.rs` fetches from datanodes → cancellable + `RecordBatch` stream. Interceptors run around parsing, before the per-statement + permission check — preserve that ordering. +- **Insert** (`instance/grpc.rs`): `handle_inserts` / `handle_row_inserts` → + `check_permission` → `operator`'s `Inserter` (schema validation, optional + auto-create, partition routing) → local `RegionServer` (standalone) or RPC to + datanodes (distributed). + +## Public surface + +- `Instance` (`instance.rs`) — the business-logic container. +- `Frontend` (`frontend.rs`) — lifecycle wrapper around `Instance` + servers + heartbeat. +- Created from `cmd`: `src/cmd/src/frontend.rs` (distributed) and + `src/cmd/src/standalone.rs` (standalone, with embedded datanode). + +## When you change X, also touch Y + +- **`servers` handler traits**: a new/changed protocol handler requires the + matching `impl` here. +- **`operator` Inserter/Deleter or `query` QueryEngine API**: update the call + sites in `instance.rs` / `instance/grpc.rs`. +- **`session::QueryContext`**: new context fields thread through most handlers. +- **`sql` statements**: new statement kinds need handling in `query_statement`. + +## Testing + +```bash +cargo nextest run -p frontend +``` + +## Gotchas + +- Keep the frontend/servers split straight: wire format and network live in + `servers`; permissions, planning, and routing live here. +- Standalone vs distributed diverge in datanode access (local `RegionServer` vs + `NodeClients` RPC), MetaClient usage, and whether heartbeat matters. In + standalone, the cache invalidator is a no-op. + +## Maintenance contract + +Update this file when you add a protocol handler, change the query/insert +lifecycle, or change how `Instance` is constructed or wired to `servers`. diff --git a/src/meta-srv/AGENTS.md b/src/meta-srv/AGENTS.md new file mode 100644 index 0000000000..e1e2358152 --- /dev/null +++ b/src/meta-srv/AGENTS.md @@ -0,0 +1,90 @@ +# meta-srv — Agent & Contributor Guide + +Navigation aid for `src/meta-srv`. Keep it short and point to code. Paths are +relative to the repo root. + +Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +## What this crate does + +Metasrv is the metadata and coordination service for distributed mode: it +persists cluster/table/region metadata, runs leader election, drives the +heartbeat loop that tracks cluster topology, and orchestrates distributed +procedures (DDL, region migration, repartition). Data models, the KV backend +abstraction, election, key encoding, and the DDL manager live in `common-meta`; +`meta-srv` implements the server, state machine, and control logic on top. + +## Module map + +| Module | Path | Purpose | +| --- | --- | --- | +| `bootstrap` | `src/meta-srv/src/bootstrap.rs` | Wires up KV backend, election, gRPC services, procedure manager | +| `metasrv` | `src/meta-srv/src/metasrv.rs`, `src/meta-srv/src/metasrv/builder.rs` | `Metasrv` struct, `MetasrvOptions`, `MetasrvBuilder`, startup | +| `state` | `src/meta-srv/src/state.rs` | Leader/Follower state machine and transitions | +| `handler` | `src/meta-srv/src/handler/` | Heartbeat handler chain (`HeartbeatHandler`): leader check, region lease, stats, mailbox | +| `service` | `src/meta-srv/src/service/` | gRPC services (`heartbeat`, `procedure`, `cluster`, `store`) and HTTP admin | +| `procedure` | `src/meta-srv/src/procedure/` | Distributed procedures: `region_migration/`, `repartition.rs`, `wal_prune/` | +| `region` | `src/meta-srv/src/region/` | Region supervisor, lease keeper, failover triggers | +| `discovery` | `src/meta-srv/src/discovery/` | Node discovery and lease-backed node info | +| `pubsub` | `src/meta-srv/src/pubsub/` | Heartbeat topic publish/subscribe support | +| `gc` | `src/meta-srv/src/gc/` | Metadata-driven garbage-collection procedures and scheduling | +| `selector` | `src/meta-srv/src/selector/` | Region placement strategies (round-robin / load-based / lease-based) | +| `peer` | `src/meta-srv/src/peer.rs` | Peer allocation through selectors | +| `cluster` | `src/meta-srv/src/cluster.rs` | `MetaPeerClient`: internal RPC with leader fallback | +| `cache_invalidator` | `src/meta-srv/src/cache_invalidator.rs` | Pushes cache-invalidation to frontends/datanodes | +| `key` | `src/meta-srv/src/key/` | Metasrv-side KV key encoding | + +## Core flows + +- **Heartbeat**: `service/heartbeat.rs` receives datanode/frontend heartbeats and + runs the handler chain in `handler/`; responses carry mailbox messages (DDL + results, cache invalidation). +- **DDL**: `service/procedure.rs` (`ddl`) is leader-only and hands off to + `common-meta`'s `DdlManager`, executed via `common-procedure`. +- **Region migration**: `procedure/region_migration/manager.rs` plus per-step + files; triggered by failover or an explicit command. +- **Leader election**: via `common-meta`'s election; transitions in `state.rs`. + +## Public surface + +Clients use the `meta-client` crate (`MetaClient`). On the server side the entry +points are the gRPC services under `src/meta-srv/src/service/` and the HTTP admin +API under `src/meta-srv/src/service/admin/`. + +## When you change X, also touch Y + +- **Heartbeat interval / leases**: timing constants live in `common-meta` + (`distributed_time_constants`); changing them affects lease and failover timing + in `handler/region_lease_handler.rs` and `region/supervisor.rs`. +- **Procedures**: state must stay persisted in the KV backend and `execute()` + must be idempotent for crash recovery. +- **Cache invalidation**: new metadata types need matching invalidation in + `cache_invalidator.rs` and the heartbeat publish handler. +- **Selector stats fields**: keep `selector/` implementations in sync with the + stats they consume. + +## Testing + +```bash +cargo nextest run -p meta-srv +# mock election / KV backend: +cargo nextest run -p meta-srv --features mock +``` + +Helpers: `src/meta-srv/src/mocks.rs`, `src/meta-srv/src/test_util.rs`, +`src/meta-srv/src/procedure/test_util.rs`. + +## Gotchas + +- Most mutating operations are leader-only — guard them (see the leader check in + `handler/` and `service/procedure.rs`). Non-leaders return "not leader". +- In-memory state is not durable; anything needed after a leader change must be + persisted to the KV backend. The leader uses a cached KV backend that resets on + leader change. +- Region lease renewal timing and the supervisor's check interval must be + coordinated, or a region can appear active on two datanodes. + +## Maintenance contract + +Update this file when you add a procedure, change the heartbeat handler chain, +change leader-only boundaries, or alter the metasrv ↔ common-meta split. diff --git a/src/metric-engine/AGENTS.md b/src/metric-engine/AGENTS.md new file mode 100644 index 0000000000..86b676db48 --- /dev/null +++ b/src/metric-engine/AGENTS.md @@ -0,0 +1,85 @@ +# metric-engine — Agent & Contributor Guide + +Navigation aid for `src/metric-engine`. Keep it short and point to code. Paths +are relative to the repo root. + +Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +## What this crate does + +The Metric Engine is optimized for Prometheus-style workloads with a huge number +of small tables. Many **logical** regions (one per metric table) share a single +**physical** pair of Mito2 regions: a data region and a metadata region. Rows +are multiplexed with metric-engine internal identity: dense primary-key mode +injects `__table_id` and `__tsid`, while the default sparse mode encodes them +into `__primary_key`. Reads still add a logical-table filter before forwarding +to the physical data region. It implements `RegionEngine` and delegates all real +storage to `mito2`. + +The architecture is documented at the top of `src/metric-engine/src/lib.rs`. + +## Module map + +| Module | Path | Purpose | +| --- | --- | --- | +| `engine` | `src/metric-engine/src/engine.rs`, `src/metric-engine/src/engine/` | `MetricEngine` (`RegionEngine` impl) and per-op handlers (`create.rs`, `put.rs`, `read.rs`, `alter.rs`, `drop.rs`, ...) | +| `metadata_region` | `src/metric-engine/src/metadata_region.rs` | K-V over a Mito2 region storing logical table/column metadata, with an LRU cache | +| `data_region` | `src/metric-engine/src/data_region.rs` | Wraps the Mito2 data region; forwards writes and manages physical columns | +| `row_modifier` | `src/metric-engine/src/row_modifier.rs` | Rewrites incoming rows for dense or sparse primary-key encoding | +| `batch_modifier` | `src/metric-engine/src/batch_modifier.rs` | RecordBatch-level TSID computation and sparse primary-key encoding | +| `state` | `src/metric-engine/src/engine/state.rs` | In-memory cache of physical columns and logical column metadata | +| `repeated_task` | `src/metric-engine/src/repeated_task.rs` | Periodic metadata-region flush task | +| `utils` | `src/metric-engine/src/utils.rs` | `RegionId` conversions (data vs metadata group), manifest encoding | +| `config` | `src/metric-engine/src/config.rs` | `EngineConfig` (metadata flush interval, sparse PK) | +| `test_util` | `src/metric-engine/src/test_util.rs` | `TestEnv` building the Mito2 + Metric stack | + +## Write path + +`MetricEngine::handle_request(Put)` (`engine/put.rs`) rejects direct writes to a +physical region, resolves the physical region for the logical id, loads logical +columns from `metadata_region`, then `row_modifier` rewrites the rows according +to the data region's primary-key encoding and forwards the request to the Mito2 +data region. + +## Read path + +`MetricEngine::handle_query` (`engine/read.rs`): a query against a logical region +is rewritten to add a `__table_id == ` filter and forwarded to the +Mito2 data region. Queries against a physical region pass straight through. + +## Public surface + +- Entry: `MetricEngine` in `src/metric-engine/src/engine.rs`, built via `try_new(mito, config)`. +- Trait: `impl RegionEngine for MetricEngine` (name = `"metric"`). +- Depends on `mito2`, `store-api`, and `mito-codec` (sparse primary key codec). + +## When you change X, also touch Y + +- **Reserved column ids / names** (`__tsid`, `__table_id`, `__primary_key`): see + `store-api`'s metric engine consts; keep them in sync with `engine.rs`. +- **Metadata K-V encoding** (`metadata_region.rs`): changes the on-disk metadata layout. +- **RegionId group mapping** (`utils.rs`): data vs metadata region derivation. +- **Physical column rules** (`engine/alter/`): a physical region allows only one field column. + +## Testing + +```bash +cargo nextest run -p metric-engine +``` + +`TestEnv` in `test_util.rs` gives you `mito()` and `metric()` handles. + +## Gotchas + +- Physical vs logical region confusion: physical regions reject direct user + writes; operate on logical region ids. +- TSID must be stable for the same tag set — it is a hash over sorted tag names + + values and may be stored in `__tsid` or encoded into `__primary_key`. +- Metadata is cached (LRU with a TTL); after an alter, stale reads are possible + until invalidation/expiry. +- Always convert ids via `utils::to_data_region_id` / `to_metadata_region_id`. + +## Maintenance contract + +Update this file when you change the logical/physical region model, the injected +columns, the metadata encoding, or the public engine entry points. diff --git a/src/mito2/AGENTS.md b/src/mito2/AGENTS.md new file mode 100644 index 0000000000..c25ede60ed --- /dev/null +++ b/src/mito2/AGENTS.md @@ -0,0 +1,88 @@ +# mito2 — Agent & Contributor Guide + +Navigation aid for `src/mito2`. Keep it short and point to code; do not duplicate +the code here. Paths are relative to the repo root. + +Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +## What this crate does + +Mito2 is GreptimeDB's primary time-series region storage engine. It owns the +write path (memtable + WAL), flushing memtables to Parquet SST files, +TWCS/windowed compaction, and the read path (multi-level merge + dedup with +snapshot isolation). It implements the `RegionEngine` trait from `store-api`. + +## Module map + +| Module | Path | Purpose | +| --- | --- | --- | +| `engine` | `src/mito2/src/engine.rs` | `MitoEngine` (the `RegionEngine` impl) and request dispatch | +| `worker` | `src/mito2/src/worker.rs`, `src/mito2/src/worker/` | Per-region worker loop; write/alter/flush handlers | +| `region` | `src/mito2/src/region.rs`, `src/mito2/src/region/version.rs` | `MitoRegion` state and copy-on-write `VersionControl` snapshots | +| `request` | `src/mito2/src/request.rs` | `WriteRequest`/`RegionRequest` types and result channels | +| `wal` | `src/mito2/src/wal.rs` | Write-ahead log wrapper over `log-store` | +| `memtable` | `src/mito2/src/memtable/` | In-memory write buffers (time-series / bulk / partition) | +| `flush` | `src/mito2/src/flush.rs` | `FlushScheduler`, `WriteBufferManager`, memtable → SST | +| `compaction` | `src/mito2/src/compaction/` | TWCS picker, strict-window manual picker, compactor, memory control | +| `access_layer` | `src/mito2/src/access_layer.rs` | SST read/write over the object store | +| `sst` | `src/mito2/src/sst/` | Parquet format, file metadata, index layout | +| `read` | `src/mito2/src/read/` | `ScanRegion`, merge, dedup, projection, streaming | +| `manifest` | `src/mito2/src/manifest/` | `RegionManifestManager`, manifest actions/edits | +| `cache` | `src/mito2/src/cache.rs` | Write/file/page caches | +| `gc` | `src/mito2/src/gc.rs`, `src/mito2/src/gc/` | Dropped-file cleanup worker | +| `schedule` | `src/mito2/src/schedule/` | Local/remote background job scheduling | +| `remap_manifest` | `src/mito2/src/remap_manifest.rs` | Manifest path remapping for region copy/migration | +| `config` | `src/mito2/src/config.rs` | `MitoConfig` tuning knobs | +| `test_util` | `src/mito2/src/test_util.rs` | `TestEnv` and builders (under the `test` feature) | + +## Write path + +`MitoEngine::handle_request` (`engine.rs`) → worker loop +(`worker/handle_write.rs`) → sequence + WAL assembly (`region_write_ctx.rs`) → +`wal.rs` → memtable (`memtable/`) → when buffer pressure trips, `flush.rs` +writes SSTs via `access_layer.rs` and appends a `RegionEdit` to the manifest +(`manifest/manager.rs`). + +## Read path + +`MitoEngine::handle_query` (`engine.rs`) → `read/scan_region.rs` takes an +immutable `Version` (`region/version.rs`) → scans memtables and Parquet SSTs +(`sst/parquet.rs`) → merges (`read/`) and dedups by sequence → projected, +filtered `RecordBatch` stream. + +## Public surface + +- Entry: `MitoEngine` in `src/mito2/src/engine.rs`, built via `MitoEngineBuilder`. +- Trait: `impl RegionEngine for MitoEngine` (`store-api`'s region engine contract). +- Consumed by `datanode` (sends `RegionRequest`s) and the query layer (scans). + +## When you change X, also touch Y + +- **Manifest format** (`manifest/action.rs`): affects crash recovery and + follower replay. Keep it backward compatible. +- **SST/Parquet layout** (`sst/`): readers must stay compatible with existing files. +- **Request types** (`request.rs`): usually tied to proto definitions consumed by `datanode`. +- **WAL/memtable encoding** (`wal/`, `memtable/`): breaks replay if changed incompatibly. + +## Testing + +```bash +cargo nextest run -p mito2 +``` + +Tests live next to the code as `*_test.rs` (e.g. `src/mito2/src/engine/flush_test.rs`). +`TestEnv` in `test_util.rs` spins up an engine over an in-process object store. + +## Gotchas + +- Sequence numbers are strictly increasing per region; dedup and snapshot reads + depend on this. Do not change assignment lightly. +- Manifest version is monotonic — never reset or skip it. +- Lock ordering: take the manifest lock before updating `version_control`; the + reverse deadlocks against concurrent flush/compaction. +- All region I/O runs on tokio workers; never `block_on` inside a worker. + +## Maintenance contract + +Update this file when you add/rename a top-level module, change the write/read +path entry points, or alter a persisted format (manifest, SST, WAL).