docs(agents): add per-crate guides, architecture invariants, and generated-files list (#8346)

* docs(agents): add per-crate guides, architecture invariants, and generated-files list Add agent/contributor navigation docs modeled on the AGENTS.md convention: - Per-crate AGENTS.md for hot crates (mito2, metric-engine, flow, frontend, meta-srv): module map, read/write paths, change-coupling points, test commands, and gotchas. - .agents/architecture-invariants.md: repo-wide rules that clippy and the style guide do not cover (format compatibility, crate layering, async runtimes, error handling, experimental gating, the DataFusion fork). - .agents/generated-files.md: tool-generated artifacts that must not be hand-edited (sqlness .result, config.md, dashboards, build.rs output, proto). - Anchor the .gitignore CLAUDE.md/AGENTS.md rules to the repo root so per-crate AGENTS.md files are tracked while root-level personal config stays ignored. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: update crate AGENTS.md and fix config.md path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(agents): fix DataFusion patch layout and SQL query lifecycle order Address review feedback on #8346: - architecture-invariants: the DataFusion sub-crates pin an exact crates.io version in [workspace.dependencies] and are redirected to the fork rev in [patch.crates-io]; the two sections hold different forms, not the same rev. - frontend: the SQL query lifecycle runs the pre_parsing/post_parsing interceptors around parsing, before the per-statement permission check. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-07-05 21:40:38 +00:00 · 2026-06-24 17:14:09 +08:00
parent db5a30c49b
commit 3b8f55e490
9 changed files with 637 additions and 3 deletions
--- a/.agents/README.md
+++ b/.agents/README.md
@@ -1,4 +1,8 @@
-# Agent Skills
+# Agent Resources
+
+Shared resources for coding agents (Codex, Claude Code, ...) and contributors.
+
+## Skills

 Shared agent skills live in `.agents/skills`.

@@ -6,3 +10,25 @@ Shared agent skills live in `.agents/skills`.
 - Claude Code reads the same skills through `.claude/skills`, which is a symlink to this directory.
 - Add or update shared skills under `.agents/skills/<skill-name>/SKILL.md`.
 - If a new skill does not appear, restart the agent or start a new thread.
+
+## Per-crate guides
+
+Hot crates carry an `AGENTS.md` next to their code as a navigation aid (module
+map, read/write paths, change-coupling points, test commands, gotchas):
+
+- [`src/mito2/AGENTS.md`](../src/mito2/AGENTS.md) — primary time-series storage engine
+- [`src/metric-engine/AGENTS.md`](../src/metric-engine/AGENTS.md) — metrics engine (logical/physical regions)
+- [`src/flow/AGENTS.md`](../src/flow/AGENTS.md) — stream processing / continuous aggregation
+- [`src/frontend/AGENTS.md`](../src/frontend/AGENTS.md) — request entry point and orchestration
+- [`src/meta-srv/AGENTS.md`](../src/meta-srv/AGENTS.md) — metadata and cluster coordination
+
+## Architecture invariants
+
+[`architecture-invariants.md`](architecture-invariants.md) lists repo-wide rules
+that are easy to violate and expensive to get wrong (format compatibility, crate
+layering, async runtimes, error handling, feature gating, the DataFusion fork).
+
+## Generated files
+
+[`generated-files.md`](generated-files.md) lists tool-generated artifacts that
+must not be hand-edited (sqlness `.result`, `config.md`, dashboards, ...).
--- a/.agents/architecture-invariants.md
+++ b/.agents/architecture-invariants.md
@@ -0,0 +1,111 @@
+# Architecture Invariants
+
+Repo-wide rules that are easy to violate and expensive to get wrong. They are
+**not** general best practices — each one is specific to GreptimeDB, has high
+blast radius, and is not caught by `cargo clippy`.
+
+This complements the other docs rather than repeating them:
+
+- `docs/style-guide.md` — micro-level code style (formatting, modules, comments).
+- `docs/rfcs/` — per-feature architecture decisions.
+- `CONTRIBUTING.md` — how to build, test, and submit.
+
+Paths below are relative to the repo root.
+
+## 1. Persisted and wire formats must stay backward/forward compatible
+
+Anything written to disk or sent over the wire outlives the process that wrote
+it: region manifests, WAL entries, SST/Parquet files and their metadata,
+metadata KV values (`common-meta` keys, metric-engine metadata), and gRPC
+messages. A node running an old version may read data written by a new one and
+vice versa.
+
+- Add fields, don't repurpose or reorder them. For serde types use
+  `#[serde(default)]` / `#[serde(alias)]`; never change the meaning of an existing
+  field or the discriminant of an existing enum variant.
+- Monotonic version counters (e.g. the mito2 manifest version) only ever move
+  forward — never reset or skip.
+- When a change touches a persisted or wire format, add a case to the
+  compatibility test suite. See `docs/rfcs/2025-07-04-compatibility-test-framework.md`
+  — compatibility has been broken on releases before (v0.14.1, v0.15.1) precisely
+  because this step was skipped.
+- Wire types are generated from the external `greptime-proto` crate; change the
+  format there first, then bump the dependency (see invariant 6's pattern).
+
+## 2. Respect crate layering and dependency direction
+
+The workspace is layered; dependencies point downward only.
+
+- `common-*` crates are the base. They must not depend on storage engines,
+  `frontend`, `datanode`, or `meta-srv`.
+- `store-api` defines the engine contract (e.g. `RegionEngine` in
+  `src/store-api/src/region_engine.rs`). Engines (`mito2`, `metric-engine`,
+  `file-engine`) implement it; `datanode` drives engines **through the trait**,
+  not through engine internals.
+- `frontend` reaches storage through `operator` / `query` / `catalog`, not by
+  depending on `datanode` internals. Standalone mode is the one bridge, via a
+  `RegionServer` wrapper (`src/frontend/src/instance/standalone.rs`).
+- Do not introduce circular dependencies. New deps go through
+  `[workspace.dependencies]` in the root `Cargo.toml`, not per-crate version
+  literals.
+
+## 3. Use the shared async runtimes; never block them
+
+Runtimes are partitioned by workload so one workload can't starve another. They
+live in `common-runtime` (`src/common/runtime/`).
+
+- Use the categorized spawns — `spawn_global`, `spawn_query`, `spawn_ingest`,
+  `spawn_compact`, `spawn_hb` — instead of constructing your own tokio runtime,
+  and pick the category that matches the work.
+- Run CPU-bound or synchronous-blocking work via `spawn_blocking_*`; never do
+  heavy CPU or blocking syscalls directly inside an async task.
+- Never call `block_on*` from inside an async context or an engine worker — it
+  deadlocks the runtime.
+
+## 4. Errors: snafu + `ErrorExt`, no panics in non-test code
+
+Each crate defines its own snafu `Error` enum and implements `ErrorExt`
+(`src/common/error/src/ext.rs`).
+
+- Set a meaningful `status_code()`. It drives the client-visible result and
+  whether the message is masked (`Internal`/`Unknown` are masked from end users).
+- Mark errors the caller may retry with the appropriate `retry_hint()`
+  (`Retryable`). Default is non-retryable.
+- In non-test code, return errors instead of `unwrap()` / `expect()` / `panic!()`.
+  Use `unimplemented!()` (not `todo!()`) for paths that won't be implemented, per
+  `docs/style-guide.md`, which also covers `with_context` vs `context`.
+
+## 5. Gate unstable features behind `experimental_` config
+
+Features whose behavior or surface may still change ship behind config keys
+prefixed `experimental_` (see existing examples in `config/datanode.example.toml`,
+`config/flownode.example.toml`, `config/standalone.example.toml`). Some can be
+overridden per-object (e.g. a flow's `WITH (experimental_... = '...')`). This lets
+unfinished work merge without freezing it into the stable config surface.
+
+When you stabilize such a feature, drop the prefix and document the migration.
+
+## 6. DataFusion is a pinned fork — two sections, two forms
+
+GreptimeDB uses a fork at `GreptimeTeam/datafusion`, wired up in the root
+`Cargo.toml` through **two sections that hold different things**:
+
+- `[workspace.dependencies]` pins each DataFusion sub-crate to an **exact
+  crates.io version** (e.g. `datafusion = "=53.1.0"`).
+- `[patch.crates-io]` redirects those same crates to the **fork at a git rev**
+  (e.g. `datafusion = { git = ".../GreptimeTeam/datafusion.git", rev = "..." }`).
+
+So:
+
+- Adding a new DataFusion sub-crate dependency means adding it to **both**
+  sections — the `=<version>` entry under `[workspace.dependencies]` and the
+  matching git-rev patch under `[patch.crates-io]`.
+- Upgrading DataFusion means bumping the version in `[workspace.dependencies]`
+  **and** the rev in `[patch.crates-io]` together, for all of them.
+
+## Maintenance contract
+
+Update this file when a new repo-wide invariant emerges (a new persisted format,
+a new runtime category, a layering rule), or when an existing one changes. Keep
+each entry high-signal: if `clippy` or `docs/style-guide.md` already enforces it,
+it does not belong here.
--- a/.agents/generated-files.md
+++ b/.agents/generated-files.md
@@ -0,0 +1,60 @@
+# Generated Files — Do Not Hand-Edit
+
+These artifacts are produced by tooling. Editing them by hand is almost always
+wrong: your change is either overwritten on regeneration or causes CI to fail.
+When the underlying source changes, regenerate them with the listed command.
+
+## sqlness `.result` files — `tests/cases/**/*.result`
+
+Generated by the sqlness runner from the sibling `.sql` files. **Never edit a
+`.result` by hand.** To update expectations, change the `.sql` (or the engine
+behavior) and re-run the test; the runner rewrites the `.result`.
+
+```bash
+cargo sqlness bare            # run all local cases
+cargo sqlness bare -t <name>  # run one case
+```
+
+Review the regenerated output with `git diff tests`.
+
+Note: `tests/cases/distributed/common` is a **symlink** to
+`tests/cases/standalone/common`. They are the same files — editing one changes
+both, and there is no need to diff standalone vs distributed.
+
+## Configuration docs — `config/config.md`
+
+Generated from the `config/*.example.toml` files and
+`config/config-docs-template.md`.
+
+```bash
+make config-docs
+```
+
+Edit the example TOMLs and/or the template, then regenerate. Do not edit
+`config/config.md` directly.
+
+## Grafana dashboards
+
+Generated artifacts; regenerate rather than editing by hand.
+
+```bash
+make dashboards
+```
+
+## `build.rs`-generated code
+
+Some crates generate code at build time into the Cargo `OUT_DIR` (e.g.
+`common-version` for build/git info, plus `common-catalog`, `common-function`'s
+system tables, `log-store`, `servers`). There is no checked-in file to edit —
+change the corresponding `build.rs` or its inputs instead.
+
+## Protobuf / gRPC types
+
+Generated from the external `greptime-proto` crate, not from sources in this
+repo. To change the wire format, change `greptime-proto` upstream and bump the
+dependency.
+
+## License headers
+
+Managed by the license-header tooling and the pre-commit hooks; let the tooling
+add/fix them rather than editing headers by hand.
--- a/.gitignore
+++ b/.gitignore
@@ -66,8 +66,9 @@ greptimedb_data
 !/.github

 # AI related
-CLAUDE.md
-AGENTS.md
+# Root-level personal/local agent config only; per-crate AGENTS.md files are tracked.
+/CLAUDE.md
+/AGENTS.md
 .codex
 .gemini
 .opencode
--- a/src/flow/AGENTS.md
+++ b/src/flow/AGENTS.md
@@ -0,0 +1,90 @@
+# flow — Agent & Contributor Guide
+
+Navigation aid for `src/flow`. Keep it short and point to code. Paths are
+relative to the repo root.
+
+Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md).
+
+## What this crate does
+
+Flownode is the stream-processing engine behind continuous aggregation /
+materialized views. It runs in two modes:
+
+- **Batching mode** (the default and the actively developed path): splits data
+  into time windows and periodically runs aggregation SQL through the frontend,
+  writing results back to a sink table.
+- **Streaming mode** (the older dataflow engine): an incremental DFIR/dataflow
+  compute graph that processes row-level diffs.
+
+A flow without an explicit `flow_type` is created as **batching**.
+
+## Module map
+
+| Module | Path | Purpose |
+| --- | --- | --- |
+| `engine` | `src/flow/src/engine.rs` | `FlowEngine` trait: create/remove/flush/insert lifecycle |
+| `adapter` | `src/flow/src/adapter.rs`, `src/flow/src/adapter/` | `StreamingEngine`, worker pool, dual-engine dispatch, table sources/sinks |
+| `batching_mode` | `src/flow/src/batching_mode.rs`, `src/flow/src/batching_mode/` | `BatchingEngine`, task scheduling, time windows, frontend client, checkpoints |
+| `compute` | `src/flow/src/compute/` | Streaming dataflow render/state |
+| `expr` | `src/flow/src/expr.rs`, `src/flow/src/expr/` | Scalar/aggregate expressions and Map-Filter-Project |
+| `plan` | `src/flow/src/plan.rs` | `TypedPlan` (reduce/join/MFP) |
+| `transform` | `src/flow/src/transform.rs` | Substrait → flow plan |
+| `df_optimizer` | `src/flow/src/df_optimizer.rs` | SQL → DataFusion logical plan → optimized plan |
+| `repr` | `src/flow/src/repr.rs` | `Row`, `DiffRow`, `Batch`, `RelationDesc` |
+| `server` | `src/flow/src/server.rs` | gRPC `Flow` service, `FlownodeBuilder`/`FlownodeInstance` |
+| `heartbeat` | `src/flow/src/heartbeat.rs` | Reports flownode state/stats to metasrv |
+
+Flow metadata lives in `common-meta`, not here:
+`src/common/meta/src/key/flow/` and `src/common/meta/src/ddl/create_flow.rs`.
+
+## Data flow
+
+`Frontend → Flownode (gRPC)` → `FlowService` (`server.rs`) →
+`FlowDualEngine` (`adapter/flownode_impl.rs`) routes by `FlowType`:
+
+- Batching: marks dirty windows; a task later runs aggregation SQL via the
+  frontend client and writes the sink table (`batching_mode/task.rs`).
+- Streaming: worker threads apply incremental diffs and push to the sink
+  (`adapter/worker.rs`, `compute/render.rs`).
+
+## Public surface
+
+- gRPC `Flow` service in `src/flow/src/server.rs`
+  (`handle_create_remove`, `handle_mirror_request`, `handle_mark_dirty_time_window`).
+- Sink writes for the streaming engine go through `FrontendInvoker`
+  (`row_inserts`, `row_deletes`).
+- `FlowEngine` trait in `src/flow/src/engine.rs`.
+- Started from the `cmd` crate via `FlownodeBuilder` / `FlownodeInstance`.
+
+## When you change X, also touch Y
+
+- **Flow definition / options**: validation in `common-meta`'s `ddl/create_flow.rs`
+  and the serialized `FlowInfoValue` in `common-meta`'s `key/flow/`.
+- **New scalar/aggregate function** (`expr/`): also wire up evaluation in
+  `compute/render.rs` (streaming) and ensure batching SQL handles it.
+- **Persisted flow metadata**: keep `FlowInfoValue` backward compatible
+  (`serde(default)` / `serde(alias)`).
+- A change to one engine often needs the mirror change in the other.
+
+## Testing
+
+```bash
+cargo nextest run -p flow
+```
+
+Helpers in `src/flow/src/test_utils.rs` (test context, test query engine).
+
+## Gotchas
+
+- Batching vs streaming differ a lot in latency, debuggability, and code path —
+  confirm which mode a flow uses before reasoning about it. Batching is the
+  default and the primary target.
+- Streaming workers are `!Send`; cross-thread interaction goes through
+  `WorkerHandle`, not the worker directly.
+- Internal flow timestamps (`repr::Timestamp`, ms) are not necessarily the
+  table's time column; window functions key off the diff timestamp.
+
+## Maintenance contract
+
+Update this file when the dual-engine routing, the gRPC surface, or the flow
+metadata contract (shared with `common-meta`) changes.
--- a/src/frontend/AGENTS.md
+++ b/src/frontend/AGENTS.md
@@ -0,0 +1,83 @@
+# frontend — Agent & Contributor Guide
+
+Navigation aid for `src/frontend`. Keep it short and point to code. Paths are
+relative to the repo root.
+
+Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md).
+
+## What this crate does
+
+Frontend is the request entry point and orchestration layer. It accepts
+multi-protocol requests (gRPC, HTTP, MySQL, PostgreSQL, InfluxDB, OTLP, Jaeger,
+Prometheus, OpenTSDB), checks permissions, parses/plans SQL, and dispatches:
+reads go to the query engine (`query` crate), writes go to the inserter/deleter
+(`operator` crate).
+
+Boundary with `servers`: the `servers` crate implements the wire protocols and
+network I/O; `frontend` provides the business logic by implementing handler
+traits (`SqlQueryHandler`, `GrpcQueryHandler`, `InfluxdbLineProtocolHandler`,
+...). In standalone mode the frontend embeds a datanode `RegionServer`; in
+distributed mode it talks to remote datanodes via `operator`/`client`.
+
+## Module map
+
+| Module | Path | Purpose |
+| --- | --- | --- |
+| `instance` | `src/frontend/src/instance.rs` | `Instance`: the core handler; implements `SqlQueryHandler`, `PrometheusHandler`, etc. |
+| `instance/builder` | `src/frontend/src/instance/builder.rs` | `FrontendBuilder` assembles `Instance` from its dependencies |
+| `instance/grpc` | `src/frontend/src/instance/grpc.rs` | `GrpcQueryHandler`: insert/delete/query/promql over gRPC |
+| `instance/standalone` | `src/frontend/src/instance/standalone.rs` | Calls the local `RegionServer` instead of RPC |
+| `instance/region_query` | `src/frontend/src/instance/region_query.rs` | Routes distributed region reads to datanodes |
+| `instance/*` | `src/frontend/src/instance/` | Per-protocol handlers (`influxdb.rs`, `promql.rs`, `otlp/`, `jaeger.rs`, `logs.rs`, `prom_store.rs`, ...) |
+| `frontend` | `src/frontend/src/frontend.rs` | `Frontend` lifecycle wrapper (`FrontendOptions`, start/shutdown) |
+| `server` | `src/frontend/src/server.rs` | `Services`: builds and wires the protocol servers |
+| `heartbeat` | `src/frontend/src/heartbeat.rs` | Heartbeat to metasrv; handles suspend / cache invalidation |
+| `service_config` | `src/frontend/src/service_config/` | Per-protocol option structs |
+
+## Request lifecycles
+
+- **SQL query** (`instance.rs`): `do_query` → `pre_parsing` interceptor → parse →
+  `post_parsing` interceptor → then per statement: `check_permission` →
+  `statement_executor.plan` (logical plan) → `query_engine.execute` → for
+  distributed reads, `region_query.rs` fetches from datanodes → cancellable
+  `RecordBatch` stream. Interceptors run around parsing, before the per-statement
+  permission check — preserve that ordering.
+- **Insert** (`instance/grpc.rs`): `handle_inserts` / `handle_row_inserts` →
+  `check_permission` → `operator`'s `Inserter` (schema validation, optional
+  auto-create, partition routing) → local `RegionServer` (standalone) or RPC to
+  datanodes (distributed).
+
+## Public surface
+
+- `Instance` (`instance.rs`) — the business-logic container.
+- `Frontend` (`frontend.rs`) — lifecycle wrapper around `Instance` + servers + heartbeat.
+- Created from `cmd`: `src/cmd/src/frontend.rs` (distributed) and
+  `src/cmd/src/standalone.rs` (standalone, with embedded datanode).
+
+## When you change X, also touch Y
+
+- **`servers` handler traits**: a new/changed protocol handler requires the
+  matching `impl` here.
+- **`operator` Inserter/Deleter or `query` QueryEngine API**: update the call
+  sites in `instance.rs` / `instance/grpc.rs`.
+- **`session::QueryContext`**: new context fields thread through most handlers.
+- **`sql` statements**: new statement kinds need handling in `query_statement`.
+
+## Testing
+
+```bash
+cargo nextest run -p frontend
+```
+
+## Gotchas
+
+- Keep the frontend/servers split straight: wire format and network live in
+  `servers`; permissions, planning, and routing live here.
+- Standalone vs distributed diverge in datanode access (local `RegionServer` vs
+  `NodeClients` RPC), MetaClient usage, and whether heartbeat matters. In
+  standalone, the cache invalidator is a no-op.
+
+## Maintenance contract
+
+Update this file when you add a protocol handler, change the query/insert
+lifecycle, or change how `Instance` is constructed or wired to `servers`.
--- a/src/meta-srv/AGENTS.md
+++ b/src/meta-srv/AGENTS.md
@@ -0,0 +1,90 @@
+# meta-srv — Agent & Contributor Guide
+
+Navigation aid for `src/meta-srv`. Keep it short and point to code. Paths are
+relative to the repo root.
+
+Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md).
+
+## What this crate does
+
+Metasrv is the metadata and coordination service for distributed mode: it
+persists cluster/table/region metadata, runs leader election, drives the
+heartbeat loop that tracks cluster topology, and orchestrates distributed
+procedures (DDL, region migration, repartition). Data models, the KV backend
+abstraction, election, key encoding, and the DDL manager live in `common-meta`;
+`meta-srv` implements the server, state machine, and control logic on top.
+
+## Module map
+
+| Module | Path | Purpose |
+| --- | --- | --- |
+| `bootstrap` | `src/meta-srv/src/bootstrap.rs` | Wires up KV backend, election, gRPC services, procedure manager |
+| `metasrv` | `src/meta-srv/src/metasrv.rs`, `src/meta-srv/src/metasrv/builder.rs` | `Metasrv` struct, `MetasrvOptions`, `MetasrvBuilder`, startup |
+| `state` | `src/meta-srv/src/state.rs` | Leader/Follower state machine and transitions |
+| `handler` | `src/meta-srv/src/handler/` | Heartbeat handler chain (`HeartbeatHandler`): leader check, region lease, stats, mailbox |
+| `service` | `src/meta-srv/src/service/` | gRPC services (`heartbeat`, `procedure`, `cluster`, `store`) and HTTP admin |
+| `procedure` | `src/meta-srv/src/procedure/` | Distributed procedures: `region_migration/`, `repartition.rs`, `wal_prune/` |
+| `region` | `src/meta-srv/src/region/` | Region supervisor, lease keeper, failover triggers |
+| `discovery` | `src/meta-srv/src/discovery/` | Node discovery and lease-backed node info |
+| `pubsub` | `src/meta-srv/src/pubsub/` | Heartbeat topic publish/subscribe support |
+| `gc` | `src/meta-srv/src/gc/` | Metadata-driven garbage-collection procedures and scheduling |
+| `selector` | `src/meta-srv/src/selector/` | Region placement strategies (round-robin / load-based / lease-based) |
+| `peer` | `src/meta-srv/src/peer.rs` | Peer allocation through selectors |
+| `cluster` | `src/meta-srv/src/cluster.rs` | `MetaPeerClient`: internal RPC with leader fallback |
+| `cache_invalidator` | `src/meta-srv/src/cache_invalidator.rs` | Pushes cache-invalidation to frontends/datanodes |
+| `key` | `src/meta-srv/src/key/` | Metasrv-side KV key encoding |
+
+## Core flows
+
+- **Heartbeat**: `service/heartbeat.rs` receives datanode/frontend heartbeats and
+  runs the handler chain in `handler/`; responses carry mailbox messages (DDL
+  results, cache invalidation).
+- **DDL**: `service/procedure.rs` (`ddl`) is leader-only and hands off to
+  `common-meta`'s `DdlManager`, executed via `common-procedure`.
+- **Region migration**: `procedure/region_migration/manager.rs` plus per-step
+  files; triggered by failover or an explicit command.
+- **Leader election**: via `common-meta`'s election; transitions in `state.rs`.
+
+## Public surface
+
+Clients use the `meta-client` crate (`MetaClient`). On the server side the entry
+points are the gRPC services under `src/meta-srv/src/service/` and the HTTP admin
+API under `src/meta-srv/src/service/admin/`.
+
+## When you change X, also touch Y
+
+- **Heartbeat interval / leases**: timing constants live in `common-meta`
+  (`distributed_time_constants`); changing them affects lease and failover timing
+  in `handler/region_lease_handler.rs` and `region/supervisor.rs`.
+- **Procedures**: state must stay persisted in the KV backend and `execute()`
+  must be idempotent for crash recovery.
+- **Cache invalidation**: new metadata types need matching invalidation in
+  `cache_invalidator.rs` and the heartbeat publish handler.
+- **Selector stats fields**: keep `selector/` implementations in sync with the
+  stats they consume.
+
+## Testing
+
+```bash
+cargo nextest run -p meta-srv
+# mock election / KV backend:
+cargo nextest run -p meta-srv --features mock
+```
+
+Helpers: `src/meta-srv/src/mocks.rs`, `src/meta-srv/src/test_util.rs`,
+`src/meta-srv/src/procedure/test_util.rs`.
+
+## Gotchas
+
+- Most mutating operations are leader-only — guard them (see the leader check in
+  `handler/` and `service/procedure.rs`). Non-leaders return "not leader".
+- In-memory state is not durable; anything needed after a leader change must be
+  persisted to the KV backend. The leader uses a cached KV backend that resets on
+  leader change.
+- Region lease renewal timing and the supervisor's check interval must be
+  coordinated, or a region can appear active on two datanodes.
+
+## Maintenance contract
+
+Update this file when you add a procedure, change the heartbeat handler chain,
+change leader-only boundaries, or alter the metasrv ↔ common-meta split.
--- a/src/metric-engine/AGENTS.md
+++ b/src/metric-engine/AGENTS.md
@@ -0,0 +1,85 @@
+# metric-engine — Agent & Contributor Guide
+
+Navigation aid for `src/metric-engine`. Keep it short and point to code. Paths
+are relative to the repo root.
+
+Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md).
+
+## What this crate does
+
+The Metric Engine is optimized for Prometheus-style workloads with a huge number
+of small tables. Many **logical** regions (one per metric table) share a single
+**physical** pair of Mito2 regions: a data region and a metadata region. Rows
+are multiplexed with metric-engine internal identity: dense primary-key mode
+injects `__table_id` and `__tsid`, while the default sparse mode encodes them
+into `__primary_key`. Reads still add a logical-table filter before forwarding
+to the physical data region. It implements `RegionEngine` and delegates all real
+storage to `mito2`.
+
+The architecture is documented at the top of `src/metric-engine/src/lib.rs`.
+
+## Module map
+
+| Module | Path | Purpose |
+| --- | --- | --- |
+| `engine` | `src/metric-engine/src/engine.rs`, `src/metric-engine/src/engine/` | `MetricEngine` (`RegionEngine` impl) and per-op handlers (`create.rs`, `put.rs`, `read.rs`, `alter.rs`, `drop.rs`, ...) |
+| `metadata_region` | `src/metric-engine/src/metadata_region.rs` | K-V over a Mito2 region storing logical table/column metadata, with an LRU cache |
+| `data_region` | `src/metric-engine/src/data_region.rs` | Wraps the Mito2 data region; forwards writes and manages physical columns |
+| `row_modifier` | `src/metric-engine/src/row_modifier.rs` | Rewrites incoming rows for dense or sparse primary-key encoding |
+| `batch_modifier` | `src/metric-engine/src/batch_modifier.rs` | RecordBatch-level TSID computation and sparse primary-key encoding |
+| `state` | `src/metric-engine/src/engine/state.rs` | In-memory cache of physical columns and logical column metadata |
+| `repeated_task` | `src/metric-engine/src/repeated_task.rs` | Periodic metadata-region flush task |
+| `utils` | `src/metric-engine/src/utils.rs` | `RegionId` conversions (data vs metadata group), manifest encoding |
+| `config` | `src/metric-engine/src/config.rs` | `EngineConfig` (metadata flush interval, sparse PK) |
+| `test_util` | `src/metric-engine/src/test_util.rs` | `TestEnv` building the Mito2 + Metric stack |
+
+## Write path
+
+`MetricEngine::handle_request(Put)` (`engine/put.rs`) rejects direct writes to a
+physical region, resolves the physical region for the logical id, loads logical
+columns from `metadata_region`, then `row_modifier` rewrites the rows according
+to the data region's primary-key encoding and forwards the request to the Mito2
+data region.
+
+## Read path
+
+`MetricEngine::handle_query` (`engine/read.rs`): a query against a logical region
+is rewritten to add a `__table_id == <logical_id>` filter and forwarded to the
+Mito2 data region. Queries against a physical region pass straight through.
+
+## Public surface
+
+- Entry: `MetricEngine` in `src/metric-engine/src/engine.rs`, built via `try_new(mito, config)`.
+- Trait: `impl RegionEngine for MetricEngine` (name = `"metric"`).
+- Depends on `mito2`, `store-api`, and `mito-codec` (sparse primary key codec).
+
+## When you change X, also touch Y
+
+- **Reserved column ids / names** (`__tsid`, `__table_id`, `__primary_key`): see
+  `store-api`'s metric engine consts; keep them in sync with `engine.rs`.
+- **Metadata K-V encoding** (`metadata_region.rs`): changes the on-disk metadata layout.
+- **RegionId group mapping** (`utils.rs`): data vs metadata region derivation.
+- **Physical column rules** (`engine/alter/`): a physical region allows only one field column.
+
+## Testing
+
+```bash
+cargo nextest run -p metric-engine
+```
+
+`TestEnv` in `test_util.rs` gives you `mito()` and `metric()` handles.
+
+## Gotchas
+
+- Physical vs logical region confusion: physical regions reject direct user
+  writes; operate on logical region ids.
+- TSID must be stable for the same tag set — it is a hash over sorted tag names
+  + values and may be stored in `__tsid` or encoded into `__primary_key`.
+- Metadata is cached (LRU with a TTL); after an alter, stale reads are possible
+  until invalidation/expiry.
+- Always convert ids via `utils::to_data_region_id` / `to_metadata_region_id`.
+
+## Maintenance contract
+
+Update this file when you change the logical/physical region model, the injected
+columns, the metadata encoding, or the public engine entry points.
--- a/src/mito2/AGENTS.md
+++ b/src/mito2/AGENTS.md
@@ -0,0 +1,88 @@
+# mito2 — Agent & Contributor Guide
+
+Navigation aid for `src/mito2`. Keep it short and point to code; do not duplicate
+the code here. Paths are relative to the repo root.
+
+Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md).
+
+## What this crate does
+
+Mito2 is GreptimeDB's primary time-series region storage engine. It owns the
+write path (memtable + WAL), flushing memtables to Parquet SST files,
+TWCS/windowed compaction, and the read path (multi-level merge + dedup with
+snapshot isolation). It implements the `RegionEngine` trait from `store-api`.
+
+## Module map
+
+| Module | Path | Purpose |
+| --- | --- | --- |
+| `engine` | `src/mito2/src/engine.rs` | `MitoEngine` (the `RegionEngine` impl) and request dispatch |
+| `worker` | `src/mito2/src/worker.rs`, `src/mito2/src/worker/` | Per-region worker loop; write/alter/flush handlers |
+| `region` | `src/mito2/src/region.rs`, `src/mito2/src/region/version.rs` | `MitoRegion` state and copy-on-write `VersionControl` snapshots |
+| `request` | `src/mito2/src/request.rs` | `WriteRequest`/`RegionRequest` types and result channels |
+| `wal` | `src/mito2/src/wal.rs` | Write-ahead log wrapper over `log-store` |
+| `memtable` | `src/mito2/src/memtable/` | In-memory write buffers (time-series / bulk / partition) |
+| `flush` | `src/mito2/src/flush.rs` | `FlushScheduler`, `WriteBufferManager`, memtable → SST |
+| `compaction` | `src/mito2/src/compaction/` | TWCS picker, strict-window manual picker, compactor, memory control |
+| `access_layer` | `src/mito2/src/access_layer.rs` | SST read/write over the object store |
+| `sst` | `src/mito2/src/sst/` | Parquet format, file metadata, index layout |
+| `read` | `src/mito2/src/read/` | `ScanRegion`, merge, dedup, projection, streaming |
+| `manifest` | `src/mito2/src/manifest/` | `RegionManifestManager`, manifest actions/edits |
+| `cache` | `src/mito2/src/cache.rs` | Write/file/page caches |
+| `gc` | `src/mito2/src/gc.rs`, `src/mito2/src/gc/` | Dropped-file cleanup worker |
+| `schedule` | `src/mito2/src/schedule/` | Local/remote background job scheduling |
+| `remap_manifest` | `src/mito2/src/remap_manifest.rs` | Manifest path remapping for region copy/migration |
+| `config` | `src/mito2/src/config.rs` | `MitoConfig` tuning knobs |
+| `test_util` | `src/mito2/src/test_util.rs` | `TestEnv` and builders (under the `test` feature) |
+
+## Write path
+
+`MitoEngine::handle_request` (`engine.rs`) → worker loop
+(`worker/handle_write.rs`) → sequence + WAL assembly (`region_write_ctx.rs`) →
+`wal.rs` → memtable (`memtable/`) → when buffer pressure trips, `flush.rs`
+writes SSTs via `access_layer.rs` and appends a `RegionEdit` to the manifest
+(`manifest/manager.rs`).
+
+## Read path
+
+`MitoEngine::handle_query` (`engine.rs`) → `read/scan_region.rs` takes an
+immutable `Version` (`region/version.rs`) → scans memtables and Parquet SSTs
+(`sst/parquet.rs`) → merges (`read/`) and dedups by sequence → projected,
+filtered `RecordBatch` stream.
+
+## Public surface
+
+- Entry: `MitoEngine` in `src/mito2/src/engine.rs`, built via `MitoEngineBuilder`.
+- Trait: `impl RegionEngine for MitoEngine` (`store-api`'s region engine contract).
+- Consumed by `datanode` (sends `RegionRequest`s) and the query layer (scans).
+
+## When you change X, also touch Y
+
+- **Manifest format** (`manifest/action.rs`): affects crash recovery and
+  follower replay. Keep it backward compatible.
+- **SST/Parquet layout** (`sst/`): readers must stay compatible with existing files.
+- **Request types** (`request.rs`): usually tied to proto definitions consumed by `datanode`.
+- **WAL/memtable encoding** (`wal/`, `memtable/`): breaks replay if changed incompatibly.
+
+## Testing
+
+```bash
+cargo nextest run -p mito2
+```
+
+Tests live next to the code as `*_test.rs` (e.g. `src/mito2/src/engine/flush_test.rs`).
+`TestEnv` in `test_util.rs` spins up an engine over an in-process object store.
+
+## Gotchas
+
+- Sequence numbers are strictly increasing per region; dedup and snapshot reads
+  depend on this. Do not change assignment lightly.
+- Manifest version is monotonic — never reset or skip it.
+- Lock ordering: take the manifest lock before updating `version_control`; the
+  reverse deadlocks against concurrent flush/compaction.
+- All region I/O runs on tokio workers; never `block_on` inside a worker.
+
+## Maintenance contract
+
+Update this file when you add/rename a top-level module, change the write/read
+path entry points, or alter a persisted format (manifest, SST, WAL).