From 5d4699db1cb3a01dec3b3ec77cef67887051d65a Mon Sep 17 00:00:00 2001 From: dennis zhuang Date: Mon, 10 Aug 2026 10:45:37 +0800 Subject: [PATCH] docs: refine coding agent maps (#8790) * docs: refine coding agent maps Signed-off-by: Dennis Zhuang * docs: trim license header guidance Signed-off-by: Dennis Zhuang * docs: update README links and project status Signed-off-by: Dennis Zhuang --------- Signed-off-by: Dennis Zhuang --- .agents/README.md | 13 +++-- .agents/architecture-invariants.md | 47 +++++++++--------- .agents/generated-files.md | 14 ++++-- .../skills/greptimedb-release-note/SKILL.md | 7 +-- AGENTS.md | 49 +++++++++++++++++-- README.md | 4 +- src/common/meta/AGENTS.md | 38 ++++++++++++++ src/flow/AGENTS.md | 21 +++++--- src/frontend/AGENTS.md | 9 ++-- src/metric-engine/AGENTS.md | 8 +-- src/operator/AGENTS.md | 40 +++++++++++++++ src/query/AGENTS.md | 39 +++++++++++++++ src/servers/AGENTS.md | 37 ++++++++++++++ tests/perf/AGENTS.md | 5 +- 14 files changed, 273 insertions(+), 58 deletions(-) create mode 100644 src/common/meta/AGENTS.md create mode 100644 src/operator/AGENTS.md create mode 100644 src/query/AGENTS.md create mode 100644 src/servers/AGENTS.md diff --git a/.agents/README.md b/.agents/README.md index d55aac50f0..8d21de0f1a 100644 --- a/.agents/README.md +++ b/.agents/README.md @@ -11,16 +11,23 @@ Shared agent skills live in `.agents/skills`. - Add or update shared skills under `.agents/skills//SKILL.md`. - If a new skill does not appear, restart the agent or start a new thread. -## Per-crate guides +## Per-directory guides -Hot crates carry an `AGENTS.md` next to their code as a navigation aid (module -map, read/write paths, change-coupling points, test commands, gotchas): +Read every `AGENTS.md` from the repository root down to the path being changed. +The root guide applies everywhere; nested guides add rules and navigation for +their subtree. +- [`src/common/meta/AGENTS.md`](../src/common/meta/AGENTS.md) — metadata keys, KV backends, DDL procedures, and caches +- [`src/query/AGENTS.md`](../src/query/AGENTS.md) — query planning, optimization, and distributed execution +- [`src/servers/AGENTS.md`](../src/servers/AGENTS.md) — wire protocols and network servers +- [`src/operator/AGENTS.md`](../src/operator/AGENTS.md) — statement, DDL/DML, and write orchestration - [`src/mito2/AGENTS.md`](../src/mito2/AGENTS.md) — primary time-series storage engine - [`src/metric-engine/AGENTS.md`](../src/metric-engine/AGENTS.md) — metrics engine (logical/physical regions) - [`src/flow/AGENTS.md`](../src/flow/AGENTS.md) — stream processing / continuous aggregation - [`src/frontend/AGENTS.md`](../src/frontend/AGENTS.md) — request entry point and orchestration - [`src/meta-srv/AGENTS.md`](../src/meta-srv/AGENTS.md) — metadata and cluster coordination +- [`tests/compatibility/AGENTS.md`](../tests/compatibility/AGENTS.md) — persisted/wire compatibility cases +- [`tests/perf/AGENTS.md`](../tests/perf/AGENTS.md) — query regression harness and case DSL ## Architecture invariants diff --git a/.agents/architecture-invariants.md b/.agents/architecture-invariants.md index b7ce954c97..78c78f3a97 100644 --- a/.agents/architecture-invariants.md +++ b/.agents/architecture-invariants.md @@ -20,15 +20,18 @@ metadata KV values (`common-meta` keys, metric-engine metadata), and gRPC messages. A node running an old version may read data written by a new one and vice versa. -- Add fields, don't repurpose or reorder them. For serde types use - `#[serde(default)]` / `#[serde(alias)]`; never change the meaning of an existing - field or the discriminant of an existing enum variant. -- Monotonic version counters (e.g. the mito2 manifest version) only ever move - forward — never reset or skip. +- Preserve the contract of the actual encoding: Protobuf field numbers/types, + serde field and variant names, positional field order, and explicitly encoded + enum discriminants. Do not treat Rust declaration order as a wire contract + for name-based formats such as JSON. +- Use `#[serde(default)]` for newly optional data and `#[serde(alias)]` for a + compatible rename. Do not reset or reuse versions within one persisted + history. - When a change touches a persisted or wire format, add a case to the - compatibility test suite. See `docs/rfcs/2025-07-04-compatibility-test-framework.md` - — compatibility has been broken on releases before (v0.14.1, v0.15.1) precisely - because this step was skipped. + compatibility test suite. Follow [`tests/compatibility/README.md`](../tests/compatibility/README.md) + and [`tests/compatibility/AGENTS.md`](../tests/compatibility/AGENTS.md). The + original compatibility RFC describes a superseded case format and is not the + current runbook. - Wire types are generated from the external `greptime-proto` crate; change the format there first, then bump the dependency (see invariant 6's pattern). @@ -54,9 +57,9 @@ The workspace is layered; dependencies point downward only. Runtimes are partitioned by workload so one workload can't starve another. They live in `common-runtime` (`src/common/runtime/`). -- Use the categorized spawns — `spawn_global`, `spawn_query`, `spawn_ingest`, - `spawn_compact`, `spawn_hb` — instead of constructing your own tokio runtime, - and pick the category that matches the work. +- Runtime construction belongs in process bootstrap or test/benchmark harnesses. + Product components use `spawn_global`, `spawn_query`, `spawn_ingest`, + `spawn_compact`, or `spawn_hb` instead of a component-local Tokio runtime. - Run CPU-bound or synchronous-blocking work via `spawn_blocking_*`; never do heavy CPU or blocking syscalls directly inside an async task. - Never call `block_on*` from inside an async context or an engine worker — it @@ -85,23 +88,19 @@ unfinished work merge without freezing it into the stable config surface. When you stabilize such a feature, drop the prefix and document the migration. -## 6. DataFusion is a pinned fork — two sections, two forms +## 6. DataFusion is a pinned fork — workspace dependencies plus patches GreptimeDB uses a fork at `GreptimeTeam/datafusion`, wired up in the root -`Cargo.toml` through **two sections that hold different things**: +`Cargo.toml` through two sections: -- `[workspace.dependencies]` pins each DataFusion sub-crate to an **exact - crates.io version** (e.g. `datafusion = "=53.1.0"`). -- `[patch.crates-io]` redirects those same crates to the **fork at a git rev** - (e.g. `datafusion = { git = ".../GreptimeTeam/datafusion.git", rev = "..." }`). +- `[workspace.dependencies]` pins directly referenced sub-crates to an exact + crates.io version. +- `[patch.crates-io]` redirects resolution to the fork. It may also contain + transitive-only sub-crates with no workspace dependency. -So: - -- Adding a new DataFusion sub-crate dependency means adding it to **both** - sections — the `=` entry under `[workspace.dependencies]` and the - matching git-rev patch under `[patch.crates-io]`. -- Upgrading DataFusion means bumping the version in `[workspace.dependencies]` - **and** the rev in `[patch.crates-io]` together, for all of them. +Add a direct dependency to `[workspace.dependencies]` and patch it if needed; a +transitive-only override needs only the patch. Upgrade all exact pins and fork +revisions together. ## 7. Enterprise-gated code is licensed differently from the rest diff --git a/.agents/generated-files.md b/.agents/generated-files.md index 1a16fcb28c..23f7e81daf 100644 --- a/.agents/generated-files.md +++ b/.agents/generated-files.md @@ -35,7 +35,14 @@ Edit the example TOMLs and/or the template, then regenerate. Do not edit ## Grafana dashboards -Generated artifacts; regenerate rather than editing by hand. +For the metrics dashboards, edit +`grafana/dashboards/metrics/cluster/dashboard.json`. The generator derives the +standalone `dashboard.json` by removing instance filters, then produces +`dashboard.yaml` and `dashboard.md` for both cluster and standalone variants. +Do not hand-edit those five derived files. + +`grafana/dashboards/events/dashboard.json` and +`grafana/dashboards/logs/dashboard.json` are not outputs of this generator. ```bash make dashboards @@ -56,5 +63,6 @@ dependency. ## License headers -Managed by the license-header tooling and the pre-commit hooks; let the tooling -add/fix them rather than editing headers by hand. +License headers are checked by `korandoru/hawkeye@v5` in +`.github/workflows/checks.yml`. For enterprise-gated files, also update both +license lists and run `make check-enterprise-license`. diff --git a/.agents/skills/greptimedb-release-note/SKILL.md b/.agents/skills/greptimedb-release-note/SKILL.md index a5cb47e9b0..893b99445c 100644 --- a/.agents/skills/greptimedb-release-note/SKILL.md +++ b/.agents/skills/greptimedb-release-note/SKILL.md @@ -167,9 +167,10 @@ The release note is also published as a blog post in **`GreptimeTeam/docs`**. git -C fetch origin main git -C worktree add -b chore/X.Y.Z-release-note /tmp/docs-release-note origin/main ``` -- **The PR body must follow the docs repo's template** (`.github/pull_request_template.md` — - "What's Changed in this PR" + a Checklist). Fill in the description; leave checklist boxes - for the reviewer. +- **Read and follow the current docs repo PR template** + (`.github/pull_request_template.md`) from the checkout used for the release. + Fill in every required section and leave reviewer-owned checklist boxes unchecked; + do not rely on hard-coded section names from an older template. - Commit with sign-off, push, open a **draft** PR, then remove the worktree: ``` git -C /tmp/docs-release-note add blog/release-X-Y-Z.md diff --git a/AGENTS.md b/AGENTS.md index a00889b855..c99856c294 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,7 +18,8 @@ provides sub-second querying at PB scale with high cost efficiency. | Build (debug) | `make build` | | Build (release) | `make build RELEASE=true` | | Run standalone | `cargo run -- standalone start` | -| Test | `cargo nextest run` (preferred over `cargo test`) | +| Targeted Rust tests | `cargo nextest run -p ` (preferred over `cargo test`) | +| Full local Rust suite | `make test` | | SQL tests | `cargo sqlness bare` (single case: `cargo sqlness bare -t `) | | Format | `make fmt` (and `make fmt-toml` for TOML) | | Lint | `make clippy` (= `cargo clippy --workspace --all-targets --all-features -- -D warnings`) | @@ -43,14 +44,30 @@ GreptimeDB is a Cargo workspace rooted at the repository; most crates live under - **Shared**: `src/common/`, `src/datatypes/`, `src/store-api/` (engine contract), `src/catalog/`, `src/table/` -Hot crates carry their own `AGENTS.md` with a module map, read/write paths, -change-coupling points, and gotchas: +Before editing a path, read every `AGENTS.md` from the repository root down to +that path. The root guide always applies; a nested guide adds or overrides rules +for its subtree. Changes spanning multiple subtrees must follow every applicable +guide. +`AGENTS.md` files are maps, not manuals. Keep them short and stable: record +where code lives, important boundaries or change-coupling, high-cost gotchas, +and the validation entry point. Implementation details belong in code. Do not +copy exhaustive test lists or workflow logic; link to the source of truth. + +High-change areas and specialized test suites carry their own `AGENTS.md` with +a module map, read/write paths, change-coupling points, and gotchas: + +- [`src/common/meta/AGENTS.md`](src/common/meta/AGENTS.md) +- [`src/query/AGENTS.md`](src/query/AGENTS.md) +- [`src/servers/AGENTS.md`](src/servers/AGENTS.md) +- [`src/operator/AGENTS.md`](src/operator/AGENTS.md) - [`src/mito2/AGENTS.md`](src/mito2/AGENTS.md) - [`src/metric-engine/AGENTS.md`](src/metric-engine/AGENTS.md) - [`src/flow/AGENTS.md`](src/flow/AGENTS.md) - [`src/frontend/AGENTS.md`](src/frontend/AGENTS.md) - [`src/meta-srv/AGENTS.md`](src/meta-srv/AGENTS.md) +- [`tests/compatibility/AGENTS.md`](tests/compatibility/AGENTS.md) +- [`tests/perf/AGENTS.md`](tests/perf/AGENTS.md) ## Read before changing code @@ -71,11 +88,35 @@ change-coupling points, and gotchas: - Error handling: `src/common/error/` (`ErrorExt`, `StatusCode`) - Protocol implementations: `src/servers/src/` +## Worktree safety + +- Check `git status --short` before editing. Preserve unrelated tracked changes + and untracked files; re-read files changed by another process. +- Do not rewrite history, force-push, or remove files in bulk unless the user + explicitly requests it. +- Update generated artifacts only through the generators documented in + [`.agents/generated-files.md`](.agents/generated-files.md). + +## Validation by change type + +Use the narrowest command that covers the change, then expand only when its +blast radius requires it. + +| Change | Minimum validation | +| --- | --- | +| One Rust crate | `cargo nextest run -p ` | +| Cross-workspace Rust behavior | `make test`; inspect `.github/workflows/rust.yml` when CI parity matters | +| SQL parsing, planning, execution, or output | `cargo sqlness bare -t `; inspect regenerated `.result` files | +| Persisted metadata or wire format | Add/run a case under `tests/compatibility/`; follow its `README.md` and `AGENTS.md` | +| Public configuration | Update example TOMLs, loading/serialization snapshots, and docs; run `make config-docs` | +| Query regression harness or DSL | Follow `tests/perf/AGENTS.md` | +| Enterprise-gated code | Build/test with `--features enterprise` where applicable and run `make check-enterprise-license` | + ## Before opening a PR 1. `make fmt` 2. `make clippy` -3. `make test` (or `cargo nextest run`) +3. `make test` 4. `make check-udeps` (run `make fix-udeps` if it reports unused dependencies). 5. If you added or changed a public configuration option, update the applicable example TOMLs, configuration-loading and serialized-config snapshot tests, diff --git a/README.md b/README.md index 019a7c848b..21980b1a22 100644 --- a/README.md +++ b/README.md @@ -203,13 +203,13 @@ See the [Contribution Guidelines](CONTRIBUTING.md) for the full developer workfl - **Kubernetes**: [GreptimeDB Operator](https://github.com/GreptimeTeam/greptimedb-operator) - **Helm Charts**: [Greptime Helm Charts](https://github.com/GreptimeTeam/helm-charts) - **Dashboard**: [Web UI](https://github.com/GreptimeTeam/dashboard) -- **gRPC Ingester**: [Go](https://github.com/GreptimeTeam/greptimedb-ingester-go), [Java](https://github.com/GreptimeTeam/greptimedb-ingester-java), [C++](https://github.com/GreptimeTeam/greptimedb-ingester-cpp), [Erlang](https://github.com/GreptimeTeam/greptimedb-ingester-erl), [Rust](https://github.com/GreptimeTeam/greptimedb-ingester-rust), [.NET](https://github.com/GreptimeTeam/greptimedb-ingester-dotnet) +- **gRPC Ingester**: [Go](https://github.com/GreptimeTeam/greptimedb-ingester-go), [Java](https://github.com/GreptimeTeam/greptimedb-ingester-java), [C++](https://github.com/GreptimeTeam/greptimedb-ingester-cpp), [Erlang](https://github.com/GreptimeTeam/greptimedb-ingester-erl), [Rust](https://github.com/GreptimeTeam/greptimedb-ingester-rust), [.NET](https://github.com/GreptimeTeam/greptimedb-ingester-dotnet), [TypeScript](https://github.com/GreptimeTeam/greptimedb-ingester-ts) - **Grafana Data Source**: [GreptimeDB Grafana data source plugin](https://github.com/GreptimeTeam/greptimedb-grafana-datasource) - **Grafana Dashboard**: [Official Dashboard for monitoring](https://github.com/GreptimeTeam/greptimedb/blob/main/grafana/README.md) ## Project Status -GreptimeDB is at [v1.0 GA](https://github.com/GreptimeTeam/greptimedb/releases/tag/v1.0.0) with stable APIs and regular releases. It runs in production at scale — [OceanBase Cloud](https://greptime.com/blogs/2025-07-22-user-case-obcloud-log-management-greptimedb) operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60% after migrating from Grafana Loki. See more in [case studies](https://greptime.com/blogs/?category=Use%20Case). +GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — [OceanBase Cloud](https://greptime.com/blogs/2025-07-22-user-case-obcloud-log-management-greptimedb) operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60% after migrating from Grafana Loki. See more in [case studies](https://greptime.com/blogs/?category=Use%20Case). Read the [v1.0 highlights](https://greptime.com/blogs/2025-11-05-greptimedb-v1-highlights) and [2026 roadmap](https://greptime.com/blogs/2026-02-11-greptimedb-roadmap-2026), or browse the [version reference](https://docs.greptime.com/nightly/reference/about-greptimedb-version). diff --git a/src/common/meta/AGENTS.md b/src/common/meta/AGENTS.md new file mode 100644 index 0000000000..fd5da08a93 --- /dev/null +++ b/src/common/meta/AGENTS.md @@ -0,0 +1,38 @@ +# common-meta — Agent & Contributor Guide + +Navigation map for `src/common/meta`. Repo-wide invariants: +[`.agents/architecture-invariants.md`](../../../.agents/architecture-invariants.md). + +`common-meta` contains shared metadata keys and values, KV backends, caches, +and durable DDL procedures. Metasrv process and service wiring live in +`src/meta-srv`. + +## Module map + +| Area | Path | Entry point | +| --- | --- | --- | +| Metadata model | `src/common/meta/src/key.rs`, `src/common/meta/src/key/` | Typed keys, values, and `TableMetadataManager` | +| KV storage | `src/common/meta/src/kv_backend.rs`, `src/common/meta/src/kv_backend/` | `KvBackend`, transactions, memory/etcd/RDS backends | +| DDL procedures | `src/common/meta/src/ddl.rs`, `src/common/meta/src/ddl/`, `src/common/meta/src/ddl_manager.rs` | Durable DDL state machines and task dispatch | +| Caches | `src/common/meta/src/cache.rs`, `src/common/meta/src/cache/`, `src/common/meta/src/cache_invalidator.rs` | Metadata caches and invalidation | +| RPC types | `src/common/meta/src/rpc.rs`, `src/common/meta/src/rpc/` | Shared metasrv request/response types | +| Recovery | `src/common/meta/src/reconciliation.rs`, `src/common/meta/src/reconciliation/` | Catalog/table/region reconciliation | + +## Change coupling + +- Key/value encoding changes require backward-compatible decoding and a case in + `tests/compatibility/`. +- Multi-key metadata changes must preserve transaction boundaries and cache + invalidation. +- Procedure changes must preserve persisted state and `TYPE_NAME`; new + procedures need loader registration in `src/common/meta/src/ddl_manager.rs`. +- `KvBackend` behavior changes should extend the shared tests in + `src/common/meta/src/kv_backend/test.rs`. + +## Testing + +```bash +cargo nextest run -p common-meta +``` + +Backend-specific coverage may also need `pg_kvbackend` or `mysql_kvbackend`. diff --git a/src/flow/AGENTS.md b/src/flow/AGENTS.md index aba019e02a..0ac0aa0e24 100644 --- a/src/flow/AGENTS.md +++ b/src/flow/AGENTS.md @@ -8,15 +8,19 @@ Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.a ## What this crate does Flownode is the stream-processing engine behind continuous aggregation / -materialized views. It runs in two modes: +materialized views. It has two execution paths: -- **Batching mode** (the default and the actively developed path): splits data +- **Batching mode** (the actively developed path): splits data into time windows and periodically runs aggregation SQL through the frontend, writing results back to a sink table. -- **Streaming mode** (the older dataflow engine): an incremental DFIR/dataflow +- **Streaming mode** (the legacy dataflow path): an incremental DFIR/dataflow compute graph that processes row-level diffs. -A flow without an explicit `flow_type` is created as **batching**. +Users cannot select a mode directly: `flow_type` is a reserved internal option. +`StatementExecutor::determine_flow_type` in `src/operator/src/statement/ddl.rs` +owns current mode selection. `FlowDualEngine` defaults missing internal +`flow_type` metadata to batching for compatibility. Read those paths before +changing routing rules. ## Module map @@ -64,7 +68,9 @@ Flow metadata lives in `common-meta`, not here: `compute/render.rs` (streaming) and ensure batching SQL handles it. - **Persisted flow metadata**: keep `FlowInfoValue` backward compatible (`serde(default)` / `serde(alias)`). -- A change to one engine often needs the mirror change in the other. +- **Shared routing, metadata, or sink contracts**: check both engines. An + engine-specific implementation change does not automatically need a mirror + change in the other path. ## Testing @@ -76,9 +82,8 @@ Helpers in `src/flow/src/test_utils.rs` (test context, test query engine). ## Gotchas -- Batching vs streaming differ a lot in latency, debuggability, and code path — - confirm which mode a flow uses before reasoning about it. Batching is the - default and the primary target. +- Batching vs streaming differ in latency, state, and execution. Confirm the + selected mode before reasoning about a flow. - Streaming workers are `!Send`; cross-thread interaction goes through `WorkerHandle`, not the worker directly. - Internal flow timestamps (`repr::Timestamp`, ms) are not necessarily the diff --git a/src/frontend/AGENTS.md b/src/frontend/AGENTS.md index 2321b65732..3afd75634d 100644 --- a/src/frontend/AGENTS.md +++ b/src/frontend/AGENTS.md @@ -36,12 +36,9 @@ distributed mode it talks to remote datanodes via `operator`/`client`. ## Request lifecycles -- **SQL query** (`instance.rs`): `do_query` → `pre_parsing` interceptor → parse → - `post_parsing` interceptor → then per statement: `check_permission` → - `statement_executor.plan` (logical plan) → `query_engine.execute` → for - distributed reads, `region_query.rs` fetches from datanodes → cancellable - `RecordBatch` stream. Interceptors run around parsing, before the per-statement - permission check — preserve that ordering. +- **SQL query** (`instance.rs`): `do_query_inner` handles parsing, interceptors, + permission checks, timeout/cancellation, and delegates planning/execution to + `StatementExecutor`. Distributed scans enter through `region_query.rs`. - **Insert** (`instance/grpc.rs`): `handle_inserts` / `handle_row_inserts` → `check_permission` → `operator`'s `Inserter` (schema validation, optional auto-create, partition routing) → local `RegionServer` (standalone) or RPC to diff --git a/src/metric-engine/AGENTS.md b/src/metric-engine/AGENTS.md index 4471de3fae..d104947897 100644 --- a/src/metric-engine/AGENTS.md +++ b/src/metric-engine/AGENTS.md @@ -30,7 +30,8 @@ The architecture is documented at the top of `src/metric-engine/src/lib.rs`. | `state` | `src/metric-engine/src/engine/state.rs` | In-memory cache of physical columns and logical column metadata | | `repeated_task` | `src/metric-engine/src/repeated_task.rs` | Periodic metadata-region flush task | | `utils` | `src/metric-engine/src/utils.rs` | `RegionId` conversions (data vs metadata group), manifest encoding | -| `config` | `src/metric-engine/src/config.rs` | `EngineConfig` (metadata flush interval, sparse PK) | +| `config` | `src/metric-engine/src/config.rs` | `EngineConfig` (metadata flush interval) | +| `engine/options` | `src/metric-engine/src/engine/options.rs` | Physical data-region options, including the default sparse primary-key encoding | | `test_util` | `src/metric-engine/src/test_util.rs` | `TestEnv` building the Mito2 + Metric stack | ## Write path @@ -77,8 +78,9 @@ cargo nextest run -p metric-engine writes; operate on logical region ids. - TSID must be stable for the same tag set — it is a hash over sorted tag names + values and may be stored in `__tsid` or encoded into `__primary_key`. -- Metadata is cached (LRU with a TTL); after an alter, stale reads are possible - until invalidation/expiry. +- Metadata reads use an LRU cache. Metadata writes take the per-region cache + write lock and invalidate the cache after the Mito2 write succeeds; preserve + that ordering when adding a metadata mutation path. - Always convert ids via `utils::to_data_region_id` / `to_metadata_region_id`. ## Maintenance contract diff --git a/src/operator/AGENTS.md b/src/operator/AGENTS.md new file mode 100644 index 0000000000..761d8e5080 --- /dev/null +++ b/src/operator/AGENTS.md @@ -0,0 +1,40 @@ +# operator — Agent & Contributor Guide + +Navigation map for `src/operator`. Repo-wide invariants: +[`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +`operator` connects frontend statements and writes to query, catalog, metasrv, +and datanodes. It owns statement dispatch, DDL task conversion, and region-level +insert/delete routing. + +## Module map + +| Area | Path | Entry point | +| --- | --- | --- | +| Statements | `src/operator/src/statement.rs`, `src/operator/src/statement/` | `StatementExecutor` and per-statement handlers | +| Inserts | `src/operator/src/insert.rs`, `src/operator/src/bulk_insert.rs` | Insert, auto-create/alter, routing, dispatch | +| Deletes | `src/operator/src/delete.rs` | Delete conversion and dispatch | +| Request conversion | `src/operator/src/req_convert.rs`, `src/operator/src/req_convert/` | Table/row/column requests to region requests | +| Region requests | `src/operator/src/region_req_factory.rs` | Region-level DDL/DML request construction | +| Procedures | `src/operator/src/procedure.rs` | Metasrv procedure administration | +| Flow | `src/operator/src/flow.rs`, `src/operator/src/statement/ddl.rs` | Flow requests and internal mode selection | + +## Change coupling + +- New statement kinds usually touch parser/AST handling, frontend permission + and interception, executor dispatch, and sqlness cases. +- DDL task changes require matching `common-meta`, metasrv, recovery, and + compatibility updates. +- Insert/delete conversion changes must cover both request formats, partition + routing, and schema-on-write behavior. +- Flow creation changes must stay aligned with `src/flow/AGENTS.md` and + `common-meta` flow metadata. + +## Testing + +```bash +cargo nextest run -p operator +cargo sqlness bare -t +``` + +Check standalone and distributed routing when changing region dispatch. diff --git a/src/query/AGENTS.md b/src/query/AGENTS.md new file mode 100644 index 0000000000..8dc03a2287 --- /dev/null +++ b/src/query/AGENTS.md @@ -0,0 +1,39 @@ +# query — Agent & Contributor Guide + +Navigation map for `src/query`. Repo-wide invariants: +[`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +`query` owns SQL/PromQL/log planning, GreptimeDB optimizer rules, DataFusion +execution, and distributed query plans. Protocols live in `servers`; statement +side effects live in `operator`. + +## Module map + +| Area | Path | Entry point | +| --- | --- | --- | +| Logical planning | `src/query/src/planner.rs`, `src/query/src/parser.rs` | `LogicalPlanner`, `QueryStatement` | +| Query engine | `src/query/src/query_engine.rs`, `src/query/src/query_engine/` | Engine traits, factory, state, serialization | +| DataFusion execution | `src/query/src/datafusion.rs`, `src/query/src/datafusion/` | Physical planning and execution | +| Optimizers | `src/query/src/optimizer.rs`, `src/query/src/optimizer/` | GreptimeDB logical/physical rules | +| Distributed plans | `src/query/src/dist_plan.rs`, `src/query/src/dist_plan/` | `MergeScan`, pruning, merge/sort, remote filters | +| Remote reads | `src/query/src/region_query.rs` | `RegionQueryHandler` boundary | +| Query languages | `src/query/src/promql/`, `src/query/src/log_query/` | PromQL and log planners | + +## Change coupling + +- Optimizer or `MergeScan` changes usually touch rule ordering, physical + planning, serialization, and distributed-plan tests together. +- User-visible SQL/PromQL/log behavior needs planner coverage and a sqlness case. +- `RegionQueryHandler` changes require the frontend implementation to move with + it. +- DataFusion dependency changes follow the pinned-fork rule in the repo-wide + invariants. + +## Testing + +```bash +cargo nextest run -p query +cargo sqlness bare -t +``` + +Check both standalone and distributed plans when changing `src/query/src/dist_plan/`. diff --git a/src/servers/AGENTS.md b/src/servers/AGENTS.md new file mode 100644 index 0000000000..2a446033f6 --- /dev/null +++ b/src/servers/AGENTS.md @@ -0,0 +1,37 @@ +# servers — Agent & Contributor Guide + +Navigation map for `src/servers`. Repo-wide invariants: +[`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md). + +`servers` owns network listeners and protocol translation. Database behavior is +provided through handler traits, mostly implemented by `frontend`. + +## Module map + +| Area | Path | Entry point | +| --- | --- | --- | +| Lifecycle | `src/servers/src/server.rs` | Shared `Server` trait | +| Handler boundary | `src/servers/src/query_handler.rs`, `src/servers/src/query_handler/` | Protocol-independent handler traits | +| gRPC / Flight | `src/servers/src/grpc.rs`, `src/servers/src/grpc/` | gRPC services, auth, cancellation, builders | +| HTTP | `src/servers/src/http.rs`, `src/servers/src/http/` | Routes, middleware, protocol endpoints, output formats | +| SQL protocols | `src/servers/src/mysql.rs`, `src/servers/src/mysql/`, `src/servers/src/postgres.rs`, `src/servers/src/postgres/` | MySQL/PostgreSQL servers and type conversion | +| Observability protocols | `src/servers/src/prom_remote_write/`, `src/servers/src/prometheus.rs`, `src/servers/src/otlp/`, `src/servers/src/otel_arrow.rs` | Prometheus and OpenTelemetry conversion | +| Resources / TLS | `src/servers/src/request_memory_limiter.rs`, `src/servers/src/tls.rs`, `src/servers/src/addrs.rs` | Memory admission, TLS, listener addresses | + +## Change coupling + +- Handler trait changes require matching frontend and mock implementations. +- gRPC wire changes start in `greptime-proto`; update builders and handlers here + after bumping it. +- Auth or `QueryContext` changes must cover HTTP, gRPC, MySQL, and PostgreSQL + entry points. +- New protocols or externally visible routes also require frontend service and + configuration wiring. + +## Testing + +```bash +cargo nextest run -p servers +``` + +Keep protocol translation here and permissions/database behavior in frontend. diff --git a/tests/perf/AGENTS.md b/tests/perf/AGENTS.md index e7c24f16e6..1f13dea58d 100644 --- a/tests/perf/AGENTS.md +++ b/tests/perf/AGENTS.md @@ -19,9 +19,10 @@ - Keep the direct-SST generator generic. Issue-specific behavior belongs in case files and thresholds, not in Rust generator logic. - Before pushing perf harness changes, run at least: - - `uv run --no-project python -m py_compile .github/scripts/query-regression-run.py .github/scripts/query-regression-summary.py .github/scripts/query-regression-pr-metadata.py tests/perf/test_query_regression_runner_compaction_toctou.py tests/perf/test_query_regression_runner_otlp_trace_load.py` - - `uv run --no-project python tests/perf/test_query_regression_runner_compaction_toctou.py && uv run --no-project python tests/perf/test_query_regression_runner_otlp_trace_load.py` + - the Python tests in the `Test query regression tooling` step of + `.github/workflows/query-regression.yml` - `cargo fmt --all -- --check` - `cargo build -p cmd --bin query_perf_fixture --features dev-tools` + - `cargo build -p cmd --bin query_regression_runner --features dev-tools` - exercise the outer lifecycle script and Rust fixture generator against all built-in cases when the DSL or workflow case selection changes.