Commit Graph
4 Commits
Author SHA1 Message Date
shuiyisongandLei, HUANG 66d38e8e1c feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add `InsertLimitInterceptor` hook to `Inserter`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: attribute insert limit checks to the target table's database

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: unify write admission through metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce write admission for pending row batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: distinguish internal requests for ingestion metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: admit split ingestion requests once per database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: OpenTSDB throws error reason

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use meter crate main rev

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude database ingest rate limit from table options

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: reserve channel 255 for internal requests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 07:27:22 +00:00
discord9 7c7132ea65 refactor(flow): execute streaming flows with DataFusion (#8976)
* test(mito2): cover regex inverted index pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): execute streaming flows with DataFusion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): remove legacy streaming runtime

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): avoid retrying stateless sink inserts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): align stateless writes with sink schema

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): reject stale stateless source schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): validate stateless flow routing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* Revert "test(mito2): cover regex inverted index pruning"

This reverts commit 79e96ac745.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preserve source timestamps in stateless flows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): address stateless review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): recover stateless flows after schema changes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): serialize schema rebuilds and validate retained sources

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): verify streaming recovery and schema changes through SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): cool down failed schema rebuilds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): explain legacy aggregate recreation requirements

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): adapt stateless provider downcast to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-17 06:44:01 +00:00
dennis zhuang 5d4699db1c docs: refine coding agent maps (#8790)
* docs: refine coding agent maps

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: trim license header guidance

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: update README links and project status

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-08-10 02:45:37 +00:00
dennis zhuang 3b8f55e490 docs(agents): add per-crate guides, architecture invariants, and generated-files list (#8346)
* docs(agents): add per-crate guides, architecture invariants, and generated-files list

Add agent/contributor navigation docs modeled on the AGENTS.md convention:

- Per-crate AGENTS.md for hot crates (mito2, metric-engine, flow, frontend,
  meta-srv): module map, read/write paths, change-coupling points, test
  commands, and gotchas.
- .agents/architecture-invariants.md: repo-wide rules that clippy and the
  style guide do not cover (format compatibility, crate layering, async
  runtimes, error handling, experimental gating, the DataFusion fork).
- .agents/generated-files.md: tool-generated artifacts that must not be
  hand-edited (sqlness .result, config.md, dashboards, build.rs output, proto).
- Anchor the .gitignore CLAUDE.md/AGENTS.md rules to the repo root so per-crate
  AGENTS.md files are tracked while root-level personal config stays ignored.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: update crate AGENTS.md and fix config.md path

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(agents): fix DataFusion patch layout and SQL query lifecycle order

Address review feedback on #8346:

- architecture-invariants: the DataFusion sub-crates pin an exact crates.io
  version in [workspace.dependencies] and are redirected to the fork rev in
  [patch.crates-io]; the two sections hold different forms, not the same rev.
- frontend: the SQL query lifecycle runs the pre_parsing/post_parsing
  interceptors around parsing, before the per-statement permission check.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-24 09:14:09 +00:00