Commit Graph
8 Commits
Author SHA1 Message Date
LFCandLei, HUANG 325415912a feat: support automatic column addition for Flight bulk inserts [Backport release/v1.3] (#9399)
feat: support automatic column addition for Flight bulk inserts (#9285)

* fix: auto-add columns when initializing bulk insert streams



* fix: reject nested columns in bulk schema auto-add



* fix: reject unknown columns in non-empty bulk inserts



---------


(cherry picked from commit 5ccfcd4644)

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-10-12 15:31:18 +08:00
LFCandshuiyisong bb6e1591d5 refactor: expose OTLP write and batching helpers for downstream routers [Backport release/v1.3] (#9462)
refactor: expose OTLP write and batching helpers for downstream routers (#9401)


(cherry picked from commit 48a2ca6340)

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
2026-10-08 15:40:45 +08:00
Weny Xu 723da69b21 feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-22 14:38:17 +00:00
shuiyisongandLei, HUANG 66d38e8e1c feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add `InsertLimitInterceptor` hook to `Inserter`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: attribute insert limit checks to the target table's database

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: unify write admission through metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce write admission for pending row batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: distinguish internal requests for ingestion metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: admit split ingestion requests once per database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: OpenTSDB throws error reason

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use meter crate main rev

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude database ingest rate limit from table options

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: reserve channel 255 for internal requests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 07:27:22 +00:00
Weny Xu be15c88e92 feat: batch ordinary table writes across HTTP protocols (#9115)
* feat: integrate table batching across HTTP protocols

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: skip empty prepared writes before batch admission

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: load batching protocols from environment and document frontend wiring

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove experimental prefix from pending rows batcher config

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style: sort frontend test dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: count batched ingestion once and update config snapshot

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-17 09:32:12 +00:00
shuiyisongandfys d4af650ec0 perf: reduce cold workspace compile time (#8801)
* refactor: remove datanode and meta-srv dep from frontend

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: use on-device protoc if possible

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: remove unused dep

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: version and docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update logs

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
2026-08-10 07:18:16 +00:00
dennis zhuang 5d4699db1c docs: refine coding agent maps (#8790)
* docs: refine coding agent maps

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: trim license header guidance

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: update README links and project status

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-08-10 02:45:37 +00:00
dennis zhuang 3b8f55e490 docs(agents): add per-crate guides, architecture invariants, and generated-files list (#8346)
* docs(agents): add per-crate guides, architecture invariants, and generated-files list

Add agent/contributor navigation docs modeled on the AGENTS.md convention:

- Per-crate AGENTS.md for hot crates (mito2, metric-engine, flow, frontend,
  meta-srv): module map, read/write paths, change-coupling points, test
  commands, and gotchas.
- .agents/architecture-invariants.md: repo-wide rules that clippy and the
  style guide do not cover (format compatibility, crate layering, async
  runtimes, error handling, experimental gating, the DataFusion fork).
- .agents/generated-files.md: tool-generated artifacts that must not be
  hand-edited (sqlness .result, config.md, dashboards, build.rs output, proto).
- Anchor the .gitignore CLAUDE.md/AGENTS.md rules to the repo root so per-crate
  AGENTS.md files are tracked while root-level personal config stays ignored.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: update crate AGENTS.md and fix config.md path

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(agents): fix DataFusion patch layout and SQL query lifecycle order

Address review feedback on #8346:

- architecture-invariants: the DataFusion sub-crates pin an exact crates.io
  version in [workspace.dependencies] and are redirected to the fork rev in
  [patch.crates-io]; the two sections hold different forms, not the same rev.
- frontend: the SQL query lifecycle runs the pre_parsing/post_parsing
  interceptors around parsing, before the per-statement permission check.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-24 09:14:09 +00:00