mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 08:09:22 +00:00
Hengfei Yang
e28de6c6c3
## Problem In local mode with the default sqlite meta store, concurrent metadata writes fail with: ``` Execution Error: error returned from database: (code: 517) database is locked ``` 517 is `SQLITE_BUSY_SNAPSHOT`: a deferred sqlite transaction pins its read snapshot at the first SELECT, and when it later performs its first write the upgrade fails immediately if any other connection committed in between — `busy_timeout` never retries it. The codebase convention is to serialize sqlite writes with `infra::table::get_lock()`, but a full audit found ~100 write functions that never take it, concentrated in the newer subsystems. ## What changed (31 files) - **Table layer, previously zero locks**: `alert_states`, `alert_incidents`, `incident_events`, `dashboards`, `folders`, `ratelimit`, `sessions`, `system_settings`, `alert_composites`, `slo` / `slos` / `slo_budget` / `slo_backfill_jobs`, `synthetics_jobs` / `synthetics_runs`. - **Partially covered modules**: the specific unlocked writers in `users`, `org_users`, `organizations`, `org_ingestion_tokens`, `synthetics_agents`, `synthetics_checks`, `compactor_manual_jobs`, `trial_quota_usage`, `alert_eval_intervals`, `search_job/search_jobs`. - **Service layer**: `db/pipeline_errors` (self-reporting flush), alert deduplication state, the composite-alert scheduler transaction in `core/alerts/scheduler/handlers.rs`, raw anomaly-detection config updates, and two direct writes in `core/alerts/composite/service.rs`. - **Bonus fixes**: `search_job_partitions::submit_partitions` ran its `FOR UPDATE` probe on the pool instead of its open transaction (on Postgres that row lock was not tied to the transaction at all); a ratelimit unit test raced the global `ORM_CLIENT` OnceCell and failed depending on suite ordering. ## Lock-placement rules (why some functions deliberately do NOT lock) The mutex is not reentrant, so the diff enforces **exactly one acquisition per call chain**: - The function that owns the transaction locks before `begin()`; helpers that receive an open transaction (`persist_update_in_transaction`, `apply_status_in_txn`, `record_evaluation_with`, the `alert_states` public wrappers, …) never lock. - Guards are dropped before coordinator emits, `db::scheduler` calls, and helpers that lock internally (e.g. `dashboards::delete_all` locks only after `distinct_values::batch_remove`, which takes the lock itself). ## Verification - Full workspace `cargo check` + clippy clean; 928 infra table tests pass (run twice for ordering stability). - Combined build with the enterprise crate passes; **companion enterprise PR uses the same branch name** `fix/sqlite-lock-remaining` per paired-CI convention. - Live smoke on a sqlite local-mode instance: 40 concurrent folder creations (previously an unlocked path) plus mixed ingestion/metadata load — all requests succeed, zero locked errors in the log. ## Note for reviewers This is the per-call-site fix consistent with the existing convention. A structural alternative (single-connection sqlite RW pool, which makes 517 impossible and would let the ~330 manual lock sites retire) was prototyped separately and can follow as its own change.