Two refinements to the `e2e` label gate (Playwright + API, OSS + ENT).
## 1. Adding the `e2e` label always runs the suite
Previously the auto-bypass could override the label — add `e2e` to an
otherwise-skippable PR and nothing ran, which was confusing. Now the
label is an explicit **"run it"**: it always runs the full suite.
Without the label, the skip logic still decides.
| Event | `e2e` label | Change | Result |
|---|---|---|---|
| PR | **yes** | anything (even docs) | **run full suite** → pass/fail
on results |
| PR | no | relevant (code) | **block** (fail, "add label") |
| PR | no | skip category (docs / unrelated workflow) | **pass, no run**
|
| push / merge queue / dispatch | n/a | relevant | run |
| push / merge queue / dispatch | n/a | nothing relevant | pass, no run
|
Gate flips from `has_changes && (non-PR || label)` to `(PR && label) ||
(non-PR && has_changes)`; the summary is reworked to match.
## 2. Unrelated workflow changes no longer require e2e
`check_changes` treated **any** `.github/workflows/**` change as
relevant, so a Dependabot bump to an unrelated workflow
(openobserve#13060, `update-translations.yml`) demanded the label. Now
workflow files are bypassed by default; only a change to the suite's
**own** workflow (`playwright.yml` / `api-testing.yml`) triggers it.
Implemented with captured non-ignored lines instead of `grep -qv`
(portable across grep builds).
## Note
This PR changes `playwright.yml` + `api-testing.yml`, so it correctly
**needs the `e2e` label** to run. Sibling PR on the other repo uses the
same branch `ci/e2e-skip-unrelated-workflows`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Why
Our OSS Playwright + API CI compute (ubicloud build + eks matrix) is a
major cost driver. Two sources of avoidable spend:
1. **Every merged PR ran the full heavy suite up to 3×** — on the PR
(label-gated), in the merge queue (`merge_group`), and *again* on `push:
main`. The `push: main` run is a duplicate of the `merge_group` run
(both build/test the exact commit landing on main) **and** it bypasses
the `e2e` label gate, clawing back the gate's savings.
2. **Regression runs every 4h regardless of whether `main` moved** —
rebuilding and re-running the full matrix on identical code
overnight/weekends.
## Changes
**`playwright.yml` + `api-testing.yml`**
- Remove the `push: main` trigger. `merge_group` remains the trusted
main-ward validation.
- **Cache fix (important):** the Rust cache was only *saved* on `push:
main` runs (`save-if: github.ref == 'refs/heads/main'`). Deleting `push`
alone would stop the cache ever refreshing → cold builds → *more*
compute. So `save-if` now also fires on the `merge_group` run, which
builds the exact soon-to-be-main commit.
**`playwright_regression.yml`**
- New `check_main_changed` gate: compares `main`'s HEAD to the last
run's `head_sha` and skips build+matrix when nothing merged since.
Manual `workflow_dispatch` always runs. The skip cascades cleanly
(`playwright_summary` already treats skipped build/ui as success → stays
green).
## Impact
- ~1 full build + 146-shard matrix eliminated per merged PR (the
`push:main` duplicate).
- Idle-period regression runs (no merges) skipped instead of running the
full matrix.
- No coverage loss: merge queue still gates every merge; cache stays
warm.
## Not in this PR (deferred per discussion)
- `db-testing.yml` / `unit-tests.yml` `push:main` removal (cheaper,
later).
- ENT merge-queue adoption.
- OSS→ENT dispatch (semgrep / endpoint-auth-probe) gating.
Paired ENT change (regression gate): openobserve/o2-enterprise
`ci/trim-redundant-runs`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
Gate the full **Playwright (UI)** and **API** test suites on PRs behind
a single **`e2e` label**, and **auto-bypass** PRs that only touch files
which can't affect those suites. Covers `playwright.yml` and
`api-testing.yml`.
## Behaviour (both suites)
| Situation | build/test jobs | summary check (`playwright_summary` /
`api_tests_summary`) | Mergeable? |
|---|---|---|---|
| PR with `e2e` label | full suite runs | green if tests pass | ✅ if
green |
| PR, no label, relevant code changed | skipped | **fails** ("add `e2e`
label") | ❌ blocked |
| PR, no label, docs/meta only | skipped | **passes** (auto-bypass) | ✅
|
| Remove the `e2e` label | skipped (fresh run) | **fails** again | ❌
until re-added |
| push to `main` / merge queue / dispatch | full suite runs | green if
tests pass | n/a |
One `e2e` label runs **both** UI + API. `push`, `merge_group`, and
`workflow_dispatch` are never label-gated.
## Why the gate lives in the summary job
A job that `if:`-skips reports `skipped`, which branch protection treats
as **pass**. So a no-label PR can't be blocked by merely skipping jobs —
the summary job **actively fails** when the change is relevant and the
`e2e` label is absent.
## Auto-bypass (`check_changes`)
No label needed for: `*.md/.mdx/.markdown/.rst/.adoc`, `docs/`,
images/media, `LICENSE/NOTICE/AUTHORS/CONTRIBUTORS/CHANGELOG/.mailmap`,
`CODEOWNERS`, GitHub issue/PR/funding templates,
`dependabot`/`renovate`,
`.gitignore/.gitattributes/.editorconfig/.gitkeep/.husky/.vscode/.idea`.
The **API** workflow additionally bypasses `web/` + UI-test dirs
(frontend changes don't affect API tests).
**Not** bypassed (still trigger): `.github/workflows/**` (incl. these
workflow files), source, `package*.json`/lockfiles, configs, `*.txt`.
## Rollout prerequisites
- Create the **`e2e`** label in this repo.
- Required status checks = **`playwright_summary`** and
**`api_tests_summary`** — the individual shard jobs must NOT be required
(they legitimately skip and would auto-pass).
Sibling PR on the other repo uses the same branch `ci/e2e-label-gate`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Why
Two CI-observability gaps, both about **per-test/per-shard counts** that
the run-level metrics couldn't show:
1. **Flaky vs failed was conflated.** "Failed Shards by Module" counted
shard-*job* failures — but a shard that **failed then passed on rerun is
flaky, not failed**. A bare count told you nothing and misclassified
recovered flakes as failures.
2. **API had no test counts at all.** API `ci_test_runs` rows carried
only run health (conclusion/duration) — `tests_total` was NULL — so the
Test Counts dashboard had nothing for API.
## What
### Per-test flaky **and** failed → `ci_test_results`
- The **merge job** parses the merged `report.json` for tests with
`status="flaky"` (failed→passed on retry) or `status="unexpected"`
(failed after retries) via the new commented
**`.github/scripts/extract-flaky.js`**, emitting **title + module +
status**.
- **report_to_openobserve** posts **one row per such test** to
**`ci_test_results`**.
- Row schema: `{run_id, repo, suite, workflow, ingest_source, status
("flaky"|"failed"), module, test_title, test_file}` — powers accurate
per-module **flaky** *and* **failed** dashboards, no misclassification.
### Per-shard API test counts → `ci_test_counts` (+ run-doc total)
- Each API pytest job emits `--junitxml` and uploads an
**`api-junit__<category>__<leg>`** artifact (category = the test shard:
integration / query_agent / regression / … ENT also cli / vortex_* /
oss_in_ent / ent_top / ent_rbac).
- **report_to_openobserve** runs
**`.github/scripts/parse-api-junit.py`**, which reads the category per
artifact and **dedups config-leg reruns by test identity** (query_agent
opt true|false, regression flag combos) so the same suite isn't counted
twice.
- The deduped **total** folds into the run doc (`ci_test_runs`,
suite=api) so API shows in the same KPI/pie panels as UI; **per-category
rows** go to the new **`ci_test_counts`** stream for the per-shard
breakdown (the API analog of UI's `ci_test_modules`).
## Notes
- **Forward-only** — historical runs never stored junit / per-test names
(only counts), so no backfill; both streams accumulate from merge.
- All **non-fatal**: any parse/ingest error only warns, never fails the
workflow. `_timestamp` unset → ingest time.
- Pairs with the same-named branch on the other repo (OSS ⇄ ENT).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
Makes the per-run test dashboards **live**. Adds non-fatal
`report_to_openobserve` jobs that emit **one retry-aware run-summary doc
per run** to OpenObserve, following the metrics work merged earlier
(regression/docgen/council). Companion PR in the other repo mirrors
this.
**Scope (this PR):**
1. **`playwright.yml` + `api-testing.yml`** → new report jobs →
`ci_test_runs` (`suite=ui` / `suite=api`).
2. **`playwright_regression.yml`** → retry-aware reporting →
`ci_regression` (OSS: augments the existing job; ENT: adds the job — was
uninstrumented).
## How it works
- **Shared `.github/scripts/o2-run-report.sh`** (used by playwright/api)
derives the run's `conclusion` from job results, wall-clock from the
jobs API, and **sums cost across all attempts**. Regression jobs do the
same inline (they also emit shard/build/test fields).
- Emits: `run_id`, `run_attempt`, `retries`, `was_retried`,
`conclusion`, `suite`, `repo`, `branch`, `author`, `commit_sha`,
`pr_number`, `trigger`, `final_duration_sec`, `total_duration_sec`,
`retry_wasted_sec`.
- **Dedup model:** re-runs append one doc per attempt; dashboards keep
the **latest attempt per `run_id`** via `row_number() OVER (PARTITION BY
run_id ORDER BY run_attempt DESC)`. Retries become an *attribute*, never
duplicate rows; `total_duration_sec` on the kept row carries cumulative
compute.
- **Safe by construction:** `continue-on-error` + the never-fail poster
— can't fail/slow a test run; skips when no tests ran; **no-ops until
`O2_REPORTING_*` secrets are set**.
## Review
AI review addressed (mktemp payload, `GH_TOKEN` guard, non-negative
duration clamp). YAML + jq unit-tested + `actionlint` clean on the added
jobs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Adds a matrix strategy to the query agent SQL regression suite jobs in
both `api-testing.yml` and `query-agent.yml`
- Each job now runs with `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=false`
and `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=true`
- Verifies both DataFusion physical optimizer paths produce identical
query results
- Matrix legs run in parallel with `fail-fast: false`
- All artifact/log names include the matrix label to avoid collisions
## Test plan
- [ ] PR CI shows two `api_query_agent` matrix legs
(single_node_opt_false and single_node_opt_true), both pass
- [ ] `/query-agent` PR comment triggers both modes and posts per-mode
results comments
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Fixes a test gap where `match_all` Tantivy queries could return wrong
results after a DataFusion upgrade without being caught by existing
assertions.
### Problem
After a DataFusion upgrade merge, the DataFusion access plan was no
longer used for `match_all` queries, causing Tantivy to return wrong
results. The existing tests only compared row counts against a DuckDB
oracle that emulates `match_all` with `LIKE '%term%'` — so wrong rows
that still matched the LIKE pattern passed undetected.
Additionally, `node/flush` does not generate Tantivy indexes
immediately. The default `ZO_MAX_FILE_RETENTION_TIME=600s` means
WAL→parquet→Tantivy conversion can lag, so FTS queries may fall back to
a non-Tantivy path during tests.
### Changes
1. **Content verification**
([conftest.py](tests/api-testing/tests/query_agent/conftest.py)): Every
hit from a `match_all` query that returns the `log` field is now checked
— each search term word must be present in `log`. A hard assertion fails
if Tantivy returns rows that don't contain the search terms.
2. **Tantivy readiness gate**
([test_2_parquet.py](tests/api-testing/tests/query_agent/test_2_parquet.py)):
After `node/flush`, the test now polls with a `match_all('warehouse')
LIMIT 1` query (up to 120s timeout) to confirm Tantivy indexes are built
before running FTS assertions. Previously only COUNT(*) was checked.
3. **Multi-word match_all emulation**
([compute_counts.py](tests/test-data/query-agent/compute_counts.py)):
`match_all('ACK batch')` now translates to `(log LIKE '%ACK%' AND log
LIKE '%batch%')` — per-token AND conditions matching Tantivy's
token-level search semantics, instead of the old phrase match `log LIKE
'%ACK batch%'`.
4. **Regenerated expected results** across all query JSON files.
## Test plan
- [ ] Run query agent test suite against an environment with Tantivy
enabled
- [ ] Verify `match_all` content assertions catch intentionally
corrupted results
- [ ] Confirm post-flush Tantivy readiness poll works within the timeout
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
## Summary
- Adds **239 new tests** covering the vortex columnar file format across
78 scenarios from the test plan
- **`tests/api-testing/tests/vortex/`** — 9 test modules (83 tests):
format conversion, schema evolution, data integrity, FTS, downsampling,
API format awareness, server config, and performance/stress
- **`tests/api-testing/tests/query_agent/test_3_vortex.py`** — 156
tests: the full 155-query SQL correctness suite + scenario 39
(self-join) run post-flush against vortex files
- **`api-testing.yml`** — adds `--ignore=tests/vortex` to the OSS
default-set run; vortex tests require the enterprise binary with
`ZO_FILE_FORMAT=vortex` and are gated to enterprise CI only
## Test plan coverage
| Module | Scenarios |
|--------|-----------|
| `test_format_conversion.py` | 1–13, 7, 8, 9, 10 |
| `test_schema_evolution.py` | 14–25 |
| `test_data_integrity.py` | 55–62 |
| `test_fts.py` | 51–54 |
| `test_downsampling.py` | 47–50 |
| `test_api_format.py` | 71–73 |
| `test_config.py` | 66–67 |
| `test_performance.py` | 74–78 |
| `test_3_vortex.py` | 26–46 (155 queries + JOIN) |
## Enterprise CI
Two dedicated jobs added in the companion `openobserve/o2-enterprise` PR
on the same branch:
- **`api_query_agent_vortex`** — runs `test_3_vortex.py` with
`ZO_FILE_FORMAT=vortex`
- **`api_vortex_format_tests`** — runs the full `tests/vortex/` suite
with 4-phase server lifecycle (parquet→vortex→parquet→vortex+compaction)
OSS CI is unchanged except for the ignore rule — no vortex tests run
against the OSS binary.
## Test checklist
- [ ] `tests/vortex/` collects cleanly: `83 tests collected`
- [ ] `tests/query_agent/test_3_vortex.py` collects cleanly: `156 tests
collected`
- [ ] Enterprise CI `api_query_agent_vortex` job passes
- [ ] Enterprise CI `api_vortex_format_tests` job passes
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- ENT CI runners are slower than local/OSS runners — the
`ingest_query_agent_data` fixture was hitting the 120s timeout before
1550 records became searchable, causing all 310 query_agent tests to
fail at setup
- Bumped `wait_until` timeout to 300s
## Test plan
- [ ] OSS `api_query_agent_tests` passes as before
- [ ] ENT `api_query_agent_tests` no longer hits WaitTimeout on ingest
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- [x] Bulk ingestion panic
- [x] Create function panic
- [x] Dashboards/Alerts/Incidents panic when list data
- [x] Match http response error code
- [x] Skip some endpoint that fuzz can't generate data
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).
**Before → After:**
| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |
**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).
## What's in the diff
- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.
## CI workflow changes
- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.
### CI infrastructure fixes
A short hardening pass landed after the initial CI run surfaced several
issues:
- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).
## Backend bugs surfaced (for backend team to track)
The rework uncovered and visibly flagged these as `xfail`/documented
test comments:
1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.
Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).
## Test bugs uncovered + fixed
- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).
## Folder layout
```
tests/api-testing/
├── support/ # NEW framework
│ ├── client.py # OpenObserveClient
│ ├── endpoints/ # 7 per-resource wrappers
│ ├── factories.py # unique_name, time_window, *_payload
│ ├── wait.py # wait_until + WaitTimeout
│ ├── sse.py # read_sse_response
│ └── fixtures.py # client, temp_user_email, temp_stream_name, ...
├── helpers/ # Non-test shared code (was scattered in tests/)
│ ├── sourcemap_helpers.py
│ └── workflow/ # workflow chain + pages/ live here now
├── fixtures/
│ └── rum/ # Standalone RUM ingest scripts (was in tests/)
└── tests/
├── alerts/ # 3 files
├── dashboards/ # 4 files
├── ingest/ # 4 files
├── orgs/ # 2 files
├── pipelines/ # 10 files
├── regression/ # 2 files (env-flag matrix targets)
├── rum/ # 3 files
├── search/ # 8 files
├── streams/ # 1 file
├── users/ # 1 file
├── workflow/ # 3 files
├── conftest.py
└── test_framework_selfcheck.py
```
## Test plan
- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)
## Note on test count
The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add merge_group event trigger to all required CI workflows so they run
when PRs enter the merge queue. Fix github.event.pull_request references
in dispatch-event-enterprise.yml to handle merge_group context.
Add comprehensive API tests for logs-to-logs joins with asymmetric data
sizes:
- 29 test cases covering INNER/LEFT/RIGHT/FULL joins, aggregations,
filters
- Subquery tests with IN/NOT IN, substr, length, CONCAT functions
- Tests run with both ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true and false
- Test data: logs_small.json (10 records) and logs_large.json (50
records)
---------
Co-authored-by: Claude <noreply@anthropic.com>
Fixes#5131
Addresses #5408
### Revamp
- Created a new table to store organizations in the `organizations`
table. Earlier, we never stored organizations in any table, rather we
used to store the organizations in the meta table as part of the `user`
json.
- The users and organization membership relations are now moved to its
own separate tables - `users` and `org_users`.
### New changes
**API changes** -
- `/organizations` - `POST` api to create an org and save it in the meta
table. In non-enterprise version, currently only the root user can
create the org. For enterprise version with rbac enabled, anyone with
`CREATE` access for organizations can create the organization and
becomes an admin. Root user can always create an organization.
- `/{org_id}/rename` - `PUT` api to rename the existing org. Only root
user and the admins of the org can access this API.
The payload for following APIs are changed -
- `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to
`{"role":"new_role"}
- `POST api/{org_id}/users/{user}` -
`{"role":"viewer","custom_role":["role_name"]}` is the new payload where
`custom_role` is optional and is array of custom roles (only works in
enterprise with rbac enabled) which will be assigned to the user. The
value of `role` can be standard role or as well as any custom role (when
it is custom role, the standard role assigned to the user is decided
based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`).
- `POST api/{org_id}/users`- the payload -
```
{"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""}
```
- `PUT api//{org_id}/users/{email_id}` the payload -
```
{"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]}
```
A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION`
(default `false`) which if true, an org will automatically created when
ingesting data to a new org. If false, user can not create org by
ingesting data to new org, they need to first call the `POST
/organizations` API.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
- Added a `cloud` feature for enhanced organizational management
capabilities.
- Introduced new methods for managing organizational users and
asynchronous message processing.
- Enhanced user invitation and membership features, including improved
role handling.
- Implemented configurable ingestion URL support across various
components.
- Added new functionality for user management and organization handling
in the super cluster architecture.
**Improvements**
- Refined user authentication and authorization processes, including
error handling.
- Updated database schema to support multi-organization architecture.
- Enhanced pagination settings and UI components for better user
experience.
- Improved handling of organization names and user roles in
notifications and alerts.
- Streamlined service account token management with visibility
enhancements.
- Enhanced the `AddUpdateOrganization` component with improved user
input handling and validation.
- Updated the `AppOrganizations` component to dynamically load the
appropriate organization management interface.
- Improved the `ServiceAccountsList` component with updated pagination
options and token handling logic.
**Bug Fixes**
- Resolved issues with user role assignments and organization management
inconsistencies.
- Corrected routing and navigation logic for login and logout processes.
- Addressed various error handling scenarios in user and organization
processes.
- Fixed issues with the visibility of service account options in the
`EditGroup` and `EditRole` components based on the cloud configuration.
**Breaking Changes**
- Significant restructuring of user and organization data models.
- Changes to authentication workflows and configuration handling.
- Removal of certain components related to user and organization
management.
- Updated the logic for determining the ingestion URL across various
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Co-authored-by: harshiv-7 <harshiv@openobserve.ai>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Zhao.wei <weizhao029@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Neha PandClaude Opus 4.7
Ashish Kolhe
udev
Yashodhan JoshiandYashodhan Joshi
Subhra264
Simha G
dependabot[bot]
sktrpathi
Ankur Srivastava
