mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 00:09:35 +00:00
4f458a0fab
## What Adds an end-to-end **traces SDR (Sensitive Data Redaction)** test suite, extending SDR coverage from logs to traces. This validates the fix in this PR branch (`feat/fix-traces-sdr-bug`) — SDR regex patterns were previously applied to logs only; these tests assert the same policies now apply to **trace span attributes** at both ingestion and query time. Targets `feat/fix-traces-sdr-bug` so the tests land together with the fix they cover. ## Tests added **6 new spec files → 12 tests** (each spec = 1 `setup: cleanup patterns` + 1 combined functional test), covering **24 field-level policy assertions** (4 sensitive fields × 6 specs). All tagged `@enterprise @sdr @traces`. | # | Spec | Policy | Timing | What it asserts | |---|------|--------|--------|-----------------| | 1 | `tracesIngestionTimeRedaction.spec.js` | Redact | Ingestion | Span-attribute values matching the pattern are stored as `[REDACTED]` | | 2 | `tracesQueryTimeRedaction.spec.js` | Redact | Query | Stored values are untouched; the search response returns `[REDACTED]` | | 3 | `tracesIngestionTimeDrop.spec.js` | Drop | Ingestion | Matching fields are removed from the record at write time | | 4 | `tracesQueryTimeDrop.spec.js` | Drop | Query | Fields present in storage are dropped from the search response | | 5 | `tracesIngestionTimeHash.spec.js` | Hash | Ingestion | Matching values are stored as `[REDACTED:<hash>]` | | 6 | `tracesQueryTimeHash.spec.js` | Hash | Query | Search response returns `[REDACTED:<hash>]`; storage untouched | Each combined test drives the full flow for **4 fields / patterns**: - **email** (`user_email` = `john.doe@example.com`) — `^[\w\.-]+@[\w\.-]+\.\w{2,}$` - **US phone** (`phone` = `5551234567`) — `^\d{10}$` - **credit card** (`cc_number` = `1234 5678 9012 3456`) — `^\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}$` - **SSN** (`ssn` = `123-45-6789`) — `^\d{3}-\d{2}-\d{4}$` (drop/hash specs use `website_url`, `user_password`, `server_ip`, `theme_color` with URL / password / IPv4 / hex-color patterns.) Each functional test follows: ingest → assert field visible (no pattern) → create 4 patterns → associate to the traces stream's fields → ingest/re-query → assert the field is redacted / dropped / hashed. ## How it works Traces reuse the **shared** pattern create/associate/verify flow with logs — only the ingest and read seams differ: - **`sdrTracesIngestionPage.js`** (new): ingests via OTLP `/api/{org}/v1/traces` (`stream-name` header). Each test field is a span attribute that flattens to an identical top-level field; every span also carries an `sdr_test_id` marker for batch isolation. - **`sdrVerificationPage` / `streamAssociationPage`**: parameterized by `streamType` — `?type=traces` search, Traces stream-type tab selection, and schema-field filtering so a field row in the (virtualized, ~20-field) traces schema is reliably found. - **`apiCleanup`**: `fetchStreams` / `deleteStream` / `cleanupStreams` / `isStreamStillDeleting` / `waitForStreamDeletion` take an optional `streamType` (default `'logs'` — no change for existing callers). ## Artifact cleanup - `cleanup.spec.js` sweeps `sdr_traces_*` streams via `streamType: 'traces'` (the logs sweep is `type=logs`-only and never listed them). - `cleanupRegexPatterns` adds the `traces_` prefix. - **Verified live**: the sweep deleted 8 leftover traces streams + 32 traces patterns; 0 remained. ## Verification - All **6 specs / 12 tests pass** against a local enterprise build carrying the fix. - The existing **logs** SDR suite still passes unchanged (no regression from the shared page-object changes). ## Companion CI wiring is a paired o2-enterprise PR (branch `test/traces-sdr-e2e`) that splits the SDR matrix into `SDR-Logs` and `SDR-Traces` shards so the new specs run as their own shard. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Chaitanya Sistla <sistlachaitanya2@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>