With `OTEL_EXPORTER_OTLP_ENDPOINT` or
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` set (and neither
`OTEL_SDK_DISABLED=true` nor `OTEL_TRACES_EXPORTER=none`), Bulwark
loads the OpenTelemetry SDK at startup and exports Next's own spans
plus a client span for every call to Stalwart, WOPI, translation, the
plugin proxy and OAuth over OTLP/HTTP. Calls to Stalwart carry a
`traceparent` header, so Stalwart's spans join the trace. All other
settings come from the standard `OTEL_*` variables.
Every span is scrubbed before export: URLs keep their origin only,
request paths become their route group, query strings, user agents and
client addresses are dropped, and addresses or credentials in other
attributes are removed. Without an endpoint the SDK is never loaded.
The endpoint is listed under the hosts that stay reachable with no
third-party traffic, and System > Monitoring shows whether tracing runs.
A second switch under System > Usage stats, "Also send anonymous error
counts", or `BULWARK_TELEMETRY_ERRORS=on|off`. Off by default, off
whenever the heartbeat is off or no third-party traffic is on, and
switching the heartbeat off switches it off too. With it on the
heartbeat becomes schema 2: the v1 fields unchanged plus `errors`, up to
50 fingerprints of the last complete UTC day with the error name, the
top frames in Bulwark's bundles (path, line, column, debug ID), the
version, a bucketed count, on how many of the last 7 days it occurred,
and a message only when it is a fixed error code. Without the consent
the heartbeat stays exactly schema 1. The admin preview shows the v2
payload as soon as the switch is flipped.
- `/api/metrics` in Prometheus text format, off until `METRICS_TOKEN`
or `METRICS_TOKEN_FILE` is set (404 before, 401 without the bearer
token): requests per route group with status class and latency,
sign-in outcomes, calls to Stalwart, WOPI, translation, the plugin
proxy and OAuth, browser errors, errors today, process memory,
event-loop lag, uptime and build info. Labels come from fixed lists.
Each replica serves its own.
- `/api/health/live` answers while the process runs;
`/api/health/ready` checks the config, the shared store, writable
data dirs and Stalwart's session resource. `/api/health` and
`/api/ready` are unchanged.
- the Docker image gets a HEALTHCHECK on the live endpoint
- adds prom-client, and the OpenTelemetry packages the tracing step
loads on demand
Admins had logs and nothing else: client errors only reached the
browser console. An admin can now set a Sentry-protocol address (DSN)
for Bugsink, GlitchTip or Sentry under System > Monitoring, or with
`ERROR_REPORTING_DSN`, `ERROR_REPORTING_DSN_FILE`,
`ERROR_REPORTING_SAMPLE_RATE` and `ERROR_REPORTING_ENVIRONMENT`.
Bulwark Lite reads the same from `errorReporting` in policy.json.
Everything stays off while no address is set.
- browsers report errors from the error boundaries, window.onerror,
unhandled rejections and failed JMAP requests (method and error type
only), with route and JMAP method breadcrumbs and the chunks' debug IDs
- the server reports what Next hands to onRequestError and what routes
log with logger.error
- every report is scrubbed first: quoted text, addresses, URLs, file
names, tokens and the rest after the first line are removed, and the
server scrubs what browsers post to /api/errors again
- reports are deduplicated and rate-limited per session and process
- errors are counted per day for "errors today" on the admin page
- startup messages and audit entries go through the logger, so
`LOG_FORMAT=json` covers them
directory-keys-shared.test.ts: the store-down case now expects the store error instead of an 'invalid' signature, since a store outage no longer counts as a bad signature.
The marker-write test from d28ee361c now expects the 503 path (a rejected promise) instead of false: refusing with "Token already used" sent admins looking for a replay that never happened, and the jti is now remembered in memory only after it was recorded, so a retry works once the store or state dir is back.
Config, policy, branding, OAuth setup, password and setup wizard writes
used to answer 500 when the shared store couldn't be reached, and the
wizard's token check failed outside any handler. They answer 503 with
Retry-After now, through the helper that already turned a read-only
refusal into 409. The step-session-secret test's wizard mock gains
wizardSessionRefusal, which the setup routes now call.
Fixes from the review of the admin part:
- GET /api/admin/policy reads the policy and its version together; an
identity check waiting on the store let another replica's change pair
the old policy with the new ETag, and a save with it then won.
- A policy save answers with the stored policy and its ETag, and the
policy, organisation, plugins and themes pages continue from that
instead of reading it back from an instance that may not have it yet.
- A stored policy that doesn't parse keeps its version, so a save can
replace it instead of answering 409 forever.
- A reload of another replica's change throws when this replica's writes
kept landing while it read, so the change poller tries again; a write
whose reload raced it reads the document again instead of applying its
older result.
- Dropping a deleted plugin from the policy rereads policy.json first in
local mode, as every other write does.
- New updateAdminConfig(key, change) builds a value on the stored one.
The policy route's test mocks gain getPolicyWithVersion.
The three admin login limiters (5 per address and 50 overall in one
reservation, 5 per known device) keep their numbers and still don't count
refused attempts, but live in the store: memory in this process as before,
or the shared store with several replicas. checkRateLimit and
checkDeviceRateLimit are async now; the login route awaits them and
answers 503 "Sign-in is temporarily unavailable" when the store can't be
reached. The two limiter tests gain await and reset the store's counters,
which vi.resetModules no longer clears.
The admin code tells a store outage by the error's name, since the store
may come from another bundle's copy of the module.
The admin pages save the whole policy from the draft they loaded. A
save from a page opened before another tab's or another admin's save,
or before a deleted plugin was dropped from the policy, used to put the
older policy back. GET /api/admin/policy now sends the policy version as
an ETag; the policy, organisation, plugins and themes pages send it back
as If-Match, and a save based on an older version answers 409 and keeps
the draft on the page. A PUT without If-Match replaces the policy as
before. Deleting a plugin or theme changes only its id in the stored
policy, and a policy import compares against what it replaces.
Test mocks of the config manager gain getPolicyVersion and updatePolicy,
which the route and the extension cleanup now call.
Linus Rath