With `OTEL_EXPORTER_OTLP_ENDPOINT` or
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` set (and neither
`OTEL_SDK_DISABLED=true` nor `OTEL_TRACES_EXPORTER=none`), Bulwark
loads the OpenTelemetry SDK at startup and exports Next's own spans
plus a client span for every call to Stalwart, WOPI, translation, the
plugin proxy and OAuth over OTLP/HTTP. Calls to Stalwart carry a
`traceparent` header, so Stalwart's spans join the trace. All other
settings come from the standard `OTEL_*` variables.
Every span is scrubbed before export: URLs keep their origin only,
request paths become their route group, query strings, user agents and
client addresses are dropped, and addresses or credentials in other
attributes are removed. Without an endpoint the SDK is never loaded.
The endpoint is listed under the hosts that stay reachable with no
third-party traffic, and System > Monitoring shows whether tracing runs.
- `/api/metrics` in Prometheus text format, off until `METRICS_TOKEN`
or `METRICS_TOKEN_FILE` is set (404 before, 401 without the bearer
token): requests per route group with status class and latency,
sign-in outcomes, calls to Stalwart, WOPI, translation, the plugin
proxy and OAuth, browser errors, errors today, process memory,
event-loop lag, uptime and build info. Labels come from fixed lists.
Each replica serves its own.
- `/api/health/live` answers while the process runs;
`/api/health/ready` checks the config, the shared store, writable
data dirs and Stalwart's session resource. `/api/health` and
`/api/ready` are unchanged.
- the Docker image gets a HEALTHCHECK on the live endpoint
- adds prom-client, and the OpenTelemetry packages the tracing step
loads on demand
Admins had logs and nothing else: client errors only reached the
browser console. An admin can now set a Sentry-protocol address (DSN)
for Bugsink, GlitchTip or Sentry under System > Monitoring, or with
`ERROR_REPORTING_DSN`, `ERROR_REPORTING_DSN_FILE`,
`ERROR_REPORTING_SAMPLE_RATE` and `ERROR_REPORTING_ENVIRONMENT`.
Bulwark Lite reads the same from `errorReporting` in policy.json.
Everything stays off while no address is set.
- browsers report errors from the error boundaries, window.onerror,
unhandled rejections and failed JMAP requests (method and error type
only), with route and JMAP method breadcrumbs and the chunks' debug IDs
- the server reports what Next hands to onRequestError and what routes
log with logger.error
- every report is scrubbed first: quoted text, addresses, URLs, file
names, tokens and the rest after the first line are removed, and the
server scrubs what browsers post to /api/errors again
- reports are deduplicated and rate-limited per session and process
- errors are counted per day for "errors today" on the admin page
- startup messages and audit entries go through the logger, so
`LOG_FORMAT=json` covers them
Linus Rath