windmill/docs/pipeline-metrics-layer.md

# Data metrics: declared measures and dimensions

## What this is

A way to declare, on the DuckDB script that materializes a DuckLake table, what the
table's canonical aggregations are:

```sql
-- materialize ducklake://sales/orders
-- measure revenue = sum(amount) where not is_refund
-- measure order_count = count(*)
-- dimension region = region
-- dimension month = date_trunc('month', ordered_at)
SELECT ...
```

Those declarations are catalogued at deploy and read back by two consumers: the
script editor (a drawer that composes a `SELECT` you can insert) and agents (a tool
that lists what is declared, so they stop guessing).

That is the whole feature. It records definitions and hands them to whoever is
writing the query. It does not rewrite, compile, or execute anything.

## What it deliberately does not do

**It does not enforce anything.** An agent or a person can ignore a declared
measure and write `sum(amount)` without the refund filter, and nothing stops them.
The bet is that most wrong numbers come from not knowing the rule, not from
defying it, and that an informed writer is most of the value.

**It is not a semantic layer.** Declarations are single-table. There is no join
planner and no fan-out-safe aggregation across grains. To combine tables, a
developer materializes a mart with a pipeline step and declares measures on that.

**There is no query-time indirection.** What you deploy is what runs. Compiling a
`{{ metric(...) }}` token into frozen SQL at deploy and substituting it on the
worker would break that: the script in the editor would stop being the script that
executes, and job logs would show SQL appearing nowhere in the source.

## Why no compiler

A compiler buys two things over handing definitions to the caller: byte-identical
SQL with no LLM in the loop, and enforcement. Both matter only once several
independent consumers depend on the same number being reproducible, which is a
situation this codebase has no evidence of yet. Until then the machinery costs a
token grammar, a deploy-time freeze keyed to script hashes, a worker substitution
path, and a second way for the editor and the runtime to disagree.

If that evidence shows up, the catalog is the right foundation to build it on: the
declarations are already structured and already validated.

## Authoring

Both annotations live in the leading comment header, next to `// materialize`:

- `// measure <name> = <aggregate> [where <predicate>]`
- `// dimension <name> = <expression>`

`expr` and `filter` are the author's own SQL, stored verbatim. The predicate is
kept separate from the aggregate rather than folded into it, because a reader
renders it as `expr FILTER (WHERE filter)`. That is what lets two measures with
different predicates sit under one `GROUP BY`; a shared `WHERE` cannot express it.

Declarations attach to the table the script materializes, so a script with no
ducklake `// materialize` target contributes nothing: there is no table to hang
them on.

## The catalog

`data_metric` holds one row per declaration, replaced wholesale for a script path
on every deploy, so it always describes deployed state. It mirrors how the `asset`
table is maintained.

Persisting rather than parsing on demand is what makes folder-scoped listing cheap.
Reading a table's declarations is a point lookup either way, but "everything
declared under `f/analytics`" would otherwise mean fetching and parsing every
script body in the folder on each agent call. Both access patterns are single index
scans:

- `idx_data_metric_table` for the editor drawer
- `idx_data_metric_folder` for the agent tool, using `text_pattern_ops` so a
  `LIKE 'f/analytics/%'` prefix becomes a range scan. This database's collation is
  not `C`, and under a locale collation the planner will not use a default-opclass
  index for prefix matching (measured at 50k rows: seq scan with the default
  opclass, index range scan with `text_pattern_ops`).

### Canonical table paths

A producer's `// materialize ducklake://sales/orders` omits the schema, while a
consumer reading `sales.main.orders` records its asset as `sales/main.orders`.
Both are canonicalized to `<lake>/<schema>.<table>` with `main` as the default, or
a consumer could never resolve the table it reads.

### Authorization

`data_metric` has no RLS. Reads are filtered by an `EXISTS` against `script` on the
authed connection, so `script`'s existing folder, group and user policies decide
what a caller sees. The producing script path is therefore part of the key: a
DuckLake path has no folder to authorize against.

## Consumers

**Script editor drawer.** A Metrics trigger sits in the editor bar of every DuckDB
script, and in the compact Helpers menu below the width threshold. It lists the
tables that declare metrics so they can be browsed, composes a plain `SELECT`
client-side from a measure/dimension selection, and offers three things to do with
it: copy it (a complete query attaching the lake under `dl`), run it in an embedded
REPL against the lake, or append it to the script (reusing whatever alias the
script already attaches, since a repeated `ATTACH` would not run). The output is
ordinary editable SQL with no link back to the catalog.

**Agent tool.** The same endpoint is exposed with `x-mcp-tool`, so an agent can ask
what a table or a folder declares and use the declared `expr`/`filter` instead of
inventing an aggregate.

## Limitations and follow-ups

- Declarations only exist for materialized DuckLake tables. There is no way to
  declare a measure over a Postgres table or an API result.
- The drawer's REPL runs a preview job per execution, so it is a place to check a
  metric rather than a dashboard. There is no charting or filter builder.
- The drawer composes SQL in TypeScript while an agent composes its own. They agree
  semantically because both read the same declaration, but not byte for byte. That
  only becomes a problem if identical SQL is ever required, which is the compiler
  above.
- Deploy hard-rejects three things, all because a reader executes the stored text:
  an unsafe lake/table path and a measure/dimension body that is not a single SQL
  expression (both are stored SQL injection), and a filtered measure whose body is
  not a single aggregate call (`sum(a)/count(b) where …` would apply `FILTER` to
  only part of the expression and silently produce the wrong number). Everything
  else is advisory:
  missing-column and non-aggregate-measure warnings come from the separate
  `check_schema_contracts` endpoint, which the editor calls fire-and-forget on save.
  A CLI or direct-API deploy skips those warnings, and a declaration whose SQL is
  otherwise wrong is only discovered when someone runs it.