Compare commits

...
Author SHA1 Message Date
Diego ImbertandClaude Opus 4.8 1d1ffed01c [ee] feat: datatable permissions via Postgres roles + RLS
Opt-in, EE-gated permissions for data tables, enforced natively by Postgres
roles and row-level security. Non-admin queries connect directly as a per-user
login role (tamper-proof against arbitrary user SQL); admins connect as the
owner and keep full access. Adds a Permissions modal (Access + Row policies
tabs) to the data table settings, get/set/sync endpoints, and enforcement in
both the postgresql and DuckDB executors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 13:59:16 +02:00
Diego ImbertandClaude Opus 4.8 e896506ae6 fix(cli): run deployed datatable migrations after workspace merge
The merge command upserted datatable_migration definitions into the target
workspace and reported the item as successfully deployed, but never ran the
migrations. For forked datatables backed by separate databases, this left the
target schema unchanged until someone manually ran `wmill datatable migrate up`,
while the CLI reported a successful merge.

Collect the datatable migrations deployed (not deleted) into the target and,
after the deploy loop, offer to run them via the existing offerToRunNewMigrations
helper — the same post-deploy run prompt the push/sync path uses (interactive
only; `--yes`/non-TTY skip the mutating run, matching push behavior). Export
parseDatatableMigrationDeployPath so the merge path can parse the deployed items.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 14:17:23 +02:00
Diego Imbert 209e150048 fix: record per-migration deployment on datatable migrations disable 2026-07-06 14:16:56 +02:00
Diego Imbert b9546b66a2 fix: keep renamed datatable migrations visible in compare view 2026-07-06 13:56:00 +02:00
Diego ImbertandClaude Opus 4.8 b2643d8b63 fix(frontend): warn before running a newly-created datatable migration out of order
The row-level Run action warns when earlier migrations are still pending, but
the create-and-run paths ran a just-created migration with `only` directly,
applying it ahead of older pending migrations without that confirmation.

Reuse the same "Run migration out of order" confirmation across all
create-and-run paths via a shared helper (datatableMigrationUtils):
- NewDataTableMigrationModal "Create and run" (and the DDL guard path)
- DatatableSchemaDiff fork→parent merge
- dbOps schema ops (DB manager create/alter/drop) — the pure factory throws a
  MigrationRunCancelled sentinel on decline, which DBTableEditor treats as a
  silent cancel

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 13:48:27 +02:00
Diego Imbert 78b68bee33 feat: add Enter shortcut to Create-a-migration in the DDL guard 2026-07-06 13:31:20 +02:00
Diego Imbert 65ead15f3a "See migration" button in the toast 2026-07-06 13:10:51 +02:00
Diego Imbert fe8e40f23a fix: surface full error detail in Database Manager DDL/query errors 2026-07-06 10:52:38 +02:00
Diego ImbertandClaude Opus 4.8 5403d1d340 fix: fail closed when applied status unreadable on datatable migration rewrite
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 10:36:11 +02:00
Diego Imbert cfb58c6562 Merge remote-tracking branch 'origin/main' into change-68b704f7 2026-07-06 10:34:02 +02:00
Diego ImbertandClaude Opus 4.8 f41e4e7f52 fix(cli): datatable migrate up/down default to main datatable, not all
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 09:58:13 +02:00
Diego Imbert 896aae6794 Merge remote-tracking branch 'origin/main' into change-68b704f7
# Conflicts:
#	backend/ee-repo-ref.txt
#	cli/src/commands/sync/sync.ts
2026-07-06 09:35:54 +02:00
Diego ImbertandClaude Opus 4.8 121e1a2906 fix(frontend): add missing @emnapi/core and @emnapi/runtime lockfile entries
Resolves npm ci EUSAGE failure: the optional cpu:wasm32 @rolldown/binding-wasm32-wasi
declares deps on @emnapi/core@1.11.2 and @emnapi/runtime@1.11.2 that had no resolved
lockfile entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 09:31:45 +02:00
Diego ImbertandClaude Opus 4.8 affcea9f17 fix: reject rewriting an already-applied datatable migration on upsert
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 08:35:59 +02:00
Diego Imbert e9775afbd3 Merge branch 'main' into change-68b704f7 2026-07-04 00:36:55 +02:00
Diego Imbert 79b265e11a fix: handle datatable migration renames on push and dedupe timestamps 2026-07-04 00:35:50 +02:00
Diego Imbert a3c553c6a6 nit 2026-07-04 00:26:29 +02:00
Diego Imbert aba8e66df8 nit 2026-07-04 00:21:51 +02:00
Diego ImbertandClaude Opus 4.8 89d5cf7bcb chore(frontend): use windmill-utils-internal 1.8.2 for migration diff drawer
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 00:19:18 +02:00
Diego ImbertandClaude Opus 4.8 260d0c2d56 fix: return datatable migration SQL from getItemValue for the diff drawer
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 00:17:33 +02:00
Diego ImbertandClaude Opus 4.8 fcf8629f92 refactor: drop unreleased _wm_migrations legacy-upgrade handling
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 00:12:16 +02:00
Diego ImbertandClaude Opus 4.8 1555ea0df9 fix(system_prompts): resolve nested local command groups in CLI docs generator
The CLI docs generator anchored on the first `new Command()` in a file and
never resolved locally-defined command groups passed as
`.command("name", localCmd)`. For datatable this flattened the nested
`migrate` group: it emitted `datatable new/up/down` plus a bare
`datatable migrate`, and mislabeled the datatable command with the migrate
group's description. jobs was broken the same way (its description was pull's,
and pull/push rendered empty).

Anchor block extraction on the `export default`ed command, recurse into
locally-defined `const x = new Command()` groups mounted as subcommands, and
render nested sub-subcommands. Regenerated docs now show
`datatable migrate new/up/down` and `jobs pull/push` with their real
options.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 00:11:34 +02:00
Diego ImbertandClaude Opus 4.8 12b9811e3c fix: scope datatable _wm_migrations by data table and cascade renames/deletes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 00:07:32 +02:00
Diego ImbertandClaude Opus 4.8 e75a2f8711 fix(datatable-migrations): validate persisted data table names as path segments
edit_datatable_config only validated rename segments, not the actual
settings.datatables keys, so a data table could be saved directly under a name
like '..' or one containing '/'. Since new tables default to
migrations_enabled = true, generate_initial_datatable_migration would then
insert a migration row and the sync export would build
migrations/datatable/<name>/... paths from that name, producing malformed or
directory-escaping export paths.

Validate every persisted data table name in edit_datatable_config (alongside
the existing rename checks) and add validate_datatable_path_segment to
generate_initial_datatable_migration for defense in depth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 23:52:46 +02:00
Diego Imbert a84aaffea3 prevent initial_datatable_migration if migrations already exist 2026-07-03 23:40:48 +02:00
Diego Imbert 0f78a202fc nit CI emnapi/core version 2026-07-03 23:39:29 +02:00
Diego Imbert 4b040b1097 fix: fail closed on migrations-status error in fork schema merge 2026-07-03 23:33:59 +02:00
Diego Imbert 50a75d042f nit npm publish 2026-07-03 23:28:16 +02:00
Diego Imbert 1684f6cf28 Merge branch 'main' into change-68b704f7 2026-07-03 23:17:30 +02:00
Diego ImbertandClaude Opus 4.8 2919efef62 fix(frontend): sync package-lock with npm 11 peer-dep resolution
npm ci failed with 'Missing: @emnapi/core@1.11.2 / @emnapi/runtime@1.11.2 from
lock file'. @napi-rs/wasm-runtime declares @emnapi/core|runtime ^1.7.1 as
peerDependencies while @rolldown/binding-wasm32-wasi pins them to exactly
1.10.0. Newer npm (bundled with node 24 in CI) installs the peer deps at the
highest match (1.11.2) alongside rolldown's nested 1.10.0, so the ideal tree
needs both versions; the committed lock only had 1.10.0.

Regenerate the lock with npm 11.18 so it carries both 1.11.2 (top-level, for
the peer deps) and 1.10.0 (nested, for rolldown's pin). Verified npm ci passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 22:18:37 +02:00
Diego Imbert f68e4882e4 feat: deploy datatable migrations on merge with explicit opt-in error 2026-07-03 21:29:36 +02:00
Diego Imbert 150a96363d fix: handle dollar-quoting and comments when splitting SQL statements 2026-07-03 21:18:14 +02:00
Diego Imbert 38d4446be3 nit 2026-07-03 21:15:48 +02:00
Diego ImbertandClaude Opus 4.8 e3b36fcead fix(datatable-migrations): never resolve instance credentials into migration job args
datatable_database_arg eagerly resolved instance data-table credentials
(including the shared instance-wide Postgres password) and passed them as the
migration job's plaintext `database` arg, landing in v2_job.args. Since the
run route has no admin gate, a non-admin could run a migration and read
args.database to recover the password, granting cross-workspace psql access to
all instance data-table DBs.

Pass a `datatable://<name>` reference for both resource-backed and instance
data tables instead; the pg executor already resolves it to real credentials
server-side at run time, so nothing sensitive is ever stored in the job args.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 20:08:38 +02:00
Diego ImbertandClaude Opus 4.8 54ed70000d chore(frontend): sync package-lock with package.json (@emnapi deps)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 20:03:42 +02:00
Diego Imbert 12a27c3e6e Merge remote-tracking branch 'origin/main' into change-68b704f7 2026-07-03 19:19:32 +02:00
Diego Imbert c4c75f3270 ee-repo-ref 2026-07-03 19:19:24 +02:00
Diego Imbert ec88990fec split 2026-07-03 19:17:46 +02:00
Diego ImbertandClaude Opus 4.8 afe025b886 fix: run DDL migration guard on the script editor Test button
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 18:00:54 +02:00
Diego ImbertandClaude Opus 4.8 072f2bf053 chore(datatable-migrations): remove unused update_datatable_migrations endpoint
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 17:52:45 +02:00
Diego Imbert f8c216d301 docs: fix generate_initial migration ordering comment to match code 2026-07-03 17:45:58 +02:00
Diego ImbertandClaude Opus 4.8 fc006e7294 fix(db-manager): fail closed when migrations-status check errors on DDL apply
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 17:44:42 +02:00
Diego ImbertandClaude Opus 4.8 91021816e7 fix(datatable-migrations): serialize run/rollback with a per-db advisory lock
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 17:41:19 +02:00
Diego Imbert 12b682ef13 windmill-utils-internal 2026-07-03 17:38:10 +02:00
Diego Imbert 2113e87495 Merge remote-tracking branch 'origin/main' into change-68b704f7
# Conflicts:
#	backend/ee-repo-ref.txt
#	frontend/src/lib/components/CompareWorkspaces.svelte
2026-07-03 14:41:46 +02:00
Diego Imbert 154f31a054 clone migrations on fork 2026-07-03 14:10:30 +02:00
Diego Imbert 7612d7b712 nit 2026-07-03 14:05:43 +02:00
Diego Imbert 7eac2577b6 feat: record fork merge as a migration when target datatable opts in 2026-07-03 14:00:59 +02:00
Diego ImbertandClaude Opus 4.8 477ba39372 feat: offer revert-and-delete when deleting an installed migration
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 13:45:21 +02:00
Diego ImbertandClaude Opus 4.8 05bf5b88b0 refactor: drop random slug from auto-generated migration names
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 13:35:30 +02:00
Diego ImbertandClaude Opus 4.8 bc53ca49fc feat: flag invalid migration name with red border, not just empty
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 13:32:09 +02:00
Diego ImbertandClaude Opus 4.8 54aed77ab7 feat(cli): push local datatable migrations before running on migrate up
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 13:29:52 +02:00
Diego Imbert a3f0b82f70 BEGIN/END scaffold in CLI 2026-07-03 12:13:48 +02:00
Diego Imbert 42a7c70dd4 refresh db manager schema on migrations 2026-07-03 11:52:21 +02:00
Diego Imbert 8470066599 border nits 2026-07-03 11:03:27 +02:00
Diego ImbertandClaude Opus 4.8 c3bb639624 feat(datatable-migrations): allow non-admins to create/run/revert migrations, gate only opt in/out
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 10:55:03 +02:00
Diego ImbertandClaude Opus 4.8 adb88e80e5 feat: autofocus migration name input and flag it red when empty
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 10:48:41 +02:00
Diego Imbert d8ed13510c BEGIN/END placeholder in down migration 2026-07-03 10:46:06 +02:00
Diego ImbertandClaude Opus 4.8 fadd515abe feat(db-manager): add Migrations button to top bar, make Refresh icon-only
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 10:39:46 +02:00
Diego ImbertandClaude Opus 4.8 6e876a9437 chore(windmill-utils-internal): bump to 1.7.1 for datatable migration deploy provider methods
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 15:55:00 +02:00
Diego ImbertandClaude Opus 4.8 74a7c843e7 refactor: move datatable migration rename/delete cascade into module
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 15:48:41 +02:00
Diego Imbert cdfd7b3d61 Refactor + handle datatable setting delete/rename 2026-06-30 13:42:52 +02:00
Diego Imbert da04ffdcc0 Merge remote-tracking branch 'origin/main' into change-68b704f7
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-api-workspaces/src/workspaces.rs
2026-06-29 16:55:41 +02:00
Diego ImbertandClaude Opus 4.8 ece1cd5800 feat: deploy and run datatable migrations on workspace merge
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 16:53:18 +02:00
Diego Imbert 26629f7dff fix: harden datatable migration version allocation and initial-migration bookkeeping, add tests 2026-06-22 15:58:28 +02:00
Diego Imbert cdc7750248 Merge remote-tracking branch 'origin/main' into change-68b704f7
# Conflicts:
#	backend/ee-repo-ref.txt
#	cli/src/commands/datatable/datatable.ts
#	cli/src/guidance/skills.gen.ts
#	system_prompts/auto-generated/cli/cli-commands.md
#	system_prompts/auto-generated/prompts.ts
#	system_prompts/auto-generated/skills/cli-commands/SKILL.md
2026-06-20 18:24:10 +02:00
Diego Imbert 8484c58559 chore: point ee-repo-ref at datatable_migration git-sync companion 2026-06-20 18:20:52 +02:00
Diego Imbert 9e6c4c88e7 feat: add datatable_migration kind to workspace diff pipeline 2026-06-20 18:16:27 +02:00
Diego Imbert 3680972816 compare paeg 2026-06-20 17:20:06 +02:00
Diego Imbert e6f6e55f6b feat: explain migrations purpose with a tooltip in the migrations modal 2026-06-20 16:55:53 +02:00
Diego Imbert ab9071ad39 fix: correct down migration for db manager alters (no double-wrap, serial) 2026-06-20 16:55:24 +02:00
Diego Imbert ad074bd405 feat: generate down migrations for db manager DDL (postgres) 2026-06-20 16:46:23 +02:00
Diego Imbert 40cc718feb fix: don't re-prompt DDL guard when creating a migration without running 2026-06-20 16:40:02 +02:00
Diego Imbert ee65da8d83 fix: make migration view editor read-only so its code can scroll 2026-06-20 16:14:53 +02:00
Diego Imbert 32732e599b feat: make datatable migrations opt-in per data table 2026-06-20 15:44:18 +02:00
Diego Imbert 56202da8b9 feat: record db manager DDL on data tables as migrations 2026-06-20 15:33:52 +02:00
Diego Imbert 4ead9cfb04 feat: refresh datatable schema after running a migration from the SQL REPL 2026-06-20 15:30:11 +02:00
Diego Imbert f066eac61d fix: re-prompt instead of stripping DDL when new-migration modal is cancelled 2026-06-20 15:10:01 +02:00
Diego Imbert cc0415b191 fix: ensure detected DDL ends with semicolon when wrapped in transaction 2026-06-20 14:54:00 +02:00
Diego Imbert d19d58ebc4 fix: validate datatable migration name and datatable at the API boundary 2026-06-20 14:51:23 +02:00
Diego Imbert f97a0b2f60 feat: default new datatable migration to a BEGIN/END transaction template 2026-06-20 14:45:00 +02:00
Diego Imbert 68d82dbe54 chore: regenerate CLI docs for datatable migrate commands 2026-06-20 14:41:44 +02:00
Diego Imbert e98dcd3579 fix: surface postgres error detail when datatable migrations fail to run 2026-06-18 23:34:55 +02:00
Diego Imbert d234001e63 feat: datatable migrate up/down default to all datatables, --datatable to target one 2026-06-18 21:07:35 +02:00
Diego Imbert 49a476fec7 feat: reject invalid datatable migrations on sync push 2026-06-18 21:04:49 +02:00
Diego Imbert dbdd09b9a2 feat: run datatable migrations as user-permissioned labeled jobs 2026-06-18 20:07:02 +02:00
Diego Imbert 52ef65c55e fix: exclude datatable migration sql files from script metadata generation 2026-06-18 19:44:16 +02:00
Diego Imbert b8c8faf629 fix: drop redundant datatable_migration label in sync output 2026-06-18 13:58:09 +02:00
Diego Imbert ed7586d61c refactor: move datatable migrations to migrations/datatable/ path 2026-06-18 10:33:25 +02:00
Diego Imbert bd5a047169 feat: sync datatable migrations as files via the workspace export 2026-06-17 19:52:28 +02:00
Diego Imbert 541a1407e0 fix: include postgres error detail in migration run/rollback failures 2026-06-17 19:35:17 +02:00
Diego Imbert 84078f33ce fix: revert created migration if create-and-run fails to run 2026-06-17 19:27:17 +02:00
Diego Imbert 2f15f39f41 fix: surface datatable migration API error details in toasts 2026-06-17 19:25:42 +02:00
Diego Imbert 7859ebfb38 feat: generate initial datatable migration via pg_dump 2026-06-17 19:24:26 +02:00
Diego Imbert d4f0083501 fix: avoid migrations list flicker on refresh after an action 2026-06-17 19:18:13 +02:00
Diego Imbert a79b9e7c43 feat: per-row revert button with out-of-order warning 2026-06-17 19:08:24 +02:00
Diego Imbert aa6ed90b60 feat: view migration content, run single migration, fix stacked modal 2026-06-17 18:36:42 +02:00
Diego Imbert 4de2dd6d59 feat: support running a single specific datatable migration 2026-06-17 18:34:52 +02:00
Diego Imbert 95ee90fc5a feat: prompt to create migration on DDL in datatable SQL editors 2026-06-17 18:14:29 +02:00
Diego Imbert a18fa7ccd6 feat: add datatable migrations management UI 2026-06-17 15:18:09 +02:00
Diego Imbert 6e1ce7fd4e feat: add datatable migrate new command to scaffold migrations 2026-06-17 11:47:47 +02:00
Diego Imbert b3002f5e24 feat: add datatable migrate up/down commands and post-push run prompt 2026-06-17 11:29:06 +02:00
Diego Imbert 1f48a470ba feat: sync datatable migrations as .up.sql/.down.sql files 2026-06-17 11:02:43 +02:00
Diego Imbert 5f2cb62527 feat: add route to run datatable migrations 2026-06-17 10:29:56 +02:00
Diego Imbert 6ba02002cd feat: add datatable_migrations table 2026-06-17 10:20:51 +02:00
83 changed files with 6767 additions and 478 deletions
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = ANY($2::text[])",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": []
},
"hash": "00e9fc3fed9379264881c58df9404ab3569a0611b33c261602d62e9c847c583e"
}
@@ -46,11 +46,11 @@
]
},
"nullable": [
false,
false,
false,
false,
false,
true,
true,
true,
true,
true,
true,
true
]
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS(SELECT 1 FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "exists",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "15c0584e9eb078442f6928adb894324c47e010e3554c14e755a7e045453745bb"
}
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT ws.datatable->'datatables' AS datatable_name\n FROM workspace_settings ws\n WHERE ws.workspace_id = $1\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable_name",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "16a67b92dbd32024838983184e6974f2ce577b78decd6b821096c9f2f252ae8b"
}
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE datatable_migrations SET datatable = $3 WHERE workspace_id = $1 AND datatable = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "2c543583bcf7bcaf2f422638ce0741a2e193f18ffa11d08e8ca49cef5c3b850c"
}
@@ -0,0 +1,24 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3 RETURNING name",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
false
]
},
"hash": "3c057e0284f1219865b8f7dc6eb3c1293c4b72d0b52abeefd1e954617984a2d8"
}
@@ -0,0 +1,36 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "code_up",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "code_down",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "44c4e9848bc97b66a49d5454d30d40dfe1ffe51839884f3e9e9360d9e17b5afd"
}
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 RETURNING timestamp, name",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "timestamp",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "name",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "564f4bdf43135c603518ef35a3e95bde7f479877018980cffe1c1e9d34aad59e"
}
@@ -15,7 +15,7 @@
]
},
"nullable": [
true
null
]
},
"hash": "5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55"
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": []
},
"hash": "5b67c3af6477d7029d118ac259a7535d4d962c328a060f33c07191ac3e033e82"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ws.datatable->'datatables'->$2 FROM workspace_settings ws WHERE ws.workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "5cca3509374b1ddd8707c930bc382734e4ca6cad2d849f75a8f8a249f83df83f"
}
@@ -0,0 +1,30 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, code_down FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 AND timestamp = $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "code_down",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
false,
true
]
},
"hash": "6ac00d65b3b7707cba9456171d4332c696e12b52066a5ae2ab10fe3b5bd0f3d0"
}
@@ -0,0 +1,19 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down) VALUES ($1, $2, $3, $4, $5, $6)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Varchar",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "760908f44cafb500e4ba2515d1b030390d9510e73e9b7df347b697d6dc7aefe6"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT (ws.datatable->'datatables'->$2->>'migrations_enabled')::boolean FROM workspace_settings ws WHERE ws.workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "bool",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "798dc8ce1c80b5ebd8120f55b6c9f909bf4babb29910514753cb822dde4e7ca9"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT MAX(timestamp) FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "max",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "95e370a82ef46d9310f77a99b437a20a5773113e290d69a016363543067baf14"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings SET datatable = jsonb_set(datatable, ARRAY['datatables', $2::text, 'migrations_enabled'], 'false'::jsonb) WHERE workspace_id = $1 AND jsonb_exists(datatable->'datatables', $2) RETURNING 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "9eea74f68a0c99d5ce6601fd854c1d18df0ab30f528ef4ce7910c43dd44f4cfb"
}
@@ -0,0 +1,46 @@
{
"db_name": "PostgreSQL",
"query": "SELECT datatable, timestamp, name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 ORDER BY datatable, timestamp ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "timestamp",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "code_up",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "code_down",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false,
false,
false,
true
]
},
"hash": "a8cec061756c7be61af829f4a5207ba6333ad5feb4557a9341427e9e68608025"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down)\n SELECT $2, datatable, timestamp, name, code_up, code_down\n FROM datatable_migrations WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "aef927110ef4cff51d3faabb0c389730dd215f4e90d105f0e86025f0ac42f020"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT pg_advisory_xact_lock(hashtext($1), hashtext($2))",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "pg_advisory_xact_lock",
"type_info": "Void"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "b3200181380df2f645bfce270dc1cf520939fa4fa4b65371d11db7416ba0b14c"
}
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down) VALUES ($1, $2, $3, 'initial', $4, NULL)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Text"
]
},
"nullable": []
},
"hash": "b96c1a720654ae8ae52e8098cc18d887920beaec97696a17da020a2adfb052f0"
}
@@ -0,0 +1,19 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO datatable_migrations (workspace_id, datatable, timestamp, name, code_up, code_down) VALUES ($1, $2, $3, $4, $5, $6) ON CONFLICT (workspace_id, datatable, timestamp) DO UPDATE SET name = EXCLUDED.name, code_up = EXCLUDED.code_up, code_down = EXCLUDED.code_down",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Int8",
"Varchar",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "c5639d48c92d6863f16f97de91eae590d0ed2f4b7831956bb429a47c478f7c1f"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ws.datatable->'datatables' FROM workspace_settings ws WHERE ws.workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "c9d9c08505e69790154876f50f90d503a92e44e291a76a53ef09ded619edfacb"
}
@@ -0,0 +1,30 @@
{
"db_name": "PostgreSQL",
"query": "SELECT datatable, timestamp FROM datatable_migrations WHERE workspace_id = $1 AND datatable = ANY($2) AND timestamp = ANY($3)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "timestamp",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"TextArray",
"Int8Array"
]
},
"nullable": [
false,
false
]
},
"hash": "cf2e74dcd0992f22eb3b62995fd0099ee5f46dafba8f323cf002e329ac69d1ac"
}
@@ -0,0 +1,41 @@
{
"db_name": "PostgreSQL",
"query": "SELECT timestamp, name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 ORDER BY timestamp ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "timestamp",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "code_up",
"type_info": "Text"
},
{
"ordinal": 3,
"name": "code_down",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false,
false,
true
]
},
"hash": "e5d8d46b9431033fa7572a880ee0e74e14afde3c02000a88e696d7b6adcad0d6"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings SET datatable = jsonb_set(datatable, ARRAY['datatables', $2::text, 'migrations_enabled'], 'true'::jsonb) WHERE workspace_id = $1 AND jsonb_exists(datatable->'datatables', $2) RETURNING 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "ec8e502f9c2926e578e02cd164a7a3e43ef12b97d55353f86a9a368617efccca"
}
@@ -0,0 +1,46 @@
{
"db_name": "PostgreSQL",
"query": "SELECT datatable, timestamp, name, code_up, code_down FROM datatable_migrations WHERE workspace_id = $1 ORDER BY datatable, timestamp",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "timestamp",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "code_up",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "code_down",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false,
false,
false,
true
]
},
"hash": "f89b024949f29eb5142744635914b94fd84c1bb64e9659f798df2888848894bd"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job SET labels = (\n SELECT array_agg(DISTINCT l)\n FROM unnest(coalesce(labels, ARRAY[]::TEXT[]) || $2) l\n ) WHERE id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"TextArray"
]
},
"nullable": []
},
"hash": "fc59d9b911529de75c466593252358581f5716b3b58dc2e6b9b9282c50b1b66b"
}
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "SELECT timestamp, name, code_up FROM datatable_migrations WHERE workspace_id = $1 AND datatable = $2 ORDER BY timestamp ASC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "timestamp",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "code_up",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false,
false
]
},
"hash": "fcb0cf55f5c9047066a6fdd62bf6f85238cbf189ccf18191493d6357e269d12d"
}
+1
View File
@@ -14473,6 +14473,7 @@ dependencies = [
"sqlx",
"strum",
"tokio",
"tokio-postgres",
"tracing",
"uuid",
"windmill-api-auth",
+1 -1
View File
@@ -1 +1 @@
0de2412ff0734b11e12ba378c9bcc373ff9ae800
71701991e6ab76c5b3033a6c765ab246a50c35dd
@@ -0,0 +1 @@
DROP TABLE datatable_migrations;
@@ -0,0 +1,21 @@
-- SQL migrations defined per data table within a workspace.
-- `datatable` is the target data table name, `name` is the migration name
-- (e.g. add_index_to_customers), and `timestamp` is the migration version
-- (YYYYMMDDHHMMSS), recorded as `version` in the data table's `_wm_migrations`
-- table once applied.
CREATE TABLE datatable_migrations (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,
datatable VARCHAR(255) NOT NULL,
timestamp BIGINT NOT NULL,
name VARCHAR(255) NOT NULL,
code_up TEXT NOT NULL,
code_down TEXT,
PRIMARY KEY (workspace_id, datatable, timestamp)
);
-- No standalone index: the (workspace_id, datatable, timestamp) primary-key btree
-- already serves both `WHERE workspace_id = $1` and `WHERE workspace_id = $1 AND
-- datatable = $2` lookups via its leading columns.
GRANT ALL ON datatable_migrations TO windmill_user;
GRANT ALL ON datatable_migrations TO windmill_admin;
@@ -47,6 +47,7 @@ serde_json.workspace = true
sha2.workspace = true
sqlx.workspace = true
tokio.workspace = true
tokio-postgres.workspace = true
tracing.workspace = true
uuid.workspace = true
strum.workspace = true
File diff suppressed because it is too large Load Diff
@@ -1,3 +1,4 @@
pub mod datatable_migrations;
pub mod deployment_requests;
pub mod workspaces;
pub mod workspaces_extra;
+239 -26
View File
@@ -43,9 +43,10 @@ use windmill_common::workspaces::GitRepositorySettings;
#[cfg(feature = "enterprise")]
use windmill_common::workspaces::WorkspaceDeploymentUISettings;
use windmill_common::workspaces::{
check_deploy_rules, check_user_against_rule, get_datatable_resource_from_db_unchecked,
validate_dev_workspace_id, validate_fork_workspace_id, validate_workspace_name, DataTable,
DataTableCatalogResourceType, DataTableForkBehavior, ProtectionRuleKind, ProtectionRules,
check_deploy_rules, check_user_against_rule, get_datatable_from_db,
get_datatable_resource_from_db_unchecked, validate_dev_workspace_id,
validate_fork_workspace_id, validate_workspace_name, DataTable, DataTableCatalogResourceType,
DataTableForkBehavior, DataTablePermissions, ProtectionRuleKind, ProtectionRules,
ProtectionRuleset, RuleCheckResult, WorkspaceGitSyncSettings, DEV_WORKSPACE_LOCK_RULE_NAME,
};
use windmill_common::workspaces::{Ducklake, DucklakeCatalogResourceType};
@@ -132,6 +133,19 @@ pub fn workspaced_service() -> Router {
get(get_datatable_table_schema),
)
.route("/edit_datatable_config", post(edit_datatable_config))
.route(
"/datatable_permissions/{datatable_name}",
get(get_datatable_permissions),
)
.route(
"/set_datatable_permissions/{datatable_name}",
post(set_datatable_permissions),
)
.route(
"/sync_datatable_permissions/{datatable_name}",
post(sync_datatable_permissions),
)
.merge(crate::datatable_migrations::routes())
.route("/git_sync_enabled", get(get_git_sync_enabled))
.route("/edit_git_sync_config", post(edit_git_sync_config))
.route("/edit_git_sync_repository", post(edit_git_sync_repository))
@@ -429,6 +443,12 @@ pub struct DucklakeSettings {
#[derive(Deserialize, Debug)]
struct EditDataTableConfig {
settings: DataTableSettings,
// Data table renames (old -> new) and deletions, tracked client-side by a
// stable id, so we can cascade or drop each data table's migrations.
#[serde(default)]
renames: Vec<crate::datatable_migrations::DatatableRename>,
#[serde(default)]
deleted_datatables: Vec<String>,
}
#[derive(Deserialize, Serialize, Debug)]
@@ -1953,8 +1973,8 @@ pub(crate) async fn resolve_pg_source_checked(
}
/// A temporary file for pg_dump output that is automatically deleted when dropped.
struct DumpFile {
path: std::path::PathBuf,
pub(crate) struct DumpFile {
pub(crate) path: std::path::PathBuf,
}
impl DumpFile {
@@ -2001,7 +2021,11 @@ impl Drop for DumpFile {
/// Run pg_dump against a PgDatabase, writing output to a temp file on disk.
/// Returns a DumpFile handle; the file is deleted when the handle is dropped.
async fn pg_dump_database(pg_db: &PgDatabase, schema_only: bool) -> Result<DumpFile> {
pub(crate) async fn pg_dump_database(
pg_db: &PgDatabase,
schema_only: bool,
exclude_tables: &[&str],
) -> Result<DumpFile> {
let dump_file = DumpFile::new()?;
let host = &pg_db.host;
@@ -2014,6 +2038,9 @@ async fn pg_dump_database(pg_db: &PgDatabase, schema_only: bool) -> Result<DumpF
if schema_only {
cmd.arg("--schema-only");
}
for table in exclude_tables {
cmd.arg(format!("--exclude-table={table}"));
}
cmd.arg("--host")
.arg(host)
.arg("--port")
@@ -2237,7 +2264,7 @@ async fn import_pg_database(
}
windmill_common::validate_dbname(&target_pg.dbname)?;
let dump_file = pg_dump_database(&source_pg, schema_only).await?;
let dump_file = pg_dump_database(&source_pg, schema_only, &[]).await?;
pg_import_dump(&target_pg, &dump_file).await?;
Ok(format!(
@@ -2259,7 +2286,7 @@ async fn export_pg_schema(
Json(req): Json<ExportPgSchemaRequest>,
) -> Result<String> {
let pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let dump_file = pg_dump_database(&pg, true).await?;
let dump_file = pg_dump_database(&pg, true, &[]).await?;
tokio::fs::read_to_string(&dump_file.path)
.await
.map_err(|e| Error::internal_err(format!("Failed to read dump file: {}", e)))
@@ -2396,13 +2423,60 @@ async fn edit_datatable_config(
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
ApiAuthed { is_admin, username, email, .. }: ApiAuthed,
Json(new_config): Json<EditDataTableConfig>,
Json(mut new_config): Json<EditDataTableConfig>,
) -> Result<String> {
require_admin(is_admin, &username)?;
let is_superadmin = require_super_admin(&db, &email).await.is_ok();
let mut tx = db.begin().await?;
let old_datatables: HashMap<String, DataTable> = serde_json::from_value(
sqlx::query_scalar!(
"SELECT ws.datatable->'datatables' FROM workspace_settings ws WHERE ws.workspace_id = $1",
&w_id
)
.fetch_one(&db)
.await?
.unwrap_or(serde_json::Value::Null),
)
.unwrap_or_default();
// Validate every persisted data table name and rename segment before
// touching anything, since they become directory segments in migration
// storage/export keys (`migrations/datatable/<name>/...`).
for name in new_config.settings.datatables.keys() {
crate::datatable_migrations::validate_datatable_path_segment(name)?;
}
for r in &new_config.renames {
crate::datatable_migrations::validate_datatable_path_segment(&r.from)?;
crate::datatable_migrations::validate_datatable_path_segment(&r.to)?;
}
// Map new name -> old name so a renamed data table inherits the previous
// flag instead of being treated as brand new.
let rename_src: HashMap<&str, &str> = new_config
.renames
.iter()
.map(|r| (r.to.as_str(), r.from.as_str()))
.collect();
// Migrations opt-in and advanced permissions are owned by their dedicated
// endpoints, not this config form: preserve each existing data table's state
// (default brand-new data tables to migrations-enabled / no permissions) so a
// plain config save can neither flip them nor desync the provisioned roles.
for (name, dt) in new_config.settings.datatables.iter_mut() {
let lookup = rename_src
.get(name.as_str())
.copied()
.unwrap_or(name.as_str());
let old = old_datatables.get(lookup);
dt.migrations_enabled = match old {
Some(old) => old.migrations_enabled,
None => Some(true),
};
dt.permissions = old.and_then(|old| old.permissions.clone());
}
let args_for_audit = format!("{:?}", new_config.settings);
audit_log(
&mut *tx,
@@ -2417,19 +2491,6 @@ async fn edit_datatable_config(
// Check that non-superadmins are not abusing Instance databases
if !is_superadmin {
let old_datatables = sqlx::query_scalar!(
r#"
SELECT ws.datatable->'datatables' AS datatable_name
FROM workspace_settings ws
WHERE ws.workspace_id = $1
"#,
&w_id
)
.fetch_one(&mut *tx)
.await?
.unwrap_or(serde_json::Value::Null);
let old_datatables: HashMap<String, DataTable> =
serde_json::from_value(old_datatables).unwrap_or_default();
for (name, dt) in new_config.settings.datatables.iter() {
if dt.database.resource_type == DataTableCatalogResourceType::Instance {
let old_dt = old_datatables.get(name);
@@ -2458,11 +2519,101 @@ async fn edit_datatable_config(
.execute(&mut *tx)
.await?;
crate::datatable_migrations::cascade_datatable_migration_renames_and_deletes(
&db,
&mut tx,
&w_id,
&new_config.renames,
&new_config.deleted_datatables,
)
.await?;
tx.commit().await?;
Ok(format!("Edit datatable config for workspace {}", &w_id))
}
async fn get_datatable_permissions(
ApiAuthed { is_admin, username, .. }: ApiAuthed,
Extension(db): Extension<DB>,
Path((w_id, datatable_name)): Path<(String, String)>,
) -> Result<Json<Option<DataTablePermissions>>> {
require_admin(is_admin, &username)?;
let datatable = get_datatable_from_db(&db, &w_id, &datatable_name).await?;
Ok(Json(datatable.permissions))
}
/// Persist the advanced-permission config for a single data table and reconcile
/// the target database (roles, grants, RLS policies) to match. Returns the
/// reconciliation log so the modal can surface warnings (e.g. a resource
/// database whose credentials lack the privileges to provision roles).
async fn set_datatable_permissions(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path((w_id, datatable_name)): Path<(String, String)>,
Json(perms): Json<DataTablePermissions>,
) -> Result<String> {
require_admin(authed.is_admin, &authed.username)?;
crate::datatable_migrations::validate_datatable_path_segment(&datatable_name)?;
// Load first so a missing data table is a clean 404, then write only the
// `permissions` subfield (the rest of the catalog entry is untouched).
let mut datatable = get_datatable_from_db(&db, &w_id, &datatable_name).await?;
let perms_json =
serde_json::to_value(&perms).map_err(|e| Error::internal_err(e.to_string()))?;
// Runtime (non-macro) query so it needs no `.sqlx` cache entry.
sqlx::query(
"UPDATE workspace_settings
SET datatable = jsonb_set(datatable, ARRAY['datatables', $2, 'permissions'], $3::jsonb)
WHERE workspace_id = $1",
)
.bind(&w_id)
.bind(&datatable_name)
.bind(perms_json)
.execute(&db)
.await?;
audit_log(
&db,
&authed,
"workspaces.set_datatable_permissions",
ActionKind::Update,
&w_id,
Some(&datatable_name),
None,
)
.await?;
datatable.permissions = Some(perms);
let log = windmill_common::datatable_permissions::reconcile_datatable_permissions(
&db,
&w_id,
&datatable_name,
&datatable,
)
.await?;
Ok(log.join("\n"))
}
/// Re-run reconciliation from the stored config, e.g. to pick up group-membership
/// changes or newly-migrated tables without editing the permissions.
async fn sync_datatable_permissions(
ApiAuthed { is_admin, username, .. }: ApiAuthed,
Extension(db): Extension<DB>,
Path((w_id, datatable_name)): Path<(String, String)>,
) -> Result<String> {
require_admin(is_admin, &username)?;
let datatable = get_datatable_from_db(&db, &w_id, &datatable_name).await?;
let log = windmill_common::datatable_permissions::reconcile_datatable_permissions(
&db,
&w_id,
&datatable_name,
&datatable,
)
.await?;
Ok(log.join("\n"))
}
#[derive(Deserialize)]
pub struct EditGitSyncConfig {
pub git_sync_settings: Option<WorkspaceGitSyncSettings>,
@@ -4048,6 +4199,15 @@ async fn clone_workspace_data(
// Clone workspace settings (merge with existing basic settings)
update_workspace_settings(tx, source_workspace_id, target_workspace_id).await?;
// Clone data table migration definitions (the settings above carry the data
// table config; this carries their migration history).
crate::datatable_migrations::clone_datatable_migrations(
tx,
source_workspace_id,
target_workspace_id,
)
.await?;
// Clone workspace environment variables
clone_workspace_env(tx, source_workspace_id, target_workspace_id).await?;
@@ -7339,6 +7499,7 @@ pub struct CompareSummary {
pub folders_changed: usize,
pub schedules_changed: usize,
pub triggers_changed: usize,
pub datatable_migrations_changed: usize,
pub conflicts: usize, // Items that are both ahead and behind
}
@@ -7530,6 +7691,15 @@ async fn compare_workspaces(
compare_two_folders(&db, &source_workspace_id, &fork_workspace_id, &item.path)
.await?,
),
"datatable_migration" => Some(
crate::datatable_migrations::compare_two_datatable_migration(
&db,
&source_workspace_id,
&fork_workspace_id,
&item.path,
)
.await?,
),
// Triggers and schedules are diffed against a hardcoded ignore list
// (mode/enabled/server_id/last_server_ping/edited_at/by/error/extra_perms/permissioned_as/email)
// so that fork-clones — which differ from the parent only in the runtime
@@ -7652,6 +7822,10 @@ async fn compare_workspaces(
.iter()
.filter(|s| s.kind.ends_with("_trigger"))
.count(),
datatable_migrations_changed: visible_diffs
.iter()
.filter(|s| s.kind == "datatable_migration")
.count(),
conflicts: visible_diffs
.iter()
.filter(|s| s.ahead > 0 && s.behind > 0)
@@ -7957,6 +8131,45 @@ async fn query_visible_items<'c>(
.fetch_all(&mut **tx)
.await?
}
"datatable_migration" => {
// Match by (datatable, timestamp), not the full path: a migration
// keeps its identity across a rename, so the candidate path's
// `name` segment can differ from the stored one. Parse each
// `<datatable>/<timestamp>_<name>` candidate, probe existence by
// (datatable, timestamp), and return the *original* candidate path
// so the visibility set stays keyed by the diff's path.
let parsed: Vec<(String, i64, String)> = paths_vec
.iter()
.filter_map(|p| {
let (dt, rest) = p.split_once('/')?;
let ts = rest.split_once('_')?.0.parse::<i64>().ok()?;
Some((dt.to_string(), ts, p.clone()))
})
.collect();
if parsed.is_empty() {
vec![]
} else {
let dts: Vec<String> = parsed.iter().map(|(d, _, _)| d.clone()).collect();
let tss: Vec<i64> = parsed.iter().map(|(_, t, _)| *t).collect();
let existing: HashSet<(String, i64)> = sqlx::query!(
"SELECT datatable, timestamp FROM datatable_migrations \
WHERE workspace_id = $1 AND datatable = ANY($2) AND timestamp = ANY($3)",
workspace_id,
&dts,
&tss,
)
.fetch_all(&mut **tx)
.await?
.into_iter()
.map(|r| (r.datatable, r.timestamp))
.collect();
parsed
.into_iter()
.filter(|(d, t, _)| existing.contains(&(d.clone(), *t)))
.map(|(_, _, p)| p)
.collect()
}
}
k if TRIGGER_OR_SCHEDULE_TABLES.contains(&k) => {
// SAFETY: `kind` comes from a hardcoded allowlist
// TRIGGER_OR_SCHEDULE_TABLES, not user input.
@@ -8024,10 +8237,10 @@ async fn existing_runnables(
}
#[derive(Debug)]
struct ItemComparison {
has_changes: bool,
exists_in_source: bool,
exists_in_fork: bool,
pub(crate) struct ItemComparison {
pub(crate) has_changes: bool,
pub(crate) exists_in_source: bool,
pub(crate) exists_in_fork: bool,
}
async fn compare_two_scripts(
+498
View File
@@ -4598,6 +4598,22 @@ paths:
properties:
settings:
$ref: "#/components/schemas/DataTableSettings"
renames:
description: data tables renamed in this save, so their migrations cascade
type: array
items:
type: object
required: [from, to]
properties:
from:
type: string
to:
type: string
deleted_datatables:
description: data tables removed in this save, so their migrations are deleted
type: array
items:
type: string
responses:
"200":
description: status
@@ -4605,6 +4621,379 @@ paths:
application/json:
schema: {}
/w/{workspace}/workspaces/run_datatable_migrations/{datatable_name}:
post:
summary: run pending datatable migrations against a datatable
operationId: runDatatableMigrations
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
- name: up_to
in: query
required: false
description: only apply pending migrations up to and including this version
schema:
type: integer
format: int64
- name: only
in: query
required: false
description: apply only this specific migration version, ignoring others
schema:
type: integer
format: int64
responses:
"200":
description: applied migrations
content:
application/json:
schema:
type: object
required: [applied]
properties:
applied:
type: array
items:
type: object
required: [version, name]
properties:
version:
type: integer
format: int64
name:
type: string
/w/{workspace}/workspaces/rollback_datatable_migrations/{datatable_name}:
post:
summary: roll back the most recently applied migration on a datatable
operationId: rollbackDatatableMigrations
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
- name: only
in: query
required: false
description: roll back this specific applied migration version instead of the latest
schema:
type: integer
format: int64
responses:
"200":
description: rolled back migrations
content:
application/json:
schema:
type: object
required: [rolled_back]
properties:
rolled_back:
type: array
items:
type: object
required: [version, name]
properties:
version:
type: integer
format: int64
name:
type: string
/w/{workspace}/workspaces/list_datatable_migrations:
get:
summary: list datatable migrations for a workspace
operationId: listDatatableMigrations
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: datatable migrations
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/DatatableMigration"
/w/{workspace}/workspaces/datatable_migrations_status/{datatable_name}:
get:
summary: list a datatable's migrations with their applied status
operationId: getDatatableMigrationsStatus
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: migrations with status
content:
application/json:
schema:
type: object
required: [enabled, migrations]
properties:
enabled:
type: boolean
migrations:
type: array
items:
$ref: "#/components/schemas/DatatableMigrationWithStatus"
error:
type: string
/w/{workspace}/workspaces/datatable_permissions/{datatable_name}:
get:
summary: get the permissions configuration of a datatable
operationId: getDatatablePermissions
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: datatable permissions or null if not configured
content:
application/json:
schema:
nullable: true
allOf:
- $ref: "#/components/schemas/DataTablePermissions"
/w/{workspace}/workspaces/set_datatable_permissions/{datatable_name}:
post:
summary: set the permissions configuration of a datatable
operationId: setDatatablePermissions
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
requestBody:
description: datatable permissions
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/DataTablePermissions"
responses:
"200":
description: reconciliation log
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/sync_datatable_permissions/{datatable_name}:
post:
summary: reconcile the datatable's database with its permissions configuration
operationId: syncDatatablePermissions
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: reconciliation log
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/enable_datatable_migrations/{datatable_name}:
post:
summary: opt a datatable in to migrations (admins / super admins only)
operationId: enableDatatableMigrations
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/disable_datatable_migrations/{datatable_name}:
post:
summary: opt a datatable out of migrations, deleting all of them (admins / super admins only)
operationId: disableDatatableMigrations
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/create_datatable_migration/{datatable_name}:
post:
summary: create a single datatable migration (version generated server-side)
operationId: createDatatableMigration
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [name, code_up]
properties:
name:
type: string
code_up:
type: string
code_down:
type: string
responses:
"200":
description: created migration
content:
application/json:
schema:
$ref: "#/components/schemas/DatatableMigration"
/w/{workspace}/workspaces/delete_datatable_migration/{datatable_name}/{timestamp}:
delete:
summary: delete a single datatable migration definition
operationId: deleteDatatableMigration
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
- name: timestamp
in: path
required: true
schema:
type: integer
format: int64
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/upsert_datatable_migration/{datatable_name}:
post:
summary: insert or update a single datatable migration at an explicit version
operationId: upsertDatatableMigration
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [timestamp, name, code_up]
properties:
timestamp:
type: integer
format: int64
name:
type: string
code_up:
type: string
code_down:
type: string
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/generate_initial_datatable_migration/{datatable_name}:
post:
summary: snapshot the current schema as an already-installed initial migration
operationId: generateInitialDatatableMigration
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: created migration
content:
application/json:
schema:
$ref: "#/components/schemas/DatatableMigration"
/w/{workspace}/workspaces/create_pg_database:
post:
summary: create a new PostgreSQL database for a datatable
@@ -29026,6 +29415,9 @@ components:
type: string
required:
- resource_type
migrations_enabled:
type: boolean
description: Whether the SQL migrations feature is opted in for this data table
forked_from:
type: object
description: Fork origin info with schema snapshot
@@ -29034,6 +29426,108 @@ components:
type: object
description: Schema snapshot at fork time
additionalProperties: true
DatatableMigration:
type: object
required: [datatable, timestamp, name, code_up]
properties:
datatable:
type: string
timestamp:
type: integer
format: int64
name:
type: string
code_up:
type: string
code_down:
type: string
DatatableMigrationWithStatus:
type: object
required: [timestamp, name, code_up, status]
properties:
timestamp:
type: integer
format: int64
name:
type: string
code_up:
type: string
code_down:
type: string
status:
type: string
enum:
- ran
- not_run
- unknown
DataTablePermissions:
type: object
required: [enabled]
properties:
enabled:
type: boolean
grants:
type: array
default: []
items:
$ref: "#/components/schemas/DataTableGrant"
policies:
type: array
default: []
items:
$ref: "#/components/schemas/DataTablePolicy"
DataTableGrant:
type: object
required: [principal, access]
properties:
principal:
$ref: "#/components/schemas/DataTablePrincipal"
table:
type: string
access:
type: string
enum:
- none
- read
- write
DataTablePrincipal:
type: object
required: [kind, name]
properties:
kind:
type: string
enum:
- user
- group
name:
type: string
description: user email or group name
DataTablePolicy:
type: object
required: [table, name, command]
properties:
table:
type: string
name:
type: string
command:
type: string
default: all
enum:
- all
- select
- insert
- update
- delete
principals:
type: array
default: []
items:
$ref: "#/components/schemas/DataTablePrincipal"
using:
type: string
check:
type: string
DataTableSchema:
type: object
required: [datatable_name, schemas]
@@ -29745,6 +30239,7 @@ components:
- folders_changed
- schedules_changed
- triggers_changed
- datatable_migrations_changed
- conflicts
properties:
total_diffs:
@@ -29783,6 +30278,9 @@ components:
triggers_changed:
type: integer
description: Number of triggers with differences (sum across all trigger kinds)
datatable_migrations_changed:
type: integer
description: Number of data table migrations with differences
conflicts:
type: integer
description: Number of items that are both ahead and behind (conflicts)
@@ -1546,6 +1546,34 @@ pub(crate) async fn tarball_workspace(
.await?;
}
{
// Data table migrations live in the `datatable_migrations` table; surface
// them in the export as `migrations/datatable/<datatable>/<version>_<name>`
// .up.sql (and .down.sql when present) so `wmill sync` treats them like any
// other workspace item.
let migrations = sqlx::query!(
"SELECT datatable, timestamp, name, code_up, code_down FROM datatable_migrations \
WHERE workspace_id = $1 ORDER BY datatable, timestamp",
&w_id
)
.fetch_all(&mut *tx)
.await?;
for m in migrations {
let base = format!(
"migrations/datatable/{}/{}_{}",
m.datatable, m.timestamp, m.name
);
archive
.write_to_archive(&m.code_up, &format!("{base}.up.sql"))
.await?;
if let Some(code_down) = m.code_down {
archive
.write_to_archive(&code_down, &format!("{base}.down.sql"))
.await?;
}
}
}
archive.finish().await?;
let file = tokio::fs::File::open(&file_path).await?;
@@ -0,0 +1,39 @@
//! OSS fallback for enterprise data-table permissions (implementation in
//! windmill-ee-private, see `datatable_permissions_ee`). The public build has no
//! RLS/role provisioning: reconciling reports that the enterprise edition is
//! required, and query-time resolution always falls through to the default
//! connection (the legacy behavior, where every workspace member shares the
//! owner role and has full access).
use crate::{
error::{Error, Result},
workspaces::{DataTable, DatatableAccessDecision},
DB,
};
/// Reconcile the target database (roles, grants, RLS policies) to match the
/// data table's stored permission config. Enterprise-only.
pub async fn reconcile_datatable_permissions(
_db: &DB,
_w_id: &str,
_datatable_name: &str,
_datatable: &DataTable,
) -> Result<Vec<String>> {
Err(Error::internal_err(
"Data table permissions require the enterprise edition".to_string(),
))
}
/// Decide which Postgres identity a query against `datatable` should run under
/// for the acting user. The public build never overrides the connection.
pub async fn resolve_datatable_access(
_db: &DB,
_w_id: &str,
_datatable_name: &str,
_datatable: &DataTable,
_base_creds: &serde_json::Value,
_acting_email: &str,
_is_workspace_admin: bool,
) -> Result<DatatableAccessDecision> {
Ok(DatatableAccessDecision::Default)
}
+7
View File
@@ -42,6 +42,13 @@ mod db_entra_ee;
mod db_iam_ee;
pub mod db_params;
#[cfg(feature = "private")]
pub mod datatable_permissions_ee;
pub mod datatable_permissions_oss;
#[cfg(feature = "private")]
pub use datatable_permissions_ee as datatable_permissions;
#[cfg(not(feature = "private"))]
pub use datatable_permissions_oss as datatable_permissions;
#[cfg(feature = "private")]
pub mod deployment_requests_ee;
pub mod deployment_requests_oss;
#[cfg(feature = "private")]
+233 -12
View File
@@ -164,6 +164,7 @@ pub enum ObjectType {
Settings,
Key,
WorkspaceDependencies,
DatatableMigration,
}
pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28719/sync-script-to-git-repo-windmill";
@@ -771,6 +772,169 @@ pub struct DataTable {
pub database: DataTableDatabase,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub forked_from: Option<DataTableForkedFrom>,
/// Whether the SQL-migrations feature is opted in for this data table.
/// Absent on data tables created before the feature: treated as enabled only
/// when migrations already exist (see `datatable_migrations_enabled`).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub migrations_enabled: Option<bool>,
/// Advanced (EE) permissions. Absent/disabled = legacy behavior: every
/// workspace member connects as the shared owner role and has full access.
/// Owned by the dedicated permissions endpoints, not `edit_datatable_config`
/// (mirrors `migrations_enabled`) so a plain config save never clobbers it or
/// desyncs the provisioned Postgres roles/policies from the stored config.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub permissions: Option<DataTablePermissions>,
}
/// Desired advanced-permission state for a data table. This is the source of
/// truth the modal edits; the EE provisioning engine reconciles the target
/// Postgres database (roles, grants, RLS policies) to match it.
#[derive(Deserialize, Serialize, Debug, Clone, Default)]
pub struct DataTablePermissions {
/// Opt-in switch. When false, no enforcement happens and provisioned roles
/// (if any) are left in place but unused.
#[serde(default)]
pub enabled: bool,
/// Table/operation access, resolved to Postgres GRANTs per principal.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub grants: Vec<DataTableGrant>,
/// Row-level security policies applied to individual tables.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub policies: Vec<DataTablePolicy>,
}
/// A Windmill principal, mirrored into a Postgres role: `wm_u_<slug>` (a LOGIN
/// role, one per user) or `wm_g_<slug>` (a NOLOGIN membership role, one per group).
#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq, Hash)]
#[serde(tag = "kind", content = "name", rename_all = "snake_case")]
pub enum DataTablePrincipal {
/// User email.
User(String),
/// Group name.
Group(String),
}
#[derive(Deserialize, Serialize, Debug, Clone)]
pub struct DataTableGrant {
pub principal: DataTablePrincipal,
/// `None` targets every table (existing tables plus future ones via
/// `ALTER DEFAULT PRIVILEGES`); `Some(t)` targets a single table.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub table: Option<String>,
pub access: DataTableAccess,
}
#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum DataTableAccess {
None,
Read,
Write,
}
/// A single RLS policy, mapped onto `CREATE POLICY <name> ON <table> FOR <command>
/// TO <principals> USING (<using>) WITH CHECK (<check>)`.
#[derive(Deserialize, Serialize, Debug, Clone)]
pub struct DataTablePolicy {
pub table: String,
pub name: String,
#[serde(default)]
pub command: DataTablePolicyCommand,
/// Principals the policy applies to (`TO` clause). Empty => `PUBLIC`.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub principals: Vec<DataTablePrincipal>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub using: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub check: Option<String>,
}
#[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum DataTablePolicyCommand {
#[default]
All,
Select,
Insert,
Update,
Delete,
}
/// Outcome of resolving which Postgres identity a data table query should run
/// under, given the acting Windmill user. Shared across CE/EE so the query
/// executor can branch on it regardless of build.
#[derive(Debug)]
pub enum DatatableAccessDecision {
/// Use the default connection (shared owner role, or resource credentials).
/// This is the legacy path and the workspace-admin bypass.
Default,
/// Connect as a per-user principal role; RLS + grants enforce access.
/// Carries the resolved `PgDatabase` credentials as JSON (host/dbname of the
/// default connection with `user`/`password` overridden).
AsPrincipal(serde_json::Value),
/// The acting user has no access to this data table at all.
Denied,
}
/// Deterministic, workspace-scoped Postgres role name for a Windmill principal.
///
/// Postgres roles are cluster-global, and every instance data table lives on the
/// same server, so the workspace id MUST be folded in to keep one workspace's
/// principal roles from colliding with (or connecting to) another's. Hashing
/// yields a stable, valid, <=63-char identifier from arbitrary emails/group
/// names. Both provisioning and query-time resolution call this, so they always
/// agree on the name for a given `(workspace, principal)`.
pub fn datatable_role_name(w_id: &str, principal: &DataTablePrincipal) -> String {
use sha2::{Digest, Sha256};
let (prefix, raw) = match principal {
DataTablePrincipal::User(email) => ("wm_u_", email.as_str()),
DataTablePrincipal::Group(group) => ("wm_g_", group.as_str()),
};
let mut hasher = Sha256::new();
hasher.update(w_id.as_bytes());
hasher.update([0u8]);
hasher.update(raw.as_bytes());
format!("{prefix}{}", hex::encode(&hasher.finalize()[..12]))
}
/// Deterministic password for a principal role, derived by HMAC-SHA256 of the
/// role name under a stable instance secret (see
/// `get_or_create_datatable_role_secret`). The role name already encodes the
/// workspace, so keying on it alone is sufficient. Nothing per-role is stored:
/// both the provisioner (which sets the password) and the executor (which
/// authenticates with it) recompute it.
pub fn derive_datatable_role_password(secret: &str, role: &str) -> String {
use hmac::{Hmac, Mac};
use sha2::Sha256;
let mut mac =
Hmac::<Sha256>::new_from_slice(secret.as_bytes()).expect("HMAC accepts any key length");
mac.update(role.as_bytes());
hex::encode(mac.finalize().into_bytes())
}
/// Fetch (or lazily create) the stable secret used to derive principal-role
/// passwords. Unlike `custom_instance_user`'s password this is never rotated —
/// rotating it would invalidate every principal password at once — so it lives
/// under its own `global_settings` key generated on first use.
pub async fn get_or_create_datatable_role_secret(db: &DB) -> Result<String> {
// `gen_random_uuid()` provides the randomness in-SQL so we don't pull in a
// CSPRNG here; two concatenated uuids give a 64-char secret. ON CONFLICT DO
// NOTHING makes concurrent first-uses converge on a single value.
// Runtime (non-macro) query so it needs no `.sqlx` cache entry.
sqlx::query(
r#"INSERT INTO global_settings (name, value)
VALUES ('datatable_permissions',
jsonb_build_object('secret', gen_random_uuid()::text || gen_random_uuid()::text))
ON CONFLICT (name) DO NOTHING"#,
)
.execute(db)
.await?;
sqlx::query_scalar::<_, Option<String>>(
"SELECT value->>'secret' FROM global_settings WHERE name = 'datatable_permissions'",
)
.fetch_one(db)
.await?
.ok_or_else(|| Error::internal_err("datatable_permissions secret missing after insert"))
}
#[derive(Deserialize, Serialize, Debug)]
@@ -822,11 +986,8 @@ fn datatable_not_found_error(name: &str, datatables: Option<&serde_json::Value>)
))
}
pub async fn get_datatable_resource_from_db_unchecked(
db: &DB,
w_id: &str,
name: &str,
) -> Result<serde_json::Value> {
/// Load a data table's catalog entry (including its permission config).
pub async fn get_datatable_from_db(db: &DB, w_id: &str, name: &str) -> Result<DataTable> {
let datatables = sqlx::query_scalar!(
r#"
SELECT ws.datatable->'datatables' AS datatables
@@ -844,26 +1005,86 @@ pub async fn get_datatable_resource_from_db_unchecked(
.and_then(|d| d.get(name))
.filter(|v| !v.is_null())
.ok_or_else(|| datatable_not_found_error(name, datatables.as_ref()))?;
let datatable = serde_json::from_value::<DataTable>(datatable.clone())?;
Ok(serde_json::from_value::<DataTable>(datatable.clone())?)
}
let db_resource = if datatable.database.resource_type == DataTableCatalogResourceType::Instance
{
/// Resolve the default connection credentials for a data table — the shared
/// owner role (instance) or the backing resource (postgres). This performs no
/// per-user authorization; permission enforcement is layered on in the `checked`
/// variant.
pub async fn datatable_base_creds(
db: &DB,
w_id: &str,
datatable: &DataTable,
) -> Result<serde_json::Value> {
if datatable.database.resource_type == DataTableCatalogResourceType::Instance {
let mut pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
pg_creds.dbname = datatable.database.resource_path.clone();
pg_creds.user = Some("custom_instance_user".to_string());
pg_creds.password = Some(get_custom_pg_instance_password(&db).await?);
serde_json::to_value(&pg_creds)
.map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))?
.map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))
} else {
transform_json_unchecked(
&serde_json::Value::String(format!("$res:{}", datatable.database.resource_path)),
w_id,
db,
)
.await?
};
.await
}
}
Ok(db_resource)
pub async fn get_datatable_resource_from_db_unchecked(
db: &DB,
w_id: &str,
name: &str,
) -> Result<serde_json::Value> {
let datatable = get_datatable_from_db(db, w_id, name).await?;
datatable_base_creds(db, w_id, &datatable).await
}
/// Like `get_datatable_resource_from_db_unchecked`, but applies the data table's
/// advanced (EE) permissions for `acting_email`: an admin (or a data table
/// without permissions) gets the default owner connection; other members get a
/// per-user principal role subject to RLS/grants; members with no access are
/// rejected. On the OSS build the EE hook is a no-op, so this equals the
/// unchecked path.
pub async fn get_datatable_resource_from_db_checked(
db: &DB,
w_id: &str,
name: &str,
acting_email: &str,
) -> Result<serde_json::Value> {
let datatable = get_datatable_from_db(db, w_id, name).await?;
let base = datatable_base_creds(db, w_id, &datatable).await?;
let is_workspace_admin = sqlx::query_scalar::<_, bool>(
"SELECT is_admin FROM usr WHERE workspace_id = $1 AND email = $2",
)
.bind(w_id)
.bind(acting_email)
.fetch_optional(db)
.await?
.unwrap_or(false);
match crate::datatable_permissions::resolve_datatable_access(
db,
w_id,
name,
&datatable,
&base,
acting_email,
is_workspace_admin,
)
.await?
{
DatatableAccessDecision::Default => Ok(base),
DatatableAccessDecision::AsPrincipal(creds) => Ok(creds),
DatatableAccessDecision::Denied => Err(Error::BadRequest(format!(
"You do not have access to data table '{name}'. Ask a workspace admin to grant \
access (data table permissions may need to be re-synced)."
))),
}
}
#[derive(Deserialize, Serialize, Debug)]
+99 -24
View File
@@ -24,30 +24,102 @@ pub use git_sync_oss::{
#[derive(Clone, Debug)]
pub enum DeployedObject {
Script { hash: ScriptHash, path: String, parent_path: Option<String> },
Flow { path: String, parent_path: Option<String>, version: i64 },
App { path: String, version: i64, parent_path: Option<String> },
RawApp { path: String, version: i64, parent_path: Option<String> },
Folder { path: String },
Resource { path: String, parent_path: Option<String> },
Variable { path: String, parent_path: Option<String> },
Schedule { path: String },
ResourceType { path: String },
User { email: String },
Group { name: String },
HttpTrigger { path: String, parent_path: Option<String> },
WebsocketTrigger { path: String, parent_path: Option<String> },
KafkaTrigger { path: String, parent_path: Option<String> },
NatsTrigger { path: String, parent_path: Option<String> },
PostgresTrigger { path: String, parent_path: Option<String> },
MqttTrigger { path: String, parent_path: Option<String> },
SqsTrigger { path: String, parent_path: Option<String> },
GcpTrigger { path: String, parent_path: Option<String> },
AzureTrigger { path: String, parent_path: Option<String> },
EmailTrigger { path: String, parent_path: Option<String> },
Settings { setting_type: String },
Key { key_type: String },
WorkspaceDependencies { path: String },
Script {
hash: ScriptHash,
path: String,
parent_path: Option<String>,
},
Flow {
path: String,
parent_path: Option<String>,
version: i64,
},
App {
path: String,
version: i64,
parent_path: Option<String>,
},
RawApp {
path: String,
version: i64,
parent_path: Option<String>,
},
Folder {
path: String,
},
Resource {
path: String,
parent_path: Option<String>,
},
Variable {
path: String,
parent_path: Option<String>,
},
Schedule {
path: String,
},
ResourceType {
path: String,
},
User {
email: String,
},
Group {
name: String,
},
HttpTrigger {
path: String,
parent_path: Option<String>,
},
WebsocketTrigger {
path: String,
parent_path: Option<String>,
},
KafkaTrigger {
path: String,
parent_path: Option<String>,
},
NatsTrigger {
path: String,
parent_path: Option<String>,
},
PostgresTrigger {
path: String,
parent_path: Option<String>,
},
MqttTrigger {
path: String,
parent_path: Option<String>,
},
SqsTrigger {
path: String,
parent_path: Option<String>,
},
GcpTrigger {
path: String,
parent_path: Option<String>,
},
AzureTrigger {
path: String,
parent_path: Option<String>,
},
EmailTrigger {
path: String,
parent_path: Option<String>,
},
Settings {
setting_type: String,
},
Key {
key_type: String,
},
WorkspaceDependencies {
path: String,
},
/// A single data table migration, identified by `<datatable>/<timestamp>_<name>`.
DatatableMigration {
path: String,
},
}
impl DeployedObject {
@@ -77,6 +149,7 @@ impl DeployedObject {
DeployedObject::Settings { .. } => "settings.yaml".to_string(),
DeployedObject::Key { .. } => "encryption_key.yaml".to_string(),
DeployedObject::WorkspaceDependencies { path, .. } => path.to_owned(),
DeployedObject::DatatableMigration { path } => path.to_owned(),
}
}
@@ -118,6 +191,7 @@ impl DeployedObject {
DeployedObject::Settings { .. } => None,
DeployedObject::Key { .. } => None,
DeployedObject::WorkspaceDependencies { .. } => None,
DeployedObject::DatatableMigration { .. } => None,
}
}
@@ -147,6 +221,7 @@ impl DeployedObject {
DeployedObject::Settings { .. } => "settings",
DeployedObject::Key { .. } => "key",
DeployedObject::WorkspaceDependencies { .. } => "workspace_dependencies",
DeployedObject::DatatableMigration { .. } => "datatable_migration",
}
.to_string()
}
+10 -2
View File
@@ -13,7 +13,7 @@ use windmill_common::error::{to_anyhow, Error, Result};
use windmill_common::utils::sanitize_string_from_password;
use windmill_common::worker::{get_memory, to_raw_value, Connection, SqlResultCollectionStrategy};
use windmill_common::workspaces::{
get_datatable_resource_from_db_unchecked, get_ducklake_from_db_unchecked,
get_datatable_resource_from_db_checked, get_ducklake_from_db_unchecked,
strip_fork_reserved_attach_args, DucklakeCatalogResourceType,
};
use windmill_common::PgDatabase;
@@ -1496,6 +1496,7 @@ pub async fn do_duckdb(
conn,
&mut hidden_passwords,
&job.workspace_id,
&job.permissioned_as_email,
)
.await?
{
@@ -1573,6 +1574,7 @@ pub async fn do_duckdb(
conn,
&mut hidden_passwords,
&job.workspace_id,
&job.permissioned_as_email,
)
.await?
{
@@ -2555,6 +2557,7 @@ async fn transform_attach_datatable(
conn: &Connection,
hidden_passwords: &mut Arc<Mutex<Vec<String>>>,
w_id: &str,
acting_email: &str,
) -> Result<Option<Vec<String>>> {
lazy_static::lazy_static! {
static ref RE: regex::Regex = regex::Regex::new(r"(?i)ATTACH\s*'datatable(://[^':]+)?'\s*AS\s+([^ ;]+)").unwrap();
@@ -2569,7 +2572,12 @@ async fn transform_attach_datatable(
Connection::Http(client) => {
get_datatable_resource_from_agent_http(client, name, w_id).await?
}
Connection::Sql(db) => get_datatable_resource_from_db_unchecked(db, w_id, name).await?,
// Enforce data table permissions (EE) as the acting user, exactly like the
// postgresql executor — otherwise a DuckDB `ATTACH 'datatable://...'` would
// run as the owner and bypass RLS/grants entirely.
Connection::Sql(db) => {
get_datatable_resource_from_db_checked(db, w_id, name, acting_email).await?
}
};
let db_type = "postgres";
+10 -3
View File
@@ -27,7 +27,7 @@ use windmill_common::error::{self, Error};
use windmill_common::worker::{
to_raw_value, Connection, SqlResultCollectionStrategy, CLOUD_HOSTED,
};
use windmill_common::workspaces::get_datatable_resource_from_db_unchecked;
use windmill_common::workspaces::get_datatable_resource_from_db_checked;
use windmill_common::{PgDatabase, PrepareQueryColumnInfo, PrepareQueryResult, DB};
use windmill_parser::{Arg, Typ};
use windmill_parser_sql::{
@@ -599,8 +599,15 @@ pub async fn do_postgresql(
.await?
}
Connection::Sql(db) => {
get_datatable_resource_from_db_unchecked(db, &job.workspace_id, &db_str)
.await?
// Enforce advanced (EE) data table permissions as the job's
// acting identity; no-op on OSS / when permissions are off.
get_datatable_resource_from_db_checked(
db,
&job.workspace_id,
&db_str,
&job.permissioned_as_email,
)
.await?
}
})
}
+71
View File
@@ -9,6 +9,13 @@ import { GlobalOptions } from "../../types.ts";
import { runCatalogQuery } from "../../utils/catalog.ts";
import { psql as psqlDatatable } from "./psql.ts";
import { serve as serveDatatable } from "./serve.ts";
import {
createMigration,
pushLocalMigrations,
rollbackMigrations,
runMigrations,
validateLocalMigrations,
} from "../datatable_migrations.ts";
const DEFAULT_DATATABLE_NAME = "main";
@@ -41,6 +48,69 @@ async function run(
await runCatalogQuery(opts, "datatable", name, sql);
}
function migrateNew(
opts: GlobalOptions & { datatable?: string },
name: string,
) {
createMigration(opts.datatable ?? DEFAULT_DATATABLE_NAME, name);
}
async function migrateUp(opts: GlobalOptions & { datatable?: string }) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const dt = opts.datatable ?? DEFAULT_DATATABLE_NAME;
// Reject malformed local migrations (duplicate timestamps, orphan downs) before
// pushing — the same check `wmill sync push` runs — so a duplicate timestamp
// can't silently overwrite one migration on upsert.
const errors = validateLocalMigrations(new Set([dt]));
if (errors.length > 0) {
log.error(
"Invalid datatable migrations, aborting:\n" +
errors.map((e) => ` - ${e}`).join("\n"),
);
process.exit(1);
}
// Push any locally-created/edited migration files first (without running
// them), so `migrate up` works even before a `wmill sync push`.
await pushLocalMigrations(workspace.workspaceId, dt);
await runMigrations(workspace.workspaceId, dt);
}
async function migrateDown(opts: GlobalOptions & { datatable?: string }) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const dt = opts.datatable ?? DEFAULT_DATATABLE_NAME;
await rollbackMigrations(workspace.workspaceId, dt);
}
const migrateCommand = new Command()
.description("manage datatable migrations")
.command("new", "scaffold a new migration (.up.sql / .down.sql files)")
.arguments("<name:string>")
.option(
"-d --datatable <datatable:string>",
"Target datatable (default: main)",
)
.action(migrateNew as any)
.command(
"up",
"apply all pending migrations to the main datatable (or one via --datatable)",
)
.option(
"-d --datatable <datatable:string>",
"Target datatable (default: main)",
)
.action(migrateUp as any)
.command(
"down",
"roll back the most recent migration on the main datatable (or one via --datatable)",
)
.option(
"-d --datatable <datatable:string>",
"Target datatable (default: main)",
)
.action(migrateDown as any);
async function create(
opts: GlobalOptions & { resource?: string; force?: boolean },
name?: string,
@@ -124,6 +194,7 @@ const command = new Command()
"Output only the final result as JSON. Useful for scripting.",
)
.action(run as any)
.command("migrate", migrateCommand)
.command(
"create",
"register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable://<name>",
+340
View File
@@ -0,0 +1,340 @@
import * as fs from "node:fs";
import * as path from "node:path";
import * as log from "../core/log.ts";
import { colors } from "@cliffy/ansi/colors";
import * as wmill from "../../gen/services.gen.ts";
import { readTextFile } from "../utils/utils.ts";
import { Confirm } from "@cliffy/prompt/confirm";
// Migrations live under <cwd>/migrations/datatable/<datatable>/, one folder per
// target data table, as `<timestamp>_<name>.up.sql` (and optional `.down.sql`).
// They are synced as ordinary workspace files (see the workspace tarball export
// and the `datatable_migration` handling in sync.ts); this module only holds the
// `wmill datatable migrate` command helpers and the per-file push primitive.
const MIGRATIONS_DIR = path.join("migrations", "datatable");
// Migration names map directly onto file names and the DB `name` column.
const MIGRATION_NAME_RE = /^[a-zA-Z0-9_-]+$/;
/** Current UTC time as a YYYYMMDDHHMMSS migration version. */
function migrationTimestamp(): string {
const d = new Date();
const p = (n: number) => String(n).padStart(2, "0");
return (
`${d.getUTCFullYear()}${p(d.getUTCMonth() + 1)}${p(d.getUTCDate())}` +
`${p(d.getUTCHours())}${p(d.getUTCMinutes())}${p(d.getUTCSeconds())}`
);
}
/**
* A migration version unique within a data table folder: the current UTC
* timestamp bumped past any existing version, so two migrations scaffolded in
* the same second don't collide on the `(datatable, timestamp)` identity used to
* upsert them.
*/
function nextMigrationTimestamp(dir: string): string {
const now = Number(migrationTimestamp());
let max = 0;
if (fs.existsSync(dir)) {
for (const file of fs.readdirSync(dir)) {
const m = file.match(/^(\d+)_.*\.(up|down)\.sql$/);
if (m) max = Math.max(max, Number(m[1]));
}
}
return String(max >= now ? max + 1 : now);
}
/**
* Scaffold a new migration under migrations/datatable/<datatable>/ as empty
* `<timestamp>_<name>.up.sql` and `.down.sql` files. Purely local no network.
*/
export function createMigration(datatable: string, name: string): void {
if (!MIGRATION_NAME_RE.test(name)) {
throw new Error(
`Invalid migration name '${name}': use only letters, digits, '_' and '-'`,
);
}
const dir = path.join(process.cwd(), MIGRATIONS_DIR, datatable);
fs.mkdirSync(dir, { recursive: true });
const timestamp = nextMigrationTimestamp(dir);
const base = `${timestamp}_${name}`;
const up = path.join(dir, `${base}.up.sql`);
const down = path.join(dir, `${base}.down.sql`);
// Frame the body in an explicit transaction so it applies atomically, matching
// the template the UI's "New migration" modal seeds.
const template = (direction: string) =>
`-- ${direction} migration: ${name}\nBEGIN;\n\n-- Add your migration here\n\nEND;\n`;
fs.writeFileSync(up, template("up"), "utf-8");
fs.writeFileSync(down, template("down"), "utf-8");
log.info(
colors.green(`Created migration ${base} in ${MIGRATIONS_DIR}/${datatable}/`),
);
for (const f of [up, down]) {
log.info(colors.gray(` ${path.relative(process.cwd(), f)}`));
}
}
/**
* Apply the workspace's pending migrations to a data table (forwards migrations
* recorded in `_wm_migrations`). Mirrors `wmill datatable migrate up`.
*/
export async function runMigrations(
workspace: string,
datatableName: string,
): Promise<void> {
const result = await wmill.runDatatableMigrations({
workspace,
datatableName,
});
const applied = result.applied ?? [];
if (applied.length === 0) {
log.info(colors.gray(`No pending migrations to run on '${datatableName}'`));
return;
}
log.info(
colors.green(`Applied ${applied.length} migration(s) to '${datatableName}':`),
);
for (const m of applied) {
log.info(colors.gray(` ${m.version} ${m.name}`));
}
}
/**
* Roll back the most recently applied migration on a data table (one step).
* Mirrors `wmill datatable migrate down`.
*/
export async function rollbackMigrations(
workspace: string,
datatableName: string,
): Promise<void> {
const result = await wmill.rollbackDatatableMigrations({
workspace,
datatableName,
});
const rolledBack = result.rolled_back ?? [];
if (rolledBack.length === 0) {
log.info(
colors.gray(`No applied migrations to roll back on '${datatableName}'`),
);
return;
}
for (const m of rolledBack) {
log.info(
colors.green(`Rolled back migration ${m.version} ${m.name} on '${datatableName}'`),
);
}
}
/**
* Validate the on-disk migration files for the given data tables (or all of
* them when `datatables` is omitted). Returns a list of human-readable problems;
* an empty list means the migrations are well-formed. Two states are invalid:
* - two up (or two down) files sharing the same timestamp, which collide on the
* `(datatable, timestamp)` identity used to upsert; and
* - a `.down.sql` with no matching `.up.sql` (an up file is mandatory).
*/
export function validateLocalMigrations(datatables?: Set<string>): string[] {
const errors: string[] = [];
const root = path.join(process.cwd(), MIGRATIONS_DIR);
if (!fs.existsSync(root)) return errors;
for (const datatable of fs.readdirSync(root)) {
if (datatables && !datatables.has(datatable)) continue;
const dtDir = path.join(root, datatable);
if (!fs.statSync(dtDir).isDirectory()) continue;
const upNamesByTs = new Map<number, string[]>();
const downNamesByTs = new Map<number, string[]>();
const upBases = new Set<string>();
const downBases: { ts: number; name: string }[] = [];
for (const file of fs.readdirSync(dtDir)) {
const m = file.match(/^(\d+)_(.*)\.(up|down)\.sql$/);
if (!m) continue;
const ts = Number(m[1]);
const name = m[2];
if (m[3] === "up") {
(upNamesByTs.get(ts) ?? upNamesByTs.set(ts, []).get(ts)!).push(name);
upBases.add(`${ts}_${name}`);
} else {
(downNamesByTs.get(ts) ?? downNamesByTs.set(ts, []).get(ts)!).push(name);
downBases.push({ ts, name });
}
}
for (const [ts, names] of upNamesByTs) {
if (names.length > 1) {
errors.push(
`${datatable}: ${names.length} up migrations share timestamp ${ts} (${names.join(", ")})`,
);
}
}
for (const [ts, names] of downNamesByTs) {
if (names.length > 1) {
errors.push(
`${datatable}: ${names.length} down migrations share timestamp ${ts} (${names.join(", ")})`,
);
}
}
for (const d of downBases) {
if (!upBases.has(`${d.ts}_${d.name}`)) {
errors.push(
`${datatable}: ${d.ts}_${d.name}.down.sql has no matching ${d.ts}_${d.name}.up.sql`,
);
}
}
}
return errors;
}
/**
* Sync a single migration to the workspace based on the current on-disk state of
* its `<datatable>/<timestamp>_<name>.up.sql` file: upsert it when the up file
* exists, otherwise delete it. Called by `wmill sync push` for each changed
* `datatable_migration` file.
*/
export async function pushMigrationFromDisk(
workspace: string,
m: { datatable: string; timestamp: number },
): Promise<void> {
const dir = path.join(process.cwd(), MIGRATIONS_DIR, m.datatable);
// Find the up file for this timestamp regardless of its name segment. A rename
// (`123_old.up.sql` -> `123_new.up.sql`) keeps the (datatable, timestamp)
// identity but changes the name; the diff sorter may process the deleted old
// path before the added new one, so keying off the passed name would delete
// the record. Scanning by timestamp upserts the surviving file instead.
const upFile = fs.existsSync(dir)
? fs.readdirSync(dir).find((f) => {
const parsed = f.match(/^(\d+)_(.*)\.up\.sql$/);
return parsed !== null && Number(parsed[1]) === m.timestamp;
})
: undefined;
if (upFile === undefined) {
log.info(colors.red(`Deleting datatable_migration ${m.datatable}/${m.timestamp}`));
await wmill.deleteDatatableMigration({
workspace,
datatableName: m.datatable,
timestamp: m.timestamp,
});
return;
}
const name = upFile.match(/^(\d+)_(.*)\.up\.sql$/)![2];
const base = `${m.timestamp}_${name}`;
const code_up = await readTextFile(path.join(dir, upFile));
const downPath = path.join(dir, `${base}.down.sql`);
const code_down = fs.existsSync(downPath) ? await readTextFile(downPath) : undefined;
log.info(colors.green(`Pushing datatable_migration ${m.datatable}/${base}`));
await wmill.upsertDatatableMigration({
workspace,
datatableName: m.datatable,
requestBody: {
timestamp: m.timestamp,
name,
code_up,
...(code_down !== undefined ? { code_down } : {}),
},
});
}
/**
* Upsert the on-disk migrations of a data table to the workspace, so a freshly
* created migration file works with `wmill datatable migrate up` even without a
* prior `wmill sync push`. Pushes only migrations that are new or edited
* (compared against the workspace's current definitions); it never deletes
* remote migrations absent on disk and never touches other item kinds.
*/
export async function pushLocalMigrations(
workspace: string,
datatableName: string,
): Promise<void> {
const dir = path.join(process.cwd(), MIGRATIONS_DIR, datatableName);
if (!fs.existsSync(dir)) return;
// Local migrations are identified by their `.up.sql` file (the up file is
// mandatory); this deliberately ignores files that were only deleted locally.
const local: { timestamp: number; name: string }[] = [];
for (const file of fs.readdirSync(dir)) {
const m = file.match(/^(\d+)_(.*)\.up\.sql$/);
if (m) local.push({ timestamp: Number(m[1]), name: m[2] });
}
if (local.length === 0) return;
const remote = await wmill.listDatatableMigrations({ workspace });
const remoteByTs = new Map(
remote
.filter((r) => r.datatable === datatableName)
.map((r) => [r.timestamp, r] as const),
);
for (const { timestamp, name } of local) {
const base = `${timestamp}_${name}`;
const code_up = await readTextFile(path.join(dir, `${base}.up.sql`));
const downPath = path.join(dir, `${base}.down.sql`);
const code_down = fs.existsSync(downPath)
? await readTextFile(downPath)
: undefined;
const r = remoteByTs.get(timestamp);
const unchanged =
r !== undefined &&
r.name === name &&
r.code_up === code_up &&
(r.code_down ?? undefined) === code_down;
if (unchanged) continue;
log.info(colors.green(`Pushing datatable_migration ${datatableName}/${base}`));
await wmill.upsertDatatableMigration({
workspace,
datatableName,
requestBody: {
timestamp,
name,
code_up,
...(code_down !== undefined ? { code_down } : {}),
},
});
}
}
/**
* After a push that introduced new migrations, list them and (interactively)
* offer to run them, equivalent to `wmill datatable migrate up` on each affected
* data table.
*/
export async function offerToRunNewMigrations(
workspace: string,
newMigrations: { datatable: string; timestamp: number; name: string }[],
opts?: { yes?: boolean; jsonOutput?: boolean },
): Promise<void> {
if (newMigrations.length === 0) return;
log.info(colors.green("New migrations were pushed:"));
for (const m of newMigrations) {
log.info(colors.gray(` ${m.datatable}: ${m.timestamp} ${m.name}`));
}
// Running migrations mutates the data tables, so skip the prompt in
// non-interactive contexts (--yes, --json, no TTY).
const interactive = !opts?.jsonOutput && !opts?.yes && !!process.stdin.isTTY;
if (!interactive) {
return;
}
const shouldRun = await Confirm.prompt({
message: "New migrations were pushed, run them?",
default: false,
});
if (!shouldRun) {
return;
}
for (const datatable of new Set(newMigrations.map((m) => m.datatable))) {
await runMigrations(workspace, datatable);
}
}
@@ -2,7 +2,7 @@ import { Command } from "@cliffy/command";
import { Confirm } from "@cliffy/prompt/confirm";
import { colors } from "@cliffy/ansi/colors";
import { sep as SEP } from "node:path";
import { GlobalOptions } from "../../types.ts";
import { GlobalOptions, isDatatableMigrationPath } from "../../types.ts";
import { SyncOptions, mergeConfigWithConfigFile } from "../../core/conf.ts";
import { resolveWorkspace } from "../../core/context.ts";
import { requireLogin } from "../../core/auth.ts";
@@ -54,6 +54,8 @@ async function walkLocalScripts(
(!isD && !exts.some((ext) => p.endsWith(ext))) ||
ignore(p, isD) ||
isFolderResourcePathAnyFormat(p) ||
// Datatable migration `.sql` files aren't Windmill scripts.
isDatatableMigrationPath(p) ||
(isScriptModulePath(p) && !isModuleEntryPoint(p)),
false,
{},
@@ -221,6 +223,8 @@ function categorizeLocalFiles(
} else if (
exts.some((ext) => p.endsWith(ext)) &&
!isFolderResourcePathAnyFormat(p) &&
// Datatable migration `.sql` files aren't Windmill scripts.
!isDatatableMigrationPath(p) &&
!(isScriptModulePath(p) && !isModuleEntryPoint(p))
) {
scripts.push(p);
+97 -9
View File
@@ -23,10 +23,17 @@ import {
showDiff,
extractNativeTriggerInfo,
redactEncryptionKey,
isDatatableMigrationPath,
parseDatatableMigrationPath,
} from "../../types.ts";
import { downloadZip } from "./pull.ts";
import { runLint, printReport, checkMissingLocks } from "../lint/lint.ts";
import { pullSharedUi, pushSharedUi } from "../shared_ui.ts";
import {
pushMigrationFromDisk,
offerToRunNewMigrations,
validateLocalMigrations,
} from "../datatable_migrations.ts";
import {
exts,
@@ -2477,7 +2484,8 @@ const isNotWmillFile = (p: string, isDirectory: boolean) => {
!p.startsWith("g" + SEP) &&
!p.startsWith("users" + SEP) &&
!p.startsWith("groups" + SEP) &&
!p.startsWith("dependencies" + SEP)
!p.startsWith("dependencies" + SEP) &&
!p.startsWith("migrations" + SEP)
);
}
@@ -2488,6 +2496,11 @@ const isNotWmillFile = (p: string, isDirectory: boolean) => {
try {
const typ = getTypeStrFromPath(p);
// Datatable migrations live under migrations/datatable/<datatable>/, outside
// the u/f/g namespaces, but are valid wmill files.
if (typ == "datatable_migration") {
return false;
}
if (
typ == "resource-type" ||
typ == "settings" ||
@@ -2519,7 +2532,8 @@ export const isWhitelisted = (p: string) => {
p == "ui" ||
p == "users" ||
p == "groups" ||
p == "dependencies"
p == "dependencies" ||
p == "migrations"
);
};
@@ -2614,6 +2628,11 @@ interface ChangeTracker {
}
async function addToChangedIfNotExists(p: string, tracker: ChangeTracker) {
// Datatable migration .sql files are not scripts; they're synced via the
// dedicated datatable_migration handler in the push loop.
if (isDatatableMigrationPath(p)) {
return;
}
const isScript = exts.some((e) => p.endsWith(e)) && !isFileResource(p) && !isFilesetResource(p);
if (isScript) {
if (isFlowPath(p)) {
@@ -3119,7 +3138,7 @@ export async function pull(
change.path.endsWith(".json")
) {
log.info(
`Editing ${getTypeStrFromPath(change.path)} ${targetPath}${
`Editing ${changeTypeLabel(change.path)}${targetPath}${
targetPath !== change.path
? colors.gray(` (workspace-specific override for ${change.path})`)
: ""
@@ -3137,7 +3156,7 @@ export async function pull(
if (opts.stateful) {
await mkdir(path.dirname(stateTarget), { recursive: true });
log.info(
`Adding ${getTypeStrFromPath(change.path)} ${targetPath}${
`Adding ${changeTypeLabel(change.path)}${targetPath}${
targetPath !== change.path
? colors.gray(` (workspace-specific override for ${change.path})`)
: ""
@@ -3146,7 +3165,7 @@ export async function pull(
}
await writeFile(target, change.content, "utf-8");
log.info(
`Writing ${getTypeStrFromPath(change.path)} ${targetPath}${
`Writing ${changeTypeLabel(change.path)}${targetPath}${
targetPath !== change.path
? colors.gray(` (workspace-specific override for ${change.path})`)
: ""
@@ -3158,7 +3177,7 @@ export async function pull(
} else if (change.name === "deleted") {
try {
log.info(
`Deleting ${getTypeStrFromPath(change.path)} ${change.path}`,
`Deleting ${changeTypeLabel(change.path)}${change.path}`,
);
await rm(target);
if (opts.stateful) {
@@ -3350,6 +3369,9 @@ export async function pull(
log.warn(`Failed to pull shared UI folder: ${e}`);
}
// Datatable migrations are part of the workspace export now, so they flow
// through the normal diff/apply above as `datatable_migration` items.
// Git-sync deployment-callback mode stops here: branch checkout + pull have
// happened, but commit + push are the caller's job. The hub script does
// them in-process with `set_gpg_signing_secret` so the agent's pre-warmed
@@ -3425,6 +3447,14 @@ export async function gitDeploy(
} as any);
}
// Display label for a change's type, with a trailing space. Datatable migrations
// are self-describing via their `migrations/datatable/...` path, so they get no
// label prefix.
function changeTypeLabel(p: string): string {
const t = getTypeStrFromPath(p);
return t === "datatable_migration" ? "" : `${t} `;
}
function prettyChanges(
changes: Change[],
specificItems?: SpecificItemsConfig,
@@ -3456,7 +3486,7 @@ function prettyChanges(
if (change.name === "added") {
log.info(
colors.green(
`+ ${getTypeStrFromPath(change.path)} ` +
`+ ${changeTypeLabel(change.path)}` +
displayPath +
colors.gray(wsNote),
) + extraNote,
@@ -3464,7 +3494,7 @@ function prettyChanges(
} else if (change.name === "deleted") {
log.info(
colors.red(
`- ${getTypeStrFromPath(change.path)} ` +
`- ${changeTypeLabel(change.path)}` +
displayPath +
colors.gray(wsNote),
),
@@ -3473,7 +3503,7 @@ function prettyChanges(
const changeType = getTypeStrFromPath(change.path);
log.info(
colors.yellow(
`~ ${changeType} ` +
`~ ${changeTypeLabel(change.path)}` +
displayPath +
colors.gray(wsNote) +
(change.codebase ? ` (codebase changed)` : ""),
@@ -4221,6 +4251,24 @@ export async function push(
));
}
// Reject malformed datatable migrations (duplicate timestamps, orphan downs)
// before touching the remote, scanning only the data tables in this push.
const migrationDatatables = new Set(
changes
.map((c) => parseDatatableMigrationPath(c.path)?.datatable)
.filter((d): d is string => !!d),
);
if (migrationDatatables.size > 0) {
const migrationErrors = validateLocalMigrations(migrationDatatables);
if (migrationErrors.length > 0) {
log.error(
"Invalid datatable migrations, aborting push:\n" +
migrationErrors.map((e) => ` - ${e}`).join("\n"),
);
process.exit(1);
}
}
if (
!opts.yes &&
!(await Confirm.prompt({
@@ -4293,6 +4341,21 @@ export async function push(
// Cache git branch at the start to avoid repeated execSync calls per change
const cachedWsNameForPush = wsNameForFiles || (isGitRepository() ? getCurrentGitBranch() : null);
// Datatable migrations are two files (.up.sql/.down.sql) for one record, so
// dedupe upsert/delete by (datatable, version) across the whole push.
const pushedMigrationKeys = new Set<string>();
// Migrations newly added by this push (an added .up.sql) — offered to run once
// the push has completed.
const newDatatableMigrations = changes
.filter((c) => c.name === "added")
.map((c) => parseDatatableMigrationPath(c.path))
.filter((p) => !!p && p.kind === "up")
.map((p) => ({
datatable: p!.datatable,
timestamp: p!.timestamp,
name: p!.name,
}));
while (queue.length > 0 || pool.size > 0) {
// Fill the pool until we reach the effective parallelism limit.
// During the folder-meta phase this is 1 (sequential) so no item change
@@ -4320,6 +4383,20 @@ export async function push(
}
for await (const change of changes) {
// A datatable migration is one record across two files; upsert/delete
// it from disk once (deduped), regardless of which file changed.
if (isDatatableMigrationPath(change.path)) {
const parsed = parseDatatableMigrationPath(change.path);
if (parsed) {
const key = `${parsed.datatable}\0${parsed.timestamp}`;
if (!pushedMigrationKeys.has(key)) {
pushedMigrationKeys.add(key);
await pushMigrationFromDisk(workspace.workspaceId, parsed);
}
}
continue;
}
let stateTarget = undefined;
if (stateful) {
try {
@@ -4974,6 +5051,16 @@ export async function push(
} catch (e) {
log.warn(`Failed to push shared UI folder: ${e}`);
}
try {
await offerToRunNewMigrations(workspace.workspaceId, newDatatableMigrations, {
yes: opts.yes,
jsonOutput: opts.jsonOutput,
});
} catch (e: any) {
log.warn(
`Failed to run new datatable migrations: ${e?.body ?? e?.message ?? e}`,
);
}
const lockJobs = await checkServerLockJobs(
workspace.workspaceId,
pushStartedAt,
@@ -5039,6 +5126,7 @@ export async function push(
} catch (e) {
log.warn(`Failed to push shared UI folder: ${e}`);
}
// No changes pushed, so no new datatable migrations to run.
if (opts.jsonOutput) {
console.log(
JSON.stringify(
+32
View File
@@ -11,10 +11,12 @@ import {
deleteItemInWorkspace,
getOnBehalfOf,
isTriggerOrScheduleKind,
parseDatatableMigrationDeployPath,
type DeployKind,
type DeployProvider,
type TriggerDeployKind,
} from "../../../windmill-utils-internal/src/deploy.ts";
import { offerToRunNewMigrations } from "../datatable_migrations.ts";
// ---------------------------------------------------------------------------
// Provider adapter — wraps CLI's standalone API functions
@@ -85,6 +87,10 @@ const provider: DeployProvider = {
createSchedule: wmill.createSchedule,
updateSchedule: wmill.updateSchedule,
deleteSchedule: wmill.deleteSchedule,
// Datatable migrations
listDatatableMigrations: wmill.listDatatableMigrations,
upsertDatatableMigration: wmill.upsertDatatableMigration,
deleteDatatableMigration: wmill.deleteDatatableMigration,
};
/**
@@ -530,6 +536,14 @@ async function mergeWorkspaces(
// 10. Deploy
let successCount = 0;
let failCount = 0;
// Datatable migrations deployed (not deleted) into the target. Deploying a
// migration only upserts its definition — the target schema is unchanged until
// the migration is run — so offer to run them afterwards (like the push path).
const deployedMigrations: {
datatable: string;
timestamp: number;
name: string;
}[] = [];
for (const diff of sorted) {
const label = `${diff.kind}:${diff.path}`;
@@ -573,6 +587,12 @@ async function mergeWorkspaces(
if (result.success) {
log.info(colors.green(`${label}`));
successCount++;
if (
!itemDeletedInSource &&
(diff.kind as DeployKind) === "datatable_migration"
) {
deployedMigrations.push(parseDatatableMigrationDeployPath(diff.path));
}
} else {
log.info(colors.red(`${label}: ${result.error}`));
failCount++;
@@ -606,6 +626,18 @@ async function mergeWorkspaces(
)
);
}
// 13. Deployed migration definitions don't touch the target schema until run;
// offer to run them on the target now (interactive only, like the push path).
if (deployedMigrations.length > 0) {
try {
await offerToRunNewMigrations(workspaceTo, deployedMigrations, {
yes: opts.yes,
});
} catch (e) {
log.warn(colors.yellow(`Failed to run deployed migrations: ${e}`));
}
}
}
export { mergeWorkspaces };
+16 -10
View File
@@ -6591,6 +6591,13 @@ datatable related commands
- \`datatable run <sql:string>\` - run a SQL query on a datatable
- \`-n --name <name:string>\` - Datatable name (default: main)
- \`-s --silent\` - Output only the final result as JSON. Useful for scripting.
- \`datatable migrate\` - manage datatable migrations
- \`datatable migrate new <name:string>\` - scaffold a new migration (.up.sql / .down.sql files)
- \`-d --datatable <datatable:string>\` - Target datatable (default: main)
- \`datatable migrate up\` - apply all pending migrations to the main datatable (or one via --datatable)
- \`-d --datatable <datatable:string>\` - Target datatable (default: main)
- \`datatable migrate down\` - roll back the most recent migration on the main datatable (or one via --datatable)
- \`-d --datatable <datatable:string>\` - Target datatable (default: main)
- \`datatable create [name:string]\` - register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable://<name>
- \`--resource <resource:string>\` - Back the datatable with an existing postgresql resource path instead of the instance database
- \`--force\` - Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved)
@@ -6859,19 +6866,18 @@ Manage jobs (list, inspect, cancel)
### jobs
Pull completed and queued jobs from workspace
**Arguments:** \`[workspace:string]\`
**Options:**
- \`-c, --completed-output <file:string>\` - Completed jobs output file (default: completed_jobs.json)
- \`-q, --queued-output <file:string>\` - Queued jobs output file (default: queued_jobs.json)
- \`--skip-worker-check\` - Skip checking for active workers before export
Manage jobs (import/export)
**Subcommands:**
- \`jobs pull\`
- \`jobs push\`
- \`jobs pull [workspace:string]\` - Pull completed and queued jobs from workspace
- \`-c, --completed-output <file:string>\` - Completed jobs output file (default: completed_jobs.json)
- \`-q, --queued-output <file:string>\` - Queued jobs output file (default: queued_jobs.json)
- \`--skip-worker-check\` - Skip checking for active workers before export
- \`jobs push [workspace:string]\` - Push completed and queued jobs to workspace
- \`-c, --completed-file <file:string>\` - Completed jobs input file (default: completed_jobs.json)
- \`-q, --queued-file <file:string>\` - Queued jobs input file (default: queued_jobs.json)
- \`--skip-worker-check\` - Skip checking for active workers before import
### lint
+36 -4
View File
@@ -274,10 +274,10 @@ export function parseFromPath(p: string, content: string): any {
return isWorkspaceDependencies(p)
? content
: p.endsWith(".yaml")
? yamlParseContent(p, content)
: p.endsWith(".json")
? JSON.parse(content)
: content;
? yamlParseContent(p, content)
: p.endsWith(".json")
? JSON.parse(content)
: content;
}
export function parseFromFile(p: string): any {
if (p.endsWith(".json")) {
@@ -288,9 +288,38 @@ export function parseFromFile(p: string): any {
throw new Error("Could not read file " + p);
}
}
/**
* Parse a `migrations/datatable/<datatable>/<timestamp>_<name>.(up|down).sql`
* path into its parts. Returns undefined for any other path.
*/
export function parseDatatableMigrationPath(p: string):
| { datatable: string; timestamp: number; name: string; kind: "up" | "down" }
| undefined {
const parts = p.split("/");
if (
parts[0] !== "migrations" ||
parts[1] !== "datatable" ||
parts.length !== 4
)
return undefined;
const m = parts[3].match(/^(\d+)_(.*)\.(up|down)\.sql$/);
if (!m) return undefined;
return {
datatable: parts[2],
timestamp: Number(m[1]),
name: m[2],
kind: m[3] as "up" | "down",
};
}
export function isDatatableMigrationPath(p: string): boolean {
return parseDatatableMigrationPath(p) !== undefined;
}
export function getTypeStrFromPath(
p: string
):
| "datatable_migration"
| "script"
| "variable"
| "flow"
@@ -316,6 +345,9 @@ export function getTypeStrFromPath(
| "settings"
| "encryption_key"
| "workspace_dependencies" {
if (isDatatableMigrationPath(p)) {
return "datatable_migration";
}
if (isScriptModulePath(p)) {
return "script";
}
+122
View File
@@ -0,0 +1,122 @@
/**
* Unit tests for datatable-migration path parsing and local validation.
*
* These exercise pure logic with no backend:
* - `parseDatatableMigrationPath` recognizes only the
* `migrations/datatable/<dt>/<timestamp>_<name>.(up|down).sql` shape.
* - `validateLocalMigrations` rejects the two invalid on-disk states a push
* must catch: two up (or two down) files sharing a timestamp, and a
* `.down.sql` with no matching `.up.sql`.
*/
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
import * as fs from "node:fs";
import * as path from "node:path";
import * as os from "node:os";
import { parseDatatableMigrationPath } from "../src/types.ts";
import { validateLocalMigrations } from "../src/commands/datatable_migrations.ts";
describe("parseDatatableMigrationPath", () => {
test("parses up and down files of the new layout", () => {
expect(
parseDatatableMigrationPath(
"migrations/datatable/mydt/20260101000001_create_users.up.sql",
),
).toEqual({
datatable: "mydt",
timestamp: 20260101000001,
name: "create_users",
kind: "up",
});
expect(
parseDatatableMigrationPath(
"migrations/datatable/my-dt/42_x.down.sql",
),
).toEqual({ datatable: "my-dt", timestamp: 42, name: "x", kind: "down" });
});
test("rejects unrelated, legacy and malformed paths", () => {
for (
const p of [
// legacy top-level layout
"datatable_migrations/mydt/20260101000001_x.up.sql",
// wrong sub-namespace / depth
"migrations/ducklake/mydt/1_x.up.sql",
"migrations/datatable/1_x.up.sql",
"migrations/datatable/mydt/sub/1_x.up.sql",
// not a migration file
"migrations/datatable/mydt/notes.txt",
"migrations/datatable/mydt/x.up.sql", // no numeric timestamp prefix
// unrelated workspace files
"f/foo/bar.script.yaml",
"u/admin/script.ts",
]
) {
expect(parseDatatableMigrationPath(p)).toBeUndefined();
}
});
});
describe("validateLocalMigrations", () => {
let prevCwd: string;
let tmp: string;
beforeEach(() => {
prevCwd = process.cwd();
tmp = fs.mkdtempSync(path.join(os.tmpdir(), "dtmig-"));
process.chdir(tmp);
});
afterEach(() => {
process.chdir(prevCwd);
fs.rmSync(tmp, { recursive: true, force: true });
});
function write(datatable: string, file: string) {
const dir = path.join(tmp, "migrations", "datatable", datatable);
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(path.join(dir, file), "-- sql\n");
}
test("accepts up+down pairs and up-only migrations", () => {
write("mydt", "20260101000001_create_users.up.sql");
write("mydt", "20260101000001_create_users.down.sql");
write("mydt", "20260101000002_add_email.up.sql"); // down is optional
expect(validateLocalMigrations()).toEqual([]);
});
test("flags two up files sharing a timestamp", () => {
write("mydt", "20260101000003_foo.up.sql");
write("mydt", "20260101000003_bar.up.sql");
const errors = validateLocalMigrations();
expect(errors.length).toBe(1);
expect(errors[0]).toContain("20260101000003");
});
test("flags two down files sharing a timestamp", () => {
write("mydt", "20260101000004_a.up.sql");
write("mydt", "20260101000004_a.down.sql");
write("mydt", "20260101000004_b.down.sql");
const errors = validateLocalMigrations();
// duplicate down + the b.down orphan (no b.up)
expect(errors.some((e) => e.includes("down") && e.includes("20260101000004"))).toBe(true);
});
test("flags a down file with no matching up", () => {
write("mydt", "20260101000005_orphan.down.sql");
const errors = validateLocalMigrations();
expect(errors.length).toBe(1);
expect(errors[0]).toContain("20260101000005_orphan");
});
test("only validates the requested datatables", () => {
write("bad", "20260101000006_x.up.sql");
write("bad", "20260101000006_y.up.sql"); // duplicate, but in 'bad'
write("good", "20260101000007_ok.up.sql");
expect(validateLocalMigrations(new Set(["good"]))).toEqual([]);
expect(validateLocalMigrations(new Set(["bad"])).length).toBe(1);
});
test("returns no errors when the migrations folder is absent", () => {
expect(validateLocalMigrations()).toEqual([]);
});
});
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "windmill-utils-internal",
"version": "1.7.0",
"version": "1.8.2",
"description": "Internal utility functions for Windmill",
"main": "dist/cjs/index.js",
"module": "dist/esm/index.js",
+124
View File
@@ -21,6 +21,7 @@ export type DeployKind =
| "resource_type"
| "folder"
| "schedule"
| "datatable_migration"
| "http_trigger"
| "websocket_trigger"
| "kafka_trigger"
@@ -161,6 +162,24 @@ export interface DeployProvider {
requestBody: any;
}): Promise<any>;
deleteFolder(p: { workspace: string; name: string }): Promise<any>;
// Datatable migrations. In the diff, an item's `path` is
// `<datatable>/<timestamp>_<name>` (see `parseDatatableMigrationDeployPath`).
listDatatableMigrations(p: { workspace: string }): Promise<any>;
upsertDatatableMigration(p: {
workspace: string;
datatableName: string;
requestBody: {
timestamp: number;
name: string;
code_up: string;
code_down?: string;
};
}): Promise<any>;
deleteDatatableMigration(p: {
workspace: string;
datatableName: string;
timestamp: number;
}): Promise<any>;
// Triggers — per-kind dispatch is delegated to the implementor so the shared
// module doesn't need to know about each of the 9 trigger services.
existsTriggerByKind(
@@ -299,6 +318,40 @@ function toError(e: unknown): string {
return err.body || err.message || String(e);
}
// A datatable-migration diff item's path is `<datatable>/<timestamp>_<name>`
// (mirrors the backend, e.g. `mydt/20260101000001_create_users`).
export function parseDatatableMigrationDeployPath(path: string): {
datatable: string;
timestamp: number;
name: string;
} {
const slash = path.indexOf("/");
const underscore = slash >= 0 ? path.indexOf("_", slash + 1) : -1;
if (slash < 0 || underscore < 0) {
throw new Error(`Invalid datatable migration path: ${path}`);
}
const datatable = path.slice(0, slash);
const timestamp = Number(path.slice(slash + 1, underscore));
const name = path.slice(underscore + 1);
if (!datatable || !Number.isFinite(timestamp) || !name) {
throw new Error(`Invalid datatable migration path: ${path}`);
}
return { datatable, timestamp, name };
}
// The backend rejects `upsertDatatableMigration` when the target data table
// hasn't opted in to migrations. Turn that opaque 400 into an explicit,
// deploy-context message (falls back to the original error otherwise).
function asMigrationsDisabledError(e: unknown, datatable: string): unknown {
const msg = (e as { body?: string; message?: string })?.body ?? ''
if (typeof msg === "string" && /migrations are not enabled/i.test(msg)) {
return new Error(
`Data table '${datatable}' has not opted in to migrations on the target workspace; enable migrations for it there before deploying its migrations.`
);
}
return e;
}
// ---------------------------------------------------------------------------
// checkItemExists
// ---------------------------------------------------------------------------
@@ -325,6 +378,12 @@ export async function checkItemExists(
return provider.existsFolder({ workspace, name: folderName(path) });
} else if (kind === "schedule") {
return provider.existsSchedule({ workspace, path });
} else if (kind === "datatable_migration") {
const { datatable, timestamp } = parseDatatableMigrationDeployPath(path);
const migrations = await provider.listDatatableMigrations({ workspace });
return (migrations as { datatable: string; timestamp: number }[]).some(
(m) => m.datatable === datatable && m.timestamp === timestamp
);
} else if (isTriggerKind(kind)) {
return provider.existsTriggerByKind(kind, { workspace, path });
}
@@ -639,6 +698,41 @@ export async function deployItem(
requestBody,
});
}
} else if (kind === "datatable_migration") {
const { datatable, timestamp } = parseDatatableMigrationDeployPath(path);
const migrations = await provider.listDatatableMigrations({
workspace: workspaceFrom,
});
const migration = (
migrations as {
datatable: string;
timestamp: number;
name: string;
code_up: string;
code_down?: string;
}[]
).find((m) => m.datatable === datatable && m.timestamp === timestamp);
if (!migration) {
throw new Error(
`Datatable migration ${path} not found in ${workspaceFrom}`
);
}
try {
await provider.upsertDatatableMigration({
workspace: workspaceTo,
datatableName: datatable,
requestBody: {
timestamp: migration.timestamp,
name: migration.name,
code_up: migration.code_up,
...(migration.code_down != null
? { code_down: migration.code_down }
: {}),
},
});
} catch (e) {
throw asMigrationsDisabledError(e, datatable);
}
} else {
throw new Error(`Unknown kind: ${kind}`);
}
@@ -684,6 +778,13 @@ export async function deleteItemInWorkspace(
await provider.deleteFolder({ workspace, name: folderName(path) });
} else if (kind === "schedule") {
await provider.deleteSchedule({ workspace, path });
} else if (kind === "datatable_migration") {
const { datatable, timestamp } = parseDatatableMigrationDeployPath(path);
await provider.deleteDatatableMigration({
workspace,
datatableName: datatable,
timestamp,
});
} else if (isTriggerKind(kind)) {
await provider.deleteTriggerByKind(kind, { workspace, path });
} else {
@@ -767,6 +868,29 @@ export async function getItemValue(
} else if (isTriggerKind(kind)) {
const trigger = await provider.getTriggerValue(kind, { workspace, path });
return stripTriggerOrScheduleRuntimeFields(trigger);
} else if (kind === "datatable_migration") {
// Surface the migration SQL so the diff drawer shows the up/down bodies a
// reviewer needs to inspect before deploying.
const { datatable, timestamp } = parseDatatableMigrationDeployPath(path);
const migrations = (await provider.listDatatableMigrations({
workspace,
})) as {
datatable: string;
timestamp: number;
name: string;
code_up: string;
code_down?: string;
}[];
const migration = migrations.find(
(m) => m.datatable === datatable && m.timestamp === timestamp
);
if (migration) {
return {
name: migration.name,
code_up: migration.code_up,
code_down: migration.code_down ?? null,
};
}
}
} catch {
// Item may not exist
+107
View File
@@ -0,0 +1,107 @@
# Data table permissions (EE)
Advanced, opt-in permissions for data tables, enforced natively by Postgres
**roles** and **row-level security (RLS)**. Enterprise-only; off by default (a
data table with no permission config keeps the legacy behavior where every
workspace member shares the owner role and has full access).
## Why roles, not session variables
Data table queries run **arbitrary user SQL** (a `postgresql` script job with
`database = "datatable://<name>"`). That rules out the usual RLS shortcut of
injecting identity via a session GUC (`SET app.user = ...`) — the user's own SQL
can overwrite it. It also rules out `SET ROLE` down to a low-privilege role — the
user can `RESET ROLE` back to the login role.
The only tamper-proof signal is the **connected role itself**. So a non-admin
query connects *directly* as a per-user login role; `current_user` is then
authoritative and cannot be escalated, because the session never held more
privilege than that role.
## Model
Windmill principals are mirrored into Postgres roles inside the data table's
physical database:
- `wm_u_<hash>` — one **LOGIN** role per user (`INHERIT`, deterministic
HMAC-derived password). The name hashes `(workspace_id, email)`; the workspace
id is folded in because Postgres roles are cluster-global and all instance data
tables share one server.
- `wm_g_<hash>` — one **NOLOGIN** role per group. User roles are granted
membership in their group roles, so group grants apply automatically. Roles are
only ever members of their own, minimally-granted groups — there is no path
upward.
- `_wm_principals(role_name, email, username)` — a mapping table (readable only
via the SECURITY DEFINER `wm_email()` helper, which returns the current role's
email) so policies can reference the acting user's email.
Two layers of control, both configured from the **Permissions** modal in the data
table settings:
1. **Access** (the `Access` tab) → Postgres `GRANT`s. Per user/group: no
access / read (`SELECT`) / read+write (`SELECT,INSERT,UPDATE,DELETE`), scoped
to one table or all tables (with `ALTER DEFAULT PRIVILEGES` so future tables
inherit the grant).
2. **Row policies** (the `Row policies` tab) → `CREATE POLICY`. RLS is enabled
**only** on tables that have a policy. Templates and raw `USING` /
`WITH CHECK` expressions (e.g. `owner = wm_email()`).
`CREATE` on `public` is **revoked** from principal roles, so a principal can
never create a table that escapes policy — all DDL flows through the migrations
path (which connects as the table owner).
### Admin bypass
We deliberately do **not** `FORCE` RLS. The table owner
(`custom_instance_user` for instance data tables; the resource user for resource
data tables) therefore bypasses RLS. Workspace admins connect as that owner, so
admins always keep full, unfiltered access. Non-admin principals are non-owners
and stay subject to RLS. This is safe because principals can never own a table.
## Enforcement path
`pg_executor` resolves `datatable://<name>` through
`get_datatable_resource_from_db_checked(db, w_id, name, acting_email)`
(`acting_email` = the job's `permissioned_as_email`):
- data table has no permissions / disabled, or the user is a **workspace
admin** → default owner connection (unchanged);
- other member → connect as their `wm_u_*` role; grants + RLS enforce;
- member with no provisioned role (instance data tables) → **denied** with a
clear error.
The EE logic lives in `windmill-common/src/datatable_permissions_ee.rs` behind the
`datatable_permissions` module switch (`_ee` under `private`, `_oss` stub
otherwise), mirroring the `pipeline_advanced` / `partition` pattern. On OSS the
resolver is a no-op, so the checked path equals the unchecked one.
## Provisioning
`set_datatable_permissions` writes the config into the catalog JSONB
(`workspace_settings.datatable`) and then **reconciles** the target database:
creates/updates roles, memberships, `_wm_principals` rows, grants, and policies.
Instance data tables provision as the Windmill superuser; resource data tables
provision with the resource's own credentials (best-effort — the reconciliation
log surfaces any privilege it lacked). `sync_datatable_permissions` re-runs
reconciliation from the stored config.
## Known limitations (v1)
- **Instance and resource data tables** are both attempted; resource support is
best-effort (needs `CREATEROLE` + table ownership on the resource credentials).
- **Group-membership drift**: roles, memberships, grants and policies are
reconciled to match the config only at save/**Sync** time. A user added to or
removed from a group (or a datatable getting new tables) takes effect on the
next Sync — reconcile is declarative and revokes removed access, so Sync both
grants and revokes. There is no per-query provisioning, so a brand-new member
is denied until an admin re-syncs.
- **New tables**: run **Sync** after schema migrations so freshly-created tables
pick up grants/policies (grants also propagate via default privileges; RLS
policies must be re-applied).
- **Enforced paths**: `postgresql` and DuckDB (`ATTACH 'datatable://...'`) script
jobs on a normal worker both resolve through the checked path. **Not** enforced
(resolve as owner): schema-introspection and the DB-manager "explore"
endpoints, Postgres-trigger capture, and **agent-worker** job execution. If you
rely on agent workers, treat data table permissions as advisory for those jobs.
- **Row policy expressions** are admin-authored raw SQL (like migrations) — they
are trusted input.
+55 -116
View File
@@ -94,7 +94,7 @@
"windmill-parser-wasm-wac": "1.668.6",
"windmill-parser-wasm-yaml": "1.593.0",
"windmill-sql-datatype-parser-wasm": "1.512.0",
"windmill-utils-internal": "^1.7.0",
"windmill-utils-internal": "1.8.2",
"xterm": "^5.3.0",
"xterm-readline": "^1.1.2",
"y-monaco": "^0.1.4",
@@ -291,7 +291,6 @@
"integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/helper-validator-identifier": "^7.28.5",
"js-tokens": "^4.0.0",
@@ -307,7 +306,6 @@
"integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=6.9.0"
}
@@ -865,7 +863,6 @@
}
],
"license": "MIT",
"peer": true,
"engines": {
"node": "^14 || ^16 || >=18"
},
@@ -875,21 +872,21 @@
}
},
"node_modules/@emnapi/core": {
"version": "1.10.0",
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz",
"integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==",
"version": "1.11.2",
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.2.tgz",
"integrity": "sha512-TC8MkTuZUtcTSiFeuC0ksCh9QIJ5+F21MvZ4Wn4ORfYaFJ/0dsiudv5tVkejgwZlwQ39jL9WWDe2lz8x0WglOA==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/wasi-threads": "1.2.1",
"@emnapi/wasi-threads": "1.2.2",
"tslib": "^2.4.0"
}
},
"node_modules/@emnapi/runtime": {
"version": "1.10.0",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz",
"integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==",
"version": "1.11.2",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.2.tgz",
"integrity": "sha512-kyOl3X0DuTiT1h2ft8r2fYO8JYtU9a9Xis/zBSiGArNaagCOWx90N1k2wxp18czFDH+OgcWGb5ZP/XMt3dcyPA==",
"dev": true,
"license": "MIT",
"optional": true,
@@ -898,9 +895,9 @@
}
},
"node_modules/@emnapi/wasi-threads": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz",
"integrity": "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==",
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.2.tgz",
"integrity": "sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==",
"dev": true,
"license": "MIT",
"optional": true,
@@ -1392,6 +1389,7 @@
"integrity": "sha512-Jer+M7DgIwT5IHfTayb4Iw/fkkxWNmC/mqn/nMh9JrbPbkxmyabfLQnhJ+JDn5HK77f84j34lubO3iqFtYAfMg==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@floating-ui/core": "^1.3.1",
"@floating-ui/dom": "^1.4.5",
@@ -1548,6 +1546,7 @@
"resolved": "https://registry.npmjs.org/@popperjs/core/-/core-2.11.8.tgz",
"integrity": "sha512-P1st0aksCrn9sGZhp8GMYwBnQsbvAWsZAX44oXNNvLHGqAOcoVxmjZiohstwQ7SqKnbR47akdNi+uleWD8+g6A==",
"license": "MIT",
"peer": true,
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/popperjs"
@@ -1774,8 +1773,8 @@
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/core": "1.10.0",
"@emnapi/runtime": "1.10.0",
"@emnapi/core": "1.11.2",
"@emnapi/runtime": "1.11.2",
"@napi-rs/wasm-runtime": "^1.1.4"
},
"engines": {
@@ -1912,6 +1911,7 @@
"integrity": "sha512-iAIPEahFgDJJyvz8g0jP08KvqnM6JvdW8YfsygZ+pMeMvyM2zssWMltcsotETvjSZ82G3VlitgDtBIvpQSZrTA==",
"devOptional": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@standard-schema/spec": "^1.0.0",
"@sveltejs/acorn-typescript": "^1.0.5",
@@ -2007,6 +2007,7 @@
"integrity": "sha512-ILXmxC7HAsnkK2eslgPetrqqW1BKSL7LktsFgqzNj83MaivMGZzluWq32m25j2mDOjmSKX7GGWahePhuEs7P/g==",
"devOptional": true,
"license": "MIT",
"peer": true,
"dependencies": {
"deepmerge": "^4.3.1",
"magic-string": "^0.30.21",
@@ -2468,8 +2469,7 @@
"resolved": "https://registry.npmjs.org/@types/minimist/-/minimist-1.2.5.tgz",
"integrity": "sha512-hov8bUuiLiyFPGyFPE1lwWhmzYbirOXQNNo40+y3zow8aFVTeyn3VWL0VFFfdNddA8S4Vf0Tc062rzyNr7Paag==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/@types/ms": {
"version": "2.1.0",
@@ -2482,8 +2482,7 @@
"resolved": "https://registry.npmjs.org/@types/normalize-package-data/-/normalize-package-data-2.4.4.tgz",
"integrity": "sha512-37i+OaWTh9qeK4LSHPsyRC7NahnGotNuZvjLSgcPzblpHB3rrCJxAOgI5gCdKm7coonsaX1Of0ILiTcnZjbfxA==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/@types/semver": {
"version": "7.7.1",
@@ -2552,6 +2551,7 @@
"integrity": "sha512-VlJEV0fOQ7BExOsHYAGrgbEiZoi8D+Bl2+f6V2RrXerRSylnp+ZBHmPvaIa8cz0Ajx7WO7Z5RqfgYg7ED1nRhA==",
"dev": true,
"license": "BSD-2-Clause",
"peer": true,
"dependencies": {
"@typescript-eslint/scope-manager": "5.62.0",
"@typescript-eslint/types": "5.62.0",
@@ -3079,6 +3079,7 @@
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz",
"integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==",
"license": "MIT",
"peer": true,
"bin": {
"acorn": "bin/acorn"
},
@@ -3132,6 +3133,7 @@
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.17.1.tgz",
"integrity": "sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.3",
"fast-uri": "^3.0.1",
@@ -3277,7 +3279,6 @@
"integrity": "sha512-3CYzex9M9FGQjCGMGyi6/31c8GJbgb0qGyrx5HWxPd0aCwh4cB2YjMb2Xf9UuoogrMrlO9cTqnB5rI5GHZTcUA==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@@ -3305,7 +3306,6 @@
"integrity": "sha512-Z7tMw1ytTXt5jqMcOP+OQteU1VuNK9Y02uuJtKQ1Sv69jXQKKg5cibLwGJow8yzZP+eAc18EmLGPal0bp36rvQ==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=8"
}
@@ -3386,8 +3386,7 @@
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-2.0.0.tgz",
"integrity": "sha512-1ugUSr8BHXRnK23KfuYS+gVMC3LB8QGH9W1iGtDPsNWoQbgtXSExkBu2aDR4epiGWZOjZsj6lDl/N/AqqTC3UA==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/base64-js": {
"version": "1.5.1",
@@ -3514,6 +3513,7 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"baseline-browser-mapping": "^2.8.9",
"caniuse-lite": "^1.0.30001746",
@@ -3711,7 +3711,6 @@
"integrity": "sha512-Rjs1H+A9R+Ig+4E/9oyB66UC5Mj9Xq3N//vcLf2WzgdTi/3gUu3Z9KoqmlrEG4VuuLK8wJHofxzdQXz/knhiYg==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"camelcase": "^6.3.0",
"map-obj": "^4.1.0",
@@ -3731,7 +3730,6 @@
"integrity": "sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=10"
},
@@ -3745,7 +3743,6 @@
"integrity": "sha512-WuyALRjWPDGtt/wzJiadO5AXY+8hZ80hVpe6MyivgraREW751X3SbhRvG3eLKOYN+8VEvqLcf3wdnt44Z4S4SA==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=10"
},
@@ -3759,7 +3756,6 @@
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
"dev": true,
"license": "(MIT OR CC0-1.0)",
"peer": true,
"engines": {
"node": ">=10"
},
@@ -3868,6 +3864,7 @@
"resolved": "https://registry.npmjs.org/chart.js/-/chart.js-4.5.1.tgz",
"integrity": "sha512-GIjfiT9dbmHRiYi6Nl2yFCq7kkwdkp1W/lp2J99rX0yo9tgJGn3lKQATztIjb5tVtevcBtIdICNWqlq5+E8/Pw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@kurkle/color": "^0.3.0"
},
@@ -4144,7 +4141,6 @@
"integrity": "sha512-kcZ6+W5QzcJ3P1Mt+83OUv/oHFqZHIx8DuxG6eZ5RGMERoLqp4BuGjhHLYGK+Kf5XVkQvqBSmAy/nGWN3qDgEA==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"import-fresh": "^3.3.0",
"js-yaml": "^4.1.0",
@@ -4209,7 +4205,6 @@
"integrity": "sha512-8HFEBPKhOpJPEPu70wJJetjKta86Gw9+CCyCnB3sui2qQfOvRyqBy4IKLKKAwdMpWb2lHXWk9Wb4Z6AmaUT1Pg==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=12"
}
@@ -4397,6 +4392,7 @@
"resolved": "https://registry.npmjs.org/cytoscape/-/cytoscape-3.34.0.tgz",
"integrity": "sha512-62rNSrioXw93uliKFBwjukeQyeWwH2PqDrTac31r2P6464u3AUvTk0xS4LVvT251g7IgkFunrI48ZEZGjywSOg==",
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10"
}
@@ -4819,6 +4815,7 @@
"resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz",
"integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==",
"license": "ISC",
"peer": true,
"engines": {
"node": ">=12"
}
@@ -4918,6 +4915,7 @@
"resolved": "https://registry.npmjs.org/date-fns/-/date-fns-2.30.0.tgz",
"integrity": "sha512-fnULvOpxnC5/Vg3NCiWelDsLiUc9bRwAPs/+LfTLNvetFCtCTN+yQz15C/fs4AwX1R9K5GLtLfn8QW+dWisaAw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/runtime": "^7.21.0"
},
@@ -4958,7 +4956,6 @@
"integrity": "sha512-VfxadyCECXgQlkoEAjeghAr5gY3Hf+IKjKb+X8tGVDtveCjN+USwprd2q3QXBR9T1+x2DG0XZF5/w+7HAtSaXA==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=10"
},
@@ -4972,7 +4969,6 @@
"integrity": "sha512-WiPxgEirIV0/eIOMcnFBA3/IJZAZqKnwAwWyvvdi4lsr1WCN22nhdf/3db3DoZcUjTV2SqfzIwNyp6y2xs3nmg==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"decamelize": "^1.1.0",
"map-obj": "^1.0.0"
@@ -4990,7 +4986,6 @@
"integrity": "sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@@ -5001,7 +4996,6 @@
"integrity": "sha512-7N/q3lyZ+LVCp7PzuxrJr4KMbBE2hW7BT7YNia330OFxIf4d3r5zVpicP2650l7CPN6RM9zOJRl3NGpqSiw3Eg==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@@ -5495,7 +5489,6 @@
"integrity": "sha512-sqQamAnR14VgCr1A618A3sGrygcpK+HEbenA/HiEAkkUwcZIIB/tgWqHFxWgOyDh4nB4JCRimh79dR5Ywc9MDQ==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"is-arrayish": "^0.2.1"
}
@@ -5593,6 +5586,7 @@
"deprecated": "This version is no longer supported. Please see https://eslint.org/version-support for other options.",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@eslint-community/eslint-utils": "^4.2.0",
"@eslint-community/regexpp": "^4.6.1",
@@ -6149,7 +6143,6 @@
"integrity": "sha512-eRnCtTTtGZFpQCwhJiUOuxPQWRXVKYDn0b2PeHfXL6/Zi53SLAzAHfVhVWK2AryC/WH05kGfxhFIPvTF0SXQzg==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">= 4.9.1"
}
@@ -6509,7 +6502,6 @@
"integrity": "sha512-NGbfmJBp9x8IxyJSd1P+otYK8vonoJactOogrVfFRIAEY1ukil8RSKDz2Yo7wh1oihl51l/r6W4epkeKJHqL8A==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"global-prefix": "^3.0.0"
},
@@ -6523,7 +6515,6 @@
"integrity": "sha512-awConJSVCHVGND6x3tmMaKcQvwXLhjdkmomy2W+Goaui8YPgYgXJZewhg3fWC+DlfqqQuWg8AwqjGTD2nAPVWg==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"ini": "^1.3.5",
"kind-of": "^6.0.2",
@@ -6539,7 +6530,6 @@
"integrity": "sha512-HxJdYWq1MTIQbJ3nw0cqssHoTNU267KlrDuGZ1WYlxDStUtKUhOaJmh112/TZmHxxUfuJqPXSOm7tDyas0OSIQ==",
"dev": true,
"license": "ISC",
"peer": true,
"dependencies": {
"isexe": "^2.0.0"
},
@@ -6589,8 +6579,7 @@
"resolved": "https://registry.npmjs.org/globjoin/-/globjoin-0.1.4.tgz",
"integrity": "sha512-xYfnw62CKG8nLkZBfWbhWwDw02CHty86jfPcc2cr3ZfeuK9ysoVPPEUxf21bAD/rWAgk52SuBrLJlefNy8mvFg==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/gopd": {
"version": "1.2.0",
@@ -6673,7 +6662,6 @@
"integrity": "sha512-VIZB+ibDhx7ObhAe7OVtoEbuP4h/MuOTHJ+J8h/eBXotJYl0fBgR72xDFCKgIh22OJZIOVNxBMWuhAr10r8HdA==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=6"
}
@@ -6910,7 +6898,6 @@
"integrity": "sha512-kyCuEOWjJqZuDbRHzL8V93NzQhwIB71oFWSyzVo+KPZI+pnQPPxucdkrOZvkLRnrf5URsQM+IJ09Dw29cRALIA==",
"dev": true,
"license": "ISC",
"peer": true,
"dependencies": {
"lru-cache": "^6.0.0"
},
@@ -6924,7 +6911,6 @@
"integrity": "sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA==",
"dev": true,
"license": "ISC",
"peer": true,
"dependencies": {
"yallist": "^4.0.0"
},
@@ -6937,8 +6923,7 @@
"resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz",
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
"dev": true,
"license": "ISC",
"peer": true
"license": "ISC"
},
"node_modules/html-tags": {
"version": "3.3.1",
@@ -6946,7 +6931,6 @@
"integrity": "sha512-ztqyC3kLto0e9WbNp0aeP+M3kTt+nbaIveGmUxAtZa+8iFgKLUOD4YKM5j+f3QD89bra7UeumolZHKuOXnTmeQ==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=8"
},
@@ -7042,7 +7026,6 @@
"integrity": "sha512-rKtvo6a868b5Hu3heneU+L4yEQ4jYKLtjpnPeUdK7h0yzXGmyBTypknlkCvHFBqfX9YlorEiMM6Dnq/5atfHkw==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=8"
}
@@ -7073,7 +7056,6 @@
"integrity": "sha512-m6FAo/spmsW2Ab2fU35JTYwtOKa2yAwXSwgjSv1TJzh4Mh7mC3lzAOVLBprb72XsTrgkEIsl7YrFNAiDiRhIGg==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=12"
},
@@ -7143,8 +7125,7 @@
"resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.2.1.tgz",
"integrity": "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/is-binary-path": {
"version": "2.1.0",
@@ -7249,7 +7230,6 @@
"integrity": "sha512-yvkRyxmFKEOQ4pNXCmJG5AEQNlXJS5LaONXo5/cLdTZdWvsZ1ioJEonLGAosKlMWE8lwUy/bJzMjcw8az73+Fg==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@@ -7260,7 +7240,6 @@
"integrity": "sha512-VRSzKkbMm5jMDoKLbltAkFQ5Qr7VDiTFGXxYFXXowVj387GeGNOCsOH6Msy00SGZ3Fp84b1Naa1psqgcCIEP5Q==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@@ -7365,8 +7344,7 @@
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
"integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/js-yaml": {
"version": "4.1.0",
@@ -7402,8 +7380,7 @@
"resolved": "https://registry.npmjs.org/json-parse-even-better-errors/-/json-parse-even-better-errors-2.3.1.tgz",
"integrity": "sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/json-refs": {
"version": "3.0.15",
@@ -7600,7 +7577,6 @@
"integrity": "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@@ -8206,8 +8182,7 @@
"resolved": "https://registry.npmjs.org/lodash.truncate/-/lodash.truncate-4.4.2.tgz",
"integrity": "sha512-jttmRe7bRse52OsWIMDLaXxWqRAmtIUccAQ3garviCqJjafXOfNMO0yMfNpdD6zbGaTU0P5Nz7e7gAT6cKmJRw==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/lodash.uniq": {
"version": "4.5.0",
@@ -8275,7 +8250,6 @@
"integrity": "sha512-hdN1wVrZbb29eBGiGjJbeP8JbKjq1urkHJ/LIP/NY48MZ1QVXUsQBV1G1zvYFHn1XE06cwjBsOI2K3Ulnj1YXQ==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=8"
},
@@ -8326,7 +8300,6 @@
"integrity": "sha512-APMBEanjybaPzUrfqU0IMU5I0AswKMH7k8OTLs0vvV4KZpExkTkY87nR/zpbuTPj+gARop7aGUbl11pnDfW6xg==",
"dev": true,
"license": "MIT",
"peer": true,
"funding": {
"type": "github",
"url": "https://github.com/sponsors/wooorm"
@@ -8567,7 +8540,6 @@
"integrity": "sha512-/d+PQ4GKmGvM9Bee/DPa8z3mXs/pkvJE2KEThngVNOqtmljC6K7NMPxtc2JeZYTmpWb9k/TmxjeL18ez3h7vCw==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@types/minimist": "^1.2.2",
"camelcase-keys": "^7.0.0",
@@ -8595,7 +8567,6 @@
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
"dev": true,
"license": "(MIT OR CC0-1.0)",
"peer": true,
"engines": {
"node": ">=10"
},
@@ -9330,7 +9301,6 @@
"integrity": "sha512-Q4r8ghd80yhO/0j1O3B2BjweX3fiHg9cdOwjJd2J76Q135c+NDxGCqdYKQ1SKBuFfgWbAUzBfvYjPUEeNgqN1A==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"arrify": "^1.0.1",
"is-plain-obj": "^1.1.0",
@@ -9396,6 +9366,7 @@
"resolved": "https://registry.npmjs.org/@codingame/monaco-vscode-editor-api/-/monaco-vscode-editor-api-25.0.0.tgz",
"integrity": "sha512-uiY06RTWFo2WZdh6OybkLlDhuG+8LlkjUDpr9/wW55uucqHo4X8fx4XKEtD98cscC+6FKQkbG2yyUiOJ/npHOw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@codingame/monaco-vscode-api": "25.0.0"
}
@@ -9632,7 +9603,6 @@
"integrity": "sha512-p2W1sgqij3zMMyRC067Dg16bfzVH+w7hyegmpIvZ4JNjqtGOVAIvLmjBx3yP7YTe9vKJgkoNOPjwQGogDoMXFA==",
"dev": true,
"license": "BSD-2-Clause",
"peer": true,
"dependencies": {
"hosted-git-info": "^4.0.1",
"is-core-module": "^2.5.0",
@@ -9984,7 +9954,6 @@
"integrity": "sha512-ayCKvm/phCGxOkYRSCM82iDwct8/EonSEgCSxWxD7ve6jHggsFl4fZVQBPRNgQoKiuV/odhFrGzQXZwbifC8Rg==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/code-frame": "^7.0.0",
"error-ex": "^1.3.1",
@@ -10306,6 +10275,7 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"nanoid": "^3.3.11",
"picocolors": "^1.1.1",
@@ -10494,6 +10464,7 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"lilconfig": "^3.0.0",
"yaml": "^2.3.4"
@@ -10883,8 +10854,7 @@
"resolved": "https://registry.npmjs.org/postcss-resolve-nested-selector/-/postcss-resolve-nested-selector-0.1.6.tgz",
"integrity": "sha512-0sglIs9Wmkzbr8lQwEyIzlDOOC9bGmfVKcJTaxv3vMmd3uo4o4DerC3En0bnmgceeql9BfC8hRkp7cg0fjdVqw==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/postcss-safe-parser": {
"version": "6.0.0",
@@ -11060,6 +11030,7 @@
"integrity": "sha512-I7AIg5boAr5R0FFtJ6rCfD+LFsWHp81dolrFD8S79U9tb8Az2nGrJncnMSnys+bpQJfRUzqs9hnA81OAA3hCuQ==",
"dev": true,
"license": "MIT",
"peer": true,
"bin": {
"prettier": "bin/prettier.cjs"
},
@@ -11363,7 +11334,6 @@
"integrity": "sha512-X1Fu3dPuk/8ZLsMhEj5f4wFAF0DWoK7qhGJvgaijocXxBmSToKfbFtqbxMO7bVjNA1dmE5huAzjXj/ey86iw9Q==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@types/normalize-package-data": "^2.4.0",
"normalize-package-data": "^3.0.2",
@@ -11383,7 +11353,6 @@
"integrity": "sha512-snVCqPczksT0HS2EC+SxUndvSzn6LRCwpfSvLrIfR5BKDQQZMaI6jPRC9dYvYFDRAuFEAnkwww8kBBNE/3VvzQ==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"find-up": "^5.0.0",
"read-pkg": "^6.0.0",
@@ -11402,7 +11371,6 @@
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
"dev": true,
"license": "(MIT OR CC0-1.0)",
"peer": true,
"engines": {
"node": ">=10"
},
@@ -11416,7 +11384,6 @@
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
"dev": true,
"license": "(MIT OR CC0-1.0)",
"peer": true,
"engines": {
"node": ">=10"
},
@@ -11459,7 +11426,6 @@
"integrity": "sha512-tYkDkVVtYkSVhuQ4zBgfvciymHaeuel+zFKXShfDnFP5SyVEP7qo70Rf1jTOTCx3vGNAbnEi/xFkcfQVMIBWag==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"indent-string": "^5.0.0",
"strip-indent": "^4.0.0"
@@ -12100,7 +12066,6 @@
"integrity": "sha512-qMCMfhY040cVHT43K9BFygqYbUPFZKHOg7K73mtTWJRb8pyP3fzf4Ixd5SzdEJQ6MRUg/WBnOLxghZtKKurENQ==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"ansi-styles": "^4.0.0",
"astral-regex": "^2.0.0",
@@ -12177,7 +12142,6 @@
"integrity": "sha512-kN9dJbvnySHULIluDHy32WHRUu3Og7B9sbY7tsFLctQkIqnMh3hErYgdMjTYuqmcXX+lK5T1lnUt3G7zNswmZA==",
"dev": true,
"license": "Apache-2.0",
"peer": true,
"dependencies": {
"spdx-expression-parse": "^3.0.0",
"spdx-license-ids": "^3.0.0"
@@ -12188,8 +12152,7 @@
"resolved": "https://registry.npmjs.org/spdx-exceptions/-/spdx-exceptions-2.5.0.tgz",
"integrity": "sha512-PiU42r+xO4UbUS1buo3LPJkjlO7430Xn5SVAhdpzzsPHsjbYVflnnFdATgabnLude+Cqu25p6N+g2lw/PFsa4w==",
"dev": true,
"license": "CC-BY-3.0",
"peer": true
"license": "CC-BY-3.0"
},
"node_modules/spdx-expression-parse": {
"version": "3.0.1",
@@ -12197,7 +12160,6 @@
"integrity": "sha512-cbqHunsQWnJNE6KhVSMsMeH5H/L9EpymbzqTQ3uLwNCLZ1Q481oWaofqH7nO6V07xlXwY6PhQdQ2IedWx/ZK4Q==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"spdx-exceptions": "^2.1.0",
"spdx-license-ids": "^3.0.0"
@@ -12208,8 +12170,7 @@
"resolved": "https://registry.npmjs.org/spdx-license-ids/-/spdx-license-ids-3.0.23.tgz",
"integrity": "sha512-CWLcCCH7VLu13TgOH+r8p1O/Znwhqv/dbb6lqWy67G+pT1kHmeD/+V36AVb/vq8QMIQwVShJ6Ssl5FPh0fuSdw==",
"dev": true,
"license": "CC0-1.0",
"peer": true
"license": "CC0-1.0"
},
"node_modules/sprintf-js": {
"version": "1.0.3",
@@ -12303,7 +12264,6 @@
"integrity": "sha512-SlyRoSkdh1dYP0PzclLE7r0M9sgbFKKMFXpFRUMNuKhQSbC6VQIGzq3E0qsfvGJaUFJPGv6Ws1NZ/haTAjfbMA==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=12"
},
@@ -12329,8 +12289,7 @@
"resolved": "https://registry.npmjs.org/style-search/-/style-search-0.1.0.tgz",
"integrity": "sha512-Dj1Okke1C3uKKwQcetra4jSuk0DqbzbYtXipzFlFMZtowbF1x7BKJwB9AayVMyFARvU8EDrZdcax4At/452cAg==",
"dev": true,
"license": "ISC",
"peer": true
"license": "ISC"
},
"node_modules/style-to-object": {
"version": "0.4.4",
@@ -12379,7 +12338,6 @@
"integrity": "sha512-78O4c6IswZ9TzpcIiQJIN49K3qNoXTM8zEJzhaTE/xRTCZswaovSEVIa/uwbOltZrk16X4jAxjaOhzz/hTm1Kw==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@csstools/css-parser-algorithms": "^2.3.1",
"@csstools/css-tokenizer": "^2.2.0",
@@ -12462,7 +12420,6 @@
}
],
"license": "MIT-0",
"peer": true,
"engines": {
"node": "^14 || ^16 || >=18"
},
@@ -12476,7 +12433,6 @@
"integrity": "sha512-TfW7/1iI4Cy7Y8L6iqNdZQVvdXn0f8B4QcIXmkIbtTIe/Okm/nSlHb4IwGzRVOd3WfSieCgvf5cMzEfySAIl0g==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"flat-cache": "^3.2.0"
},
@@ -12489,8 +12445,7 @@
"resolved": "https://registry.npmjs.org/known-css-properties/-/known-css-properties-0.29.0.tgz",
"integrity": "sha512-Ne7wqW7/9Cz54PDt4I3tcV+hAyat8ypyOGzYRJQfdxnnjeWsTxt1cy8pjvvKeI5kfXuyvULyeeAvwvvtAX3ayQ==",
"dev": true,
"license": "MIT",
"peer": true
"license": "MIT"
},
"node_modules/stylelint/node_modules/postcss-selector-parser": {
"version": "6.1.2",
@@ -12513,7 +12468,6 @@
"integrity": "sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=8"
}
@@ -12654,7 +12608,6 @@
"integrity": "sha512-zFObLMyZeEwzAoKCyu1B91U79K2t7ApXuQfo8OuxwXLDgcKxuwM+YvcbIhm6QWqz7mHUH1TVytR1PwVVjEuMig==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"has-flag": "^4.0.0",
"supports-color": "^7.0.0"
@@ -12684,6 +12637,7 @@
"resolved": "https://registry.npmjs.org/svelte/-/svelte-5.53.5.tgz",
"integrity": "sha512-YkqERnF05g8KLdDZwZrF8/i1eSbj6Eoat8Jjr2IfruZz9StLuBqo8sfCSzjosNKd+ZrQ8DkKZDjpO5y3ht1Pow==",
"license": "MIT",
"peer": true,
"dependencies": {
"@jridgewell/remapping": "^2.3.4",
"@jridgewell/sourcemap-codec": "^1.5.0",
@@ -12781,21 +12735,6 @@
}
}
},
"node_modules/svelte-check/node_modules/picomatch": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz",
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
"dev": true,
"license": "MIT",
"optional": true,
"peer": true,
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/svelte-eslint-parser": {
"version": "0.43.0",
"resolved": "https://registry.npmjs.org/svelte-eslint-parser/-/svelte-eslint-parser-0.43.0.tgz",
@@ -13025,8 +12964,7 @@
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/svg-tags/-/svg-tags-1.0.0.tgz",
"integrity": "sha512-ovssysQTa+luh7A5Weu3Rta6FJlFBBbInjOh722LIt6klpU2/HtdUbszju/G4devcvk8PGt7FCLv5wftu3THUA==",
"dev": true,
"peer": true
"dev": true
},
"node_modules/svgo": {
"version": "3.3.2",
@@ -13077,7 +13015,6 @@
"integrity": "sha512-9kY+CygyYM6j02t5YFHbNz2FN5QmYGv9zAjVp4lCDjlCw7amdckXlEt/bjMhUIfj4ThGRE4gCUH5+yGnNuPo5A==",
"dev": true,
"license": "BSD-3-Clause",
"peer": true,
"dependencies": {
"ajv": "^8.0.1",
"lodash.truncate": "^4.4.2",
@@ -13105,6 +13042,7 @@
"integrity": "sha512-6A2rnmW5xZMdw11LYjhcI5846rt9pbLSabY5XPxo+XWdxwZaFEn47Go4NzFiHu9sNNmr/kXivP1vStfvMaK1GQ==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@alloc/quick-lru": "^5.2.0",
"arg": "^5.0.2",
@@ -13357,6 +13295,7 @@
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
"devOptional": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=12"
},
@@ -13419,7 +13358,6 @@
"integrity": "sha512-jRKj0n0jXWo6kh62nA5TEh3+4igKDXLvzBJcPpiizP7oOolUrYIxmVBG9TOtHYFHoddUk6YvAkGeGoSVTXfQXQ==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=12"
},
@@ -13537,6 +13475,7 @@
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
"dev": true,
"license": "Apache-2.0",
"peer": true,
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
@@ -13773,7 +13712,6 @@
"integrity": "sha512-DpKm2Ui/xN7/HQKCtpZxoRWBhZ9Z0kqtygG8XCgNQ8ZlDnxuQmWhj566j8fN4Cu3/JmbhsDo7fcAJq4s9h27Ew==",
"dev": true,
"license": "Apache-2.0",
"peer": true,
"dependencies": {
"spdx-correct": "^3.0.0",
"spdx-expression-parse": "^3.0.0"
@@ -13827,6 +13765,7 @@
"integrity": "sha512-MFtjBYgzmSxmgA4RAfjIyXWpGe1oALnjgUTzzV7QLx/TKxCzjtMH6Fd9/eVK+5Fg1qNoz5VAwsmMs/NofrmJvw==",
"devOptional": true,
"license": "MIT",
"peer": true,
"dependencies": {
"lightningcss": "^1.32.0",
"picomatch": "^4.0.4",
@@ -14396,9 +14335,9 @@
"integrity": "sha512-uHNL8F72/Tf96xF3hOHnPDjkEyqXw7fNjcPJiUhth9sTQkcwUIoJMOdwm8/cs+j9kKVRJ4tgNYMHEBLylazp6g=="
},
"node_modules/windmill-utils-internal": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/windmill-utils-internal/-/windmill-utils-internal-1.7.0.tgz",
"integrity": "sha512-K5kAiJKhavfGatmicbJyqT+KFspzFZK5Ou14pHj4Xg6Q2Z1a4ZgziBdrRNsaeAuBhB62yXWZA0OXDzg936Ymmw==",
"version": "1.8.2",
"resolved": "https://registry.npmjs.org/windmill-utils-internal/-/windmill-utils-internal-1.8.2.tgz",
"integrity": "sha512-Otdn4iCE0QiOtMKPdHX5I89oECfZa3vmO2jdLdSETf7MBSv/gw9gvQm3oUQ29ymIHXLeuhQk/ebPD+WYolQJEA==",
"license": "Apache 2.0"
},
"node_modules/word-wrap": {
@@ -14534,7 +14473,6 @@
"integrity": "sha512-+QU2zd6OTD8XWIJCbffaiQeH9U73qIqafo1x6V1snCWYGJf6cVE0cDR4D8xRzcEnfI21IFrUPzPGtcPf8AC+Rw==",
"dev": true,
"license": "ISC",
"peer": true,
"dependencies": {
"imurmurhash": "^0.1.4",
"signal-exit": "^4.0.1"
@@ -14730,7 +14668,6 @@
"integrity": "sha512-y11nGElTIV+CT3Zv9t7VKl+Q3hTQoT9a1Qzezhhl6Rp21gJ/IVTW7Z3y9EWXhuUBC2Shnf+DX0antecpAwSP8w==",
"dev": true,
"license": "ISC",
"peer": true,
"engines": {
"node": ">=10"
}
@@ -14750,6 +14687,7 @@
"resolved": "https://registry.npmjs.org/yjs/-/yjs-13.6.27.tgz",
"integrity": "sha512-OIDwaflOaq4wC6YlPBy2L6ceKeKuF7DeTxx+jPzv1FHn9tCZ0ZwSRnUBxD05E3yed46fv/FWJbvR+Ud7x0L7zw==",
"license": "MIT",
"peer": true,
"dependencies": {
"lib0": "^0.2.99"
},
@@ -14785,6 +14723,7 @@
"resolved": "https://registry.npmjs.org/zod/-/zod-4.1.12.tgz",
"integrity": "sha512-JInaHOamG8pt5+Ey8kGmdcAcg3OL9reK8ltczgHTAwNhMys/6ThXHityHxVV2p3fkw/c+MAvBHFVYHFZDmjMCQ==",
"license": "MIT",
"peer": true,
"funding": {
"url": "https://github.com/sponsors/colinhacks"
}
+1 -1
View File
@@ -169,7 +169,7 @@
"windmill-parser-wasm-wac": "1.668.6",
"windmill-parser-wasm-yaml": "1.593.0",
"windmill-sql-datatype-parser-wasm": "1.512.0",
"windmill-utils-internal": "^1.7.0",
"windmill-utils-internal": "1.8.2",
"xterm": "^5.3.0",
"xterm-readline": "^1.1.2",
"y-monaco": "^0.1.4",
@@ -646,7 +646,7 @@
deploymentStatus[statusKey] = { status: 'deployed' }
} else {
deploymentStatus[statusKey] = { status: 'failed', error: result.error }
sendUserToast(`Failed to deploy ${statusKey}: ${result.error}`)
sendUserToast(`Failed to deploy ${statusKey}: ${result.error}`, 'error')
}
}
@@ -696,7 +696,10 @@
if (!aIsFolder && bIsFolder) return 1
return 0
})
const to = mergeIntoParent ? parent : current
let anyFailed = false
// Datatables whose migrations deployed cleanly — candidates for a run prompt.
const deployedMigrationDatatables = new Set<string>()
for (const itemKey of sortedItems) {
const deployable = deployableItems.find((d) => d.key === itemKey)
@@ -705,16 +708,19 @@
continue
}
const to = mergeIntoParent ? parent : current
const from = mergeIntoParent ? current : parent
await deploy(deployable.kind, deployable.path, to, from, itemKey)
if (deploymentStatus[itemKey]?.status === 'failed') {
anyFailed = true
} else if (deployable.kind === 'datatable_migration') {
deployedMigrationDatatables.add(deployable.path.split('/')[0])
}
}
deploying = false
deselectAll()
await maybePromptRunMigrations(deployedMigrationDatatables, to)
// If every selected item deployed cleanly and the direction was
// merge-into-parent, resolve any open deployment request for this fork.
if (!anyFailed && mergeIntoParent) {
@@ -750,6 +756,51 @@
onChanged?.()
}
/**
* After a deploy, offer to run the migrations of every cloned datatable that
* received one. `forked_from` is set on the fork's datatable config only when
* the datatable was cloned into a separate database — shared-DB datatables
* have already had the schema change applied and must not be re-run.
*/
async function maybePromptRunMigrations(
deployedMigrationDatatables: Set<string>,
runTargetWorkspace: string
) {
if (deployedMigrationDatatables.size === 0) return
try {
const forkSettings = await WorkspaceService.getPublicSettings({
workspace: currentWorkspaceId
})
const datatables = forkSettings.datatable?.datatables ?? {}
const cloned = [...deployedMigrationDatatables].filter(
(dt) => datatables[dt]?.forked_from != null
)
if (cloned.length === 0) return
runMigrationsDatatables = cloned.sort()
runMigrationsTargetWorkspace = runTargetWorkspace
runMigrationsModalOpen = true
} catch (e) {
console.error('Failed to determine cloned datatables for migration run prompt', e)
}
}
async function runDeployedMigrations() {
runMigrationsModalOpen = false
for (const dt of runMigrationsDatatables) {
try {
const res = await WorkspaceService.runDatatableMigrations({
workspace: runMigrationsTargetWorkspace,
datatableName: dt
})
sendUserToast(
`Ran ${res.applied.length} migration${res.applied.length !== 1 ? 's' : ''} on ${dt}`
)
} catch (e: any) {
sendUserToast(`Failed to run migrations on ${dt}: ${e.body ?? e.message ?? e}`, true)
}
}
}
function toggleKey(key: string) {
if (selectedItems.includes(key)) {
selectedItems = selectedItems.filter((i) => i !== key)
@@ -930,6 +981,17 @@
let deploymentRequestPanel: DeploymentRequestPanel | undefined = $state(undefined)
let hasOpenDeploymentRequest = $state(false)
// After deploying datatable migrations to a cloned (separate-DB) datatable, we
// offer to run them in the target workspace. Shared-DB datatables are skipped:
// the schema change is already physically applied, so re-running is redundant.
let runMigrationsModalOpen = $state(false)
let runMigrationsDatatables = $state<string[]>([])
let runMigrationsTargetWorkspace = $state('')
let runMigrationsTargetWorkspaceName = $derived(
$userWorkspaces.find((w) => w.id == runMigrationsTargetWorkspace)?.name ??
runMigrationsTargetWorkspace
)
/** Display labels for trigger/schedule kinds in the merge UI. */
const KIND_DISPLAY_NAMES: Record<string, string> = {
schedule: 'Schedule',
@@ -942,7 +1004,8 @@
sqs_trigger: 'SQS trigger',
gcp_trigger: 'GCP trigger',
azure_trigger: 'Azure trigger',
email_trigger: 'Email trigger'
email_trigger: 'Email trigger',
datatable_migration: 'Data table migration'
}
// Human label for a diff kind, lowercased for inline use in the hidden-items
@@ -1453,9 +1516,7 @@
/>
</div>
<div class="bg-surface-tertiary p-4 rounded-md border">
<DatatableSchemaDiff {currentWorkspaceId} {parentWorkspaceId} />
</div>
<DatatableSchemaDiff {currentWorkspaceId} {parentWorkspaceId} />
{#if pinnedItems.length > 0}
<div class="bg-surface-tertiary p-4 rounded-md border flex flex-col gap-2">
@@ -1536,6 +1597,26 @@
</div>
</ConfirmationModal>
<ConfirmationModal
open={runMigrationsModalOpen}
title="Run datatable migrations?"
confirmationText="Run migrations"
onConfirmed={runDeployedMigrations}
onCanceled={() => (runMigrationsModalOpen = false)}
>
<div class="flex flex-col gap-2">
<p>
Run the deployed migrations in <b>{runMigrationsTargetWorkspaceName}</b> now? These data tables
use a separate database, so the schema changes won't apply until the migrations are run.
</p>
<ul class="list-disc pl-5 text-sm font-mono text-secondary">
{#each runMigrationsDatatables as dt (dt)}
<li>{dt}</li>
{/each}
</ul>
</div>
</ConfirmationModal>
<ConfirmationModal
open={!!createConfirm}
title="Create in {createConfirm?.onCurrent ? parentWorkspaceId : currentWorkspaceId}?"
+6 -4
View File
@@ -370,7 +370,7 @@
refresh?.()
sendUserToast(`Schema '${schemaKey}' deleted successfully`)
} catch (e) {
let msg: string | undefined = (e as Error).message
let msg: string | undefined = (e as any).body ?? (e as Error).message
if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : undefined
sendUserToast(msg ?? 'Action failed!', true)
}
@@ -436,7 +436,7 @@
refresh?.()
sendUserToast(`Table '${tableKey}' deleted successfully`)
} catch (e) {
let msg: string | undefined = (e as Error).message
let msg: string | undefined = (e as any).body ?? (e as Error).message
if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : undefined
sendUserToast(msg ?? 'Action failed!', true)
}
@@ -502,7 +502,7 @@
refresh?.()
sendUserToast(`Table '${tableKey}' deleted successfully`)
} catch (e) {
let msg: string | undefined = (e as Error).message
let msg: string | undefined = (e as any).body ?? (e as Error).message
if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : undefined
sendUserToast(msg ?? 'Action failed!', true)
}
@@ -605,7 +605,9 @@
onConfirm={async ({ values }) => {
if (dbTableEditorState.alterTableKey && dbTableEditorAlterTableData.current) {
let diff = diffTableEditorValues(dbTableEditorAlterTableData.current, values)
await dbSchemaOps.onAlter({ schema: selected.schemaKey, values: diff })
// Reverse diff (new → old) so the migration's down undoes the alter.
let reverse = diffTableEditorValues(values, dbTableEditorAlterTableData.current)
await dbSchemaOps.onAlter({ schema: selected.schemaKey, values: diff, reverse })
} else {
await dbSchemaOps.onCreate({ values, schema: selected.schemaKey })
}
@@ -20,6 +20,10 @@
import type { SelectedTable } from './DBManager.svelte'
import { getDbFeatures } from './apps/components/display/dbtable/dbFeatures'
import { resource } from 'runed'
import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from './common/confirmationModal/asyncConfirmationModal.svelte'
import Portal from '$lib/components/Portal.svelte'
import { outOfOrderRunMessage } from './workspaceSettings/datatableMigrationUtils'
interface Props {
input?: DbInput
@@ -52,6 +56,8 @@
let dbSchema: DBSchema | undefined = $derived(input && $dbSchemas[getDbSchemasPath(input)])
const outOfOrderModal = createAsyncConfirmationModal()
function getDbSchemasPath(input: DbInput): string {
switch (input.type) {
case 'database':
@@ -163,7 +169,13 @@
})}
dbSchemaOps={dbSchemaOpsWithPreviewScripts({
input: _input,
workspace: $workspaceStore
workspace: $workspaceStore,
confirmRunOutOfOrder: (pending) =>
outOfOrderModal.ask({
title: 'Run migration out of order',
confirmationText: 'Run anyway',
children: outOfOrderRunMessage(pending)
})
})}
initialTableKey={input.specificTable}
initialSchemaKey={input.specificSchema}
@@ -192,6 +204,7 @@
onData={(data) => {
replResultData = data
}}
onSchemaChange={() => refresh()}
placeholderTableName={sortArray(
Object.keys(
dbSchema?.schema[
@@ -214,3 +227,7 @@
</Pane>
</Splitpanes>
{/if}
<Portal>
<ConfirmationModal {...outOfOrderModal.props} />
</Portal>
@@ -16,6 +16,7 @@
Upload
} from 'lucide-svelte'
import DBManagerContent from './DBManagerContent.svelte'
import DataTableMigrationsButton from './workspaceSettings/DataTableMigrationsButton.svelte'
import { resource } from 'runed'
import { untrack } from 'svelte'
import type { DbManagerUriState } from './dbManagerDrawerModel.svelte'
@@ -105,6 +106,11 @@
return toSourceIdentifier(input.resourcePath)
}
function refreshManager() {
dbManagerContent?.refresh()
dbManagerContent?.dbManager()?.dbTable()?.refresh()
}
async function handleExportSchema() {
const source = currentSourceIdentifier()
if (!source || !$workspaceStore) return
@@ -201,6 +207,13 @@
{/key}
{/if}
{#snippet actions()}
{#if uriState.isDatatableInput && uriState.selectedDatatable && $workspaceStore}
<DataTableMigrationsButton
workspace={$workspaceStore}
datatable={uriState.selectedDatatable}
onSchemaChanged={refreshManager}
/>
{/if}
{#if enableImportExport}
<Button startIcon={{ icon: Download }} onClick={handleExportSchema}>Export</Button>
<Button startIcon={{ icon: Upload }} onClick={() => (importDrawerOpen = true)}>
@@ -209,16 +222,13 @@
{/if}
<Button
loading={dbManagerContent?.isLoading() ?? false}
on:click={() => {
dbManagerContent?.refresh()
dbManagerContent?.dbManager()?.dbTable()?.refresh()
}}
on:click={refreshManager}
startIcon={{ icon: RefreshCcw }}
iconOnly
title="Refresh"
size="xs"
color="light"
>
Refresh
</Button>
/>
<Button
on:click={() => (expand = !expand)}
@@ -234,7 +244,9 @@
<DrawerContent title="Export Schemas" on:close={() => (exportDrawerOpen = false)}>
{#if exportResult}
<div class="flex flex-col gap-2 h-full relative">
<pre class="overflow-auto text-xs bg-surface-secondary p-4 rounded flex-1">{exportResult}</pre>
<pre class="overflow-auto text-xs bg-surface-secondary p-4 rounded flex-1"
>{exportResult}</pre
>
<Button
size="xs"
color="light"
@@ -52,6 +52,7 @@
import Popover from './meltComponents/Popover.svelte'
import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte'
import { sendUserToast } from '$lib/toast'
import { MigrationRunCancelled } from './dbOps'
import { getFlatTableNamesFromSchema, type DBSchema } from '$lib/stores'
import { twMerge } from 'tailwind-merge'
import DarkModeObserver from './DarkModeObserver.svelte'
@@ -460,9 +461,13 @@
askingForConfirmation && (askingForConfirmation.loading = true)
await onConfirm({ values })
} catch (e) {
let msg: string | undefined = (e as Error)?.message
if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : 'An error occurred'
sendUserToast(msg, true)
// User declined the out-of-order run warning: silent cancel,
// leave the editor open so they can adjust or run earlier first.
if (!(e instanceof MigrationRunCancelled)) {
let msg: string | undefined = (e as any)?.body ?? (e as Error)?.message
if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : 'An error occurred'
sendUserToast(msg, true)
}
}
askingForConfirmation = undefined
},
@@ -472,7 +477,7 @@
...(preview && { codeContent: preview.sql, alert: preview.alert })
}
} catch (e) {
let msg: string | undefined = (e as Error)?.message
let msg: string | undefined = (e as any)?.body ?? (e as Error)?.message
if (typeof msg !== 'string') msg = e ? JSON.stringify(e) : 'An error occurred'
sendUserToast(msg, true)
} finally {
@@ -196,10 +196,17 @@
import Drawer from '$lib/components/common/drawer/Drawer.svelte'
import SimpleEditor from '$lib/components/SimpleEditor.svelte'
import { sendUserToast } from '$lib/toast'
import { userWorkspaces } from '$lib/stores'
import { runScriptAndPollResult } from '$lib/components/jobs/utils'
import YAML from 'yaml'
import DrawerContent from './common/drawer/DrawerContent.svelte'
import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from './common/confirmationModal/asyncConfirmationModal.svelte'
import Portal from '$lib/components/Portal.svelte'
import {
pendingMigrations,
outOfOrderRunMessage
} from './workspaceSettings/datatableMigrationUtils'
import Alert from './common/alert/Alert.svelte'
import ResizeTransitionWrapper from './common/ResizeTransitionWrapper.svelte'
@@ -213,6 +220,11 @@
let loading = $state(true)
let error: string | undefined = $state(undefined)
let diffs: DatatableDiff[] = $state([])
// Number of forked datatables this schema-diff section applies to: those that
// have NOT opted in to the migrations feature. When a datatable enables
// migrations, its changes flow through the normal item diff instead, so it is
// excluded here and the whole section hides once none remain.
let applicableCount = $state(0)
let expandedDatatables: Set<string> = $state(new Set())
// Drawer state
@@ -223,6 +235,7 @@
let migrationSql = $state('')
let migrationRunning = $state(false)
let confirmDeployOpen = $state(false)
const outOfOrderModal = createAsyncConfirmationModal()
async function loadDiffs() {
loading = true
@@ -233,7 +246,10 @@
workspace: currentWorkspaceId
})
const datatables = forkSettings.datatable?.datatables ?? {}
const forkedEntries = Object.entries(datatables).filter(([_, dt]) => dt.forked_from != null)
const forkedEntries = Object.entries(datatables).filter(
([_, dt]) => dt.forked_from != null && dt.migrations_enabled !== true
)
applicableCount = forkedEntries.length
if (forkedEntries.length === 0) {
loading = false
return
@@ -343,14 +359,68 @@
const dtName = drawerDiff.datatableName
try {
await runScriptAndPollResult({
// If the target data table opted in to migrations, record this merge as a
// tracked migration (named after the fork) and run it, instead of applying
// raw SQL that would bypass the target's migration history.
// Don't swallow a status-check failure by defaulting to raw apply: that
// would apply the DDL untracked (schema drift) — exactly what this feature
// prevents. Let the error propagate (fail closed, handled by the outer
// catch); only fall back to raw apply when the API explicitly returns
// enabled === false.
const status = await WorkspaceService.getDatatableMigrationsStatus({
workspace: targetWorkspace,
requestBody: {
args: { database: `datatable://${dtName}` },
language: 'postgresql',
content: migrationSql
}
datatableName: dtName
})
if (status.enabled) {
// The merge migration gets the highest timestamp, so any still-pending
// migration on the target is earlier: running only the merge applies it
// out of order. Warn like the row-level Run action does.
const pending = pendingMigrations(status.migrations)
if (pending.length > 0) {
const confirmed = await outOfOrderModal.ask({
title: 'Run migration out of order',
confirmationText: 'Run anyway',
children: outOfOrderRunMessage(pending.length)
})
if (!confirmed) {
migrationRunning = false
return
}
}
const forkName =
$userWorkspaces.find((w) => w.id === currentWorkspaceId)?.name || currentWorkspaceId
const migName = `merge_${forkName}`.replace(/[^a-zA-Z0-9_-]+/g, '_')
const created = await WorkspaceService.createDatatableMigration({
workspace: targetWorkspace,
datatableName: dtName,
requestBody: { name: migName, code_up: migrationSql }
})
try {
await WorkspaceService.runDatatableMigrations({
workspace: targetWorkspace,
datatableName: dtName,
only: created.timestamp
})
} catch (runErr: any) {
// Undo the insertion so the user can fix and retry from a clean state.
await WorkspaceService.deleteDatatableMigration({
workspace: targetWorkspace,
datatableName: dtName,
timestamp: created.timestamp
}).catch(() => {})
throw runErr
}
} else {
await runScriptAndPollResult({
workspace: targetWorkspace,
requestBody: {
args: { database: `datatable://${dtName}` },
language: 'postgresql',
content: migrationSql
}
})
}
} catch (e: any) {
sendUserToast(e?.body ?? e?.message ?? String(e), true)
migrationRunning = false
@@ -394,102 +464,107 @@
}
</script>
<h3 class="text-sm font-semibold">Datatable schema changes</h3>
{#if loading}
<div class="flex items-center gap-2 text-xs text-tertiary py-2">
<Loader2 class="w-4 h-4 animate-spin" /> Loading datatable diffs...
</div>
{:else if error}
<div class="text-xs text-red-500 py-2">Failed to load datatable diffs: {error}</div>
{:else if diffs.length > 0}
<div class="flex flex-col gap-2 mt-3 mb-1">
{#each diffs as diff}
<ResizeTransitionWrapper class="border rounded-md" innerClass="w-full" vertical>
<button
class="w-full flex items-center justify-between px-3 py-2 hover:bg-surface-hover"
onclick={() => toggleExpanded(diff.datatableName)}
>
<span class="text-xs font-medium">{diff.datatableName}</span>
<div class="flex items-center gap-2 text-2xs text-tertiary">
{#if diff.aheadChanges.length > 0}
<span class="text-blue-500">{diff.aheadChanges.length} ahead</span>
{/if}
{#if diff.behindChanges.length > 0}
<span class="text-orange-500">{diff.behindChanges.length} behind</span>
{/if}
{#if expandedDatatables.has(diff.datatableName)}
<ChevronDown class="w-3 h-3" />
{:else}
<ChevronRight class="w-3 h-3" />
{/if}
</div>
</button>
{#if applicableCount > 0}
<div class="bg-surface-tertiary p-4 rounded-md border">
<h3 class="text-sm font-semibold">Datatable schema changes</h3>
{#if loading}
<div class="flex items-center gap-2 text-xs text-tertiary py-2">
<Loader2 class="w-4 h-4 animate-spin" /> Loading datatable diffs...
</div>
{:else if error}
<div class="text-xs text-red-500 py-2">Failed to load datatable diffs: {error}</div>
{:else if diffs.length > 0}
<div class="flex flex-col gap-2 mt-3 mb-1">
{#each diffs as diff}
<ResizeTransitionWrapper class="border rounded-md" innerClass="w-full" vertical>
<button
class="w-full flex items-center justify-between px-3 py-2 hover:bg-surface-hover"
onclick={() => toggleExpanded(diff.datatableName)}
>
<span class="text-xs font-medium">{diff.datatableName}</span>
<div class="flex items-center gap-2 text-2xs text-tertiary">
{#if diff.aheadChanges.length > 0}
<span class="text-blue-500">{diff.aheadChanges.length} ahead</span>
{/if}
{#if diff.behindChanges.length > 0}
<span class="text-orange-500">{diff.behindChanges.length} behind</span>
{/if}
{#if expandedDatatables.has(diff.datatableName)}
<ChevronDown class="w-3 h-3" />
{:else}
<ChevronRight class="w-3 h-3" />
{/if}
</div>
</button>
{#if expandedDatatables.has(diff.datatableName)}
<div class="border-t divide-y">
{#if diff.aheadChanges.length > 0}
<div class="px-3 py-1.5">
<div class="text-2xs font-semibold text-blue-500 mb-1">Fork changes (ahead)</div>
{#each diff.aheadChanges as change}
<div class="flex items-center gap-2 text-xs py-0.5">
{#if change.kind === 'added'}
<Plus class="w-3 h-3 text-green-500 shrink-0" />
{:else if change.kind === 'removed'}
<Minus class="w-3 h-3 text-red-500 shrink-0" />
{:else}
<Pencil class="w-3 h-3 text-yellow-500 shrink-0" />
{/if}
<span class="text-tertiary">{change.schemaName}.</span>
<span class="font-medium">{change.tableName}</span>
<span class="text-tertiary text-2xs grow">{operationSummary(change)}</span>
<Button
size="xs"
variant="subtle"
startIcon={{ icon: Eye }}
onclick={() => openReview(change, diff, 'ahead')}
{#if expandedDatatables.has(diff.datatableName)}
<div class="border-t divide-y">
{#if diff.aheadChanges.length > 0}
<div class="px-3 py-1.5">
<div class="text-2xs font-semibold text-blue-500 mb-1">Fork changes (ahead)</div
>
Review
</Button>
{#each diff.aheadChanges as change}
<div class="flex items-center gap-2 text-xs py-0.5">
{#if change.kind === 'added'}
<Plus class="w-3 h-3 text-green-500 shrink-0" />
{:else if change.kind === 'removed'}
<Minus class="w-3 h-3 text-red-500 shrink-0" />
{:else}
<Pencil class="w-3 h-3 text-yellow-500 shrink-0" />
{/if}
<span class="text-tertiary">{change.schemaName}.</span>
<span class="font-medium">{change.tableName}</span>
<span class="text-tertiary text-2xs grow">{operationSummary(change)}</span>
<Button
size="xs"
variant="subtle"
startIcon={{ icon: Eye }}
onclick={() => openReview(change, diff, 'ahead')}
>
Review
</Button>
</div>
{/each}
</div>
{/each}
{/if}
{#if diff.behindChanges.length > 0}
<div class="px-3 py-1.5">
<div class="text-2xs font-semibold text-orange-500 mb-1">
Parent changes (behind)
</div>
{#each diff.behindChanges as change}
<div class="flex items-center gap-2 text-xs py-0.5">
{#if change.kind === 'added'}
<Plus class="w-3 h-3 text-green-500 shrink-0" />
{:else if change.kind === 'removed'}
<Minus class="w-3 h-3 text-red-500 shrink-0" />
{:else}
<Pencil class="w-3 h-3 text-yellow-500 shrink-0" />
{/if}
<span class="text-tertiary">{change.schemaName}.</span>
<span class="font-medium">{change.tableName}</span>
<span class="text-tertiary text-2xs grow">{operationSummary(change)}</span>
<Button
size="xs"
variant="subtle"
startIcon={{ icon: Eye }}
onclick={() => openReview(change, diff, 'behind')}
>
Review
</Button>
</div>
{/each}
</div>
{/if}
</div>
{/if}
{#if diff.behindChanges.length > 0}
<div class="px-3 py-1.5">
<div class="text-2xs font-semibold text-orange-500 mb-1">
Parent changes (behind)
</div>
{#each diff.behindChanges as change}
<div class="flex items-center gap-2 text-xs py-0.5">
{#if change.kind === 'added'}
<Plus class="w-3 h-3 text-green-500 shrink-0" />
{:else if change.kind === 'removed'}
<Minus class="w-3 h-3 text-red-500 shrink-0" />
{:else}
<Pencil class="w-3 h-3 text-yellow-500 shrink-0" />
{/if}
<span class="text-tertiary">{change.schemaName}.</span>
<span class="font-medium">{change.tableName}</span>
<span class="text-tertiary text-2xs grow">{operationSummary(change)}</span>
<Button
size="xs"
variant="subtle"
startIcon={{ icon: Eye }}
onclick={() => openReview(change, diff, 'behind')}
>
Review
</Button>
</div>
{/each}
</div>
{/if}
</div>
{/if}
</ResizeTransitionWrapper>
{/each}
</ResizeTransitionWrapper>
{/each}
</div>
{:else}
<span class="text-xs text-secondary"> No changes detected </span>
{/if}
</div>
{:else}
<span class="text-xs text-secondary"> No changes detected </span>
{/if}
<Drawer bind:open={drawerOpen} size="900px">
@@ -580,3 +655,7 @@
>{migrationSql}</pre
>
</ConfirmationModal>
<Portal>
<ConfirmationModal {...outOfOrderModal.props} />
</Portal>
@@ -0,0 +1,159 @@
<script lang="ts">
import { Button } from './common'
import Modal2 from './common/modal/Modal2.svelte'
import NewDataTableMigrationModal from './workspaceSettings/NewDataTableMigrationModal.svelte'
import DataTableMigrationsButton from './workspaceSettings/DataTableMigrationsButton.svelte'
import { splitSqlStatements, isDdlStatement } from './sqlDdl'
import { CornerDownLeft } from 'lucide-svelte'
let { workspace, datatable }: { workspace: string; datatable: string } = $props()
type Choice = 'run' | 'migrate' | 'cancel'
let promptStatement = $state<string | undefined>(undefined)
let promptOpen = $state(false)
let resolvePrompt: ((choice: Choice) => void) | undefined = undefined
let resolveMigrationClosed: ((created: boolean) => void) | undefined = undefined
// Set when a migration is created *and run* during a guard() call, so the
// caller can refresh the schema afterwards.
let migrationRan = false
let newMigrationModal = $state<NewDataTableMigrationModal | undefined>(undefined)
// Triggerless Migrations modal, opened programmatically from the "See migration"
// toast action after a migration is created here.
let migrationsModal = $state<DataTableMigrationsButton | undefined>(undefined)
function finishPrompt(choice: Choice) {
const r = resolvePrompt
resolvePrompt = undefined
promptOpen = false
promptStatement = undefined
r?.(choice)
}
// Closing the modal (X / escape) while a prompt is pending counts as cancel.
$effect(() => {
if (!promptOpen && resolvePrompt) {
finishPrompt('cancel')
}
})
// Enter confirms the primary action ("Create a migration"). Ignore modified
// Enter so editor shortcuts (Cmd/Ctrl+Enter) keep working underneath.
function onKeyDown(event: KeyboardEvent) {
if (!promptOpen || event.metaKey || event.ctrlKey || event.altKey) return
if (event.key === 'Enter') {
event.stopPropagation()
event.preventDefault()
finishPrompt('migrate')
}
}
function promptDdl(statement: string): Promise<Choice> {
return new Promise((resolve) => {
resolvePrompt = resolve
promptStatement = statement
promptOpen = true
})
}
function handleMigrationClosed(result: { created: boolean; ran: boolean }) {
if (result.ran) migrationRan = true
const r = resolveMigrationClosed
resolveMigrationClosed = undefined
r?.(result.created)
}
// Open the prefilled new-migration modal. Resolves with whether a migration
// was actually created (false if the user cancelled / closed it).
function openMigrationModal(statement: string): Promise<boolean> {
return new Promise((resolve) => {
resolveMigrationClosed = (created: boolean) => resolve(created)
newMigrationModal?.open({ codeUp: statement })
})
}
/**
* Inspect `code` for DDL statements. For each one, prompt the user to run it
* anyway or turn it into a migration (prompts shown one at a time). Returns
* whether to proceed and the code to run (with migrated statements stripped).
*/
export async function guard(
code: string
): Promise<{ proceed: boolean; code: string; ranMigration: boolean }> {
migrationRan = false
const statements = splitSqlStatements(code)
if (!statements.some((s) => isDdlStatement(s))) {
return { proceed: true, code, ranMigration: false }
}
const kept: string[] = []
for (const statement of statements) {
if (!isDdlStatement(statement)) {
kept.push(statement)
continue
}
// Re-prompt for this statement until the user makes a terminal choice;
// cancelling the migration modal returns to the prompt with the DDL intact.
for (;;) {
const choice = await promptDdl(statement)
if (choice === 'cancel') {
return { proceed: false, code, ranMigration: migrationRan }
}
if (choice === 'run') {
kept.push(statement)
break
}
// migrate: only strip the statement once a migration is actually
// created; if the modal was cancelled, loop back to the prompt.
const created = await openMigrationModal(statement)
if (created) {
break
}
}
}
return { proceed: true, code: kept.join(';\n'), ranMigration: migrationRan }
}
</script>
<svelte:window onkeydown={onKeyDown} />
<Modal2
title="Schema change detected"
fixedWidth="md"
fixedHeight="adaptive"
bind:isOpen={promptOpen}
closeOnOutsideClick={false}
>
<div class="flex flex-col gap-3 w-full">
<p class="text-sm text-secondary">
This looks like a schema-changing (DDL) statement. Schema changes are best tracked as
migrations rather than run ad-hoc. Create a migration for it instead?
</p>
<pre
class="text-xs whitespace-pre-wrap font-mono bg-surface-secondary rounded p-3 max-h-48 overflow-auto"
>{promptStatement ?? ''}</pre
>
<div class="flex justify-end gap-2 pt-2">
<Button variant="default" size="sm" on:click={() => finishPrompt('run')}>Run anyway</Button>
<Button
variant="accent"
size="sm"
shortCut={{ Icon: CornerDownLeft, withoutModifier: true }}
on:click={() => finishPrompt('migrate')}
>
Create a migration
</Button>
</div>
</div>
</Modal2>
<NewDataTableMigrationModal
bind:this={newMigrationModal}
{workspace}
{datatable}
onClose={handleMigrationClosed}
onSeeMigration={(m) => migrationsModal?.openMigration(m.timestamp)}
/>
<DataTableMigrationsButton bind:this={migrationsModal} hideTrigger {workspace} {datatable} />
+39 -1
View File
@@ -43,6 +43,7 @@
import { editorFontSize } from '$lib/editorFontSize.svelte'
import { createHash as randomHash } from '$lib/editorLangUtils'
import { workspaceStore } from '$lib/stores'
import DdlMigrationGuard from './DdlMigrationGuard.svelte'
import {
type Preview,
ResourceService,
@@ -222,6 +223,35 @@
let lang = $state(scriptLangToEditorLang(untrack(() => scriptLang)))
// On a postgres script targeting a datatable, DDL statements are intercepted
// on run (cmd+enter) and offered as migrations instead.
let datatableForMigrations = $derived(
scriptLang === 'postgresql' &&
typeof args?.database === 'string' &&
args.database.startsWith('datatable://')
? args.database.slice('datatable://'.length).split('/')[0]
: undefined
)
let ddlGuard = $state<DdlMigrationGuard | undefined>(undefined)
// Run the DDL migration guard against the current code. Returns false when the
// user cancels (the run must be aborted); may rewrite the code (migrated
// statements stripped). Exported so run paths that bypass the Monaco
// Cmd+Enter binding (e.g. the Test button) can guard too.
export async function guardDdlBeforeRun(): Promise<boolean> {
if (datatableForMigrations && ddlGuard) {
const res = await ddlGuard.guard(getCode())
if (!res.proceed) return false
if (res.code !== getCode()) setCode(res.code)
}
return true
}
async function runCmdEnterWithDdlGuard() {
if (!(await guardDdlBeforeRun())) return
cmdEnterAction?.()
}
let filePath = $state(computePath(untrack(() => path)))
let initialPath: string | undefined = $state(untrack(() => path))
@@ -1639,7 +1669,8 @@
editor?.addCommand(KeyMod.CtrlCmd | KeyCode.Enter, function () {
updateCode()
shouldBindKey && cmdEnterAction && cmdEnterAction()
if (!shouldBindKey || !cmdEnterAction) return
void runCmdEnterWithDdlGuard()
})
editor?.addCommand(KeyMod.CtrlCmd | KeyMod.Shift | KeyCode.Digit7, function () {
@@ -2209,6 +2240,13 @@
<svelte:window onkeydown={onKeyDown} />
<EditorTheme />
{#if datatableForMigrations && $workspaceStore}
<DdlMigrationGuard
bind:this={ddlGuard}
workspace={$workspaceStore}
datatable={datatableForMigrations}
/>
{/if}
{#if !editor}
<div class="inset-0 absolute overflow-clip">
<FakeMonacoPlaceHolder {code} lineNumbersWidth={51} />
@@ -793,7 +793,17 @@
args = nargs
}
export async function runTest(opts?: { cascade?: boolean }) {
export async function runTest(opts?: { cascade?: boolean; skipDdlGuard?: boolean }) {
// Intercept DDL statements (offer to turn them into data table migrations)
// on every run path, not just the editor's Cmd+Enter. `skipDdlGuard` is set
// by the Cmd+Enter action, which already guarded before calling us.
if (!opts?.skipDdlGuard) {
if ((await editor?.guardDdlBeforeRun()) === false) return
// The guard may have rewritten the code (migrated statements stripped);
// `editorCode` is kept in sync by the editor binding, so mirror the
// on:change handler and pull it into `code` before we run.
if (activeModuleTab === null) code = editorCode
}
// When the caller forces a cascade choice (e.g. the canvas runnable
// menu's "Run + trigger N downstream"), also flip the persistent
// `cascadeDownstream` state so the split button's label/icon reflect
@@ -2677,7 +2687,8 @@
} else {
await inferModuleSchema()
}
runTest()
// The Editor already ran the DDL guard before invoking this action.
runTest({ skipDdlGuard: true })
}}
formatAction={async () => {
if (activeModuleTab === null) {
@@ -95,6 +95,7 @@
loadAsync = false,
key,
disabled = false,
readOnly = false,
minHeight = 1000,
renderLineHighlight = 'none',
suggestion
@@ -123,6 +124,9 @@
initialCursorPos?: IPosition
key?: string
disabled?: boolean
/** Read-only Monaco mode: not editable, but still scrollable/selectable
* (unlike `disabled`, which makes the editor non-interactive). */
readOnly?: boolean
minHeight?: number
renderLineHighlight?: 'all' | 'line' | 'gutter' | 'none'
suggestion?: string
@@ -239,6 +243,9 @@
lineNumbers: $relativeLineNumbers ? 'relative' : 'on'
})
})
$effect(() => {
editor?.updateOptions({ readOnly })
})
function onVimDisable() {
vimDisposable?.dispose()
@@ -342,6 +349,7 @@
),
model,
...(yPadding !== undefined ? { padding: { bottom: yPadding, top: yPadding } } : {}),
readOnly,
renderLineHighlight,
lineDecorationsWidth: 0,
lineNumbersMinChars: 2,
+35 -48
View File
@@ -1,49 +1,3 @@
<script module lang="ts">
// code may be composed of many sql statements separated by ';'
// this function splits them while taking into account that ';' may not
// be the end of a statement (string or escaped)
function splitSqlStatements(code: string) {
const statements: string[] = []
let currentStatement = ''
let inSingleQuote = false
let inDoubleQuote = false
let inBacktick = false
for (let i = 0; i < code.length; i++) {
const char = code[i]
const prevChar = i > 0 ? code[i - 1] : null
if (char === "'" && !inDoubleQuote && !inBacktick && prevChar !== '\\') {
inSingleQuote = !inSingleQuote
} else if (char === '"' && !inSingleQuote && !inBacktick && prevChar !== '\\') {
inDoubleQuote = !inDoubleQuote
} else if (char === '`' && !inSingleQuote && !inDoubleQuote && prevChar !== '\\') {
inBacktick = !inBacktick
}
if (char === ';' && !inSingleQuote && !inDoubleQuote && !inBacktick) {
statements.push(currentStatement.trim())
currentStatement = ''
} else {
currentStatement += char
}
}
if (currentStatement.trim()) {
statements.push(currentStatement.trim())
}
return statements
}
function pruneComments(code: string) {
return code
.replace(/--.*?(\r?\n|$)/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.trim()
}
</script>
<script lang="ts">
import { CornerDownLeft, Loader2 } from 'lucide-svelte'
import Button from './common/button/Button.svelte'
@@ -57,15 +11,29 @@
import { getDatabaseArg, getDbType } from './dbOps'
import type { DbInput } from './dbTypes'
import { wrapDucklakeQuery } from './ducklake'
import { splitSqlStatements, pruneComments } from './sqlDdl'
import DdlMigrationGuard from './DdlMigrationGuard.svelte'
type Props = {
input: DbInput
onData: (data: Record<string, any>[]) => void
placeholderTableName?: string
/** Called after a migration is run via the DDL guard, so the schema view
* can be refreshed to reflect the applied change. */
onSchemaChange?: () => void
}
let { input, onData, placeholderTableName }: Props = $props()
let { input, onData, placeholderTableName, onSchemaChange }: Props = $props()
let dbType = $derived(getDbType(input))
// A datatable REPL targets `datatable://<name>`; surface DDL statements as
// migration prompts instead of running them ad-hoc.
let datatableName = $derived(
input.type === 'database' && input.resourcePath.startsWith('datatable://')
? input.resourcePath.slice('datatable://'.length).split('/')[0]
: undefined
)
let ddlGuard = $state<DdlMigrationGuard | undefined>(undefined)
const DEFAULT_SQL = 'SELECT * FROM _'
let code = $state(DEFAULT_SQL)
$effect(() => {
@@ -81,6 +49,18 @@
async function run({ doPostgresRowToJsonFix }: { doPostgresRowToJsonFix?: boolean } = {}) {
if (isRunning || !$workspaceStore) return
const READ_OPS = ['SELECT', 'WITH', 'SHOW', 'EXPLAIN', 'DESCRIBE']
// On a datatable, intercept DDL statements and offer to make them
// migrations; the user may strip some, leaving the rest to run.
if (datatableName && ddlGuard && !doPostgresRowToJsonFix) {
const res = await ddlGuard.guard(code)
// A migration that was created and run changes the schema; refresh it.
if (res.ranMigration) onSchemaChange?.()
if (!res.proceed) return
if (res.code !== code) code = res.code
if (pruneComments(code).trim() === '') return
}
isRunning = true
try {
const statements = splitSqlStatements(pruneComments(code))
@@ -155,7 +135,10 @@
isRunning = false
return await run({ doPostgresRowToJsonFix: true })
}
sendUserToast('Error running query: ' + (e.message ?? e.error.message), true)
sendUserToast(
'Error running query: ' + (e?.body ?? e?.message ?? e?.error?.message ?? e),
true
)
} finally {
isRunning = false
}
@@ -197,3 +180,7 @@
/>
</Pane>
</Splitpanes>
{#if datatableName && $workspaceStore}
<DdlMigrationGuard bind:this={ddlGuard} workspace={$workspaceStore} datatable={datatableName} />
{/if}
@@ -50,6 +50,7 @@
| 'azure_trigger'
| 'email_trigger'
| 'data_pipeline'
| 'datatable_migration'
triggerKind?: string | undefined
summary?: string | undefined
path: string
@@ -55,6 +55,7 @@
| 'azure_trigger'
| 'email_trigger'
| 'data_pipeline'
| 'datatable_migration'
/** For 'trigger' kind, specifies the specific trigger type (routes, schedules, etc.) */
triggerKind?: string | undefined
/** For 'raw_app_file' kind: the file name/path, used to pick an
@@ -128,6 +129,8 @@
<Calendar {size} class="text-gray-400" />
{:else if effectiveKind === 'data_pipeline'}
<Workflow {size} class="text-indigo-500" />
{:else if effectiveKind === 'datatable_migration'}
<Database {size} class="text-violet-500" />
{:else}
<div style="width: {size}px;"></div>
{/if}
+145 -23
View File
@@ -9,6 +9,8 @@ import type { DBSchema, SQLSchema } from '$lib/stores'
import { stringifySchema } from './copilot/lib'
import type { DbInput, DbType } from './dbTypes'
import { assert } from '$lib/utils'
import { WorkspaceService } from '$lib/gen'
import { pendingMigrations } from './workspaceSettings/datatableMigrationUtils'
import {
buildTableEditorValues,
type TableEditorValues
@@ -226,7 +228,12 @@ export type IDbSchemaOps = {
onDelete: (params: { tableKey: string; schema?: string }) => Promise<void>
onCreate: (params: { values: TableEditorValues; schema?: string }) => Promise<void>
previewCreateSql: (params: { values: TableEditorValues; schema?: string }) => Promise<string>
onAlter: (params: { values: AlterTableValues; schema?: string }) => Promise<void>
onAlter: (params: {
values: AlterTableValues
/** Reverse diff (new → old), used to generate the down migration. */
reverse?: AlterTableValues
schema?: string
}) => Promise<void>
previewAlterSql: (params: { values: AlterTableValues; schema?: string }) => Promise<string>
onCreateSchema: (params: { schema: string }) => Promise<void>
onDeleteSchema: (params: { schema: string }) => Promise<void>
@@ -237,30 +244,146 @@ export type IDbSchemaOps = {
}) => Promise<TableEditorValues>
}
/** Thrown by a schema op when the user declines the out-of-order run warning.
* Callers should treat it as a silent cancel (no error toast). */
export class MigrationRunCancelled extends Error {
constructor() {
super('Migration run cancelled')
this.name = 'MigrationRunCancelled'
}
}
export function dbSchemaOpsWithPreviewScripts({
workspace,
input
input,
confirmRunOutOfOrder
}: {
workspace: string
input: DbInput
/** Asked before running a just-created migration ahead of `pendingCount`
* still-pending earlier ones. Return false to abort (throws MigrationRunCancelled). */
confirmRunOutOfOrder?: (pendingCount: number) => Promise<boolean>
}): IDbSchemaOps {
const dbType = getDbType(input)
const dbArg = getDatabaseArg(input)
const language = getLanguageByResourceType(dbType)
const ducklake = input.type === 'ducklake' ? input.ducklake : undefined
// When managing a data table, schema changes are recorded as migrations
// instead of being run ad-hoc, so the manager stays the source of truth.
const datatableName =
input.type === 'database' && input.resourcePath.startsWith('datatable://')
? input.resourcePath.slice('datatable://'.length)
: undefined
function makeMarker(op: string, payload: Record<string, unknown>): string {
if (ducklake) payload.ducklake = ducklake
return `-- WM_INTERNAL_DB_${op} ${JSON.stringify(payload)}`
}
// Auto-generated migration name, e.g. `create_customers`. The server allocates
// a unique timestamp (bumping on collision), so the name itself need not be unique.
function migrationName(op: string, target: string): string {
const safe = target.replace(/[^a-zA-Z0-9_-]+/g, '_').replace(/^_+|_+$/g, '')
return safe ? `${op}_${safe}` : op
}
// A dropped SERIAL column reports its default as `nextval()` of an owned
// sequence that is dropped along with the column. When the down re-adds such a
// column, recreate it as its SERIAL type so a fresh sequence is created
// instead of referencing the gone one.
const SERIAL_FOR: Record<string, string> = {
BIGINT: 'BIGSERIAL',
INT8: 'BIGSERIAL',
INTEGER: 'SERIAL',
INT: 'SERIAL',
INT4: 'SERIAL',
SMALLINT: 'SMALLSERIAL',
INT2: 'SMALLSERIAL'
}
function reverseSerialFix(reverse: AlterTableValues): AlterTableValues {
return {
...reverse,
operations: reverse.operations.map((op) => {
if (op.kind !== 'addColumn' || !/nextval\s*\(/i.test(op.column.defaultValue ?? '')) {
return op
}
const serial = SERIAL_FOR[(op.column.datatype ?? '').toUpperCase()]
return serial
? { ...op, column: { ...op.column, datatype: serial, defaultValue: undefined } }
: op
})
}
}
// Frame a single (or multi-) statement body in an explicit, `;`-terminated
// transaction, matching the data table migration convention. Some expanded
// markers (e.g. ALTER TABLE) already come wrapped in their own transaction, so
// avoid nesting BEGIN/COMMIT in that case.
function wrapMigration(sql: string): string {
const t = sql.trim()
if (/^BEGIN\b/i.test(t)) return t
return `BEGIN;\n\n${t.endsWith(';') ? t : `${t};`}\n\nEND;`
}
// Apply a DDL marker. For a data table that has migrations enabled this
// creates a migration and runs it (rolling the record back if the run fails);
// otherwise it runs ad-hoc via the internal-db job as before. `downContent`,
// when provided, is expanded into the migration's down SQL (Postgres only).
async function applyDdl(migName: string, content: string, downContent?: string): Promise<void> {
// A DDL edit on a migrations-enabled data table must be captured as a
// migration. Don't swallow a status-check failure by defaulting to ad-hoc:
// that would run the change untracked (schema drift) — exactly what this
// feature prevents. Let the error propagate (fail closed); only fall back to
// ad-hoc when there's no data table, or `enabled === false` is returned.
const status = datatableName
? await WorkspaceService.getDatatableMigrationsStatus({ workspace, datatableName })
: undefined
if (!datatableName || !status?.enabled) {
await runScriptAndPollResult({ workspace, requestBody: { args: dbArg, content, language } })
return
}
// The new migration gets the highest timestamp, so any still-pending
// migration is earlier: running only this one applies it out of order.
// Warn like the row-level Run action does (skipped if no confirm hook).
if (confirmRunOutOfOrder) {
const pending = pendingMigrations(status.migrations).length
if (pending > 0 && !(await confirmRunOutOfOrder(pending))) {
throw new MigrationRunCancelled()
}
}
const codeUp = wrapMigration(await expandMarker(workspace, language, content))
// Down migrations are only generated for Postgres for now.
let codeDown: string | undefined
if (downContent && dbType === 'postgresql') {
const downSql = (await expandMarker(workspace, language, downContent)).trim()
if (downSql) codeDown = wrapMigration(downSql)
}
const created = await WorkspaceService.createDatatableMigration({
workspace,
datatableName,
requestBody: { name: migName, code_up: codeUp, ...(codeDown ? { code_down: codeDown } : {}) }
})
try {
await WorkspaceService.runDatatableMigrations({
workspace,
datatableName,
only: created.timestamp
})
} catch (e) {
await WorkspaceService.deleteDatatableMigration({
workspace,
datatableName,
timestamp: created.timestamp
}).catch(() => {})
throw e
}
}
return {
onDelete: async ({ tableKey, schema }) => {
const content = makeMarker('DROP_TABLE', { table: tableKey, schema })
await runScriptAndPollResult({
workspace,
requestBody: { args: { ...dbArg }, language, content }
})
await applyDdl(migrationName('drop', tableKey), content)
},
onCreate: async ({ values, schema }) => {
const content = makeMarker('CREATE_TABLE', {
@@ -269,10 +392,8 @@ export function dbSchemaOpsWithPreviewScripts({
foreignKeys: values.foreignKeys,
schema
})
await runScriptAndPollResult({
workspace,
requestBody: { args: dbArg, content, language }
})
const downContent = makeMarker('DROP_TABLE', { table: values.name, schema })
await applyDdl(migrationName('create', values.name), content, downContent)
},
previewCreateSql: async ({ values, schema }) => {
const content = makeMarker('CREATE_TABLE', {
@@ -283,16 +404,21 @@ export function dbSchemaOpsWithPreviewScripts({
})
return expandMarker(workspace, language, content)
},
onAlter: async ({ values, schema }) => {
onAlter: async ({ values, reverse, schema }) => {
const content = makeMarker('ALTER_TABLE', {
name: values.name,
operations: values.operations,
schema
})
await runScriptAndPollResult({
workspace,
requestBody: { args: dbArg, content, language }
})
// The down is the same alter run in the opposite direction.
const downContent = reverse
? makeMarker('ALTER_TABLE', {
name: reverse.name,
operations: reverseSerialFix(reverse).operations,
schema
})
: undefined
await applyDdl(migrationName('alter', values.name), content, downContent)
},
previewAlterSql: async ({ values, schema }) => {
const content = makeMarker('ALTER_TABLE', {
@@ -304,17 +430,13 @@ export function dbSchemaOpsWithPreviewScripts({
},
onCreateSchema: async ({ schema }) => {
const content = makeMarker('CREATE_SCHEMA', { schema })
await runScriptAndPollResult({
workspace,
requestBody: { args: { ...dbArg }, language, content }
})
const downContent = makeMarker('DROP_SCHEMA', { schema })
await applyDdl(migrationName('create_schema', schema), content, downContent)
},
onDeleteSchema: async ({ schema }) => {
const content = makeMarker('DROP_SCHEMA', { schema })
await runScriptAndPollResult({
workspace,
requestBody: { args: { ...dbArg }, language, content }
})
const downContent = makeMarker('CREATE_SCHEMA', { schema })
await applyDdl(migrationName('drop_schema', schema), content, downContent)
},
onFetchTableEditorDefinition: async ({ table, schema, colDefs }) => {
let foreignKeys: import('./apps/components/display/dbtable/tableEditor').TableEditorForeignKey[] =
+142
View File
@@ -0,0 +1,142 @@
// Heuristics for splitting a SQL script into individual statements and
// detecting Data Definition Language (DDL) statements. Shared by the datatable
// SQL REPL and the postgres editor to steer schema changes into migrations.
// A dollar-quote tag: `$$` or `$name$` (the optional tag follows unquoted
// identifier rules, so it starts with a letter/underscore — `$1` is a
// parameter placeholder, not a dollar quote).
const DOLLAR_QUOTE_START = /^\$([A-Za-z_][A-Za-z0-9_]*)?\$/
// code may be composed of many sql statements separated by ';'
// this splits them while taking into account that ';' may appear inside a
// string, quoted identifier, dollar-quoted body ($$ ... $$), or a comment
// (-- ... or /* ... */) and is then not the end of a statement.
export function splitSqlStatements(code: string): string[] {
const statements: string[] = []
let currentStatement = ''
let inSingleQuote = false
let inDoubleQuote = false
let inBacktick = false
let inLineComment = false
let inBlockComment = false
let dollarTag: string | null = null
for (let i = 0; i < code.length; i++) {
const char = code[i]
const prevChar = i > 0 ? code[i - 1] : null
const nextChar = i + 1 < code.length ? code[i + 1] : null
// Inside a protected region: append verbatim (';' is not a separator)
// and only watch for the region's end.
if (inLineComment) {
currentStatement += char
if (char === '\n') inLineComment = false
continue
}
if (inBlockComment) {
// Look ahead for the close so the '*' of the opening '/*' can't be
// reused (e.g. `/*/` stays open).
if (char === '*' && nextChar === '/') {
currentStatement += '*/'
i += 1
inBlockComment = false
} else {
currentStatement += char
}
continue
}
if (dollarTag !== null) {
if (char === '$' && code.startsWith(dollarTag, i)) {
currentStatement += dollarTag
i += dollarTag.length - 1
dollarTag = null
} else {
currentStatement += char
}
continue
}
if (inSingleQuote) {
currentStatement += char
if (char === "'" && prevChar !== '\\') inSingleQuote = false
continue
}
if (inDoubleQuote) {
currentStatement += char
if (char === '"' && prevChar !== '\\') inDoubleQuote = false
continue
}
if (inBacktick) {
currentStatement += char
if (char === '`' && prevChar !== '\\') inBacktick = false
continue
}
// Not in any protected region: detect the start of one.
if (char === '-' && nextChar === '-') {
inLineComment = true
currentStatement += char
continue
}
if (char === '/' && nextChar === '*') {
// Consume both chars so the '*' of '/*' can't double as a '*/' close.
inBlockComment = true
currentStatement += '/*'
i += 1
continue
}
if (char === '$') {
const m = DOLLAR_QUOTE_START.exec(code.slice(i))
if (m) {
dollarTag = m[0]
currentStatement += dollarTag
i += dollarTag.length - 1
continue
}
}
if (char === "'") {
inSingleQuote = true
currentStatement += char
continue
}
if (char === '"') {
inDoubleQuote = true
currentStatement += char
continue
}
if (char === '`') {
inBacktick = true
currentStatement += char
continue
}
if (char === ';') {
statements.push(currentStatement.trim())
currentStatement = ''
} else {
currentStatement += char
}
}
if (currentStatement.trim()) {
statements.push(currentStatement.trim())
}
return statements.filter((s) => s.length > 0)
}
export function pruneComments(code: string): string {
return code
.replace(/--.*?(\r?\n|$)/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.trim()
}
// Schema-changing keywords. GRANT/REVOKE are included since permission changes
// also belong in migrations rather than ad-hoc execution.
const DDL_KEYWORDS = ['CREATE', 'ALTER', 'DROP', 'TRUNCATE', 'RENAME', 'COMMENT', 'GRANT', 'REVOKE']
/** Heuristic: a statement is DDL if its first keyword is schema-changing. */
export function isDdlStatement(statement: string): boolean {
const firstWord = pruneComments(statement).trim().split(/\s+/)[0]?.toUpperCase()
return !!firstWord && DDL_KEYWORDS.includes(firstWord)
}
@@ -0,0 +1,596 @@
<script lang="ts">
import { Button } from '../common'
import Modal2 from '../common/modal/Modal2.svelte'
import Tabs from '../common/tabs/Tabs.svelte'
import Tab from '../common/tabs/Tab.svelte'
import TabContent from '../common/tabs/TabContent.svelte'
import SimpleEditor from '../SimpleEditor.svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import {
ChevronDown,
Play,
Trash2,
Plus,
Undo2,
Loader2,
Camera,
Settings,
Power,
PowerOff
} from 'lucide-svelte'
import { WorkspaceService, type DatatableMigrationWithStatus } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
import NewDataTableMigrationModal from './NewDataTableMigrationModal.svelte'
import Tooltip from '../Tooltip.svelte'
import Portal from '$lib/components/Portal.svelte'
import DropdownV2 from '../DropdownV2.svelte'
import { superadmin, userStore } from '$lib/stores'
let {
workspace,
datatable,
disabled = false,
hideTrigger = false,
onSchemaChanged
}: {
workspace: string
datatable: string
disabled?: boolean
/** Mount the modals without the "Migrations" trigger button, so a caller can
* drive them programmatically via `openMigration` (e.g. the DDL guard). */
hideTrigger?: boolean
/** Called after a migration run/revert actually changes the data table
* schema, so an open viewer (e.g. the database manager) can refresh. */
onSchemaChanged?: () => void
} = $props()
let listOpen = $state(false)
let migrations = $state<DatatableMigrationWithStatus[]>([])
let enabled = $state(true)
let loadError = $state<string | undefined>(undefined)
let loading = $state(false)
let busy = $state(false)
// Only workspace admins and super admins can opt a data table in or out.
const canManage = $derived(!!$userStore?.is_admin || !!$superadmin)
let newMigrationModal = $state<NewDataTableMigrationModal | undefined>(undefined)
let newMigrationOpen = $state(false)
let viewOpen = $state(false)
let viewMigration = $state<DatatableMigrationWithStatus | undefined>(undefined)
let viewTab = $state('up')
let settingsDropdownOpen = $state(false)
let deleteOpen = $state(false)
let deleteTarget = $state<DatatableMigrationWithStatus | undefined>(undefined)
function openView(m: DatatableMigrationWithStatus) {
viewMigration = m
viewTab = 'up'
viewOpen = true
}
const confirmationModal = createAsyncConfirmationModal()
const hasPending = $derived(migrations.some((m) => m.status !== 'ran'))
async function loadMigrations() {
// Only show the full-list spinner on the initial load. Refreshes after an
// action (run/revert/delete) update the keyed list in place to avoid a
// flicker, with the buttons already gated by `busy`.
if (migrations.length === 0) {
loading = true
}
try {
const res = await WorkspaceService.getDatatableMigrationsStatus({
workspace,
datatableName: datatable
})
enabled = res.enabled
migrations = res.migrations
loadError = res.error
} catch (e: any) {
sendUserToast(`Failed to load migrations: ${e?.body ?? e?.message ?? e}`, true)
} finally {
loading = false
}
}
function openList() {
listOpen = true
loadMigrations()
}
// Open the list modal and the detail view for a specific migration. Used to
// jump to a just-created migration from the "See migration" toast action.
export async function openMigration(timestamp: number) {
listOpen = true
await loadMigrations()
const m = migrations.find((x) => x.timestamp === timestamp)
if (m) openView(m)
}
async function runUpTo(upTo: number | undefined) {
busy = true
try {
const res = await WorkspaceService.runDatatableMigrations({
workspace,
datatableName: datatable,
upTo
})
sendUserToast(
res.applied.length > 0
? `Applied ${res.applied.length} migration(s)`
: 'No pending migrations to run'
)
await loadMigrations()
if (res.applied.length > 0) onSchemaChanged?.()
} catch (e: any) {
sendUserToast(`Failed to run migrations: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
async function runOnly(version: number) {
busy = true
try {
const res = await WorkspaceService.runDatatableMigrations({
workspace,
datatableName: datatable,
only: version
})
sendUserToast(
res.applied.length > 0 ? `Applied ${res.applied[0].name}` : 'Migration already applied'
)
await loadMigrations()
if (res.applied.length > 0) onSchemaChanged?.()
} catch (e: any) {
sendUserToast(`Failed to run migration: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
// Run a single migration. Warn first if earlier migrations haven't run, since
// this one might depend on them.
async function runMigration(m: DatatableMigrationWithStatus) {
const earlierPending = migrations.filter(
(x) => x.timestamp < m.timestamp && x.status !== 'ran'
).length
if (earlierPending > 0) {
const confirmed = await confirmationModal.ask({
title: 'Run migration out of order',
confirmationText: 'Run anyway',
children: `${earlierPending} earlier migration(s) have not been run yet. "${m.name}" might depend on them. Run it anyway?`
})
if (!confirmed) return
}
await runOnly(m.timestamp)
}
async function revertOnly(version: number) {
busy = true
try {
const res = await WorkspaceService.rollbackDatatableMigrations({
workspace,
datatableName: datatable,
only: version
})
sendUserToast(
res.rolled_back.length > 0
? `Reverted ${res.rolled_back[0].name}`
: 'Migration was not applied'
)
await loadMigrations()
if (res.rolled_back.length > 0) onSchemaChanged?.()
} catch (e: any) {
sendUserToast(`Failed to revert migration: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
// Revert a single migration. Warn first if later migrations have run, since
// they might depend on this one.
async function revertMigration(m: DatatableMigrationWithStatus) {
const laterApplied = migrations.filter(
(x) => x.timestamp > m.timestamp && x.status === 'ran'
).length
if (laterApplied > 0) {
const confirmed = await confirmationModal.ask({
title: 'Revert migration out of order',
confirmationText: 'Revert anyway',
children: `${laterApplied} later migration(s) have already been run and might depend on "${m.name}". Reverting it may break them. Revert anyway?`
})
if (!confirmed) return
}
await revertOnly(m.timestamp)
}
async function generateInitial() {
const confirmed = await confirmationModal.ask({
title: 'Generate initial migration',
confirmationText: 'Generate',
children: `This snapshots the current schema of "${datatable}" with pg_dump as an "initial" migration and marks it as already applied, so it is never re-run on this data table. It has no down migration. Use this to start tracking migrations on an existing data table.`
})
if (!confirmed) return
busy = true
try {
await WorkspaceService.generateInitialDatatableMigration({
workspace,
datatableName: datatable
})
sendUserToast('Initial migration generated')
await loadMigrations()
} catch (e: any) {
sendUserToast(`Failed to generate initial migration: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
async function enableMigrations() {
busy = true
try {
await WorkspaceService.enableDatatableMigrations({ workspace, datatableName: datatable })
sendUserToast('Migrations enabled')
await loadMigrations()
} catch (e: any) {
sendUserToast(`Failed to enable migrations: ${e?.body ?? e?.message ?? e}`, true)
return
} finally {
busy = false
}
// Right after enabling, offer to snapshot the current schema as the initial migration.
await generateInitial()
}
async function disableMigrations() {
const confirmed = await confirmationModal.ask({
title: 'Disable migrations',
confirmationText: 'Disable and delete',
children: `Disabling migrations for "${datatable}" will delete ALL of its migrations. This cannot be undone. The data table's data and schema are not affected. Continue?`
})
if (!confirmed) return
busy = true
try {
await WorkspaceService.disableDatatableMigrations({ workspace, datatableName: datatable })
sendUserToast('Migrations disabled')
await loadMigrations()
} catch (e: any) {
sendUserToast(`Failed to disable migrations: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
// Delete a migration's definition. Does not touch the data table's schema or
// its `_wm_migrations` bookkeeping.
async function performDelete(m: DatatableMigrationWithStatus) {
busy = true
try {
await WorkspaceService.deleteDatatableMigration({
workspace,
datatableName: datatable,
timestamp: m.timestamp
})
await loadMigrations()
} catch (e: any) {
sendUserToast(`Failed to delete migration: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
// Revert an installed migration (run its down) then delete its definition.
// The delete is skipped if the revert fails, so we never drop the definition
// of a migration that is still installed on the data table.
async function revertAndDelete(m: DatatableMigrationWithStatus) {
busy = true
try {
await WorkspaceService.rollbackDatatableMigrations({
workspace,
datatableName: datatable,
only: m.timestamp
})
await WorkspaceService.deleteDatatableMigration({
workspace,
datatableName: datatable,
timestamp: m.timestamp
})
await loadMigrations()
onSchemaChanged?.()
sendUserToast(`Reverted and deleted "${m.name}"`)
} catch (e: any) {
sendUserToast(`Failed to revert and delete migration: ${e?.body ?? e?.message ?? e}`, true)
await loadMigrations()
} finally {
busy = false
}
}
async function deleteMigration(m: DatatableMigrationWithStatus) {
// An installed migration can be reverted first, so offer that as a third
// choice instead of a plain confirm.
if (m.status === 'ran') {
deleteTarget = m
deleteOpen = true
return
}
const body =
m.status === 'unknown'
? `The applied status of "${m.name}" is unknown. If it is installed on the data table, deleting it may leave the data table in a broken state. Delete anyway?`
: `Delete the definition of "${m.name}"? It has not been run, so this only removes the migration definition.`
const confirmed = await confirmationModal.ask({
title: 'Delete migration',
confirmationText: 'Delete',
children: body
})
if (!confirmed) return
await performDelete(m)
}
const statusColor = {
ran: 'bg-green-500',
not_run: 'bg-orange-500',
unknown: 'bg-gray-400'
}
const statusTitle = {
ran: 'Already ran',
not_run: 'Not run',
unknown: 'Status unknown'
}
</script>
{#if !hideTrigger}
<Button
variant="default"
size="sm"
{disabled}
endIcon={{ icon: ChevronDown }}
on:click={openList}
>
Migrations
</Button>
{/if}
<Modal2
bind:isOpen={listOpen}
title="Migrations — {datatable}"
fixedWidth="md"
fixedHeight="lg"
closeOnOutsideClick={!newMigrationOpen &&
!viewOpen &&
!deleteOpen &&
!confirmationModal.props.open &&
!settingsDropdownOpen}
>
{#snippet headerLeft()}
<Tooltip>
Each schema edit made in the database manager is captured here as a migration. Tracking schema
changes as migrations makes it easy to export data tables to other workspaces, and to
reproduce their schema when forking a workspace.
</Tooltip>
{/snippet}
{#snippet headerRight()}
{#if enabled && canManage}
<DropdownV2
bind:open={settingsDropdownOpen}
items={[
{
displayName: 'Disable migrations',
icon: PowerOff,
type: 'delete',
action: () => disableMigrations()
}
]}
>
{#snippet buttonReplacement()}
<Button
variant="subtle"
size="xs"
iconOnly
startIcon={{ icon: Settings }}
title="Migration settings"
disabled={busy}
/>
{/snippet}
</DropdownV2>
{/if}
{/snippet}
<div class="flex flex-col gap-2 w-full grow min-h-0">
{#if loading}
<div class="flex items-center justify-center grow text-tertiary">
<Loader2 size={18} class="animate-spin" />
</div>
{:else if !enabled}
<div class="flex flex-col items-center justify-center gap-3 grow text-sm text-tertiary">
<span>Migrations are disabled for this data table.</span>
{#if canManage}
<Button
variant="subtle"
size="sm"
startIcon={{ icon: Power }}
disabled={busy}
on:click={enableMigrations}
>
Enable migrations
</Button>
{/if}
</div>
{:else}
{#if loadError}
<div class="text-xs text-red-500">
Could not read applied status from the data table: {loadError}
</div>
{/if}
<div class="flex flex-col grow min-h-0 overflow-auto border rounded-md divide-y">
{#if migrations.length === 0}
<div class="flex flex-col items-center gap-3 p-6 text-sm text-tertiary">
<span>No migrations yet</span>
<Button
variant="subtle"
size="xs"
startIcon={{ icon: Camera }}
disabled={busy}
on:click={generateInitial}
>
Generate initial migration
</Button>
</div>
{:else}
{#each migrations as m (m.timestamp)}
<div class="flex items-center gap-3 px-3 py-2">
<div
class="shrink-0 w-2.5 h-2.5 rounded-full {statusColor[m.status]}"
title={statusTitle[m.status]}
></div>
<button
type="button"
class="flex flex-col min-w-0 grow text-left rounded -mx-1 px-1 py-0.5 hover:bg-surface-hover"
title="View migration"
onclick={() => openView(m)}
>
<span class="text-sm text-primary truncate">{m.name}</span>
<span class="text-xs text-hint">{m.timestamp}</span>
</button>
<Button
variant="subtle"
size="xs"
iconOnly
startIcon={{ icon: Play }}
title="Run this migration"
disabled={busy || m.status === 'ran'}
on:click={() => runMigration(m)}
/>
<Button
variant="subtle"
size="xs"
iconOnly
startIcon={{ icon: Undo2 }}
title="Revert this migration"
disabled={busy || m.status !== 'ran'}
on:click={() => revertMigration(m)}
/>
<Button
variant="subtle"
size="xs"
iconOnly
color="red"
startIcon={{ icon: Trash2 }}
title="Delete migration"
disabled={busy}
on:click={() => deleteMigration(m)}
/>
</div>
{/each}
{/if}
</div>
<div class="flex justify-between gap-2 pt-2">
<Button
variant="default"
size="sm"
startIcon={{ icon: Plus }}
on:click={() => {
newMigrationOpen = true
newMigrationModal?.open()
}}>New</Button
>
<Button
variant="accent"
size="sm"
startIcon={{ icon: Play }}
disabled={busy || !hasPending}
on:click={() => runUpTo(undefined)}
>
Run all
</Button>
</div>
{/if}
</div>
</Modal2>
<NewDataTableMigrationModal
bind:this={newMigrationModal}
{workspace}
{datatable}
onCreated={loadMigrations}
onClose={() => (newMigrationOpen = false)}
onSeeMigration={(m) => openMigration(m.timestamp)}
/>
<Modal2
bind:isOpen={viewOpen}
title="Migration — {viewMigration?.name ?? ''}"
fixedWidth="md"
fixedHeight="lg"
>
{#if viewMigration}
<div class="flex flex-col gap-3 w-full grow min-h-0">
<span class="text-xs text-hint">{viewMigration.timestamp}</span>
<Tabs bind:selected={viewTab} class="grow min-h-0">
<Tab value="up" label="Up" />
<Tab value="down" label="Down" />
{#snippet content()}
<TabContent value="up" class="h-80 border rounded-md overflow-hidden">
<SimpleEditor class="h-full" lang="sql" code={viewMigration?.code_up ?? ''} readOnly />
</TabContent>
<TabContent value="down" class="h-80 border rounded-md overflow-hidden">
{#if viewMigration?.code_down}
<SimpleEditor class="h-full" lang="sql" code={viewMigration.code_down} readOnly />
{:else}
<div class="p-6 text-center text-sm text-tertiary">No down migration</div>
{/if}
</TabContent>
{/snippet}
</Tabs>
</div>
{/if}
</Modal2>
<Modal2 bind:isOpen={deleteOpen} title="Delete migration" fixedWidth="sm" fixedHeight="adaptive">
<div class="flex flex-col gap-3 w-full">
<p class="text-sm text-secondary">
"{deleteTarget?.name}" is installed on the data table. Revert it first to undo its schema
change, or delete only the definition and leave the schema as-is — deleting without reverting
means it can no longer be reverted.
</p>
<div class="flex justify-end gap-2 pt-2">
<Button variant="default" size="sm" disabled={busy} on:click={() => (deleteOpen = false)}>
Cancel
</Button>
<Button
variant="default"
color="red"
size="sm"
disabled={busy}
on:click={() => {
const m = deleteTarget
deleteOpen = false
if (m) performDelete(m)
}}
>
Delete
</Button>
<Button
variant="accent"
size="sm"
disabled={busy}
on:click={() => {
const m = deleteTarget
deleteOpen = false
if (m) revertAndDelete(m)
}}
>
Revert and delete
</Button>
</div>
</div>
</Modal2>
<Portal>
<ConfirmationModal {...confirmationModal.props} />
</Portal>
@@ -0,0 +1,380 @@
<script lang="ts">
import { Button } from '../common'
import TextInput from '../text_input/TextInput.svelte'
import Modal2 from '../common/modal/Modal2.svelte'
import Tabs from '../common/tabs/Tabs.svelte'
import Tab from '../common/tabs/Tab.svelte'
import TabContent from '../common/tabs/TabContent.svelte'
import Toggle from '../Toggle.svelte'
import Select from '../select/Select.svelte'
import Tooltip from '../Tooltip.svelte'
import { Shield, Plus, Trash2, RefreshCw, Loader2, TriangleAlert } from 'lucide-svelte'
import {
WorkspaceService,
type DataTablePermissions,
type DataTableGrant,
type DataTablePolicy
} from '$lib/gen'
import { sendUserToast } from '$lib/toast'
import { enterpriseLicense, superadmin, userStore } from '$lib/stores'
let {
workspace,
datatable,
disabled = false
}: {
workspace: string
datatable: string
disabled?: boolean
} = $props()
let open = $state(false)
let loading = $state(false)
let busy = $state(false)
let log = $state<string | undefined>(undefined)
let selectedTab = $state('access')
// Advanced permissions are an enterprise feature; only workspace admins manage them.
const ee = $derived(!!$enterpriseLicense)
const canManage = $derived(!!$userStore?.is_admin || !!$superadmin)
let enabled = $state(false)
let grants = $state<DataTableGrant[]>([])
let policies = $state<DataTablePolicy[]>([])
const kindItems = [
{ label: 'User', value: 'user' },
{ label: 'Group', value: 'group' }
]
const accessItems = [
{ label: 'No access', value: 'none' },
{ label: 'Read', value: 'read' },
{ label: 'Read & write', value: 'write' }
]
const commandItems = [
{ label: 'All', value: 'all' },
{ label: 'Select', value: 'select' },
{ label: 'Insert', value: 'insert' },
{ label: 'Update', value: 'update' },
{ label: 'Delete', value: 'delete' }
]
async function load() {
loading = true
log = undefined
try {
const res = await WorkspaceService.getDatatablePermissions({
workspace,
datatableName: datatable
})
enabled = res?.enabled ?? false
grants = res?.grants ?? []
policies = res?.policies ?? []
} catch (e: any) {
sendUserToast(`Failed to load permissions: ${e?.body ?? e?.message ?? e}`, true)
} finally {
loading = false
}
}
function openModal() {
open = true
load()
}
async function save() {
busy = true
try {
const body: DataTablePermissions = { enabled, grants, policies }
log = await WorkspaceService.setDatatablePermissions({
workspace,
datatableName: datatable,
requestBody: body
})
sendUserToast('Permissions saved')
} catch (e: any) {
sendUserToast(`Failed to save permissions: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
async function sync() {
busy = true
try {
log = await WorkspaceService.syncDatatablePermissions({
workspace,
datatableName: datatable
})
sendUserToast('Permissions re-synced')
} catch (e: any) {
sendUserToast(`Failed to sync permissions: ${e?.body ?? e?.message ?? e}`, true)
} finally {
busy = false
}
}
function addGrant() {
grants = [
...grants,
{ principal: { kind: 'user', name: '' }, table: undefined, access: 'read' }
]
}
function removeGrant(i: number) {
grants = grants.filter((_, j) => j !== i)
}
function addPolicy() {
policies = [...policies, { table: '', name: '', command: 'all', principals: [] }]
}
function removePolicy(i: number) {
policies = policies.filter((_, j) => j !== i)
}
function addPolicyPrincipal(p: DataTablePolicy) {
p.principals = [...(p.principals ?? []), { kind: 'user', name: '' }]
}
function removePolicyPrincipal(p: DataTablePolicy, i: number) {
p.principals = (p.principals ?? []).filter((_, j) => j !== i)
}
</script>
<Button
variant="default"
size="sm"
{disabled}
startIcon={{ icon: Shield }}
title="Manage data table permissions"
on:click={openModal}
>
Permissions
</Button>
<Modal2 bind:isOpen={open} title="Permissions {datatable}" fixedWidth="lg" fixedHeight="lg">
{#snippet headerLeft()}
<Tooltip>
Restrict who can read and write this data table's rows, enforced natively by Postgres roles
and row-level security. Opt in below, then grant access per user/group and optionally add
row-level policies. Workspace admins always keep full access.
</Tooltip>
{/snippet}
<div class="flex flex-col gap-3 w-full grow min-h-0">
{#if loading}
<div class="flex items-center justify-center grow text-tertiary">
<Loader2 size={18} class="animate-spin" />
</div>
{:else}
<div class="flex items-center justify-between gap-2">
<Toggle
size="sm"
checked={enabled}
disabled={!ee || !canManage}
eeOnly
options={{
right: 'Advanced permissions',
rightTooltip:
'When off, every workspace member has full access (legacy behavior). When on, access is restricted to the grants and policies below.'
}}
on:change={(e) => (enabled = e.detail)}
/>
</div>
{#if !ee}
<div class="flex items-center gap-2 text-xs text-yellow-600 dark:text-yellow-400">
<TriangleAlert size={14} />
Data table permissions require an enterprise license.
</div>
{/if}
<Tabs bind:selected={selectedTab} class="grow min-h-0">
<Tab value="access" label="Access" />
<Tab value="policies" label="Row policies" />
{#snippet content()}
<TabContent value="access" class="grow min-h-0 overflow-auto pt-3">
<div class="flex flex-col gap-2">
<div class="text-xs text-tertiary">
Grant a user or group access to a specific table, or to all tables (leave the table
field empty).
</div>
{#each grants as grant, i (i)}
<div class="flex items-center gap-2">
<Select
items={kindItems}
bind:value={grant.principal.kind}
disabled={!canManage}
class="w-28"
/>
<TextInput
bind:value={grant.principal.name}
inputProps={{
placeholder: grant.principal.kind === 'user' ? 'email' : 'group name',
disabled: !canManage
}}
/>
<TextInput
bind:value={grant.table}
inputProps={{ placeholder: 'all tables', disabled: !canManage }}
/>
<Select
items={accessItems}
bind:value={grant.access}
disabled={!canManage}
class="w-40"
/>
<Button
variant="subtle"
size="xs"
iconOnly
color="red"
startIcon={{ icon: Trash2 }}
disabled={!canManage}
on:click={() => removeGrant(i)}
/>
</div>
{/each}
<div>
<Button
variant="subtle"
size="xs"
startIcon={{ icon: Plus }}
disabled={!canManage}
on:click={addGrant}
>
Add grant
</Button>
</div>
</div>
</TabContent>
<TabContent value="policies" class="grow min-h-0 overflow-auto pt-3">
<div class="flex flex-col gap-3">
<div class="text-xs text-tertiary">
Row-level policies filter which rows a principal can see or change. Use
<code>wm_email()</code> for the current user's email, and standard SQL in the USING
(read/delete) and WITH CHECK (insert/update) expressions, e.g.
<code>owner = wm_email()</code>.
</div>
{#each policies as policy, i (i)}
<div class="flex flex-col gap-2 border rounded-md p-3">
<div class="flex items-center gap-2">
<TextInput
bind:value={policy.table}
inputProps={{ placeholder: 'table', disabled: !canManage }}
/>
<TextInput
bind:value={policy.name}
inputProps={{ placeholder: 'policy name', disabled: !canManage }}
/>
<Select
items={commandItems}
bind:value={policy.command}
disabled={!canManage}
class="w-32"
/>
<Button
variant="subtle"
size="xs"
iconOnly
color="red"
startIcon={{ icon: Trash2 }}
disabled={!canManage}
on:click={() => removePolicy(i)}
/>
</div>
<div class="flex flex-col gap-1 pl-1">
<span class="text-xs text-hint">Applies to</span>
{#each policy.principals ?? [] as principal, j (j)}
<div class="flex items-center gap-2">
<Select
items={kindItems}
bind:value={principal.kind}
disabled={!canManage}
class="w-28"
/>
<TextInput
bind:value={principal.name}
inputProps={{
placeholder: principal.kind === 'user' ? 'email' : 'group name',
disabled: !canManage
}}
/>
<Button
variant="subtle"
size="xs"
iconOnly
color="red"
startIcon={{ icon: Trash2 }}
disabled={!canManage}
on:click={() => removePolicyPrincipal(policy, j)}
/>
</div>
{/each}
<div>
<Button
variant="subtle"
size="xs"
startIcon={{ icon: Plus }}
disabled={!canManage}
on:click={() => addPolicyPrincipal(policy)}
>
Add principal
</Button>
</div>
</div>
<TextInput
bind:value={policy.using}
inputProps={{
placeholder: 'USING expression (e.g. owner = wm_email())',
disabled: !canManage
}}
/>
<TextInput
bind:value={policy.check}
inputProps={{
placeholder: 'WITH CHECK expression (insert/update)',
disabled: !canManage
}}
/>
</div>
{/each}
<div>
<Button
variant="subtle"
size="xs"
startIcon={{ icon: Plus }}
disabled={!canManage}
on:click={addPolicy}
>
Add policy
</Button>
</div>
</div>
</TabContent>
{/snippet}
</Tabs>
{#if log}
<div
class="text-xs font-mono whitespace-pre-wrap max-h-32 overflow-auto border rounded-md p-2 text-secondary"
>
{log}
</div>
{/if}
<div class="flex justify-between gap-2 pt-2 border-t">
<Button
variant="subtle"
size="sm"
startIcon={{ icon: RefreshCw }}
disabled={busy || !ee || !canManage || !enabled}
title="Re-apply the stored config (e.g. after group changes or new tables)"
on:click={sync}
>
Sync
</Button>
<Button variant="accent" size="sm" disabled={busy || !ee || !canManage} on:click={save}>
Save
</Button>
</div>
{/if}
</div>
</Modal2>
@@ -1,6 +1,12 @@
<script lang="ts" module>
import { randomUUID } from '$lib/utils/uuid'
export type DataTableSettingsType = {
dataTables: {
// Stable client-side id so the UI can track renames (A -> B) across a
// save rather than seeing them as a delete + add. Never sent to the
// backend config.
id: string
name: string
database: {
resource_type: 'postgresql' | 'instance'
@@ -16,6 +22,7 @@
if (settings?.datatables) {
for (const [name, rest] of Object.entries(settings.datatables)) {
s.dataTables.push({
id: randomUUID(),
name,
...rest
})
@@ -70,6 +77,8 @@
import CustomInstanceDbSelect from './CustomInstanceDbSelect.svelte'
import { Popover } from '../meltComponents'
import ExploreAssetButton from '../ExploreAssetButton.svelte'
import DataTableMigrationsButton from './DataTableMigrationsButton.svelte'
import DataTablePermissionsButton from './DataTablePermissionsButton.svelte'
import { deepEqual } from 'fast-equals'
import { clone } from '$lib/utils'
import SettingsFooter from './SettingsFooter.svelte'
@@ -114,6 +123,7 @@
? `${random_adj()}_datatable`
: 'main'
tempSettings.dataTables.push({
id: randomUUID(),
name,
database: {
resource_type: $isCustomInstanceDbEnabled ? 'instance' : 'postgresql',
@@ -140,9 +150,19 @@
if (!confirm) return
}
const settings = convertDataTableSettingsToBackend(tempSettings)
// Track renames/deletions by stable id (against the saved baseline) so
// the backend can cascade or delete each data table's migrations.
const savedById = new Map(dataTableSettings.dataTables.map((d) => [d.id, d.name]))
const tempIds = new Set(tempSettings.dataTables.map((d) => d.id))
const renames = tempSettings.dataTables
.filter((d) => savedById.has(d.id) && savedById.get(d.id) !== d.name)
.map((d) => ({ from: savedById.get(d.id)!, to: d.name }))
const deleted_datatables = dataTableSettings.dataTables
.filter((d) => !tempIds.has(d.id))
.map((d) => d.name)
await WorkspaceService.editDataTableConfig({
workspace: $workspaceStore!,
requestBody: { settings }
requestBody: { settings, renames, deleted_datatables }
})
dataTableSettings = clone(tempSettings)
sendUserToast('Data table settings saved successfully')
@@ -158,7 +178,7 @@
const map: Record<string, boolean> = {}
for (let i = 0; i < tempSettings.dataTables.length; i++) {
let temp = tempSettings.dataTables[i]
let dt = dataTableSettings.dataTables.find((d) => d.name === temp.name)
let dt = dataTableSettings.dataTables.find((d) => d.id === temp.id)
map[temp.name] = !deepEqual(dt, temp)
}
return map
@@ -218,7 +238,7 @@
</Cell>
</Row>
{/if}
{#each tempSettings.dataTables as dataTable, dataTableIndex}
{#each tempSettings.dataTables as dataTable, dataTableIndex (dataTable.id)}
<Row>
<Cell first class="w-48 relative">
<TextInput bind:value={dataTable.name} inputProps={{ placeholder: 'Name', id: 'name' }} />
@@ -279,8 +299,18 @@
</div>
</Cell>
<Cell class="w-12">
<Cell class="whitespace-nowrap">
<div class="flex gap-2">
<DataTableMigrationsButton
workspace={$workspaceStore ?? ''}
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
<DataTablePermissionsButton
workspace={$workspaceStore ?? ''}
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
{#if dirtyMap[dataTable.name]}
<Popover
openOnHover
@@ -0,0 +1,230 @@
<script lang="ts">
import { Button } from '../common'
import Modal2 from '../common/modal/Modal2.svelte'
import Tabs from '../common/tabs/Tabs.svelte'
import Tab from '../common/tabs/Tab.svelte'
import TabContent from '../common/tabs/TabContent.svelte'
import Toggle from '../Toggle.svelte'
import TextInput from '../text_input/TextInput.svelte'
import SimpleEditor from '../SimpleEditor.svelte'
import { WorkspaceService, type DatatableMigration } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
import { tick } from 'svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import Portal from '$lib/components/Portal.svelte'
import { fetchPendingMigrations, outOfOrderRunMessage } from './datatableMigrationUtils'
let {
workspace,
datatable,
onCreated,
onClose,
onSeeMigration
}: {
workspace: string
datatable: string
/** Called after a successful create; `ran` is true when it was also run. */
onCreated?: (ran: boolean) => void
/** Called whenever the modal closes. `result` reports whether the close
* was due to a create (and whether that create was also run) vs a cancel —
* computed synchronously so callers don't depend on onCreated/onClose order. */
onClose?: (result: { created: boolean; ran: boolean }) => void
/** When set, the success toast gets a "See migration" action that calls this
* with the created migration so the caller can open it in the Migrations modal. */
onSeeMigration?: (migration: DatatableMigration) => void
} = $props()
let isOpen = $state(false)
// The reason the modal is about to close, set synchronously before `isOpen`
// flips so the onClose effect reports it reliably regardless of effect timing.
let closeResult = { created: false, ran: false }
let prevOpen = false
$effect(() => {
if (prevOpen && !isOpen) {
onClose?.(closeResult)
closeResult = { created: false, ran: false }
}
prevOpen = isOpen
})
let tab = $state('up')
let name = $state('')
let nameInput = $state<TextInput>()
// A valid migration name is non-empty and limited to letters, digits, '_' and '-'.
const MIGRATION_NAME_RE = /^[a-zA-Z0-9_-]+$/
let nameInvalid = $derived(!MIGRATION_NAME_RE.test(name.trim()))
let codeUp = $state('')
let enableDown = $state(false)
let codeDown = $state('')
let creating = $state(false)
const confirmationModal = createAsyncConfirmationModal()
// Frame the migration body in an explicit transaction so it applies atomically.
function wrapInTransaction(body: string): string {
return `BEGIN;\n\n${body}\n\nEND;`
}
// A statement inside BEGIN; ... END; must be `;`-terminated. The SQL splitter
// strips the trailing `;` when extracting a detected DDL statement, so re-add
// it when missing.
function ensureTrailingSemicolon(body: string): string {
const trimmed = body.trimEnd()
return trimmed.endsWith(';') ? trimmed : `${trimmed};`
}
const PLACEHOLDER = wrapInTransaction('-- Add your migration here')
export function open(prefill?: { name?: string; codeUp?: string; codeDown?: string }) {
name = prefill?.name ?? ''
// Start from the transaction template; when prefilled from detected DDL,
// wrap that DDL in the same BEGIN; ... END; frame.
codeUp = prefill?.codeUp
? wrapInTransaction(ensureTrailingSemicolon(prefill.codeUp))
: PLACEHOLDER
codeDown = prefill?.codeDown ?? PLACEHOLDER
enableDown = (prefill?.codeDown ?? '') !== ''
tab = 'up'
isOpen = true
// Focus the name field once the modal content has rendered.
tick().then(() => nameInput?.focus())
}
async function create(run: boolean) {
if (name.trim() === '') {
sendUserToast('Migration name is required', true)
return
}
if (!MIGRATION_NAME_RE.test(name.trim())) {
sendUserToast("Invalid migration name: use only letters, digits, '_' and '-'", true)
return
}
if (run) {
// A new migration gets the highest timestamp, so any still-pending
// migration is earlier: running only this one applies it out of order.
// Warn just like the row-level Run action does. Best-effort: if the
// status can't be fetched, fall through and let the run/rollback handle it.
let pending: Awaited<ReturnType<typeof fetchPendingMigrations>> = []
try {
pending = await fetchPendingMigrations(workspace, datatable)
} catch {}
if (pending.length > 0) {
const confirmed = await confirmationModal.ask({
title: 'Run migration out of order',
confirmationText: 'Run anyway',
children: outOfOrderRunMessage(pending.length)
})
if (!confirmed) return
}
}
creating = true
try {
const created = await WorkspaceService.createDatatableMigration({
workspace,
datatableName: datatable,
requestBody: {
name: name.trim(),
code_up: codeUp,
code_down: enableDown ? codeDown : undefined
}
})
if (run) {
try {
await WorkspaceService.runDatatableMigrations({
workspace,
datatableName: datatable,
only: created.timestamp
})
} catch (runErr: any) {
// The migration was created but failed to run; undo the insertion so
// the user can fix the SQL and retry from a clean state.
await WorkspaceService.deleteDatatableMigration({
workspace,
datatableName: datatable,
timestamp: created.timestamp
}).catch(() => {})
sendUserToast(
`Migration failed to run and was reverted: ${runErr?.body ?? runErr?.message ?? runErr}`,
true
)
return
}
}
closeResult = { created: true, ran: run }
onCreated?.(run)
sendUserToast(
run ? 'Migration created and run' : 'Migration created',
'success',
onSeeMigration
? [{ label: 'See migration', callback: () => onSeeMigration?.(created) }]
: []
)
isOpen = false
} catch (e: any) {
sendUserToast(`Failed to create migration: ${e?.body ?? e?.message ?? e}`, true)
} finally {
creating = false
}
}
</script>
<!-- closeOnOutsideClick is off: the "Create and run" split-button menu renders in a
portal outside the modal, so an outside-click close would fire on its items and be
mistaken for a cancel. Close via the header X or Escape instead. -->
<Modal2
bind:isOpen
title="New migration — {datatable}"
fixedWidth="md"
fixedHeight="adaptive"
closeOnOutsideClick={false}
>
<div class="flex flex-col gap-3 w-full grow min-h-0">
<TextInput
bind:this={nameInput}
bind:value={name}
error={nameInvalid}
inputProps={{ placeholder: 'Migration name (e.g. add_index_to_customers)' }}
/>
<Tabs bind:selected={tab} class="grow min-h-0">
<Tab value="up" label="Up" />
<Tab value="down" label="Down" />
{#snippet content()}
<TabContent value="up" class="h-80 border rounded-md overflow-hidden">
<SimpleEditor class="h-full" lang="sql" bind:code={codeUp} />
</TabContent>
<TabContent value="down" class="h-80">
<div class="flex flex-col gap-2 h-full">
<Toggle
bind:checked={enableDown}
options={{ right: 'Enable down migration' }}
size="sm"
/>
{#if enableDown}
<div class="grow min-h-0 border rounded-md overflow-hidden">
<SimpleEditor class="h-full" lang="sql" bind:code={codeDown} />
</div>
{/if}
</div>
</TabContent>
{/snippet}
</Tabs>
<div class="flex justify-end pt-2">
<Button
variant="accent"
size="sm"
disabled={creating}
on:click={() => create(true)}
dropdownItems={[
{
label: 'Create without running',
onClick: () => create(false)
}
]}
>
Create and run
</Button>
</div>
</div>
</Modal2>
<Portal>
<ConfirmationModal {...confirmationModal.props} />
</Portal>
@@ -0,0 +1,30 @@
import { WorkspaceService, type DatatableMigrationWithStatus } from '$lib/gen'
/**
* Migrations that are defined but not yet applied. A newly-created migration
* always gets the highest timestamp, so every pending migration is "earlier":
* running the new one on its own would apply it ahead of them (out of order).
*/
export function pendingMigrations(
migrations: DatatableMigrationWithStatus[]
): DatatableMigrationWithStatus[] {
return migrations.filter((m) => m.status !== 'ran')
}
/** Fetch the data table's migration status and return the pending ones. */
export async function fetchPendingMigrations(
workspace: string,
datatableName: string
): Promise<DatatableMigrationWithStatus[]> {
const { migrations } = await WorkspaceService.getDatatableMigrationsStatus({
workspace,
datatableName
})
return pendingMigrations(migrations)
}
/** Confirmation copy shown before running a just-created migration ahead of
* `count` still-pending earlier ones (mirrors the row-level Run warning). */
export function outOfOrderRunMessage(count: number): string {
return `${count} earlier migration(s) have not been run yet. This migration might depend on them. Run it anyway?`
}
+2
View File
@@ -40,6 +40,8 @@ export type Kind =
| 'gcp_trigger'
| 'azure_trigger'
| 'email_trigger'
// Data table migration, diffed per `<datatable>/<timestamp>_<name>` path.
| 'datatable_migration'
// Legacy generic kind used by the cross-workspace `DeployWorkspace` UI,
// which carries the trigger sub-kind in `additionalInformation`.
| 'trigger'
+7 -2
View File
@@ -16,7 +16,8 @@ import {
SqsTriggerService,
UserService,
VariableService,
WebsocketTriggerService
WebsocketTriggerService,
WorkspaceService
} from '$lib/gen'
import {
fetchProtectionRulesForWorkspace,
@@ -234,7 +235,11 @@ function makeProvider(): DeployProvider {
getSchedule: (p) => ScheduleService.getSchedule(p),
createSchedule: (p) => ScheduleService.createSchedule(p),
updateSchedule: (p) => ScheduleService.updateSchedule(p),
deleteSchedule: (p) => ScheduleService.deleteSchedule(p)
deleteSchedule: (p) => ScheduleService.deleteSchedule(p),
// Datatable migrations
listDatatableMigrations: (p) => WorkspaceService.listDatatableMigrations(p),
upsertDatatableMigration: (p) => WorkspaceService.upsertDatatableMigration(p),
deleteDatatableMigration: (p) => WorkspaceService.deleteDatatableMigration(p)
}
}
@@ -77,6 +77,13 @@ datatable related commands
- `datatable run <sql:string>` - run a SQL query on a datatable
- `-n --name <name:string>` - Datatable name (default: main)
- `-s --silent` - Output only the final result as JSON. Useful for scripting.
- `datatable migrate` - manage datatable migrations
- `datatable migrate new <name:string>` - scaffold a new migration (.up.sql / .down.sql files)
- `-d --datatable <datatable:string>` - Target datatable (default: main)
- `datatable migrate up` - apply all pending migrations to the main datatable (or one via --datatable)
- `-d --datatable <datatable:string>` - Target datatable (default: main)
- `datatable migrate down` - roll back the most recent migration on the main datatable (or one via --datatable)
- `-d --datatable <datatable:string>` - Target datatable (default: main)
- `datatable create [name:string]` - register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable://<name>
- `--resource <resource:string>` - Back the datatable with an existing postgresql resource path instead of the instance database
- `--force` - Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved)
@@ -345,19 +352,18 @@ Manage jobs (list, inspect, cancel)
### jobs
Pull completed and queued jobs from workspace
**Arguments:** `[workspace:string]`
**Options:**
- `-c, --completed-output <file:string>` - Completed jobs output file (default: completed_jobs.json)
- `-q, --queued-output <file:string>` - Queued jobs output file (default: queued_jobs.json)
- `--skip-worker-check` - Skip checking for active workers before export
Manage jobs (import/export)
**Subcommands:**
- `jobs pull`
- `jobs push`
- `jobs pull [workspace:string]` - Pull completed and queued jobs from workspace
- `-c, --completed-output <file:string>` - Completed jobs output file (default: completed_jobs.json)
- `-q, --queued-output <file:string>` - Queued jobs output file (default: queued_jobs.json)
- `--skip-worker-check` - Skip checking for active workers before export
- `jobs push [workspace:string]` - Push completed and queued jobs to workspace
- `-c, --completed-file <file:string>` - Completed jobs input file (default: completed_jobs.json)
- `-q, --queued-file <file:string>` - Queued jobs input file (default: queued_jobs.json)
- `--skip-worker-check` - Skip checking for active workers before import
### lint
+16 -10
View File
@@ -2753,6 +2753,13 @@ datatable related commands
- \`datatable run <sql:string>\` - run a SQL query on a datatable
- \`-n --name <name:string>\` - Datatable name (default: main)
- \`-s --silent\` - Output only the final result as JSON. Useful for scripting.
- \`datatable migrate\` - manage datatable migrations
- \`datatable migrate new <name:string>\` - scaffold a new migration (.up.sql / .down.sql files)
- \`-d --datatable <datatable:string>\` - Target datatable (default: main)
- \`datatable migrate up\` - apply all pending migrations to the main datatable (or one via --datatable)
- \`-d --datatable <datatable:string>\` - Target datatable (default: main)
- \`datatable migrate down\` - roll back the most recent migration on the main datatable (or one via --datatable)
- \`-d --datatable <datatable:string>\` - Target datatable (default: main)
- \`datatable create [name:string]\` - register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable://<name>
- \`--resource <resource:string>\` - Back the datatable with an existing postgresql resource path instead of the instance database
- \`--force\` - Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved)
@@ -3021,19 +3028,18 @@ Manage jobs (list, inspect, cancel)
### jobs
Pull completed and queued jobs from workspace
**Arguments:** \`[workspace:string]\`
**Options:**
- \`-c, --completed-output <file:string>\` - Completed jobs output file (default: completed_jobs.json)
- \`-q, --queued-output <file:string>\` - Queued jobs output file (default: queued_jobs.json)
- \`--skip-worker-check\` - Skip checking for active workers before export
Manage jobs (import/export)
**Subcommands:**
- \`jobs pull\`
- \`jobs push\`
- \`jobs pull [workspace:string]\` - Pull completed and queued jobs from workspace
- \`-c, --completed-output <file:string>\` - Completed jobs output file (default: completed_jobs.json)
- \`-q, --queued-output <file:string>\` - Queued jobs output file (default: queued_jobs.json)
- \`--skip-worker-check\` - Skip checking for active workers before export
- \`jobs push [workspace:string]\` - Push completed and queued jobs to workspace
- \`-c, --completed-file <file:string>\` - Completed jobs input file (default: completed_jobs.json)
- \`-q, --queued-file <file:string>\` - Queued jobs input file (default: queued_jobs.json)
- \`--skip-worker-check\` - Skip checking for active workers before import
### lint
@@ -82,6 +82,13 @@ datatable related commands
- `datatable run <sql:string>` - run a SQL query on a datatable
- `-n --name <name:string>` - Datatable name (default: main)
- `-s --silent` - Output only the final result as JSON. Useful for scripting.
- `datatable migrate` - manage datatable migrations
- `datatable migrate new <name:string>` - scaffold a new migration (.up.sql / .down.sql files)
- `-d --datatable <datatable:string>` - Target datatable (default: main)
- `datatable migrate up` - apply all pending migrations to the main datatable (or one via --datatable)
- `-d --datatable <datatable:string>` - Target datatable (default: main)
- `datatable migrate down` - roll back the most recent migration on the main datatable (or one via --datatable)
- `-d --datatable <datatable:string>` - Target datatable (default: main)
- `datatable create [name:string]` - register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable://<name>
- `--resource <resource:string>` - Back the datatable with an existing postgresql resource path instead of the instance database
- `--force` - Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved)
@@ -350,19 +357,18 @@ Manage jobs (list, inspect, cancel)
### jobs
Pull completed and queued jobs from workspace
**Arguments:** `[workspace:string]`
**Options:**
- `-c, --completed-output <file:string>` - Completed jobs output file (default: completed_jobs.json)
- `-q, --queued-output <file:string>` - Queued jobs output file (default: queued_jobs.json)
- `--skip-worker-check` - Skip checking for active workers before export
Manage jobs (import/export)
**Subcommands:**
- `jobs pull`
- `jobs push`
- `jobs pull [workspace:string]` - Pull completed and queued jobs from workspace
- `-c, --completed-output <file:string>` - Completed jobs output file (default: completed_jobs.json)
- `-q, --queued-output <file:string>` - Queued jobs output file (default: queued_jobs.json)
- `--skip-worker-check` - Skip checking for active workers before export
- `jobs push [workspace:string]` - Push completed and queued jobs to workspace
- `-c, --completed-file <file:string>` - Completed jobs input file (default: completed_jobs.json)
- `-q, --queued-file <file:string>` - Queued jobs input file (default: queued_jobs.json)
- `--skip-worker-check` - Skip checking for active workers before import
### lint
+76 -10
View File
@@ -340,13 +340,54 @@ def extract_description(section: str) -> str | None:
return ''.join(_unquote_js_string(p) for p in parts).strip() or None
def parse_command_block(content: str, file_path: Path | None = None) -> dict:
def extract_named_command_block(content: str, var_name: str) -> str | None:
"""Return the chained-call body of `const <var_name> = new Command() ...`,
from just after `new Command()` up to the next top-level statement.
Returns None when the var isn't a *direct* `new Command()` (e.g. it's wrapped
in a helper call like `auditListOptions(new Command()...)`), so callers can
fall back to a looser match.
"""
m = re.search(
r'const\s+' + re.escape(var_name) + r'\s*=\s*new\s+Command\(\)'
r'([\s\S]*?)(?=\n(?:const|let|var|async|function|export)\b)',
content,
)
return m.group(1) if m else None
def extract_exported_command_block(content: str) -> str | None:
"""Return the chained-call body of the command that is `export default`ed.
A command file may define helper `new Command()` groups (assigned to local
consts and mounted as nested subcommands via `.command("x", localCmd)`)
*before* the exported command. Anchoring on the first `new Command()` in the
file would merge those helpers into the top-level command, so resolve the
exported variable first and only then fall back to the first `new Command()`
(which covers inline/wrapped exports).
"""
export_match = re.search(r'export\s+default\s+(\w+)\s*;', content)
if export_match:
block = extract_named_command_block(content, export_match.group(1))
if block is not None:
return block
command_match = re.search(
r'(?:const\s+command\s*=\s*)?new\s+Command\(\)([\s\S]*?)(?=export\s+default)',
content,
)
return command_match.group(1) if command_match else None
def parse_command_block(
content: str, file_path: Path | None = None, block: str | None = None
) -> dict:
"""
Parse a Cliffy Command() definition block and extract metadata.
Returns a dict with: description, options, subcommands, arguments, alias
If file_path is provided, imported subcommands will be resolved by parsing
the imported files.
the imported files. `block` may be passed to parse a specific pre-extracted
command body (used to recurse into locally-defined nested command groups).
"""
result = {
'description': '',
@@ -357,15 +398,11 @@ def parse_command_block(content: str, file_path: Path | None = None) -> dict:
}
# Find the command block
command_match = re.search(
r'(?:const\s+command\s*=\s*)?new\s+Command\(\)([\s\S]*?)(?=export\s+default)',
content
)
if not command_match:
if block is None:
block = extract_exported_command_block(content)
if block is None:
return result
block = command_match.group(1)
# Find where subcommands start
first_subcommand_pos = block.find('.command(')
if first_subcommand_pos == -1:
@@ -451,12 +488,31 @@ def parse_command_block(content: str, file_path: Path | None = None) -> dict:
'name': cmd_name,
'description': imported_cmd.get('description', ''),
'arguments': imported_cmd.get('arguments', ''),
'options': imported_cmd.get('options', [])
'options': imported_cmd.get('options', []),
'subcommands': imported_cmd.get('subcommands', []),
})
continue
except Exception as e:
print(f" Warning: Could not parse imported command {second_arg}: {e}")
cmd_desc = ''
elif second_arg and re.search(
r'const\s+' + re.escape(second_arg) + r'\s*=\s*new\s+Command\(\)', content
):
# Locally-defined command group mounted as a subcommand
# (e.g. `.command("migrate", migrateCommand)`): recurse into its
# definition so its own subcommands/options are captured.
nested_block = extract_named_command_block(content, second_arg)
if nested_block is not None:
nested = parse_command_block(content, file_path, block=nested_block)
result['subcommands'].append({
'name': cmd_name,
'description': nested.get('description', ''),
'arguments': nested.get('arguments', ''),
'options': nested.get('options', []),
'subcommands': nested.get('subcommands', []),
})
continue
cmd_desc = ''
else:
cmd_desc = ''
@@ -628,6 +684,16 @@ def generate_cli_commands_markdown(cli_data: dict) -> str:
for opt in sub['options']:
md += f" - `{opt['flag']}` - {opt['description']}\n"
# Nested sub-subcommands (e.g. `datatable migrate new`)
for subsub in sub.get('subcommands', []):
ss_args = f" {subsub['arguments']}" if subsub.get('arguments') else ""
md += f" - `{cmd['name']} {sub_name} {subsub['name']}{ss_args}`"
if subsub.get('description'):
md += f" - {subsub['description']}"
md += "\n"
for opt in subsub.get('options', []):
md += f" - `{opt['flag']}` - {opt['description']}\n"
md += "\n"
return md