mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-20 08:03:28 +00:00
9519b19f4777caada4fbb2d2d06e761eab7cfa3d
108
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Ruben FiszelandClaude Opus 5.5
|
26856b8061 |
fix: cancel an abandoned pg query and close failed connections cleanly (#11511)
* fix: cancel the query of a timed out or cancelled postgresql job Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpWeGy8jRMtWqvuRBpgu4 * test: fail the cancelled-query test unless the cancel happened Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpWeGy8jRMtWqvuRBpgu4 * fix: leave a failed pg job's connection clean for a pooler before closing it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpWeGy8jRMtWqvuRBpgu4 * fix: cancel a failed pg job's unread results and roll back only inside a transaction Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpWeGy8jRMtWqvuRBpgu4 * fix: redo a pg connection's closing reset when a late cancel lands on it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpWeGy8jRMtWqvuRBpgu4 * fix: let a late cancel land on the settle query without skipping the reset Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpWeGy8jRMtWqvuRBpgu4 --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5.5
|
d4a423f567 |
fix: bound the pg cached-connection reset probe so a vanished server fails fast (#11505)
Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5.5
|
013b302d48 |
perf: keep a pg executor connection per database (#11492)
* fix: stop the pg executor cache from pinning a pooler slot Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test: retry while the evicted pg backend exits Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf: keep a pg executor connection per database Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: close a pg connection whose query was cut off by a timeout or cancel Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: keep a pg connection only after its query read every result Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: free idle pg connections and retry when a connect is refused Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: retry a pg connect only when a connection cap refused it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: detect pg connection caps by SQLSTATE and label cloud connections as new Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5.5
|
b7f74cd562 |
fix: never reuse a pg connection a script left inside a transaction (#11497)
* fix: never reuse a pg connection a script left inside a transaction Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: close a pg connection whose job failed instead of caching it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * docs: describe every caller of PgConnectionLease::discard Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5.5
|
d7227726be |
feat: log pg job progress, warn on stalls, detect dead db connections (#11491)
* feat: log pg job progress, warn on stalls, detect dead db connections Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: log only slow pg steps to keep job log writes off fast statements Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi * fix: say when a slow connect included a failed cached-connection reset Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjj2iG8kKPCaFuK6bqXvi --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5.5
|
eed7e7d9e6 |
fix: stop the pg executor cache from pinning a pooler slot (#11490)
* fix: stop the pg executor cache from pinning a pooler slot Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test: retry while the evicted pg backend exits Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5.5
|
14801fdd68 |
fix: unstick postgres jobs on large results with custom-typed columns (#11475)
* fix: unstick postgres jobs on large results with custom-typed columns Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: carry the parked-rows delivery fix in the postgres fork Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: describe postgres queries before streaming instead of parking rows Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf: skip the postgres describe for statements that return no rows Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: look for returning in the whole postgres statement Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: read the leading postgres keyword past nested block comments Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: end leading postgres line comments at CR too Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5.5
|
ae093b7666 |
fix: read inet and cidr results with their prefix, including arrays (#11476)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
 
|
0e807fb1dd |
feat: put a data table's connection under Postgres roles (#11020)
* feat(datatables): put a data table's connection under Postgres roles
A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.
A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.
Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.
A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.
Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): gate the paths that reach a whole database as admin
Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.
All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping
A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.
Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.
Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): fail loudly where a role or a pointer can be left half-recorded
Three ways the feature could end up in a state nobody could see or undo.
Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.
Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.
`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.
Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): unbreak two operator messages and two comments that described other code
The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.
`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): serialize role catalog mutations, and state each helper's authorization contract
The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.
Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.
The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.
Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.
Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): give the role catalog its own row, out of reach of the config machinery
Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.
`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.
Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.
Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.
Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced
The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".
The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): close the last ways a role or a pointer can be left pointing at nothing
The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.
Three ways a save could leave something resolving to nothing:
A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.
Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.
Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.
Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* refactor(datatables): put the role catalog in its own table, not in global_settings
Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.
The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.
One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.
Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): validate a rename against the save it describes, and re-check under the locks
Three from the round, all about deciding on state that could already have moved.
A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.
Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.
The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.
Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically
The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".
The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.
Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): refuse to copy a data table that is under roles
pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.
Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): refuse the clone's database too, not only its data
A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.
Refuse in both, so the clone stops before a database exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* nit worker error msg
* fix pg_dump stuck on version 17 on nix
* fix(datatables): refuse a malformed role annotation instead of ignoring it
`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.
A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.
Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): refuse a malformed role query string instead of ignoring it
`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.
The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): carry the role annotation into the row_to_json retry
The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".
Carry the leading comment block over. The retry itself is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* chore(datatables): don't mount the roles UI until the ACL editor lands
Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.
The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): honour `-- role: x`, and fix the DuckDB attach test
Two review findings, both real.
`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.
`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): clone a fork's pointer instead of failing after the copy
Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.
Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.
Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): refuse to roll back the catalog while roles exist
The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.
Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* refactor(datatables): settle clonability in one place, before anything is created
A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.
That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.
Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): let a retried clone reclaim its own leftover database
A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.
`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Revert "fix(datatables): let a retried clone reclaim its own leftover database"
This reverts commit
|
||
|
Ruben FiszelandClaude Opus 5
|
201d7c4eb2 |
fix: bound postgres result collection so an oversized result cannot OOM the worker (#10644)
* fix: bound postgres result collection so it cannot OOM the worker Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: render the sql result limit exactly so the error can be set verbatim Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: point the fraction rationale at the renderer that still emits them Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: stop re-parsing every collected row to rebuild it as a RawValue Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style: drop a dangling doc line and an unrelated rustfmt reflow Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
c054018c9b |
require an explicit user for azure workload identity on postgres (#10521)
* fix: name the pg login in the job log for token auth modes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: route remaining pg login defaults through login_name Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: require an explicit user for azure workload identity on postgres Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
9f3d15583a |
fix: log the db auth mode used and hint at the ms_entraid sentinel (#10508)
* fix: surface which auth mode a sql connection used and hint at ms_entraid Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope the ms_entraid hint to azure hosts and pin the sentinel trim Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
beef6e295c |
feat: azure workload identity auth for mssql and postgres resources (#10470)
* feat: azure workload identity auth for mssql and postgres resources * refactor: keep mssql config lines untouched by the auth-mode change * fix: single-flight token refresh, cache eviction and identity-aware pg cache key * fix: back off after a failed entra id refresh and normalize blank pg identity fields * fix: re-check the fallback token lifetime after a failed refresh * refactor: select workload identity with a sentinel password instead of resource fields * fix: log the workload identity mode on the postgres path too |
||
|
Ruben FiszelandClaude Opus 4.8
|
6e42633643 |
fix(pg): actionable error when s3object input exceeds jsonb 256MB cap (#10228)
Native SQL PostgreSQL scripts with an `(s3object)` input materialize the whole referenced file into a single jsonb parameter. PostgreSQL hard-caps a jsonb value's element payload at 256MB, so a large file fails with an opaque `total size of jsonb array elements exceeds the maximum of 268435455 bytes`. `materialize_s3object_args` now reports the largest materialized payload, and that specific server error is rewritten into guidance explaining the input is materialized (not streamed) and pointing large-file users at DuckDB, which reads S3 natively and streams. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
7d5009e392 |
fix: heartbeat job ping during s3object materialization in SQL executors (#10152)
* fix: heartbeat job ping during s3object materialization in SQL executors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrFtZjXc8GB6VtSMVJFXjE * chore: update ee-repo-ref.txt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrFtZjXc8GB6VtSMVJFXjE * chore: update ee-repo-ref to e19948fa2974a7d89bec12957fc6d9fa0a421da8 This commit updates the EE repository reference after PR #668 was merged in windmill-ee-private. Previous ee-repo-ref: a3828dcd67f026c0e983a1a5dc5c5b33af3c3120 New ee-repo-ref: e19948fa2974a7d89bec12957fc6d9fa0a421da8 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
 hugocasaandClaude Opus 4.8
|
bf6be967fa |
fix: honor verify-ca/verify-full sslmode for postgres connections (#9835)
* fix: enforce tls verification for postgres verify-ca/verify-full sslmode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: make PG_ACCEPT_INVALID_CERTS value-based and keep cache key well-formed Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: grandfather existing postgres resources via per-resource trust_cert flag Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope pg trust_cert migration to resource data, drop schema patch Hub resource type sync (windmill cache-rt + startup SYNC_CACHED_RT) only touches the admins workspace and is opt-in, so the schema is left to the hub; the migration just grandfathers existing resource values so the upgrade is non-breaking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: replace pg trust_cert with verify-*-scoped accept_invalid_certs, drop migration Per-resource accept_invalid_certs (default false for new resources) replaces the trust_cert flag and grandfather migration. It only applies to verify-ca/verify-full; unset falls back to legacy behavior (verify only when a root cert is present) so existing and git-synced resources are not broken on upgrade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: warn in job logs when a verify-* postgres resource skips cert verification Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 Diego Imbertandclaude[bot]
|
28c8b5c60f |
feat: CLI datatable serve / psql (#9267)
* feat(cli): add datatable list and run commands * feat(cli): render datatable query results as a table * feat(cli): serve datatables as a postgres-wire endpoint * feat(cli): add 'datatable psql' to launch psql against the proxy * feat(cli): route datatable serve by client-supplied database name * override database list + password option * fix: support extended queries in datatable serve * fix: correct cloud size threshold log and parse CLI descriptions with parens/trailing comma * refactor: extract raw_output envelope encoding into pg_raw_output module --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.7
|
aedf369174 |
fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999)
* fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(pg): wrap encoder errors with arg context, add fallback test Followups on #8999 review: - Wrap rust-postgres "error serializing parameter N" failures with the arg name, JSON value kind, and asserted Postgres type plus a hint about an explicit cast — so users see actionable context instead of an opaque WrongType. - Drift-prevention meta-test: assert otyp_to_pg_type and convert_val agree on the Type for every recognised arg_t when the JSON value matches its natural Rust kind. Catches future drift if either side changes. - Integration test for the prepare + query_raw fallback path: confirms unrecognised arg_t (custom enum) is routed through prepare and the server-resolved type appears in the failure surface — flips into a test failure if a regression accidentally routes unrecognised types through query_typed_raw. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add otyp_inferred flag + regex-based placeholder renumbering Two follow-ups from the review of #8999: 1. **Issue #1 (Number/Bool + explicit text decl in WHERE)** Add `Arg::otyp_inferred: bool` to the parser. The PG SQL parser sets it `true` only at the "no info → fall back to text" site (bare `$N`, no inline cast, no `-- $N (TYPE)` decl). All other arg sources keep it `false`. In `convert_val` this flag distinguishes: - explicit text-like target (`-- $1 (text)` or `$1::text`) — coerce `Bool`/`Number` → `Box<String>` so `WHERE text_col = $1` works (`text = text` operator). Pre-#8988 behaviour, restored. - parser-default text (bare `$N`) — bind the value's natural Rust type so the regression case (`Value::Bool` against a real `bool` column via `CAST AS bool`) keeps working. `Arg` is in `windmill-parser`; the new field has `#[serde(default)]` so persisted signatures stay backward-compatible. 2. **Issue #4 ($5/$50 substring rewrite collision)** Replace the per-index `String::replace` chain (which turned `$50` into `$10` when oidx=5 was processed first) with a single regex pass. `\d+` is greedy, so `$5` and `$50` match as distinct units; indices outside the mapping are left intact. 3. Tests: - parser: `test_parse_pgsql_otyp_inferred_flag` covers bare/inline- cast/decl/mixed shapes. - executor unit: `convert_val_bool_against_every_arg_t` and `convert_val_*_number_*` split each text-like target into explicit vs inferred expectations. - executor unit: `renumber_sparse_placeholders_no_collision`. - integration: `test_postgresql_arg_type_combinations` adds 4 cases covering decl(text)+Number/Bool in WHERE, bare $1+Bool, and sparse positional args ($5/$50). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg+sdk): enum support, extended String arms, position-aware $N rewrite, SDK quality Backend: 1. **`AnyTextValue` ToSql/FromSql wrapper**: vanilla `tokio_postgres`'s `ToSql for String` / `FromSql for String` reject `Kind::Enum` and `Kind::Domain` even though the wire format is plain UTF-8. The wrapper accepts those kinds in both directions. End result: explicit `$1::my_enum` / `CAST($1 AS my_enum)` casts now round-trip without the ugly `CAST($1::text AS my_enum)` workaround, AND `SELECT enum_col` results come back as JSON strings instead of erroring at the FromSql layer. 2. **#10 — Value::String → numeric/real/double/oid/bool**. Without these arms, a string-encoded value (`"3.14"`, `"true"`) for a non-text / non-temporal arg_t fell through to `Box<String> + TEXT`, which then failed at the server (no implicit cast text→numeric in expression context). Now strings are parsed into the matching native type with clear error messages on parse failure. 3. **Position-aware `$N` rewrite**: replaces the regex-based renumbering (which fixed the `$5/$50` substring collision but still walked through string literals and comments, mangling `'price: $5'` etc.) with a walk over `parse_pg_statement_arg_positions` — the same string/comment/dollar-quote-aware tokenizer used for index discovery. Adds `parse_pg_statement_arg_positions` to the parser's public API. SDK: 4. **BigInt support**: `JSON.stringify(BigInt)` throws. The SDK now stringifies bigints before serialisation; the executor accepts numeric strings into BIGINT arg slots via the existing `Value::String → INT8` parsing arm. SDK-side `inferSqlType` is split so `BigInt` always resolves to `BIGINT` (was reaching `Number.isInteger(BigInt)` which returns false → wrong default). 5. **Homogeneous array auto-tag**: `${[1,2,3]}` against an `int[]` column now emits `$1::BIGINT[]` instead of `$1::JSON`. Detection covers primitive types only (number / bigint / string / boolean); mixed or nested arrays still fall back to JSON. Mixed int/float widens to `DOUBLE PRECISION[]`. 6. **`.query()` positional bug**: previously the `.query()` method abused the template-tag builder, which appended `$N::TYPE` after the user's literal SQL string instead of binding by position (`SELECT $1, $2` became `SELECT $1, $2$1::BIGINT`). Now `.query()` builds the executor-shaped content directly: a `-- $N argN (TYPE)` declaration block followed by the user's SQL verbatim. Tests: - Parser: `test_parse_pg_statement_arg_positions_skips_strings_and_comments` asserts string literals, comments, and dollar-quoted blocks don't produce positions (so renumbering doesn't mangle them). - Executor unit: `renumber_sparse_placeholders_no_collision_no_string_mangling` uses the new position-aware path and includes string-literal + comment + `$$…$$` cases. Existing convert_val tests grow to cover new String→numeric/real/double/oid/bool arms. - Integration: `test_postgresql_arg_type_combinations` adds 13 cases (enum round-trip both directions, string→numeric/real/double/bool/oid, string-literal `$N` non-mangling). The prepare-fallback test now asserts SUCCESS (not failure) for enum encoding via AnyTextValue. - SDK: new `typescript-client/tests/sqlUtils.test.ts` (42 tests) exhaustively covering inferSqlType primitives + arrays, parseTypeAnnotation, datatable() template tag (with all the new shapes — BigInt, homogeneous arrays, RawSql, schema preamble), datatable().query() positional, and ducklake() shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): replace DISCARD ALL with curated reset (preserves typeinfo cache) Found while exhaustively probing custom-type DX: every cached-connection reuse was running `DISCARD ALL`, whose included `DEALLOCATE ALL` deallocates *all* prepared statements server-side — including the typeinfo statements that tokio_postgres caches per-Client to resolve custom enum / domain Oids. tokio_postgres still held `Statement` objects whose names the server had forgotten, so the next custom-type query failed with intermittent "prepared statement \"sN\" does not exist" errors. The failure was easy to reproduce: any sequence that forced typeinfo lookup for two different custom-type kinds on the same cached connection (e.g. enum followed by domain) would hit it. Replace `DISCARD ALL` with a curated reset that explicitly targets the state we actually care about, *without* touching prepared statements: RESET ALL — GUC parameters (search_path, application _name, statement_timeout, …) RESET SESSION AUTHORIZATION — undoes both `SET SESSION AUTHORIZATION` and `SET ROLE` (RESET ALL does NOT — these aren't GUC parameters, so without this an elevated role from a previous job would silently leak) UNLISTEN * — drops LISTEN registrations CLOSE ALL — closes open cursors Trade-off: temp tables, advisory locks (session-scoped), and user-created PREPARE statements may persist across cached-connection reuse — rare in datatable / PG-script workloads. tokio_postgres's typeinfo cache survives intact, so custom enum / domain queries are fast on subsequent reuse. Tests: - `test_postgresql_custom_types_on_cached_connection` — runs 10× alternating enum + domain queries on a cached connection. Pre-fix this failed with `prepared statement "sN" does not exist` after the first reuse; post-fix passes. - `test_postgresql_set_role_does_not_leak_across_cached_connection` — switches `SET ROLE` and `SET SESSION AUTHORIZATION` to a non-postgres role, then runs a follow-up job and asserts current_user/session_user are restored. Specifically catches the case where someone might switch back to `RESET ALL` alone (which doesn't cover SET ROLE / SESSION AUTHORIZATION) and silently introduce a permission-leak vector. - All existing session-isolation tests (`test_postgresql_cached_connection_resets_session`, `test_postgresql_single_worker_session_isolation`, `test_postgresql_100_jobs_cached`) continue to pass. Found via end-to-end probing of datatable / PG-script DX, not previously covered: the existing isolation tests only did `SET ROLE postgres`, the connecting user, so the leak was invisible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): address PR #8999 review (cubic + claude) cubic (P1, real bug): - `convert_vec_val` for `timetz` array asserted `Type::TIMETZ_ARRAY`, but chrono `NaiveTime` only encodes for TIME (same caveat as the scalar arm). Switch to `Type::TIME_ARRAY`; rely on PG's implicit `time→timetz` assignment cast at the column site. Add an explicit unit test. claude (#1, silent failure → explicit error): - `Bool` + explicit `(char)` / `(character)` decl previously silently bound BOOL, hoping the server would cast at the use site — but PG has no implicit `bool→char` and the resulting error ("operator does not exist: bool = char") was opaque. Now error at bind time with an actionable hint to use `bool` decl or pass the value as a "t"/"f" string. claude (#2, asymmetry doc): - Object/Array still coerce to text on `matches!(typ, Typ::Str(_))` (covers both explicit AND inferred-default text), unlike Bool/Number which key on `explicit_text_target`. The asymmetry is intentional (no implicit `jsonb → text` cast in expression context vs PG having implicit `bool/int → text` casts) — added a body comment so future maintainers don't try to "align" them. claude (#3, perf): - `parse_pg_statement_arg_indices` and `parse_pg_statement_arg_positions` walked the SQL tokenizer twice. Fold into a single pass that derives the index set from the position list. claude (#4, fmt drift): - `cargo fmt` over the parser crates I touched with perl scripts in the earlier commit (windmill-parser-{sql,bash,ts,go,php,java,csharp,nu,py, rust,graphql,yaml,r}). Net cosmetic. claude (#5, parseTypeAnnotation): - One-line caveat in the SDK's `parseTypeAnnotation` that the returned string is presence-only (e.g. `${x}::DOUBLE PRECISION` returns `"DOUBLE"`, `CAST(${x} AS int)` returns `"int)"` — neither matches a real PG type, but the only consumer just checks `!== undefined`). While here — discovered + fixed independently while exhaustively probing DX: - **Replace `DISCARD ALL` with curated reset** (`RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`). DISCARD's `DEALLOCATE ALL` killed tokio_postgres' typeinfo cache, producing intermittent `prepared statement "sN" does not exist` errors on custom-type queries after cached-conn reuse. New regression tests: `test_postgresql_custom_types_on_cached_connection` and `test_postgresql_set_role_does_not_leak_across_cached_connection` (the latter catches the case where someone might switch back to `RESET ALL` alone and silently introduce a permission-leak vector — RESET ALL doesn't cover SET ROLE / SET SESSION AUTHORIZATION). - **ISO-8601 timestamp results** (`pg_cell_to_json_value`). Pre-fix `TIMESTAMP` was rendered with a space separator ("2024-01-15 10:30:00") and `TIMESTAMPTZ` with " UTC" suffix ("2024-01-15 10:30:00 UTC") — neither parseable by `date-fns parseISO`, JavaScript `new Date()` is lenient enough to handle them but several frontend `App*Input.svelte` components use parseISO and fail silently. Switched to ISO-8601 with `T` separator and `+00:00` offset; arg-parsing path still accepts the legacy " UTC" suffix for back-compat. Test coverage: - 17/17 unit (`pg_executor::tests`) - 9/9 integration (`backend/tests/worker.rs`, `test_postgresql_*`) - 27/27 parser (`windmill-parser-sql`) - 42/42 SDK (`typescript-client/tests/sqlUtils.test.ts`) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): bounded one-shot warning on numeric precision loss + ISO-8601 + NaN handling Found while probing PG-script DX with millions of numeric cells: 1. **Numeric precision-loss warning**: `numeric` results are still serialised as JSON Number (back-compat — switching to JSON String would silently break user code doing arithmetic on results), but we now detect `Decimal -> f64 -> Decimal` round-trip failure and emit a single job-log warning recommending a `::text` cast in the SQL. Bounded by `NUMERIC_PRECISION_CHECK_BUDGET = 256` cells per query (one atomic load + one fetch_sub on the hot path; first lossy value short-circuits to a single load thereafter). Worst-case overhead on a 1M-cell numeric-heavy query: ~25µs of checks + 5ns × N atomic loads (vs. ~100ms unbounded). 2. **ISO-8601 timestamps**: `pg_cell_to_json_value` previously returned `"2024-01-15 10:30:00"` (TIMESTAMP) and `"2024-01-15 10:30:00 UTC"` (TIMESTAMPTZ) — neither parseable by date-fns `parseISO`, which is what the apps `App*Input.svelte` components use, so timestamp values silently failed to round-trip into date pickers. Switch to ISO-8601 (`T` separator + `+00:00` offset) on the result side; arg-parser continues to accept the legacy `" UTC"`-suffixed format for back-compat. 3. **Float NaN / Infinity results**: `Number::from_f64` returns None for NaN / ±Inf, which `pg_cell_to_json_value` was raising as "invalid json-float" — failing the *entire* query if any cell held one of these special values. Now serialise them as JSON strings ("NaN", "Infinity", "-Infinity") and let the rest of the row come through. Arg-side: `s.parse::<f64>()` already accepts the same strings. Tests: - `decimal_fits_f64_losslessly_predicate` — covers fits / doesn't-fit cases for the precision-loss predicate. - `precision_check_budget_caps_per_query_overhead` — locks in the budget cap and the loss-flag short-circuit. - All 9 PG integration tests + 17 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add pg_advisory_unlock_all to reset; warn on missing args; honor decl defaults While probing PG-script DX further found three more frictions: 1. **Advisory lock leak** (cubic P2): switching from `DISCARD ALL` to `RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;` meant session-scoped advisory locks (`pg_advisory_lock`) leaked across cached-connection reuse. Add `SELECT pg_advisory_unlock_all()` to the chain — `DISCARD ALL` covered this implicitly via `DISCARD PLANS / DEALLOCATE / pg_advisory_unlock_all` and we lost it in the switch. 2. **Missing-arg silent NULL**: an arg declared in the SQL (e.g. `-- $1 amount (numeric)`) but not provided in the args object was bound as NULL with no error / warning. Misspelling the key in the args object silently produced a row of NULLs — a notorious DX debugging trap. Now: collect the names of declared-but-missing args during dispatch and emit a single one-shot warning to the job logs at end-of-query naming each one. Bound NULL is preserved for back-compat. 3. **Declaration defaults ignored**: `-- $1 a (int) = 5` carries `arg.default = Some(Number(5))`, but the dispatch fell straight to NULL when the arg was missing. Now: respect the default — user-supplied value > declaration default > NULL. Also fixes the warning logic above (only warn for args that *don't* have a default). Tests: existing 19 unit + 9 integration pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): multi-word PG types with [] suffix lost the array-ness; array arms accept stringified values Two more frictions found while probing SDK end-to-end against a real datatable resource: 1. **Multi-word array types lose the [] suffix in the parser**. `transform_types_with_spaces` recognises aliases for "double precision", "character varying", "timestamp with time zone", etc. but its return type was `&'a str` — only the bare alias, never with a trailing `[]`. The `RE_CODE_PGSQL` regex's `\w+` captures stop at the first space, so the regex's own `(?:\[\])?` array-suffix branch sees only `"double"` (not `"double precision[]"`); the `[]` was silently lost. Result: `$1::double precision[]` (which the SDK now emits for homogeneous float arrays via the new auto-tag) routed through `Value::Array → Type::JSONB` and the server failed with "cannot cast type jsonb to double precision[]". Fix: switch `transform_types_with_spaces` to return `Cow<'a, str>` and re-check the trailing bytes after a multi-word match. If they start with `[]`, return `format!("{alias}[]")` — Owned. Single-word types and the no-match path keep returning Borrowed slices, so no allocation in the hot path. 2. **Array arms in `convert_vec_val` rejected stringified values for numeric / int* / bool / oid / real / double**. The scalar `convert_val` already parses strings into the matching native type for these arg_ts, but the array variant only accepted JSON-native counterparts. Sending `["1.5", "2.5", "3.5"]` against `$1::numeric[]` (e.g. via `unnest` for bulk loading, or `JSON.stringify(BigInt[])` round-trip) failed with "Mixed types in array". Now the array arms mirror the scalar ones — `as_<native>().or_else(|| as_str().and_then(parse))` — so both shapes round-trip cleanly. Tests: 19 unit + 9 integration pass; existing parser tests cover the multi-word array forms (the regex-cap behaviour didn't break for single-word types, and Cow plumbing is transparent to all callers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(parsers): add otyp_inferred field to Arg literals in tests + 3 missed src files CI failures: the perl-driven sweep that added `otyp_inferred: false` to every `Arg { ... }` literal when I introduced the field in the parser schema covered `src/lib.rs` files but missed: - parsers/windmill-parser-bash/src/lib.rs (mass-edited but a later format pass un-applied a few sites) - parsers/windmill-parser-go/src/lib.rs (same) - parsers/windmill-parser-graphql/src/lib.rs (same) - parsers/windmill-parser-nu/tests/tests.rs (test file — not swept the first time) - parsers/windmill-parser-ts/tests/tests.rs (test file — same) Also tightened the regex to handle `oidx: None` without the trailing comma (some test files had the field as the last initialiser line). `cargo build --features <CI feature combo> --workspace --all-targets` is clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(sdk): Date → TIMESTAMPTZ; NaN / ±Infinity → string Two more frictions found while running the actual SDK end-to-end against a live datatable resource: 1. **JS `Date`** fell into the typeof "object" branch and was tagged `::JSON`. It worked accidentally for `${date}::timestamptz` via PG's `json → text → timestamptz` implicit cast chain, but `${date}` against a `timestamptz` column without a user-supplied cast bound the value as a JSON string and the comparison `timestamptz = json` failed. Now: `inferSqlType` recognises `Date` and tags `::TIMESTAMPTZ`; `serializeArgValue` emits `Date.toISOString()` so the executor's `Value::String → TIMESTAMPTZ` arm parses it cleanly. 2. **JS `NaN` / `±Infinity`** silently became NULL. `JSON.stringify(NaN)` returns `"null"` per the JS spec, so the value reached the executor as JSON null — the SDK's `::DOUBLE PRECISION` tag then bound a NULL double. Fix: detect non-finite numbers in `serializeArgValue` and stringify them as `"NaN" / "Infinity" / "-Infinity"`. The executor's `Value::String → FLOAT8` arm (`f64::from_str`) accepts these literals directly, and the result-side already renders the values as JSON strings (matching round-trip). SDK unit tests grow from 42 → 44 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(pg): integration coverage for multi-word arrays + stringified array elements Locks in the two array fixes from the previous commit (`fix(pg): multi-word PG types with [] suffix lost the array-ness`) with end-to-end cases in `test_postgresql_arg_type_combinations`: - `double precision[]`, `character varying[]`, `timestamp without time zone[]` — verifies the parser keeps the `[]` suffix after multi-word alias resolution. - `numeric[]` / `int[]` / `bool[]` from stringified primitives — verifies the array arms of `convert_vec_val` apply the same string-coercion the scalar arms do. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: fix indentation drift on otyp_inferred lines cargo fmt cleanup of leftover indentation where the perl-driven sweep that introduced the otyp_inferred field landed at the wrong column. No behaviour change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.7
|
7bdfc0ea06 |
fix: avoid named prepared statements in datatable/PG executor (#8988)
Always send queries as unnamed prepared statements (query_typed_raw) when arg types resolve via otyp_to_pg_type. This eliminates the intermittent "prepared statement \"sN\" does not exist" error reported on datatable scripts whose Postgres connection sits behind a transaction-mode pooler (PgBouncer/Supabase pooler/RDS Proxy), where prepare and execute can land on different backend connections. The previous code only used the unnamed-statement path when the parser detected at least one explicitly typed arg; datatable-generated SQL with bare $1/$2 (relying on inline ::cast hints) fell back to prepare + query_raw and accumulated named statements (s0, s1, ..., s882, ...) on the cached connection, which the pooler then dropped. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.5
|
c0eeea9c83 |
feat: support S3Object input args in native SQL scripts (#8954)
* feat: support S3Object input args in native SQL scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: review fixes from local-review Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * update parser --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.7
|
53badf1a8c |
fix: track dollar-quoted strings in SQL block splitter (#8891)
* fix: track dollar-quoted strings in SQL block splitter Queries like `CREATE FUNCTION ... AS $$ ... ; ... $$ LANGUAGE plpgsql;` were being shredded on every `;` inside the function body because the SQL splitter's state machine didn't recognize PostgreSQL dollar-quoted strings. Add an `InDollarQuote(tag)` state so `$$ ... $$` and `$tag$ ... $tag$` regions are treated as a single quoted span. Opt-in via a new `track_dollar_quotes` flag on `parse_sql_blocks`; enabled for PostgreSQL and DuckDB, disabled for MySQL/Oracle/BigQuery/ Snowflake which don't support the syntax. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: make windmill-parser-wasm a self-contained workspace The wasm parser crate is excluded from the backend workspace (its nightly-only `cargo-features = ["panic-immediate-abort"]` would break stable cargo on the whole workspace), but its manifest still used `.workspace = true` inheritance — which fails with "failed to find a workspace root" once the parent no longer considers it a member. Declare the crate as its own workspace by adding `[workspace]`, `[workspace.package]`, and `[workspace.dependencies]` tables. Mirror the relevant entries from the parent `backend/Cargo.toml` (same version specs, same path targets) so resolution stays byte-identical to what the parent would have produced. Also: - Teach `.github/change-versions.sh` (+ mac variant) to update this crate's own `Cargo.toml` version and bulk-bump the `windmill-*` entries in its `Cargo.lock` on each release. - Bump the frontend's pinned `windmill-parser-wasm-regex` to 1.688.0 to match the freshly-built package, and refresh `package-lock.json`. - Regenerate the wasm crate's `Cargo.lock` from scratch (first build under the new workspace re-resolves the full graph; target-gated deps from sibling crates like `windmill-parser-py-imports` are now recorded in the lockfile but not compiled when targeting wasm32). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.7
|
2d4fadb590 |
feat: add s3 stream progress logs to other DB executors (#8898)
Extract the MSSQL s3 ingest+upload logging pattern into a reusable `s3_stream_and_upload_with_logs` helper and apply it to the PostgreSQL, MySQL, OracleDB, BigQuery, and Snowflake executors. Each s3 streamed query now emits periodic progress lines, an ingest-done line, and an upload+transcode-done line to the job output, matching MSSQL. `convert_json_line_stream` now returns `BoxStream<'static, _>` so the output stream can be forwarded to `s3.upload` from inside the generic helper without lifetime gymnastics; the two existing callers already boxed the result, so behavior is unchanged. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.7
|
43a6b57581 |
perf: speed up mssql s3 ingest and add phase logs to job output (#8884)
* perf: speed up mssql s3 ingest and add phase logs to job output Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: drop mssql s3 progress interval to 10s for better visibility Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: drop transient tests that compared against removed code path Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Diego Imbert
|
06fe809ecc |
fix: DB Manager delete/update for timestamp and serial types (#8830)
* Fix time(stamp)(tz) comparisons in pg_executor * fix serial bug * UPDATE and DELETE use primary key only |
||
|
Diego Imbert
|
3d43d31aba |
fix: refresh custom instance user password if auth failed (#8787)
* Refresh custom instance user pwd if connection failed * No longer need to check on startup * nit: unneeded inner function * fix |
||
|
Diego ImbertandClaude Opus 4.6
|
3d02be98f7 |
fix: normalize multi-word pg types in build_parameters to fix float8 serialization (#8778)
Multi-word Postgres type names like "double precision" caused the SQL parser regex to fail (no spaces allowed in type group), falling back to otyp="text". When Postgres inferred float8 for the column, the text-typed null couldn't serialize, breaking DB Manager inserts/updates. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
aff95c33b2 |
fix: create pg connection for cloud-hosted jobs instead of panicking (#8710)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
1a0f580f3d | nit | ||
|
Ruben FiszelandClaude Opus 4.6
|
6d58d1a74d |
fix: pipeline DISCARD ALL with first query on cached pg connections (#8707)
* perf: pipeline DISCARD ALL with first query on cached pg connections Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: use RESET ALL instead of DISCARD ALL for lighter session reset Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add integration test for pg session reset on cached connections Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: release MutexGuard before caching so pg connection cache actually works The old code shadowed the MutexGuard variable without dropping it, so try_lock() in the post-query caching path always failed — connection caching was effectively dead code. Restructure to explicitly drop the guard before connecting. Also adds a CACHE_HITS counter and clear_pg_cache() helper so the integration test can verify the cached-connection path is exercised. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add single-worker session isolation test for SET ROLE + search_path Pushes 3 jobs into the queue before starting the worker so a single worker processes them all sequentially (matching production). Verifies SET ROLE and SET search_path do not leak between jobs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add RESET ROLE to session reset (RESET ALL does not undo SET ROLE) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use DISCARD ALL for full session reset and retry on stale connections - Switch from pipelined RESET ROLE; RESET ALL to eager DISCARD ALL when validating cached connections. This resets everything: role, GUCs, prepared statements, temp tables, advisory locks, LISTEN registrations. - DISCARD ALL also serves as a health check: if it fails, the stale connection is discarded and a fresh one is created transparently. - Extract new_pg_connection() helper to avoid duplicating the connect + spawn-connection-task logic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add 100-job single-worker cache stress test Runs 100 varied PG jobs (plain SELECTs, SET ROLE, SET search_path, multi-statement) through one worker. Verifies all succeed, 99 hit the cache, and no session state leaks between jobs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Diego Imbert
|
8581a3300d |
fix: Run typed pg queries in a single protocol conversation (#8679)
* Run typed pg queries in one protocol conversation * Update pg queries for db manager to use explicit type syntax * Unused import |
||
|
56253c04cb |
feat: IAM RDS auth for PostgreSQL worker resources (#8573)
* feat: add IAM RDS auth support for PostgreSQL worker resources Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: use Config builder for IAM RDS connections Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR review feedback for IAM RDS auth Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update ee-repo-ref to ebea6ef1e5bfcfc3f0151da9687dac6c61bbfab6 This commit updates the EE repository reference after PR #493 was merged in windmill-ee-private. Previous ee-repo-ref: 1228561a98c5195bb97a81d4a57ce2bb2ecfca79 New ee-repo-ref: ebea6ef1e5bfcfc3f0151da9687dac6c61bbfab6 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Diego ImbertandRuben Fiszel
|
762fd3d993 |
Fix python datatable client requiring explicit types (#8086)
* Support arg type decl in postgres * Python datatable client no longer requires explicit arg typing * compilation fix * Set correct type in statement exec * reset to main * Explicit pg arg types * remove code duplication * update parser js * FLOAT8 doesn't have space --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Ruben FiszelandClaude Opus 4.6
|
6bf544f507 |
refactor: extract object store into dedicated crate with filesystem backend (#7996)
* refactor: extract object store code into windmill-object-store crate with filesystem backend Consolidate all object_store-dependent code from windmill-common into a new windmill-object-store crate. Add a filesystem-backed object store implementation using LocalFileSystem for dev/testing without cloud credentials. Includes 30 comprehensive tests covering render_endpoint, lfs_to_object_store_resource, duckdb_connection_settings, error mapping, and filesystem-backed integration tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * all * all * fix: fix raw_app hardcoded path, add missing ObjectStoreResource import, and add tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: move S3ModeFormat to windmill-types, make windmill-parser-sql optional, restore debug logs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Diego Imbert
|
3affbb3321 |
feat: type-checked data tables v0 (#7381)
* data tables settings ui * install runed * zod 4 fixes * use new toJSONSchema * Migrate ducklake catalogs to more generic custom instance databases * fix compilation * Safety conversion for old duckdb ffi * data tables settings * ts client basis * inline run works * datatables work * Revert "datatables work" This reverts commit |
||
|
Diego Imbert
|
ca45937166 |
Refactor PgDatabase to share common code (#7386)
* Refactor PgDatabase to share common code * TokioPgConnection refactor |
||
|
Diego Imbert
|
9bbab3321e |
feat: Data tables (#7226)
* data tables settings ui * install runed * zod 4 fixes * use new toJSONSchema * Migrate ducklake catalogs to more generic custom instance databases * fix compilation * Safety conversion for old duckdb ffi * data tables settings * ts client basis * inline run works * datatables work * Revert "datatables work" This reverts commit |
||
|
Diego Imbert
|
adfb7e809a | fix column order set with s3 mode (#7289) | ||
|
Diego Imbert
|
2cae72c9db |
fix: fix parse_postgres_uri not decoding password (#7157)
* nit overscroll auto * decode password in parse_postgres_url * don't encode host in pg_executor * nit |
||
|
Diego Imbert
|
c079155e3e |
SQL result_collection flag (#7041)
* Return array of results duckdb * Migration script to add result_collection=legacy * migration script fixes * app_version_lite not necessary * New annotations macro that supports custom types * pass unit tests * nit style * result_collection almost works for postgres * fix last_statement * frontend suggesitons * fix column_order making columns disappear * added version check for duckdb FFI lib to avoid crashes when changing FFI interface on Windows agent workers * result_collection for duckdb * Correct legacy behavior in DuckDB * mysql result_collection * mssql collection_strategy * result_collection for oracle * snowflake result_collection * fix errors * mistake, .clone() causing deadlock * fix frontend errors on oracle / mssql * fix legacy behavior on mssql * add exception catch in migration * missing app_script update * change cache layout to avoid worker pulling old scripts |
||
|
hugocasa
|
b972eb9721 | feat: add support for contextual vars in SQL (#6791) | ||
|
Alexander Petric
|
1c9faf9d03 | fix: support pg jsonb array (#6788) | ||
|
Diego Imbert
|
686cf8dd69 |
Timestamp input in db manager (#6754)
* nullable by default * unrelated nits * native timezone input postgres * Fix timestamptz NaiveDateTime conversion is pg_executor * remove consolelog |
||
|
Diego Imbert
|
59cdb141c3 |
NULL Toggle in InsertRow drawer (#6729)
* NULL toggle in InsertRow * fix long type parsing in postgres * nits * graphite catch * lazy_static * support for time/timestamp/tz long forms in pg parser * graphite suggestion |
||
|
Diego Imbert
|
fdefd4be93 | feat: duckdb sql lang support (#5761) | ||
|
Ruben Fiszel
|
5b96bccedd |
feat: add aws oidc support for instance s3 storage (#5810)
* backend * iterate * all * all * all * iterate * revert * all * add tracing to get of authed client * all * all * lal * all * update * fix * push * all * all * revert * frontend * fix checks * avoid deadlock * safer * fix * fix |
||
|
Diego Imbert
|
b29c6e7636 |
fix: S3 sql mode returns S3Object (#5764)
* fix upload_s3_file not checking response * Return S3Object instead of path string |
||
|
Diego Imbert
|
c7886ea07a |
feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate * S3 works with new syntax * snowflake s3 streaming support * postgres s3 support * fix postgres stream format * mysql s3 streaming * mssql s3 streaming * new s3 mode syntax * optional folder param * rename folder to prefix * json_stream_arr_values * cargo toml rollback * convert_ndjson with datafusion * format conversion kinda works * Fixed not finishing the datafusion writer * support for pg and mssql * fix file ext * bigquery conversion and works with s3 streaming * fix s3 flag parser * snowflake s3 streaming support * factor out duplicate code * remove anyhow * Err case for parse s3 mode * Send error to mpsc * bigquery s3 streaming fix for huge queries * remove extra stuff * snowflake s3 streaming support * small regex mistake * cfg(not(feature = "parquet")) * fix CI (unused import) * error handling fix (graphite) |
||
|
Ruben Fiszel
|
a3c76fb10c | fix: improve error display of nativets exceptions | ||
|
Ruben Fiszel
|
63fa499015 | feat: agent workers v2 using http (#5588) | ||
|
Ruben Fiszel
|
3edca4bc91 |
perf: optimize number of queries needed for job run (#5504)
* optPerf * update sqlx * update sqlx |