Files
windmill/frontend/src/lib/components/ResourcePicker.svelte
T
3d4f4c6c38 feat: Fork datatables (#8339)
* export_datatable_schema

* Propose to fork the datatable on ws fork

* dump datatable

* Dockerfile

* Fix import_datatable_dump

* datatable schema fork works!

* Option to copy both schema and data

* Datatable fork behavior

* nit ui

* use psql instead

* remove fork_datatable route

* feat: add fork_pg_database and export_pg_schema routes with DB Manager UI

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: pluralize "schema" to "schemas" in DB Manager export/import UI

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add import mode select (schema only vs schema + data) to DB Manager import

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Select schema or schema+data when important database

* fix: prepend $res: prefix to resource paths in DB Manager import/export

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: dynamic import button label based on selected mode

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* nits

* feat: add warning alert when schema+data import mode is selected

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* nit hide on cloud hosted

* refactor: remove fork_behavior from datatable settings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: split CreateWorkspace into layout wrapper and CreateWorkspaceInner

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: instantiate CreateWorkspaceInner in globalForkModal

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* nit icons

* Data table fork UI

* feat: pass per-datatable fork behaviors from UI to backend during workspace fork

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix fork overwriting all datatables

* UI nits

* custom instance db refactor

* custom instance db wizard btn for all in dropdown

* nit

* Delete custom instance database button

* Disable forking for resource datatables

* Big import buttons when db empty

* Revert "Disable forking for resource datatables"

This reverts commit 9561cc8fd4.

* feat: add non_diffable flag to resource table

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add resource-type datatable fork with CREATE DATABASE

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: tag forked datatables with nonDiffable and forkedFrom

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: diff datatable and ducklake settings individually on workspace merge

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: skip non_diffable resources and datatables in workspace diff

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: default datatable fork behavior to keep_original

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: make grant permissions non-fatal in instance datatable fork

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: make datatable and ducklake diffs visible in workspace comparison

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: remove datatable fork logic from workspace fork route

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: correct ahead/behind logic for datatable and ducklake diffs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Revert "fix: correct ahead/behind logic for datatable and ducklake diffs"

This reverts commit 6b50884dc6.

* revert: remove datatable and ducklake settings diffing logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add datatable clone UI with step-by-step confirmation modal

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract datatable fork UI into ForkDatatableSection component

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit

* fix: run datatable cloning before workspace fork creation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit disable fork admins

* nit fix switching workspace prematurely

* fix: use source workspace for forkPgDatabase calls during fork

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: update forked workspace datatable settings after fork creation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add forked_from field to DataTable and set it for instance forks

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit onFinish

* fix: add forked_from to DataTableSettings OpenAPI schema

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: track datatable table DDL changes in workspace_diff

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Revert "feat: track datatable table DDL changes in workspace_diff"

This reverts commit 7526dd68b9.

* feat: add get_datatable_full_schema endpoint and snapshot schema on fork

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix duplicate migration key

* fix: set forked_from on datatable config for both instance and resource types

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nits

* feat: drop forked databases on workspace deletion with confirmation UI

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract drop_forked_datatable_databases from delete_workspace

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: cast pg char columns to text in FK schema query

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: show dbname instead of resource type in fork deletion modal

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ui nit

* refactor: extract drop_custom_instance_database into windmill-common

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add datatable schema diff section to merge UI

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* UI

* feat: add review drawer with YAML diff and SQL migration runner

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: use Monaco DiffEditor for YAML diff in review drawer

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit

* Revert "refactor: use Monaco DiffEditor for YAML diff in review drawer"

This reverts commit a86008ba4c.

* Revert "feat: add review drawer with YAML diff and SQL migration runner"

This reverts commit 0a0deb5ddb.

* feat: add review drawer with DiffEditor and SQL migration runner

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ui nits

* fix: show diff between forked_from schema and changed side

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: re-fetch target live schema after migration for correct baseline

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* revert

* nit auto next

* feat: add confirmation modal before deploying migration to parent

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: handle missing columns/foreignKeys in schema conversion

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nits

* refactor: use temp file on disk for pg_dump instead of in-memory string

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Don't replace postgres dbname

* fix: add validation to drop_custom_instance_database and use source db for CREATE/DROP

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: type DataTable.forked_from as DataTableForkedFrom struct

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: simplify fork_pg_database to take source + target_dbname

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* dead code

* feat: enforce schema_and_data admin-only and extract create_custom_instance_database

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: rename fork_pg_database to import_pg_database with source/target/override params

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit

* refactor: remove original_dbname/original_resource from forked_from, resolve from parent

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit

* fix: resolve forked dbname from fork workspace when dropping resource databases

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nits

* fix: always clean up global_settings even if database doesn't exist

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: check datatable resource_type from config instead of URL prefix

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: wrap PG default value expressions in braces to prevent CAST quoting

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Revert "fix: wrap PG default value expressions in braces to prevent CAST quoting"

This reverts commit 77f5a2c4e8.

* refactor: reuse columnDefToTableEditorValuesColumn for default value handling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: store raw API schema in forked_from to avoid double transformation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Revert "fix: store raw API schema in forked_from to avoid double transformation"

This reverts commit e326197a20.

* Revert "refactor: reuse columnDefToTableEditorValuesColumn for default value handling"

This reverts commit bd8f071d9f.

* fix: validate dbname with strict regex to prevent SQL injection

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix default value

* always validate dbname

* refactor: move get_datatable_full_schema structs and logic to query_builders.rs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: split import_pg_database into create_pg_database + import_pg_database

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract drop_forked_datatable_databases into its own route

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: require admin when using $res: resource paths in import_pg_database

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use UserDB for $res: resource access and restrict dbname creation

- resolve_pg_source_checked uses UserDB (row-level security) for $res: paths
- transform_json_unchecked is now pub(crate) to prevent misuse
- Non-superadmins can only create databases with wm_fork_ prefix
- datatable:// remains accessible to everyone

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: refuse to drop forked databases unless name starts with wm_fork_

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: remove resolve_pg_source, use resolve_pg_source_checked everywhere

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix permissions

* sqlx prepare

* compilation nits

* sqlx prepare

* sqlx prepare

* wrong route syntax

* fix: allow workspace owner to edit datatable config for fork setup

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Revert "fix: allow workspace owner to edit datatable config for fork setup"

This reverts commit ab683e637b.

* refactor: move datatable fork setup into create_workspace_fork backend

Instead of updating datatable settings from the frontend after fork
creation (which required admin/owner access), pass forked_datatables
info to create_workspace_fork and handle it atomically in the same
transaction. Removes applyPostForkDatatableUpdates from frontend.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: snapshot schema in backend during fork instead of frontend

The schema snapshot is now taken by the backend in apply_forked_datatable
via snapshot_datatable_schema, which connects to the parent workspace's
datatable and runs pg_get_full_schema. This removes the need for the
frontend to call getDatatableFullSchema and pass the schema through.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use get_resource_value_interpolated_internal for $res: to resolve $var: references

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit

* sqlx prepare

* fix: add permission check to drop_forked_datatable_databases, validate dbnames, restrict temp file perms

- drop_forked_datatable_databases: same permission as delete_workspace
  (fork owner or super admin)
- validate_dbname on target_dbname_override and ForkedDatatableInfo.new_dbname
- Enforce wm_fork_ prefix on forked datatable new_dbname
- DumpFile: set /tmp/windmill/ to 0700 and create files with 0600

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit CLI

* Rename to ws_specific

* sqlx prepare

* nit always validate dbname

* fix: include foreign keys in CREATE TABLE migration for added tables

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: detect nextval defaults and use SERIAL/BIGSERIAL types in CREATE TABLE

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Update frontend/src/lib/components/DBManagerDrawer.svelte

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/lib.rs

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/lib.rs

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

* fix: sort foreign keys by constraint name for deterministic schema output

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* sqlx prepare

* rename migration to update timestamp

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-04-07 21:03:06 +00:00

334 lines
8.6 KiB
Svelte

<script lang="ts">
import { ResourceService, WorkspaceService } from '$lib/gen'
import { workspaceStore } from '$lib/stores'
import { onMount, untrack } from 'svelte'
import AppConnect from './AppConnectDrawer.svelte'
import ResourceEditorDrawer from './ResourceEditorDrawer.svelte'
import { Button } from './common'
import { Loader2, Pen, Plus, RotateCw } from 'lucide-svelte'
import { sendUserToast } from '$lib/toast'
import Select from './select/Select.svelte'
import ExploreAssetButton, { assetCanBeExplored } from './ExploreAssetButton.svelte'
import DropdownV2 from './DropdownV2.svelte'
interface Props {
initialValue?: string | undefined
value?: string | undefined
valueType?: string | undefined
resourceType?: string | undefined
disabled?: boolean
disablePortal?: boolean
showSchemaExplorer?: boolean
selectFirst?: boolean
expressOAuthSetup?: boolean
defaultValues?: Record<string, any> | undefined
placeholder?: string | undefined
selectInputClass?: string
class?: string
onClear?: () => void
excludedValues?: string[]
datatableAsPgResource?: boolean
workspace?: string | undefined
disableChatOffset?: boolean
}
let {
initialValue = $bindable(undefined),
value = $bindable(undefined),
valueType = $bindable(undefined),
resourceType = undefined,
disabled = false,
disablePortal = false,
showSchemaExplorer = false,
selectFirst = false,
expressOAuthSetup = false,
defaultValues = undefined,
placeholder = undefined,
selectInputClass = '',
class: className = '',
onClear = undefined,
excludedValues = undefined,
datatableAsPgResource = false,
workspace = undefined,
disableChatOffset = false
}: Props = $props()
let effectiveWorkspace = $derived(workspace ?? $workspaceStore!)
if (initialValue && value == undefined) {
value = initialValue
}
onMount(() => {
setTimeout(() => {
if (Object.keys(defaultValues ?? {}).length > 0 && resourceType) {
resourceEditor?.initNew?.(resourceType, defaultValues)
}
}, 500)
})
// let valueSelect = $state(
// initialValue || value
// ? {
// value: value ?? initialValue,
// label: value ?? initialValue,
// type: valueType
// }
// : undefined
// )
$effect(() => {
if (value === undefined) {
if (initialValue) {
if (initialValue != value) {
value = initialValue
}
} else {
}
} else {
}
})
let collection = $state(value ? [{ value, label: value, type: valueType }] : [])
export async function askNewResource() {
appConnect?.open?.(resourceType)
}
export async function refreshResources() {
await loadResources(resourceType)
}
let loading = $state(true)
async function loadResources(resourceType: string | undefined) {
loading = true
try {
const resourceTypesToQuery =
resourceType === 'snowflake' ? ['snowflake', 'snowflake_oauth'] : [resourceType]
const resources = await Promise.all(
resourceTypesToQuery.map((rt) =>
ResourceService.listResource({
workspace: effectiveWorkspace,
resourceType: rt
})
)
)
const nc = resources
.flat()
.filter((x) => x.resource_type != 'state' && x.resource_type != 'cache')
.filter((x) => !excludedValues || !excludedValues.includes(x.path))
.map((x) => ({
value: x.path,
label: x.path,
type: x.resource_type
}))
if (datatableAsPgResource && resourceType === 'postgresql') {
try {
const datatables = (
await WorkspaceService.listDataTables({
workspace: effectiveWorkspace
})
).map((d) => d.name)
for (const dt of datatables) {
nc.push({
value: `datatable://${dt}`,
label: `datatable://${dt}`,
type: 'postgresql'
})
}
} catch (e) {
console.error('Failed to load data tables', e)
}
}
// TODO check if this is needed
if (!nc.find((x) => x.value == value) && (initialValue || value)) {
nc.push({ value: value ?? initialValue!, label: value ?? initialValue!, type: '' })
}
collection = nc
if (collection.length == 1 && selectFirst && (value == undefined || value == '')) {
value = collection[0].value
valueType = collection[0].type
}
} catch (e) {
sendUserToast('Failed to load resource types', true)
console.error(e)
}
loading = false
}
let previousResourceType = untrack(() => resourceType)
$effect(() => {
effectiveWorkspace && resourceType
untrack(() => {
if (previousResourceType != resourceType) {
previousResourceType = resourceType
value = undefined
}
})
untrack(() => loadResources(resourceType))
})
$effect(() => {
excludedValues
if (effectiveWorkspace && resourceType && !disabled) {
untrack(() => loadResources(resourceType))
}
})
let appConnect: AppConnect | undefined = $state()
let resourceEditor: ResourceEditorDrawer | undefined = $state()
let hovering = $state(false)
let isDatatableSelected = $derived(value?.startsWith('datatable://') ?? false)
</script>
<AppConnect
on:refresh={async (e) => {
await loadResources(resourceType)
value = e.detail
valueType = collection.find((x) => x?.value == value)?.type
}}
bind:this={appConnect}
{expressOAuthSetup}
{workspace}
{disableChatOffset}
/>
<ResourceEditorDrawer
bind:this={resourceEditor}
{workspace}
{disableChatOffset}
on:refresh={async (e) => {
await loadResources(resourceType)
if (e.detail) {
value = e.detail
valueType = collection.find((x) => x?.value == value)?.type
// valueSelect = { value: e.detail, label: e.detail, type: valueType ?? '' }
}
}}
/>
<!-- {JSON.stringify({ value, collection })} -->
<div class="flex flex-col w-full items-start {className}">
<div
class="flex flex-row w-full items-center relative"
role="group"
onmouseenter={() => (hovering = true)}
onmouseleave={() => (hovering = false)}
>
<Select
{disabled}
{disablePortal}
bind:value={
() => value,
(v) => {
value = v
valueType = collection.find((x) => x?.value == v)?.type
}
}
onClear={() => {
initialValue = undefined
value = undefined
valueType = undefined
onClear?.()
}}
items={collection}
clearable
class="text-clip grow min-w-0"
inputClass={selectInputClass}
placeholder={placeholder ?? `${resourceType ?? 'any'} resource`}
itemLabelWrapperClasses="flex-1"
id="resource-picker-select"
>
{#snippet endSnippet({ item, close })}
{#if !item.value?.startsWith('datatable://')}
<Button
{disabled}
variant="subtle"
size="xs2"
wrapperClasses="-mr-2 pl-1 -my-2"
btnClasses="hover:bg-surface-tertiary"
on:click={() => (resourceEditor?.initEdit?.(item.value ?? ''), close())}
startIcon={{ icon: Pen }}
iconOnly
/>
{/if}
{/snippet}
{#snippet bottomSnippet({ close })}
<div class="flex bg-surface border-t divide-x">
{#if resourceType?.includes(',')}
<DropdownV2
enableFlyTransition
items={resourceType?.split(',').map((rt) => ({
displayName: `${rt} resource`,
icon: Plus,
action: () => (appConnect?.open?.(rt), close())
})) ?? []}
>
{#snippet buttonReplacement()}
<Button
{disabled}
color="light"
variant="contained"
wrapperClasses="flex-1"
btnClasses="rounded-none mt-0.5"
size="sm"
startIcon={{ icon: Plus }}
>
Add a resource
</Button>
{/snippet}
</DropdownV2>
{:else}
<Button
{disabled}
color="light"
variant="contained"
wrapperClasses="flex-1"
btnClasses="rounded-none"
size="sm"
on:click={() => (appConnect?.open?.(resourceType), close())}
startIcon={{ icon: Plus }}
>
Add a {resourceType} resource
</Button>
{/if}
<Button
variant="contained"
color="light"
btnClasses="rounded-none"
size="sm"
on:click={() => {
loadResources(resourceType)
}}
startIcon={loading ? { icon: Loader2, classes: 'animate-spin' } : { icon: RotateCw }}
iconOnly
/>
</div>
{/snippet}
</Select>
{#if value && hovering && !isDatatableSelected}
<div class="absolute {disabled ? 'right-2' : 'right-10'} z-20">
<Button
variant="subtle"
size="xs2"
wrapperClasses="pl-1"
btnClasses="hover:bg-surface-tertiary"
on:click={() => resourceEditor?.initEdit?.(value ?? '')}
startIcon={{ icon: Pen }}
iconOnly
/>
</div>
{/if}
</div>
{#if showSchemaExplorer && value && assetCanBeExplored({ kind: 'resource', path: value }, { resource_type: resourceType })}
<ExploreAssetButton
class="mt-1"
_resourceMetadata={{ resource_type: resourceType }}
asset={{ kind: 'resource', path: value }}
/>
{/if}
</div>