Files
windmill/frontend/src/lib/components/assets/JobAssetsViewer.svelte
T
Diego Imbert c6c7f3415a Specific asset tables (#7323)
* data tables settings ui

* install runed

* zod 4 fixes

* use new toJSONSchema

* Migrate ducklake catalogs to more generic custom instance databases

* fix compilation

* Safety conversion for old duckdb ffi

* data tables settings

* ts client basis

* inline run works

* datatables work

* Revert "datatables work"

This reverts commit 6e1588d59e.

* datatables work (without leaking pg credentials)

* println

* separate sqlUtils.ts

* nit

* Separate custom instance db Select and Wizard components

* nit

* nit wording

* add tags to custom instance dbs

* error when trying to use ducklake as datatable or opposite

* show status in dropdown

* data table instance setup works

* sqk function for ducklake

* factorize logic

* fix temp reactivity

* Data table assetexplore

* Migrate S3 permissions to modal

* Revert "Migrate S3 permissions to modal"

This reverts commit 0631d03cb0.

* nit query -> fetch

* Custom instance setup new look

* run_language_executor separate fn

* run_inline param

* nit wording

* Better typed client

* Data tables display as assets in frontend

* asset db icon

* nit

* cleaner errors

* nit

* Fix sed calls in mac

* run_inline_script_preview in python client

* basic python datatable client

* datatable and datalake parser in python

* ducklake client python

* nit fix

* Fix migration producing NULL instead of {} when no custom databases

* merge conflict fail

* python ducklake client arg fix

* parse or infer sql types in ts client

* ts asset parser, detect datatable & ducklake R/W

* fix sql repl for other read ops than select

* export type SqlTemplateFunction

* rename list_custom_instance_pg_databases

* typecheck datatable and ducklake name in Typescript

* Fix typecheck datatable and ducklake in TS

* declare module overriding instead of extending

* infer_sql_type in python client

* SqlQuery object in python

* fix merge conflicts

* update const_format

* CI fix

* factor out to var_identifiers

* sqlx prepare

* unnecessary security (admin is required)

* clearer comment

* ee repo ref

* nit snake case

* claude step 1: detect var declarations

* move detect_sql_access_type to common mod

* claude step 2: detect when saved vars are queried

* Revert "claude step 2: detect when saved vars are queried"

This reverts commit 1e1f930568.

* Revert "claude step 1: detect var declarations"

This reverts commit f866f4819d.

* remove ducklake/datatable and default

* detect data table assigns in var_identifiers

* Python parser successfully infers R/W/RW from ducklake / datatable

* still register ducklake/datatable if not used as unknown R/W

* Go to settings button in Assets Dropdown on not found

* nit

* sqlx prepare fail

* manual fix, somehow sqlx prepare won't do it

* fix frontend ci

* ee repo ref

* ducklake_user doesnt exist in unit tests

* nit fix

* ui nit

* nit

* nit missing clone

* fork ducklakes and datatables

* fix surface hover bug

* stupid mistake

* better deeply reactive mutable derived

* Ducklake picker

* Editor bar data tables

* DuckDB supports datatables

* datatable in duckdb asset parser

* duckdb asset parser var_identifiers

* Revert "duckdb asset parser var_identifiers"

This reverts commit 88068b1a77.

* sqlx prepare

* Box pin in test_workflow_as_code to fix stack overflow

* stash

* sql asset parser parses most s3 literals

* nit

* Detect attach + handle returning RW

* detect assets used with dot notation

* detect implicit access with USE dl; syntax

* Add assets as unknown if var was never used

* Support default ducklake/datatable main in parser

* ignore asset parsing errors in frontend (avoid flow layout shift)

* super weird duplication (merge conflict ?)

* nits

* fix duckdb parser detecting too much as asset when RW ctx is unknown

* fix transparent assets btn

* missing arg

* nit styling

* asset parser specific table parsing

* fix resource specific table parsing

* More concise asset display in flows + better icons

* fix assets page filtering out resources with added table

* Fix frontend to support specific table assets

* Open DB Manager to specific table

* Specific table parser in Python and TS + unit tests

* Fix UPDATE setting access to None

* fix flow edge rendering on top of output picker

* python parser fix var override bug

* add ts test

* fix compilation

* sqlx prepare

* update parsers version

* fix missing schema key onDelete
2025-12-11 13:34:17 +00:00

104 lines
3.2 KiB
Svelte

<script lang="ts">
import { ResourceService, type Job } from '$lib/gen'
import { inferAssets } from '$lib/infer'
import { workspaceStore } from '$lib/stores'
import { usePromise } from '$lib/svelte5Utils.svelte'
import { pruneNullishArray, uniqueBy } from '$lib/utils'
import DbManagerDrawer from '../DBManagerDrawer.svelte'
import ResourceEditorDrawer from '../ResourceEditorDrawer.svelte'
import S3FilePicker from '../S3FilePicker.svelte'
import AssetButtons from './AssetButtons.svelte'
import {
formatAssetKind,
getFlowModuleAssets,
parseInputArgsAssets,
type AssetWithAccessType
} from './lib'
type Props = {
job: Job
}
let { job }: Props = $props()
async function extractAssets(job: Job): Promise<AssetWithAccessType[]> {
if (job.job_kind === 'flow') {
const additionalAssetsMap = {
// TODO : Transitive assets
}
return uniqueBy(
pruneNullishArray([
...(job.raw_flow?.modules.flatMap((m) => getFlowModuleAssets(m, additionalAssetsMap)) ??
[]),
...parseInputArgsAssets(job.args ?? {})
]),
(x) => x.kind + x.path
)
}
if (job.job_kind === 'script') {
let inferAssetsResult = await inferAssets(job.language!, job.raw_code ?? '')
let assets = inferAssetsResult.status === 'ok' ? inferAssetsResult.assets : []
return [...assets, ...parseInputArgsAssets(job.args ?? {})]
}
return []
}
let assets = usePromise(() => extractAssets(job), { loadInit: false })
$effect(() => {
job.id
$workspaceStore
assets.refresh()
})
let resourceDataCache: Record<string, string | undefined> = $state({})
$effect(() => {
for (const asset of assets.value ?? []) {
if (asset.kind == 'resource') {
let truncatedPath = asset.path.split('/').slice(0, 3).join('/')
if (truncatedPath in resourceDataCache) continue
resourceDataCache[truncatedPath] = undefined // avoid fetching multiple times because of async
ResourceService.getResource({ path: truncatedPath, workspace: $workspaceStore! })
.then((r) => (resourceDataCache[truncatedPath] = r.resource_type))
.catch((err) => console.error("Couldn't fetch resource", truncatedPath, err))
}
}
})
let s3FilePicker: S3FilePicker | undefined = $state()
let dbManagerDrawer: DbManagerDrawer | undefined = $state()
let resourceEditorDrawer: ResourceEditorDrawer | undefined = $state()
</script>
{#if assets.value && assets.value.length > 0}
<ul class="flex flex-col divide-y mt-1">
{#each assets.value ?? [] as asset}
<li class="flex justify-between py-3 leading-4 text-sm pl-4">
<div class="flex flex-col flex-1 truncate">
{asset.path}
<span class="text-2xs text-primary">
{formatAssetKind({
...asset,
...(asset.kind === 'resource'
? { metadata: { resource_type: resourceDataCache[asset.path] } }
: {})
})}
</span>
</div>
<AssetButtons
{asset}
{resourceDataCache}
{dbManagerDrawer}
{resourceEditorDrawer}
{s3FilePicker}
/>
</li>
{/each}
</ul>
{:else}
<div class="text-sm text-primary">No assets found</div>
{/if}
<S3FilePicker bind:this={s3FilePicker} readOnlyMode />
<DbManagerDrawer bind:this={dbManagerDrawer} />
<ResourceEditorDrawer bind:this={resourceEditorDrawer} />