Files
windmill/backend/windmill-worker/src/agent_workers.rs
T
Diego Imbert a45c9ca525 feat: Ducklake native support (#6268)
* upgrade duckdb

* basic ducklake works

* ducklake works with custom db catalogs

* fix: pwsh skip already installed modules outside of cache (#6037)

* improve query performance of user stats

* separate ducklake_catalog db

* ducklake settings

* DucklakeSettings frontend

* Ducklake ws settings saved in backend

* fetch ducklake catalog resource

* Ducklake works with configured s3 storage

* Ducklake as asset

* ducklake asset icon

* Fix duckdb array and object args not working properly (#6254)

* Fix bug with comments in duckdb

* Avoid multiple queries when doing ATTACH ducklake

* trunc sig no longer needed now that comments are trimmed

* cache DuckdbConnectionSettingsResponse

* duplicated code

* transform_attach_ducklake contributes to duckdb_connection_settings_cache

* eliminate the need for used_storages

* nit

* cleaner management of the bigquery credentials file

* DBManagerDrawer refactor to prepare for Ducklake

* get ducklake schema

* implement delete for ducklake

* load column metadata for ducklake

* Select query works for ducklake, basic db explorer works !

* duckdb count query

* Support all db ops for ducklake

* clean migrations

* SQL repl for Ducklake

* fix broken database studio

* nit

* assert function

* Ducklake in Editor Bar

* default ducklake syntax + allow extra args

* DucklakeCatalogWizard UI

* nit + remove extra $

* modal when databases do not exist

* cannot be windmill

* Ducklake works safely with instance database

* Avoid sending instance db credentials on network

* resource leak security

* remove fetch_attach_db_conn_str

* prevent instance pg password leak

* hide asset usage count when not available

* case unsensitivity duckdb

* warnings

* disable instance catalog

* use shorthand syntax when inserting with EditorBar

* Instance ducklake catalog is now safe to use

* use safer argon2 pwd

* update package json parsers

* update package json

* better msgs

* tooltips

* disable explore button until saved

* nit

* fix warnings

* better ducklake_user password management

* nit

* Sanitize passwords from errors in ducklake

* DisplayResult broken in job result

* remove superadmin requirement to check databases_exist

* duckdb_connection_settings_v2_inner

* Ducklake works on agent worker (finally)

* ci

* #[allow(dead_code)]

* fix openapi missing response

* Separate +Database button for DuckDB in EditorBar

* Fix dropdown in ducklake settings

* Attempt to fix migration race condition in CI

* update sqlx failing for some offline queries

* avoid temp password for ducklake_user

* nits

* ducklake settings nits

* update duckdb default script

* fix sql repl resetting text on refresh

* avoid pgcrypto extension

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-08-06 13:55:36 +00:00

68 lines
1.8 KiB
Rust

use reqwest::header::HeaderMap;
use uuid::Uuid;
use windmill_common::{
agent_workers::QueueInitJob, worker::HttpClient, workspaces::DucklakeWithConnData,
};
use windmill_queue::{JobAndPerms, JobCompleted};
pub async fn queue_init_job(client: &HttpClient, content: &str) -> anyhow::Result<Uuid> {
client
.post(
"/api/agent_workers/queue_init_job",
None,
&QueueInitJob { content: content.to_string() },
)
.await
.and_then(|x: String| Uuid::parse_str(&x).map_err(|e| anyhow::anyhow!(e)))
}
pub async fn queue_periodic_job(client: &HttpClient, content: &str) -> anyhow::Result<Uuid> {
client
.post(
"/api/agent_workers/queue_periodic_job",
None,
&QueueInitJob { content: content.to_string() },
)
.await
.and_then(|x: String| Uuid::parse_str(&x).map_err(|e| anyhow::anyhow!(e)))
}
pub async fn pull_job(
client: &HttpClient,
headers: Option<HeaderMap>,
body: Option<bool>,
) -> anyhow::Result<Option<JobAndPerms>> {
client
.post("/api/agent_workers/pull_job", headers, &body)
.await
}
pub async fn send_result(client: &HttpClient, jc: JobCompleted) -> anyhow::Result<String> {
client
.post(
&format!(
"/api/w/{}/agent_workers/send_result/{}",
jc.job.workspace_id, jc.job.id
),
None,
&jc,
)
.await
}
#[allow(dead_code)]
pub async fn get_ducklake_from_agent_http(
client: &HttpClient,
name: &str,
w_id: &str,
) -> anyhow::Result<DucklakeWithConnData> {
client
.get(&format!(
"/api/w/{}/agent_workers/get_ducklake/{}",
w_id, &name
))
.await
}
pub const UPDATE_PING_URL: &str = "/api/agent_workers/update_ping";