starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-10-26 00:03:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge origin/main into scim-handle-active-false

Browse Source
This commit is contained in:
hugocasa
2026-03-24 17:55:22 +01:00
parent c1a6a66969 a26a2e8092
commit 9da8bb52e2
236 changed files with 15615 additions and 1748 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/commit/SKILL.md
+59
View File
@@ -0,0 +1,59 @@
---
name: commit
description: Create a git commit with conventional commit format. MUST use anytime you want to commit changes.
---
# Git Commit Skill
Create a focused, single-line commit following conventional commit conventions.
## Instructions
1. **Analyze changes**: Run `git status` and `git diff` to understand what was modified
2. **Stage only modified files**: Add files individually by name. NEVER use `git add -A` or `git add .`
3. **Write commit message**: Follow the conventional commit format as a single line
## Conventional Commit Format
```
<type>: <description>
```
### Types
- `feat`: New feature or capability
- `fix`: Bug fix
- `refactor`: Code change that neither fixes a bug nor adds a feature
- `docs`: Documentation only changes
- `style`: Formatting, missing semicolons, etc (no code change)
- `test`: Adding or correcting tests
- `chore`: Maintenance tasks, dependency updates, etc
- `perf`: Performance improvement
### Rules
- Message MUST be a single line (no multi-line messages)
- Description should be lowercase, imperative mood ("add" not "added")
- No period at the end
- Keep under 72 characters total
### Examples
```
feat: add token usage tracking for AI providers
fix: resolve null pointer in job executor
refactor: extract common validation logic
docs: update API endpoint documentation
chore: upgrade sqlx to 0.7
```
## Execution Steps
1. Run `git status` to see all changes
2. Run `git diff` to understand the changes in detail
3. Run `git log --oneline -5` to see recent commit style
4. Stage ONLY the modified/relevant files: `git add <file1> <file2> ...`
5. Create the commit with conventional format:
```bash
git commit -m "<type>: <description>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
6. Run `git status` to verify the commit succeeded
.agents/skills/local-review/SKILL.md
+97
View File
@@ -0,0 +1,97 @@
---
name: local-review
description: Code review a pull request for bugs and CLAUDE.md compliance. MUST use when asked to review code.
---
# Local Code Review Skill
Review a pull request for real bugs and CLAUDE.md compliance violations. This review targets HIGH SIGNAL issues only.
## Review Philosophy
- **Only flag issues you are certain about.** If you are not sure an issue is real, do not flag it. False positives erode trust and waste reviewer time.
- Think like a senior engineer doing a final review — flag things that would cause incidents, not things that are merely imperfect.
## What to Flag
- Code that won't compile or parse (syntax errors, type errors, missing imports)
- Code that will definitely produce wrong results regardless of inputs
- Clear, unambiguous CLAUDE.md violations (quote the exact rule being violated)
- Security issues in introduced code (injection, auth bypass, data exposure)
- Incorrect logic that will fail in production
## What NOT to Flag
- Code style or quality concerns
- Potential issues that depend on specific inputs or runtime state
- Subjective suggestions or improvements
- Pre-existing issues not introduced by this PR
- Pedantic nitpicks a senior engineer wouldn't flag
- Issues a linter or type checker will catch
- General quality concerns unless explicitly prohibited in CLAUDE.md
- Issues silenced via lint ignore comments
## Execution Steps
1. **Determine the PR scope**:
- If an argument is provided, use it as the PR number or branch
- Otherwise, detect from the current branch vs main
- Run `gh pr view` if a PR exists, or use `git diff main...HEAD`
2. **Find relevant CLAUDE.md files**:
- Read the root `CLAUDE.md`
- Check for CLAUDE.md files in directories containing changed files
3. **Get the diff and metadata**:
- `gh pr diff` or `git diff main...HEAD` for the full diff
- `gh pr view` or `git log main..HEAD --oneline` for context
4. **Read changed files** where the diff alone is insufficient to understand context
5. **Review for**:
- CLAUDE.md compliance — check each rule against the changed code
- Bugs and logic errors — will this code work correctly?
- Security issues — injection, auth, data exposure in new code
6. **Self-validate each finding**: Before reporting, ask yourself:
- "Is this definitely a real issue, not a false positive?"
- "Would a senior engineer flag this in review?"
- If the answer to either is no, discard the finding
7. **Output findings** to the terminal (default) or post as PR comments (with `--comment` flag)
## Output Format
```
## Code review
Found N issues:
1. <description> (<reason: CLAUDE.md adherence | bug | security>)
<file_path:line_number>
2. <description> (<reason>)
<file_path:line_number>
```
If no issues are found:
```
## Code review
No issues found. Checked for bugs and CLAUDE.md compliance.
```
## Posting Comments (--comment flag)
If the user passes `--comment`, post findings as inline PR comments using:
```bash
gh pr review --comment --body "<summary>"
```
Or for inline comments on specific lines:
```bash
gh api repos/{owner}/{repo}/pulls/{pr}/reviews -f body="<summary>" -f event="COMMENT" -f comments="[...]"
```
.agents/skills/native-trigger/SKILL.md
+777
View File
@@ -0,0 +1,777 @@
# Skill: Adding Native Trigger Services
This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications.
## Architecture Overview
The native trigger system consists of:
1. **Database Layer** - PostgreSQL tables and enum types
2. **Backend Rust Implementation** - Core trait, handlers, and service modules in the `windmill-native-triggers` crate
3. **Frontend Svelte Components** - Configuration forms and UI components
### Key Files
| Component | Path |
|-----------|------|
| Core module with `External` trait | `backend/windmill-native-triggers/src/lib.rs` |
| Generic CRUD handlers | `backend/windmill-native-triggers/src/handler.rs` |
| Background sync logic | `backend/windmill-native-triggers/src/sync.rs` |
| OAuth/workspace integration | `backend/windmill-native-triggers/src/workspace_integrations.rs` |
| Re-export shim (windmill-api) | `backend/windmill-api/src/native_triggers/mod.rs` |
| TriggerKind enum | `backend/windmill-common/src/triggers.rs` |
| JobTriggerKind enum | `backend/windmill-common/src/jobs.rs` |
| Frontend service registry | `frontend/src/lib/components/triggers/native/utils.ts` |
| Frontend trigger utilities | `frontend/src/lib/components/triggers/utils.ts` |
| Trigger badges (icons + counts) | `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte` |
| Workspace integrations UI | `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte` |
| OAuth config form component | `frontend/src/lib/components/workspaceSettings/OAuthClientConfig.svelte` |
| OpenAPI spec | `backend/windmill-api/openapi.yaml` |
| Reference: Nextcloud module | `backend/windmill-native-triggers/src/nextcloud/` |
| Reference: Google module | `backend/windmill-native-triggers/src/google/` |
### Crate Structure
The native trigger code lives in the `windmill-native-triggers` crate (`backend/windmill-native-triggers/`). The `windmill-api` crate re-exports everything via a shim:
```rust
// backend/windmill-api/src/native_triggers/mod.rs
pub use windmill_native_triggers::*;
```
All new service modules go in `backend/windmill-native-triggers/src/`.
---
## Core Concepts
### The `External` Trait
Every native trigger service implements the `External` trait defined in `lib.rs`:
```rust
#[async_trait]
pub trait External: Send + Sync + 'static {
// Associated types:
type ServiceConfig: Debug + DeserializeOwned + Serialize + Send + Sync;
type TriggerData: Debug + Serialize + Send + Sync;
type OAuthData: DeserializeOwned + Serialize + Clone + Send + Sync;
type CreateResponse: DeserializeOwned + Send + Sync;
// Constants:
const SUPPORT_WEBHOOK: bool;
const SERVICE_NAME: ServiceName;
const DISPLAY_NAME: &'static str;
const TOKEN_ENDPOINT: &'static str;
const REFRESH_ENDPOINT: &'static str;
const AUTH_ENDPOINT: &'static str;
// Required methods:
async fn create(&self, w_id, oauth_data, webhook_token, data, db, tx) -> Result<Self::CreateResponse>;
async fn update(&self, w_id, oauth_data, external_id, webhook_token, data, db, tx) -> Result<serde_json::Value>;
async fn get(&self, w_id, oauth_data, external_id, db, tx) -> Result<Self::TriggerData>;
async fn delete(&self, w_id, oauth_data, external_id, db, tx) -> Result<()>;
async fn exists(&self, w_id, oauth_data, external_id, db, tx) -> Result<bool>;
async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors);
fn external_id_and_metadata_from_response(&self, resp) -> (String, Option<serde_json::Value>);
// Methods with defaults:
async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned>;
fn service_config_from_create_response(&self, data, resp) -> Option<serde_json::Value>;
fn additional_routes(&self) -> axum::Router;
async fn http_client_request<T, B>(&self, url, method, workspace_id, tx, db, headers, body) -> Result<T>;
}
```
Key design points:
- **`update()` returns `serde_json::Value`** - the resolved service_config to store. Each service is responsible for building the final config.
- **`maintain_triggers()`** - periodic background maintenance. Each service implements its own strategy (Nextcloud: reconcile with external state; Google: renew expiring channels).
- **No `list_all()` in the trait** - services that need it (Nextcloud) implement it privately; services that don't (Google) use different maintenance strategies.
- **No `get_external_id_from_trigger_data()` or `extract_service_config_from_trigger_data()`** - removed in favor of the `maintain_triggers` pattern.
### Create Lifecycle: Two Paths
The `create_native_trigger` handler in `handler.rs` supports two creation flows, controlled by `service_config_from_create_response()`:
**Path A: Short (Google pattern)** - `service_config_from_create_response()` returns `Some(config)`:
1. `create()` registers on external service
2. `external_id_and_metadata_from_response()` extracts the ID
3. `service_config_from_create_response()` builds the config directly from input data + response metadata
4. Stores trigger in DB -- done, no extra round-trip
Use this when the external_id is known before the create call (e.g., Google generates the channel_id as a UUID upfront and includes it in the webhook URL).
**Path B: Long (Nextcloud pattern)** - `service_config_from_create_response()` returns `None` (default):
1. `create()` registers on external service (webhook URL has no external_id yet)
2. `external_id_and_metadata_from_response()` extracts the ID
3. `update()` is called to fix the webhook URL with the now-known external_id
4. `update()` returns the resolved service_config
5. Stores trigger in DB
Use this when the external_id is assigned by the remote service and the webhook URL needs to be corrected after creation.
### OAuth Token Storage (Three-Table Pattern)
OAuth tokens are stored across three tables, NOT in `workspace_integrations.oauth_data` directly:
| Table | What's Stored |
|-------|---------------|
| `workspace_integrations` | `oauth_data` JSON with `base_url`, `client_id`, `client_secret`, `instance_shared` flag; `resource_path` pointing to the variable |
| `variable` | Encrypted `access_token` (at the path stored in `resource_path`), linked to `account` via `account` column |
| `account` | `refresh_token`, keyed by `workspace_id` + `client` (service name) + `is_workspace_integration = true` |
The `decrypt_oauth_data()` function in `lib.rs` assembles these into a unified struct:
```rust
pub struct OAuthConfig {
pub base_url: String,
pub access_token: String, // decrypted from variable
pub refresh_token: Option<String>, // from account table
pub client_id: String, // from oauth_data or instance settings
pub client_secret: String, // from oauth_data or instance settings
}
```
Instance-level sharing: when `oauth_data.instance_shared == true`, `client_id` and `client_secret` are read from global settings instead of workspace_integrations.
### URL Resolution
The `resolve_endpoint()` helper handles both absolute and relative OAuth URLs:
```rust
pub fn resolve_endpoint(base_url: &str, endpoint: &str) -> String {
if endpoint.starts_with("http://") || endpoint.starts_with("https://") {
endpoint.to_string() // Google: absolute URLs
} else {
format!("{}{}", base_url, endpoint) // Nextcloud: relative paths
}
}
```
### ServiceName Methods
`ServiceName` is the central registry enum. Each variant must implement these match arms:
| Method | Purpose |
|--------|---------|
| `as_str()` | Lowercase identifier (e.g., `"google"`) |
| `as_trigger_kind()` | Maps to `TriggerKind` enum |
| `as_job_trigger_kind()` | Maps to `JobTriggerKind` enum |
| `token_endpoint()` | OAuth token endpoint (relative or absolute) |
| `auth_endpoint()` | OAuth authorization endpoint |
| `oauth_scopes()` | Space-separated OAuth scopes |
| `resource_type()` | Resource type for token storage (e.g., `"gworkspace"`) |
| `extra_auth_params()` | Extra OAuth params (e.g., Google needs `access_type=offline`, `prompt=consent`) |
| `integration_service()` | Maps to the workspace integration service (usually `*self`) |
| `TryFrom<String>` | Parse from string |
| `Display` | Delegates to `as_str()` |
---
## Step-by-Step Implementation Guide
### Step 1: Database Migration
Create a new migration file: `backend/migrations/YYYYMMDDHHMMSS_newservice_trigger.up.sql`
```sql
-- Add the service to the native_trigger_service enum
ALTER TYPE native_trigger_service ADD VALUE IF NOT EXISTS 'newservice';
-- Add to TRIGGER_KIND enum (used for trigger tracking)
ALTER TYPE TRIGGER_KIND ADD VALUE IF NOT EXISTS 'newservice';
-- Add to job_trigger_kind enum (used for job tracking)
ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'newservice';
```
Also create the corresponding down migration.
### Step 2: Update windmill-common Enums
#### `backend/windmill-common/src/triggers.rs`
Add variant to `TriggerKind` enum, and update `to_key()` and `fmt()` implementations.
#### `backend/windmill-common/src/jobs.rs`
Add variant to `JobTriggerKind` enum and update the `Display` implementation.
### Step 3: Backend Service Module
Create a new directory: `backend/windmill-native-triggers/src/newservice/`
#### `mod.rs` - Type Definitions
```rust
use serde::{Deserialize, Serialize};
pub mod external;
// pub mod routes; // Only if you need additional service-specific routes
/// OAuth data deserialized from the three-table pattern.
/// The actual structure is built by decrypt_oauth_data() from variable + account + workspace_integrations.
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct NewServiceOAuthData {
pub base_url: String, // from workspace_integrations.oauth_data
pub access_token: String, // decrypted from variable table
pub refresh_token: Option<String>, // from account table
// Note: client_id and client_secret are in OAuthConfig, not here
// unless the service needs them at runtime for API calls
}
/// Configuration provided by user when creating/updating a trigger.
/// Stored as JSON in native_trigger.service_config.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct NewServiceConfig {
// Service-specific configuration fields
pub folder_path: String,
pub file_filter: Option<String>,
}
/// Data retrieved from the external service about a trigger.
/// Returned by the get() method and shown in the UI.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct NewServiceTriggerData {
pub folder_path: String,
pub file_filter: Option<String>,
// Fields that shouldn't affect service_config comparison should use #[serde(skip_serializing)]
}
/// Response from external service when creating a trigger/webhook.
#[derive(Debug, Deserialize)]
pub struct CreateTriggerResponse {
pub id: String,
}
/// Handler struct (stateless, used for routing)
#[derive(Copy, Clone)]
pub struct NewService;
```
#### `external.rs` - External Trait Implementation
```rust
use async_trait::async_trait;
use reqwest::Method;
use sqlx::PgConnection;
use std::collections::HashMap;
use windmill_common::{
error::{Error, Result},
BASE_URL, DB,
};
use crate::{
generate_webhook_service_url, External, NativeTrigger, NativeTriggerData, ServiceName,
sync::{SyncError, TriggerSyncInfo},
};
use super::{NewService, NewServiceConfig, NewServiceOAuthData, NewServiceTriggerData, CreateTriggerResponse};
#[async_trait]
impl External for NewService {
type ServiceConfig = NewServiceConfig;
type TriggerData = NewServiceTriggerData;
type OAuthData = NewServiceOAuthData;
type CreateResponse = CreateTriggerResponse;
const SERVICE_NAME: ServiceName = ServiceName::NewService;
const DISPLAY_NAME: &'static str = "New Service";
const SUPPORT_WEBHOOK: bool = true;
const TOKEN_ENDPOINT: &'static str = "/oauth/token";
const REFRESH_ENDPOINT: &'static str = "/oauth/token";
const AUTH_ENDPOINT: &'static str = "/oauth/authorize";
async fn create(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
webhook_token: &str,
data: &NativeTriggerData<Self::ServiceConfig>,
db: &DB,
tx: &mut PgConnection,
) -> Result<Self::CreateResponse> {
let base_url = &*BASE_URL.read().await;
// external_id is None during create (we get it from the response)
let webhook_url = generate_webhook_service_url(
base_url, w_id, &data.script_path, data.is_flow,
None, Self::SERVICE_NAME, webhook_token,
);
let url = format!("{}/api/webhooks/create", oauth_data.base_url);
let payload = serde_json::json!({
"callback_url": webhook_url,
"folder_path": data.service_config.folder_path,
});
let response: CreateTriggerResponse = self
.http_client_request(&url, Method::POST, w_id, tx, db, None, Some(&payload))
.await?;
Ok(response)
}
/// Update returns the resolved service_config as JSON.
/// For services using the update+get pattern, call self.get() and serialize.
async fn update(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
webhook_token: &str,
data: &NativeTriggerData<Self::ServiceConfig>,
db: &DB,
tx: &mut PgConnection,
) -> Result<serde_json::Value> {
let base_url = &*BASE_URL.read().await;
let webhook_url = generate_webhook_service_url(
base_url, w_id, &data.script_path, data.is_flow,
Some(external_id), Self::SERVICE_NAME, webhook_token,
);
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
let payload = serde_json::json!({
"callback_url": webhook_url,
"folder_path": data.service_config.folder_path,
});
let _: serde_json::Value = self
.http_client_request(&url, Method::PUT, w_id, tx, db, None, Some(&payload))
.await?;
// Fetch back the updated state to get the resolved config
let trigger_data = self.get(w_id, oauth_data, external_id, db, tx).await?;
serde_json::to_value(&trigger_data)
.map_err(|e| Error::InternalErr(format!("Failed to serialize trigger data: {}", e)))
}
async fn get(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
db: &DB,
tx: &mut PgConnection,
) -> Result<Self::TriggerData> {
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
self.http_client_request::<_, ()>(&url, Method::GET, w_id, tx, db, None, None).await
}
async fn delete(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
db: &DB,
tx: &mut PgConnection,
) -> Result<()> {
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
let _: serde_json::Value = self
.http_client_request::<_, ()>(&url, Method::DELETE, w_id, tx, db, None, None)
.await
.or_else(|e| match &e {
Error::InternalErr(msg) if msg.contains("404") => Ok(serde_json::Value::Null),
_ => Err(e),
})?;
Ok(())
}
async fn exists(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
db: &DB,
tx: &mut PgConnection,
) -> Result<bool> {
match self.get(w_id, oauth_data, external_id, db, tx).await {
Ok(_) => Ok(true),
Err(Error::NotFound(_)) => Ok(false),
Err(e) => Err(e),
}
}
/// Background maintenance. Choose the right pattern for your service:
/// - For services with queryable external state: use reconcile_with_external_state()
/// - For channel-based services with expiration: implement renewal logic
async fn maintain_triggers(
&self,
db: &DB,
workspace_id: &str,
triggers: &[NativeTrigger],
oauth_data: &Self::OAuthData,
synced: &mut Vec<TriggerSyncInfo>,
errors: &mut Vec<SyncError>,
) {
// Option A: Reconcile with external state (Nextcloud pattern)
// Fetch all triggers from external service and compare with DB
let external_triggers = match self.list_all(workspace_id, oauth_data, db).await {
Ok(triggers) => triggers,
Err(e) => {
errors.push(SyncError {
resource_path: format!("workspace:{}", workspace_id),
error_message: format!("Failed to list triggers: {}", e),
error_type: "api_error".to_string(),
});
return;
}
};
// Convert to (external_id, config_json) pairs
let external_pairs: Vec<(String, serde_json::Value)> = external_triggers
.into_iter()
.map(|t| (t.id.clone(), serde_json::to_value(&t).unwrap_or_default()))
.collect();
crate::sync::reconcile_with_external_state(
db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
).await;
}
fn external_id_and_metadata_from_response(
&self,
resp: &Self::CreateResponse,
) -> (String, Option<serde_json::Value>) {
(resp.id.clone(), None)
}
// service_config_from_create_response: NOT overridden (returns None).
// This means the handler uses the update+get pattern after create.
// Override and return Some(...) to skip the update+get cycle (Google pattern).
}
impl NewService {
/// Private helper to list all triggers from the external service.
async fn list_all(
&self,
w_id: &str,
oauth_data: &<Self as External>::OAuthData,
db: &DB,
) -> Result<Vec<<Self as External>::TriggerData>> {
// Implementation depends on the external service's API
todo!()
}
}
```
### Step 4: Update lib.rs Registry
In `backend/windmill-native-triggers/src/lib.rs`:
```rust
// Service modules - add new services here:
#[cfg(feature = "native_trigger")]
pub mod newservice; // <-- Add this
// ServiceName enum - add variant:
pub enum ServiceName {
Nextcloud,
Google,
NewService, // <-- Add this
}
// Then add match arms in ALL ServiceName methods:
// as_str(), as_trigger_kind(), as_job_trigger_kind(), token_endpoint(),
// auth_endpoint(), oauth_scopes(), resource_type(), extra_auth_params(),
// integration_service(), TryFrom<String>, Display
```
### Step 5: Update handler.rs Routes
In `backend/windmill-native-triggers/src/handler.rs`:
```rust
pub fn generate_native_trigger_routers() -> Router {
// ...
#[cfg(feature = "native_trigger")]
{
use crate::newservice::NewService;
return router
.nest("/nextcloud", service_routes(NextCloud))
.nest("/google", service_routes(Google))
.nest("/newservice", service_routes(NewService)); // <-- Add this
}
// ...
}
```
### Step 6: Update sync.rs
In `backend/windmill-native-triggers/src/sync.rs`:
```rust
pub async fn sync_all_triggers(db: &DB) -> Result<BackgroundSyncResult> {
// ...
#[cfg(feature = "native_trigger")]
{
use crate::newservice::NewService;
// ... existing service syncs ...
// New service sync
let (service_name, result) = sync_service_triggers(db, NewService).await;
total_synced += result.synced_triggers.len();
total_errors += result.errors.len();
service_results.insert(service_name, result);
}
// ...
}
```
### Step 7: Frontend Service Registry
In `frontend/src/lib/components/triggers/native/utils.ts`:
Add to `NATIVE_TRIGGER_SERVICES`, `getTriggerIconName()`, and `getServiceIcon()`.
### Step 8: Frontend Trigger Form Component
Create: `frontend/src/lib/components/triggers/native/services/newservice/NewServiceTriggerForm.svelte`
### Step 9: Frontend Icon Component
Create: `frontend/src/lib/components/icons/NewServiceIcon.svelte`
### Step 10: Update NativeTriggerEditor
Check `frontend/src/lib/components/triggers/native/NativeTriggerEditor.svelte` to ensure it dynamically loads form components based on service name.
### Step 11: Workspace Integration UI
Add your service to the `supportedServices` map in `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte`:
```typescript
const supportedServices: Record<string, ServiceConfig> = {
// ... existing services ...
newservice: {
name: 'newservice',
displayName: 'New Service',
description: 'Connect to New Service for triggers',
icon: NewServiceIcon,
docsUrl: 'https://www.windmill.dev/docs/integrations/newservice',
requiresBaseUrl: false, // false for cloud services, true for self-hosted
setupInstructions: [
'Step 1: Create an OAuth app on the service',
'Step 2: Configure the redirect URI shown below',
'Step 3: Enter the client credentials below'
]
}
}
```
### Step 12: Update `frontend/src/lib/components/triggers/utils.ts`
Update ALL of these maps/functions:
1. `triggerIconMap` - import and add icon
2. `triggerDisplayNamesMap` - add display name
3. `triggerTypeOrder` in `sortTriggers()` - add type
4. `getLightConfig()` - add case for your service
5. `getTriggerLabel()` - add case for your service
6. `jobTriggerKinds` - add to array
7. `countPropertyMap` - add count property
8. `triggerSaveFunctions` - add save function
### Step 13: Update TriggersBadge Component
In `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte`:
1. Import the icon
2. Add to `baseConfig` with `countKey` (the dynamic `availableNativeServices` loop does NOT set `countKey`)
3. Add to the `allTypes` array
### Step 14: Update TriggersWrapper.svelte
In `frontend/src/lib/components/triggers/TriggersWrapper.svelte`:
Add a `{:else if selectedTrigger.type === 'yourservice'}` case that renders `<NativeTriggersPanel service="yourservice" ...>` with the same props pattern as the existing native trigger cases (e.g., `nextcloud`).
### Step 15: Update AddTriggersButton.svelte
In `frontend/src/lib/components/triggers/AddTriggersButton.svelte`:
1. Add `yourserviceAvailable` state variable
2. Add `setYourserviceState()` async function using `isServiceAvailable('yourservice', $workspaceStore!)`
3. Call it at module level
4. Add a dropdown entry to `addTriggerItems` with `hidden: !yourserviceAvailable`
### Step 16: Update TriggersEditor.svelte Delete Handling
In `frontend/src/lib/components/triggers/TriggersEditor.svelte`:
Add your service to the `nativeTriggerServices` map in `deleteDeployedTrigger()`. Native triggers use `NativeTriggerService.deleteNativeTrigger({ workspace, serviceName, externalId })` instead of the standard `path`-based delete.
### Step 17: Update OpenAPI Spec and Regenerate Types
Add to `JobTriggerKind` enum in `backend/windmill-api/openapi.yaml`, then:
```bash
cd frontend && npm run generate-backend-client
```
---
## Special Patterns
### Unified Service with `trigger_type` (Google Pattern)
When a single service handles multiple trigger types (e.g., Google Drive + Calendar share OAuth and API patterns), use a single `ServiceName` variant with a discriminator field:
```rust
pub enum GoogleTriggerType { Drive, Calendar }
pub struct GoogleServiceConfig {
pub trigger_type: GoogleTriggerType,
// Drive-specific fields (only used when trigger_type = Drive)
pub resource_id: Option<String>,
pub resource_name: Option<String>,
// Calendar-specific fields (only used when trigger_type = Calendar)
pub calendar_id: Option<String>,
pub calendar_name: Option<String>,
// Metadata set after creation
pub google_resource_id: Option<String>,
pub expiration: Option<String>,
}
```
Branch in trait methods based on `trigger_type`. Frontend uses a `ToggleButtonGroup` to switch between types. This keeps the codebase simpler (one service, one OAuth flow, one set of routes).
See `backend/windmill-native-triggers/src/google/` for the reference implementation.
### Skipping update+get After Create (Google Pattern)
Override `service_config_from_create_response()` to return `Some(config)` when the external_id is known before the create call:
```rust
fn service_config_from_create_response(
&self,
data: &NativeTriggerData<Self::ServiceConfig>,
resp: &Self::CreateResponse,
) -> Option<serde_json::Value> {
// Clone input config, add metadata from response
let mut config = data.service_config.clone();
config.google_resource_id = Some(resp.resource_id.clone());
config.expiration = Some(resp.expiration.clone());
Some(serde_json::to_value(&config).unwrap())
}
```
### Services with Absolute OAuth Endpoints (Google)
Unlike self-hosted services where OAuth endpoints are relative paths appended to `base_url`, services like Google have absolute URLs:
```rust
// Nextcloud: relative paths
ServiceName::Nextcloud => "/apps/oauth2/api/v1/token",
// Google: absolute URLs
ServiceName::Google => "https://oauth2.googleapis.com/token",
```
The `resolve_endpoint()` function handles both. For services with absolute endpoints:
- `base_url` can be empty
- `requiresBaseUrl: false` in the frontend workspace integration config
- Add `extra_auth_params()` if needed (Google requires `access_type=offline` and `prompt=consent`)
### Channel-Based Push Notifications with Renewal (Google Pattern)
For services using expiring watch channels instead of persistent webhooks:
1. Store expiration in `service_config` (as part of `ServiceConfig`)
2. In `maintain_triggers()`, implement renewal logic instead of using `reconcile_with_external_state()`:
```rust
async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors) {
for trigger in triggers {
if should_renew_channel(trigger) {
self.renew_channel(db, trigger, oauth_data).await;
}
}
}
```
3. Renewal: best-effort stop old channel, create new one with same external_id, update service_config with new expiration
4. Google example: Drive channels expire in 24h (renew when <1h left), Calendar channels expire in 7 days (renew when <1 day left)
### reconcile_with_external_state (Nextcloud Pattern)
The reusable function in `sync.rs` compares external triggers with DB state:
- Triggers missing externally: sets error "Trigger no longer exists on external service"
- Triggers present externally: clears errors, updates service_config if it differs
Usage in `maintain_triggers()`:
```rust
let external_pairs: Vec<(String, serde_json::Value)> = /* fetch from external */;
crate::sync::reconcile_with_external_state(
db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
).await;
```
### Webhook Payload Processing
Override `prepare_webhook()` to parse service-specific payloads into script/flow args:
```rust
async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned> {
let mut args = HashMap::new();
args.insert("event_type".to_string(), Box::new(headers.get("x-event-type").cloned()) as _);
args.insert("payload".to_string(), Box::new(serde_json::from_str::<serde_json::Value>(&body)?) as _);
Ok(PushArgsOwned { extra: None, args })
}
```
Then register in `prepare_native_trigger_args()` in `lib.rs`:
```rust
pub async fn prepare_native_trigger_args(service_name, db, w_id, headers, body) -> Result<Option<PushArgsOwned>> {
match service_name {
ServiceName::Google => { /* ... */ Ok(Some(args)) }
ServiceName::NewService => { /* ... */ Ok(Some(args)) }
ServiceName::Nextcloud => Ok(None), // Uses default body parsing
}
}
```
### Instance-Level OAuth Credentials
When `workspace_integrations.oauth_data.instance_shared == true`, `decrypt_oauth_data()` reads `client_id` and `client_secret` from instance-level global settings instead of workspace-level. This allows admins to share OAuth app credentials across workspaces.
The frontend handles this via the `generate_instance_connect_url` endpoint in `workspace_integrations.rs`.
---
## Testing Checklist
- [ ] Database migration runs successfully
- [ ] `cargo check -p windmill-native-triggers --features native_trigger` passes
- [ ] `npx svelte-check --threshold error` passes (in frontend/)
- [ ] Service appears in workspace integrations list
- [ ] OAuth flow completes successfully
- [ ] Can create a new trigger
- [ ] Can view trigger details
- [ ] Can update trigger configuration
- [ ] Can delete trigger
- [ ] Webhook receives and processes payloads
- [ ] Background sync works correctly (reconciliation or channel renewal)
- [ ] Error handling works (expired tokens, service unavailable)
---
## Reference Implementations
### Nextcloud (Self-Hosted, Update+Get Pattern)
| File | Purpose |
|------|---------|
| `nextcloud/mod.rs` | Types: NextCloudOAuthData, NextcloudServiceConfig, NextCloudTriggerData |
| `nextcloud/external.rs` | External trait: uses update+get pattern, reconcile_with_external_state for sync |
| `nextcloud/routes.rs` | Additional route: `GET /events` |
Key patterns: relative OAuth endpoints, base_url required, list_all + reconcile for sync, update returns JSON from get().
### Google (Cloud, Unified Service, Short Create)
| File | Purpose |
|------|---------|
| `google/mod.rs` | Types: GoogleServiceConfig with trigger_type discriminator, GoogleTriggerType enum |
| `google/external.rs` | External trait: overrides service_config_from_create_response, channel renewal for sync |
| `google/routes.rs` | Additional routes: `GET /calendars`, `GET /drive/files`, `GET /drive/shared_drives` |
Key patterns: absolute OAuth endpoints, empty base_url, trigger_type for Drive/Calendar, expiring watch channels with renewal, service_config_from_create_response skips update+get, get() reconstructs data from stored service_config (no external "get channel" API).
.agents/skills/pr/SKILL.md
+109
View File
@@ -0,0 +1,109 @@
---
name: pr
description: Open a draft pull request on GitHub. MUST use when you want to create/open a PR.
---
# Pull Request Skill
Create a draft pull request with a clear title and explicit description of changes.
## Instructions
1. **Analyze branch changes**: Understand all commits since diverging from main
2. **Push to remote**: Ensure all commits are pushed
3. **Create draft PR**: Always open as draft for review before merging
## PR Title Format
Follow conventional commit format for the PR title:
```
<type>: <description>
```
### Types
- `feat`: New feature or capability
- `fix`: Bug fix
- `refactor`: Code restructuring
- `docs`: Documentation changes
- `chore`: Maintenance tasks
- `perf`: Performance improvements
### Title Rules
- Keep under 70 characters
- Use lowercase, imperative mood
- No period at the end
- If `*_ee.rs` files were modified, prefix with `[ee]`: `[ee] <type>: <description>`
## PR Body Format
The body MUST be explicit about what changed. Structure:
```markdown
## Summary
<Clear description of what this PR does and why>
## Changes
- <Specific change 1>
- <Specific change 2>
- <Specific change 3>
## Test plan
- [ ] <How to verify change 1>
- [ ] <How to verify change 2>
---
Generated with [Claude Code](https://claude.com/claude-code)
```
## Execution Steps
1. Run `git status` to check for uncommitted changes
2. Run `git log main..HEAD --oneline` to see all commits in this branch
3. Run `git diff main...HEAD` to see the full diff against main
4. Check if remote branch exists and is up to date:
```bash
git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
```
5. Push to remote if needed: `git push -u origin HEAD`
6. Create draft PR using gh CLI:
```bash
gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
## Summary
<description>
## Changes
- <change 1>
- <change 2>
## Test plan
- [ ] <test 1>
- [ ] <test 2>
---
Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"
```
7. Return the PR URL to the user
## EE Companion PR (when `*_ee.rs` files were modified)
The `*_ee.rs` files in the windmill repo are **symlinks** to `windmill-ee-private` — changes won't appear in `git diff` of the windmill repo. Instead, check the EE repo for uncommitted or unpushed changes.
Follow the full EE PR workflow in `docs/enterprise.md`. The key PR-specific details:
1. Find the EE repo/worktree: see "Finding the EE Repo" in `docs/enterprise.md`
2. Check for changes: `git -C <ee-path> status --short`
- If there are no changes in the EE repo, skip this entire section
3. Follow steps 1–5 from the "EE PR Workflow" in `docs/enterprise.md`
4. Create the companion PR (title does NOT get the `[ee]` prefix):
```bash
gh pr create --draft --repo windmill-labs/windmill-ee-private --title "<type>: <description>" --body "$(cat <<'EOF'
Companion PR for windmill-labs/windmill#<PR_NUMBER>
---
Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"
```
5. Commit `ee-repo-ref.txt` and push the updated windmill branch
.agents/skills/refine/SKILL.md
+38
View File
@@ -0,0 +1,38 @@
---
name: refine
description: End-of-session reflection. Reviews friction encountered during the session and proposes updates to docs/ to capture lessons learned.
---
# Refine Skill
Reflect on the current session and update documentation with lessons learned.
## Instructions
1. **Identify friction**: Review what happened in this session:
- Run `git diff main...HEAD --stat` to see what files were touched
- Think about: what was slow, what failed, what required multiple attempts, what information was missing or hard to find
2. **Read current docs**: Read the docs that were relevant to this session:
- `docs/validation.md`
- `docs/enterprise.md`
- `docs/autonomous-mode.md`
- Any skills that were invoked
3. **Propose updates**: For each piece of friction, decide if it warrants a doc update:
- **Missing knowledge**: Information you had to discover that should be documented
- **Wrong guidance**: Instructions that led you astray
- **Missing validation rule**: A check that should be in the validation matrix
- **New pattern**: A codebase pattern worth capturing for next time
4. **Apply updates**: Edit the relevant `docs/` files. Keep changes minimal and specific — add only what would have saved time this session.
5. **Report**: Summarize what was added/changed and why.
## Rules
- Only add knowledge confirmed by this session — no speculative additions
- Keep docs concise — add a line or two, not a paragraph
- If a whole new doc is needed, create it in `docs/` and add a pointer in `CLAUDE.md`
- Don't update skills unless a coding pattern was genuinely wrong
- Don't add things Claude already knows — only Windmill-specific knowledge
.agents/skills/rust-backend/SKILL.md
+107
View File
@@ -0,0 +1,107 @@
---
name: rust-backend
description: Rust coding guidelines for the Windmill backend. MUST use when writing or modifying Rust code in the backend directory.
---
# Windmill Rust Patterns
Apply these Windmill-specific patterns when writing Rust code in `backend/`.
## Error Handling
Use `Error` from `windmill_common::error`. Return `Result<T, Error>` or `JsonResult<T>`:
```rust
use windmill_common::error::{Error, Result};
pub async fn get_job(db: &DB, id: Uuid) -> Result<Job> {
sqlx::query_as!(Job, "SELECT id, workspace_id FROM v2_job WHERE id = $1", id)
.fetch_optional(db)
.await?
.ok_or_else(|| Error::NotFound("job not found".to_string()))?;
}
```
Never panic in library code. Reserve `.unwrap()` for compile-time guarantees.
## SQLx Patterns
**Never use `SELECT *`** — always list columns explicitly. Critical for backwards compatibility when workers lag behind API version:
```rust
// Correct
sqlx::query_as!(Job, "SELECT id, workspace_id, path FROM v2_job WHERE id = $1", id)
// Wrong — breaks when columns are added
sqlx::query_as!(Job, "SELECT * FROM v2_job WHERE id = $1", id)
```
Use batch operations to avoid N+1:
```rust
// Preferred — single query with IN clause
sqlx::query!("SELECT ... WHERE id = ANY($1)", &ids[..]).fetch_all(db).await?
```
Use transactions for multi-step operations. Parameterize all queries.
## JSON Handling
Prefer `Box<serde_json::value::RawValue>` over `serde_json::Value` when storing/passing JSON without inspection:
```rust
pub struct Job {
pub args: Option<Box<serde_json::value::RawValue>>,
}
```
Only use `serde_json::Value` when you need to inspect or modify the JSON.
## Serde Optimizations
```rust
#[derive(Serialize, Deserialize)]
pub struct Job {
#[serde(skip_serializing_if = "Option::is_none")]
pub parent_job: Option<Uuid>,
#[serde(skip_serializing_if = "Vec::is_empty")]
pub tags: Vec<String>,
#[serde(default)]
pub priority: i32,
}
```
## Async & Concurrency
Never block the async runtime. Use `spawn_blocking` for CPU-intensive work:
```rust
let result = tokio::task::spawn_blocking(move || expensive_computation(&data)).await?;
```
**Mutex selection**: Prefer `std::sync::Mutex` (or `parking_lot::Mutex`) for data protection. Only use `tokio::sync::Mutex` when holding locks across `.await` points.
Use `tokio::sync::mpsc` (bounded) for channels. Avoid `std::thread::sleep` in async contexts.
## Module Structure & Visibility
- Use `pub(crate)` instead of `pub` when possible
- Place new code in the appropriate crate based on functionality
- API endpoints go in `windmill-api/src/` organized by domain
- Shared functionality goes in `windmill-common/src/`
## Code Navigation
Always use rust-analyzer LSP for go-to-definition, find-references, and type info. Do not guess at module paths.
## Axum Handlers
Destructure extractors directly in function signatures:
```rust
async fn process_job(
Extension(db): Extension<DB>,
Path((workspace, job_id)): Path<(String, Uuid)>,
Query(pagination): Query<Pagination>,
) -> Result<Json<Job>> { ... }
```
.agents/skills/svelte-frontend/SKILL.md
+80
View File
@@ -0,0 +1,80 @@
---
name: svelte-frontend
description: Svelte coding guidelines for the Windmill frontend. MUST use when writing or modifying code in the frontend directory.
---
# Windmill Svelte Patterns
Apply these Windmill-specific patterns when writing Svelte code in `frontend/`. For general Svelte 5 syntax (runes, snippets, event handling), use the Svelte MCP server.
## Windmill UI Components (MUST use)
Always use Windmill's design-system components. Never use raw HTML elements.
### Buttons — `<Button>`
```svelte
<script>
import { Button } from '$lib/components/common'
import { ChevronLeft } from 'lucide-svelte'
</script>
<Button variant="default" onclick={handleClick}>Label</Button>
<Button startIcon={{ icon: ChevronLeft }} iconOnly onclick={prev} />
```
Props: `variant?: 'accent' | 'accent-secondary' | 'default' | 'subtle'`, `unifiedSize?: 'sm' | 'md' | 'lg'`, `startIcon?: { icon: SvelteComponent }`, `iconOnly?: boolean`, `disabled?: boolean`
### Text inputs — `<TextInput>`
```svelte
<script>
import { TextInput } from '$lib/components/common'
</script>
<TextInput bind:value={val} placeholder="Enter value" />
```
Props: `value?: string | number` (bindable), `placeholder?: string`, `disabled?: boolean`, `error?: string | boolean`, `size?: 'sm' | 'md' | 'lg'`
### Selects — `<Select>`
```svelte
<script>
import Select from '$lib/components/select/Select.svelte'
</script>
<Select items={[{ label: 'Jan', value: 1 }]} bind:value={selected} />
```
Props: `items?: Array<{ label?: string; value: any }>`, `value` (bindable), `placeholder?: string`, `clearable?: boolean`, `size?: 'sm' | 'md' | 'lg'`
### Icons — `lucide-svelte`
Never write inline SVGs. Import from `lucide-svelte`:
```svelte
<script>
import { ChevronLeft, X } from 'lucide-svelte'
</script>
<ChevronLeft size={16} />
```
## Form Components
Form components (TextInput, Toggle, Select, etc.) should use the unified size system when placed together.
## Styling
- Use Tailwind CSS for all styling — no custom CSS
- Use Windmill's theming classes for colors/surfaces (see `frontend/brand-guidelines.md`)
- Read component props JSDoc before using them
## Svelte MCP Server
Use the Svelte MCP tools when working on Svelte code:
1. **list-sections**: Call first to discover available docs
2. **get-documentation**: Fetch relevant sections based on use_cases
3. **svelte-autofixer**: MUST use on all Svelte code before finalizing — keep calling until no issues
4. **playground-link**: Only after user confirms and code was NOT written to project files
.github/workflows/claude.yml
+4 -4
View File
@@ -13,10 +13,10 @@ on:
jobs:
check-membership:
if: |
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '/ai')) ||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/ai')) ||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '/ai')) ||
(github.event_name == 'issues' && contains(github.event.issue.body, '/ai'))
(github.event_name == 'issue_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
(github.event_name == 'pull_request_review_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
(github.event_name == 'pull_request_review' && startsWith(github.event.review.body, '/ai') && !startsWith(github.event.review.body, '/ai-fast')) ||
(github.event_name == 'issues' && startsWith(github.event.issue.body, '/ai') && !startsWith(github.event.issue.body, '/ai-fast'))
uses: ./.github/workflows/check-org-membership.yml
secrets:
access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
CHANGELOG.md
+25
View File
@@ -1,5 +1,30 @@
# Changelog
## [1.663.0](https://github.com/windmill-labs/windmill/compare/v1.662.0...v1.663.0) (2026-03-24)
### Features
* add summary field for native triggers ([#8476](https://github.com/windmill-labs/windmill/issues/8476)) ([5089a45](https://github.com/windmill-labs/windmill/commit/5089a458819abbc6f241bc354bebb91520bd1a52))
* add typed request body to OpenAPI spec generation ([#8481](https://github.com/windmill-labs/windmill/issues/8481)) ([37ebaf4](https://github.com/windmill-labs/windmill/commit/37ebaf4d0ac342703498733f97778a552f979f6a))
* **cli:** better stale scripts detection [#3](https://github.com/windmill-labs/windmill/issues/3) ([#8480](https://github.com/windmill-labs/windmill/issues/8480)) ([9643006](https://github.com/windmill-labs/windmill/commit/9643006f1e90b991b334bb58caf62301bc26d09d))
* Debounce node ([#8324](https://github.com/windmill-labs/windmill/issues/8324)) ([5d1c54d](https://github.com/windmill-labs/windmill/commit/5d1c54d9b33d6ff6f2c98481a2740d1e7629cdfa))
* surface permissioned_as selector in trigger editor UI ([#8475](https://github.com/windmill-labs/windmill/issues/8475)) ([f035b53](https://github.com/windmill-labs/windmill/commit/f035b538bbd786445526339f88be8f33a3628105))
### Bug Fixes
* clean up stale dependency map entries for renamed scripts ([#8492](https://github.com/windmill-labs/windmill/issues/8492)) ([47c0c36](https://github.com/windmill-labs/windmill/commit/47c0c363f4fc1d9af7efd07ea172e32989ce50d2))
* **cli:** add Svelte 5 event delegation guidance and safe push to raw-app skill ([#8466](https://github.com/windmill-labs/windmill/issues/8466)) ([911df95](https://github.com/windmill-labs/windmill/commit/911df958e78d2dab9823dfa7d7e5c9824fc2d565))
* Fix worker panic when job_isolation changed to unshare at runtime ([#8490](https://github.com/windmill-labs/windmill/issues/8490)) ([cbe47c0](https://github.com/windmill-labs/windmill/commit/cbe47c0b6c22f79452d020777e481ee26970f25b))
* improve SQS retries ([3c8d351](https://github.com/windmill-labs/windmill/commit/3c8d351c9722a089133871019d27cf3bc3cdc159))
* Move database manager SQL queries to backend ([#8306](https://github.com/windmill-labs/windmill/issues/8306)) ([aa30fd2](https://github.com/windmill-labs/windmill/commit/aa30fd252dcf40233d191c43a6293fb9feabf010))
* prevent SQL injection in job query parameters ([#8494](https://github.com/windmill-labs/windmill/issues/8494)) ([54f5a19](https://github.com/windmill-labs/windmill/commit/54f5a19377e9df712e18f85f896e21b1776981ed))
* respect NO_COLOR env variable for stdout log output ([#8483](https://github.com/windmill-labs/windmill/issues/8483)) ([f329ee7](https://github.com/windmill-labs/windmill/commit/f329ee7aaefbae0ad344743c40825440a936bd30))
* show effective isolation level on workers page ([#8491](https://github.com/windmill-labs/windmill/issues/8491)) ([37886ed](https://github.com/windmill-labs/windmill/commit/37886edda1443293806a9b1b810196b72e076b12))
* skip debounce arg accumulation when batch table is empty (CE) ([#8485](https://github.com/windmill-labs/windmill/issues/8485)) ([010753c](https://github.com/windmill-labs/windmill/commit/010753c73ac85237af50acadf9c08567b1bc993c))
* stop_after_if with empty error_message prevents flow from stopping ([#8464](https://github.com/windmill-labs/windmill/issues/8464)) ([1503bf9](https://github.com/windmill-labs/windmill/commit/1503bf948e3340b8a6933d71885f8f2cb8dc1867))
## [1.662.0](https://github.com/windmill-labs/windmill/compare/v1.661.0...v1.662.0) (2026-03-20)
backend/.sqlx/query-6f9386dfcb4c201525722aee3caa25bf2f3a35d90f7354c7d3aef8a3538a03a7.json → backend/.sqlx/query-1048d1c95270ce1f36c02bce31a2bc8a88935c613bd213b7156299811377db8e.json
Generated
+4 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO native_trigger (\n external_id,\n workspace_id,\n service_name,\n script_path,\n is_flow,\n webhook_token_hash,\n service_config\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7\n )\n ON CONFLICT (external_id, workspace_id, service_name)\n DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, error = NULL, updated_at = NOW()\n ",
"query": "\n INSERT INTO native_trigger (\n external_id,\n workspace_id,\n service_name,\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n summary\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7, $8\n )\n ON CONFLICT (external_id, workspace_id, service_name)\n DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, summary = $8, error = NULL, updated_at = NOW()\n ",
"describe": {
"columns": [],
"parameters": {
@@ -21,10 +21,11 @@
"Varchar",
"Bool",
"Varchar",
"Jsonb"
"Jsonb",
"Varchar"
]
},
"nullable": []
},
"hash": "6f9386dfcb4c201525722aee3caa25bf2f3a35d90f7354c7d3aef8a3538a03a7"
"hash": "1048d1c95270ce1f36c02bce31a2bc8a88935c613bd213b7156299811377db8e"
}
backend/.sqlx/query-bac545933a627a62b7845d8aab80702443285e4d1d11e5a0f4cd2a3d4add51bb.json → backend/.sqlx/query-15014ce696cf2af4f719a537a4e3ca5b322cc130a35a91f8b8854f5ebdf25ad2.json
Generated
+9 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND external_id = $3\n ",
"query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at,\n summary\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND external_id = $3\n ",
"describe": {
"columns": [
{
@@ -62,6 +62,11 @@
"ordinal": 9,
"name": "updated_at",
"type_info": "Timestamptz"
},
{
"ordinal": 10,
"name": "summary",
"type_info": "Varchar"
}
],
"parameters": {
@@ -91,8 +96,9 @@
true,
true,
false,
false
false,
true
]
},
"hash": "bac545933a627a62b7845d8aab80702443285e4d1d11e5a0f4cd2a3d4add51bb"
"hash": "15014ce696cf2af4f719a537a4e3ca5b322cc130a35a91f8b8854f5ebdf25ad2"
}
backend/.sqlx/query-9360d00990822f153ff09c7905ae3180f07d02f38ac12d07a5664d93f160e7ee.json → backend/.sqlx/query-1cb21a66ffc89ebe53fd8f58690eec4cf11cb4b7816738202b474e8d3ffef427.json
Generated
+21 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n route_path,\n http_method AS \"http_method: _\",\n request_type AS \"request_type: _\",\n workspaced_route,\n summary,\n description,\n authentication_method AS \"authentication_method: _\",\n authentication_resource_path\n FROM\n http_trigger\n WHERE\n path ~ ANY($1) AND\n route_path ~ ANY($2) AND\n workspace_id = $3\n ",
"query": "\n SELECT\n route_path,\n http_method AS \"http_method: _\",\n request_type AS \"request_type: _\",\n workspaced_route,\n summary,\n description,\n authentication_method AS \"authentication_method: _\",\n authentication_resource_path,\n script_path,\n is_flow,\n wrap_body\n FROM\n http_trigger\n WHERE\n path ~ ANY($1) AND\n route_path ~ ANY($2) AND\n workspace_id = $3\n ",
"describe": {
"columns": [
{
@@ -80,6 +80,21 @@
"ordinal": 7,
"name": "authentication_resource_path",
"type_info": "Varchar"
},
{
"ordinal": 8,
"name": "script_path",
"type_info": "Varchar"
},
{
"ordinal": 9,
"name": "is_flow",
"type_info": "Bool"
},
{
"ordinal": 10,
"name": "wrap_body",
"type_info": "Bool"
}
],
"parameters": {
@@ -97,8 +112,11 @@
true,
true,
false,
true
true,
false,
false,
false
]
},
"hash": "9360d00990822f153ff09c7905ae3180f07d02f38ac12d07a5664d93f160e7ee"
"hash": "1cb21a66ffc89ebe53fd8f58690eec4cf11cb4b7816738202b474e8d3ffef427"
}
backend/.sqlx/query-25ac66a1022c41267df199a95f532b0f778c25fbe0f7a7f9734c1f7e536ed6ce.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "25ac66a1022c41267df199a95f532b0f778c25fbe0f7a7f9734c1f7e536ed6ce"
}
backend/.sqlx/query-2d523cd0d5b7107b15846b885fa40af492d4c8a8871cef972980150f319fe6ff.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO raw_script_temp (workspace_id, hash, content, created_at)\n VALUES ($1, $2, $3, NOW())\n ON CONFLICT (workspace_id, hash) DO UPDATE SET created_at = NOW()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Bpchar",
"Text"
]
},
"nullable": []
},
"hash": "2d523cd0d5b7107b15846b885fa40af492d4c8a8871cef972980150f319fe6ff"
}
backend/.sqlx/query-32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9.json
Generated
+28
View File
@@ -0,0 +1,28 @@
{
"db_name": "PostgreSQL",
"query": "SELECT parent_job, flow_step_id FROM v2_job WHERE id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "flow_step_id",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
true,
true
]
},
"hash": "32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9"
}
backend/.sqlx/query-372ec62fad93831b00d44f6c52a148b6bfa6008e49b2a80bec06c76d9432ec77.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n f.schema AS \"schema: serde_json::Value\",\n fv.value->>'preprocessor_module' IS NOT NULL AS \"has_preprocessor: bool\"\n FROM flow f\n LEFT JOIN flow_version fv ON fv.id = f.versions[array_length(f.versions, 1)]\n AND fv.workspace_id = f.workspace_id\n WHERE f.path = $1 AND f.workspace_id = $2 AND NOT f.archived",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "schema: serde_json::Value",
"type_info": "Json"
},
{
"ordinal": 1,
"name": "has_preprocessor: bool",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
true,
null
]
},
"hash": "372ec62fad93831b00d44f6c52a148b6bfa6008e49b2a80bec06c76d9432ec77"
}
backend/.sqlx/query-4461fe84370e7f07b4423ea5e71b913dd6ee9c655f9ec7087cc0fec9b9c3099a.json
Generated
+27
View File
@@ -0,0 +1,27 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH step_index AS (\n SELECT idx::text AS idx\n FROM v2_job_status,\n jsonb_array_elements(flow_status->'modules') WITH ORDINALITY arr(elem, idx)\n WHERE id = $1\n AND elem->>'id' = $5\n LIMIT 1\n ), completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n $3::text::jsonb,\n CASE WHEN si.idx IS NOT NULL\n THEN jsonb_set(\n s.flow_status,\n ARRAY['modules', (si.idx::int - 1)::text],\n $6::jsonb\n )\n ELSE s.flow_status\n END,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n LEFT JOIN step_index si ON true\n WHERE q.id = $1\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id = $1\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n VALUES ($4, $1, $2)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "x",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Text",
"Text",
"Text",
"Jsonb"
]
},
"nullable": [
null
]
},
"hash": "4461fe84370e7f07b4423ea5e71b913dd6ee9c655f9ec7087cc0fec9b9c3099a"
}
backend/.sqlx/query-4538bea4159677d8e653159d7d01649cae08e6ffef668a7cbac49b312bf30766.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO v2_job (id, kind, tag, created_by, permissioned_as, permissioned_as_email, workspace_id, parent_job)\n VALUES ($1, 'noop', 'deno', 'test-user', 'u/test-user', 'test@windmill.dev', $2, $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Uuid"
]
},
"nullable": []
},
"hash": "4538bea4159677d8e653159d7d01649cae08e6ffef668a7cbac49b312bf30766"
}
backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json
Generated
+35
View File
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "first_started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "job_id_to_debounce",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618"
}
backend/.sqlx/query-33367c42e87e78ae987c0966dc4d445c5eff75b2e2843ffd7a46b03cbaea9ae8.json → backend/.sqlx/query-4615b37cb848f9589622426d291c721e532b230c527deb701e24605c7027e38b.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT \n path,\n summary,\n description\n FROM\n flow\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"query": "SELECT\n path,\n summary,\n description\n FROM\n flow\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"describe": {
"columns": [
{
@@ -31,5 +31,5 @@
false
]
},
"hash": "33367c42e87e78ae987c0966dc4d445c5eff75b2e2843ffd7a46b03cbaea9ae8"
"hash": "4615b37cb848f9589622426d291c721e532b230c527deb701e24605c7027e38b"
}
backend/.sqlx/query-6009cf60c43608bb1c7924dbd0a9cdc01986d787eb24f0fffe4550d939851e22.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT debounced_times FROM debounce_key WHERE key = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false
]
},
"hash": "6009cf60c43608bb1c7924dbd0a9cdc01986d787eb24f0fffe4550d939851e22"
}
backend/.sqlx/query-dc36b46b9eb80cb7c92fa72519d117eda99a6f482a073ccd36a6431ef689a3fd.json → backend/.sqlx/query-69b44efb0144fececccafc8a77d040649bb17e239e5afbc27b915efc4c95d54e.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT \n path,\n summary,\n description\n FROM\n script\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"query": "SELECT\n path,\n summary,\n description\n FROM\n script\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"describe": {
"columns": [
{
@@ -31,5 +31,5 @@
false
]
},
"hash": "dc36b46b9eb80cb7c92fa72519d117eda99a6f482a073ccd36a6431ef689a3fd"
"hash": "69b44efb0144fececccafc8a77d040649bb17e239e5afbc27b915efc4c95d54e"
}
backend/.sqlx/query-1a69ef11a3f361f105c2a8af7b7fa182f3953150ade1756259b31a50e9308fce.json → backend/.sqlx/query-6dafcc89668fb0e5740f23264b515b1724c36031da39d53ae6c329a479bdf8aa.json
Generated
+9 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND script_path = $3\n AND is_flow = $4\n LIMIT 1\n ",
"query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at,\n summary\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND script_path = $3\n AND is_flow = $4\n LIMIT 1\n ",
"describe": {
"columns": [
{
@@ -62,6 +62,11 @@
"ordinal": 9,
"name": "updated_at",
"type_info": "Timestamptz"
},
{
"ordinal": 10,
"name": "summary",
"type_info": "Varchar"
}
],
"parameters": {
@@ -92,8 +97,9 @@
true,
true,
false,
false
false,
true
]
},
"hash": "1a69ef11a3f361f105c2a8af7b7fa182f3953150ade1756259b31a50e9308fce"
"hash": "6dafcc89668fb0e5740f23264b515b1724c36031da39d53ae6c329a479bdf8aa"
}
backend/.sqlx/query-79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073.json
Generated
+35
View File
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "first_started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "job_id_to_debounce",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073"
}
backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json
Generated
+35
View File
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "first_started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "job_id_to_debounce",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b"
}
backend/.sqlx/query-88ec0ddcc86fb67089b551ccbce7b932800e33cd462a651f7e6716929ee9b6f2.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT content FROM raw_script_temp WHERE hash = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "content",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Bpchar"
]
},
"nullable": [
false
]
},
"hash": "88ec0ddcc86fb67089b551ccbce7b932800e33cd462a651f7e6716929ee9b6f2"
}
backend/.sqlx/query-909a095d679ee7b8266fc93fe0c598e4b6583daa2b0d7c593fc72ff9b8d1d06b.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n schema AS \"schema: serde_json::Value\",\n has_preprocessor\n FROM script\n WHERE path = $1 AND workspace_id = $2\n AND NOT archived AND NOT deleted\n ORDER BY created_at DESC\n LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "schema: serde_json::Value",
"type_info": "Json"
},
{
"ordinal": 1,
"name": "has_preprocessor",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
true,
true
]
},
"hash": "909a095d679ee7b8266fc93fe0c598e4b6583daa2b0d7c593fc72ff9b8d1d06b"
}
backend/.sqlx/query-950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb.json
Generated
+46
View File
@@ -0,0 +1,46 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH RECURSIVE chain AS (\n SELECT\n j.id,\n j.parent_job,\n j.flow_step_id,\n 1 AS depth\n FROM v2_job j\n WHERE j.id = $1\n UNION ALL\n SELECT\n pj.id,\n pj.parent_job,\n pj.flow_step_id,\n c.depth + 1\n FROM chain c\n JOIN v2_job pj ON pj.id = c.parent_job\n WHERE c.parent_job IS NOT NULL\n )\n SELECT\n c.id,\n c.parent_job,\n c.flow_step_id,\n EXISTS(SELECT 1 FROM v2_job_queue q WHERE q.id = c.parent_job) AS \"parent_in_queue!\",\n EXISTS(\n SELECT 1 FROM v2_job sib\n WHERE sib.parent_job = c.parent_job\n AND sib.id != c.id\n AND sib.id IN (SELECT sq.id FROM v2_job_queue sq)\n ) AS \"has_other_active_siblings!\"\n FROM chain c\n WHERE c.depth >= 1\n ORDER BY c.depth ASC\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 2,
"name": "flow_step_id",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "parent_in_queue!",
"type_info": "Bool"
},
{
"ordinal": 4,
"name": "has_other_active_siblings!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
null,
null,
null,
null,
null
]
},
"hash": "950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb"
}
backend/.sqlx/query-c7cae4cf872fce0a989cf89aa35929218a9d459ee1c2b36a28b110e9741ab623.json → backend/.sqlx/query-96f6163a164b9ffb4ec52372d7fea158db101f54f0908551b5a4f5e6655e122b.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT content FROM script WHERE path = $1 AND workspace_id = $2\n AND archived = false ORDER BY created_at DESC LIMIT 1\n ",
"query": "\n SELECT content FROM script WHERE path = $1 AND workspace_id = $2\n AND archived = false ORDER BY created_at DESC LIMIT 1\n ",
"describe": {
"columns": [
{
@@ -19,5 +19,5 @@
false
]
},
"hash": "c7cae4cf872fce0a989cf89aa35929218a9d459ee1c2b36a28b110e9741ab623"
"hash": "96f6163a164b9ffb4ec52372d7fea158db101f54f0908551b5a4f5e6655e122b"
}
backend/.sqlx/query-d5661c7557cf3a8dee7cf799cd364d21d38edb827d2c08b0ca7d72311b78d574.json → backend/.sqlx/query-a32d7ba43745226fd65328475731526e0b20ea6eeafeb937eb01cdc2cdfcb859.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false ORDER BY created_at DESC LIMIT 1",
"query": "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false AND archived = false ORDER BY created_at DESC LIMIT 1",
"describe": {
"columns": [
{
@@ -19,5 +19,5 @@
false
]
},
"hash": "d5661c7557cf3a8dee7cf799cd364d21d38edb827d2c08b0ca7d72311b78d574"
"hash": "a32d7ba43745226fd65328475731526e0b20ea6eeafeb937eb01cdc2cdfcb859"
}
backend/.sqlx/query-a9c2019c7bafb172dfb353870238c02673113b806bc3c7a71354c34255ecd31c.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT scheduled_for FROM v2_job_queue WHERE id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "scheduled_for",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
false
]
},
"hash": "a9c2019c7bafb172dfb353870238c02673113b806bc3c7a71354c34255ecd31c"
}
backend/.sqlx/query-a115d8ea786907561afdbbc07d11dc715d80b00c0e79b61b0057a3ae3886a85e.json → backend/.sqlx/query-b0775af41a9b54cce040bf37cae770e63dab12a1383a0855d105c061e4e4ca48.json
Generated
+9 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n nt.external_id,\n nt.workspace_id,\n nt.service_name AS \"service_name!: ServiceName\",\n nt.script_path,\n nt.is_flow,\n nt.webhook_token_hash,\n nt.service_config,\n nt.error,\n nt.created_at,\n nt.updated_at\n FROM\n native_trigger nt\n WHERE\n nt.workspace_id = $1 AND\n nt.service_name = $2 AND\n ($5::text IS NULL OR nt.script_path = $5) AND\n ($6::bool IS NULL OR nt.is_flow = $6) AND\n (\n (nt.is_flow = false AND EXISTS (\n SELECT 1 FROM script s\n WHERE s.workspace_id = nt.workspace_id\n AND s.path = nt.script_path\n ))\n OR\n (nt.is_flow = true AND EXISTS (\n SELECT 1 FROM flow f\n WHERE f.workspace_id = nt.workspace_id\n AND f.path = nt.script_path\n ))\n )\n LIMIT $3\n OFFSET $4\n ",
"query": "\n SELECT\n nt.external_id,\n nt.workspace_id,\n nt.service_name AS \"service_name!: ServiceName\",\n nt.script_path,\n nt.is_flow,\n nt.webhook_token_hash,\n nt.service_config,\n nt.error,\n nt.created_at,\n nt.updated_at,\n nt.summary\n FROM\n native_trigger nt\n WHERE\n nt.workspace_id = $1 AND\n nt.service_name = $2 AND\n ($5::text IS NULL OR nt.script_path = $5) AND\n ($6::bool IS NULL OR nt.is_flow = $6) AND\n (\n (nt.is_flow = false AND EXISTS (\n SELECT 1 FROM script s\n WHERE s.workspace_id = nt.workspace_id\n AND s.path = nt.script_path\n ))\n OR\n (nt.is_flow = true AND EXISTS (\n SELECT 1 FROM flow f\n WHERE f.workspace_id = nt.workspace_id\n AND f.path = nt.script_path\n ))\n )\n LIMIT $3\n OFFSET $4\n ",
"describe": {
"columns": [
{
@@ -62,6 +62,11 @@
"ordinal": 9,
"name": "updated_at",
"type_info": "Timestamptz"
},
{
"ordinal": 10,
"name": "summary",
"type_info": "Varchar"
}
],
"parameters": {
@@ -94,8 +99,9 @@
true,
true,
false,
false
false,
true
]
},
"hash": "a115d8ea786907561afdbbc07d11dc715d80b00c0e79b61b0057a3ae3886a85e"
"hash": "b0775af41a9b54cce040bf37cae770e63dab12a1383a0855d105c061e4e4ca48"
}
backend/.sqlx/query-b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status SET flow_status = (\n SELECT jsonb_set(\n flow_status,\n ARRAY['modules', (idx - 1)::text],\n $2::jsonb\n )\n FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n LIMIT 1\n ) WHERE id = $1 AND (\n SELECT COUNT(*) FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n ) > 0",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Jsonb",
"Text"
]
},
"nullable": []
},
"hash": "b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c"
}
backend/.sqlx/query-40a8bf6a5a42c275d73221bc5f386f2e18cb911352551d0a34bf1933e558674e.json → backend/.sqlx/query-bf224f6441c36187f1402f9f01bfe15bb9edfa1dc9052f8a829e486b7334d708.json
Generated
+4 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n UPDATE native_trigger\n SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, error = NULL, updated_at = NOW()\n WHERE\n workspace_id = $5\n AND service_name = $6\n AND external_id = $7\n ",
"query": "\n UPDATE native_trigger\n SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, summary = $8, error = NULL, updated_at = NOW()\n WHERE\n workspace_id = $5\n AND service_name = $6\n AND external_id = $7\n ",
"describe": {
"columns": [],
"parameters": {
@@ -21,10 +21,11 @@
}
}
},
"Text"
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "40a8bf6a5a42c275d73221bc5f386f2e18cb911352551d0a34bf1933e558674e"
"hash": "bf224f6441c36187f1402f9f01bfe15bb9edfa1dc9052f8a829e486b7334d708"
}
backend/.sqlx/query-c79501b30fc28a3ae761579d05f2296e848554e62a00ff7c164109bf3d97f44f.json
Generated
+40
View File
@@ -0,0 +1,40 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH RECURSIVE chain AS (\n SELECT\n j.id,\n j.parent_job,\n j.flow_step_id,\n 1 AS depth\n FROM v2_job j\n WHERE j.id = $1\n UNION ALL\n SELECT\n pj.id,\n pj.parent_job,\n pj.flow_step_id,\n c.depth + 1\n FROM chain c\n JOIN v2_job pj ON pj.id = c.parent_job\n WHERE c.parent_job IS NOT NULL\n )\n SELECT\n c.id,\n c.parent_job,\n c.flow_step_id,\n EXISTS(SELECT 1 FROM v2_job_queue q WHERE q.id = c.parent_job) AS \"parent_in_queue!\"\n FROM chain c\n WHERE c.depth >= 1\n ORDER BY c.depth ASC\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 2,
"name": "flow_step_id",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "parent_in_queue!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
null,
null,
null,
null
]
},
"hash": "c79501b30fc28a3ae761579d05f2296e848554e62a00ff7c164109bf3d97f44f"
}
backend/.sqlx/query-ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS(\n SELECT 1 FROM v2_job\n WHERE parent_job = $1 AND id != $2\n AND id IN (SELECT id FROM v2_job_queue)\n ) as has",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "has",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid",
"Uuid"
]
},
"nullable": [
null
]
},
"hash": "ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2"
}
backend/.sqlx/query-f9eabfab66ae102c8a61ae5e6349b5b167ff7939828a747ad09bb06c9d6c6d8d.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_queue SET scheduled_for = $1 WHERE id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Timestamptz",
"Uuid"
]
},
"nullable": []
},
"hash": "f9eabfab66ae102c8a61ae5e6349b5b167ff7939828a747ad09bb06c9d6c6d8d"
}
backend/Cargo.lock
Generated
+131 -128
View File
@@ -234,9 +234,9 @@ dependencies = [
[[package]]
name = "arc-swap"
version = "1.8.2"
version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f3647c145568cec02c42054e07bdf9a5a698e15b466fb2341bfc393cd24aa5"
checksum = "a07d1f37ff60921c83bdfc7407723bdefe89b44b98a9b772f225c8f9d67141a6"
dependencies = [
"rustversion",
]
@@ -2536,12 +2536,6 @@ version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
[[package]]
name = "convert_case"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e"
[[package]]
name = "convert_case"
version = "0.6.0"
@@ -4887,19 +4881,6 @@ dependencies = [
"syn 1.0.109",
]
[[package]]
name = "derive_more"
version = "0.99.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f"
dependencies = [
"convert_case 0.4.0",
"proc-macro2",
"quote",
"rustc_version 0.4.1",
"syn 2.0.117",
]
[[package]]
name = "derive_more"
version = "1.0.0"
@@ -4927,6 +4908,7 @@ dependencies = [
"proc-macro2",
"quote",
"syn 2.0.117",
"unicode-xid",
]
[[package]]
@@ -7419,13 +7401,13 @@ dependencies = [
[[package]]
name = "ipconfig"
version = "0.3.2"
version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b58db92f96b720de98181bbbe63c831e87005ab460c1bf306eb2622b4707997f"
checksum = "2d72a21f6a71a6c4c3160e095e8925861f5119dd26ef71acee1b9146f74f76c8"
dependencies = [
"socket2 0.5.10",
"socket2 0.6.3",
"widestring",
"windows-sys 0.48.0",
"windows-sys 0.61.2",
"winreg",
]
@@ -7446,9 +7428,9 @@ dependencies = [
[[package]]
name = "iri-string"
version = "0.7.10"
version = "0.7.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c91338f0783edbd6195decb37bae672fd3b165faffb89bf7b9e6942f8b1a731a"
checksum = "d8e7418f59cc01c88316161279a7f665217ae316b388e58a0d10e29f54f1e5eb"
dependencies = [
"memchr",
"serde",
@@ -7538,7 +7520,7 @@ dependencies = [
"cesu8",
"cfg-if",
"combine",
"jni-sys",
"jni-sys 0.3.1",
"log",
"thiserror 1.0.69",
"walkdir",
@@ -7547,9 +7529,31 @@ dependencies = [
[[package]]
name = "jni-sys"
version = "0.3.0"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8eaf4bc02d17cbdd7ff4c7438cafcdf7fb9a4613313ad11b4f8fefe7d3fa0130"
checksum = "41a652e1f9b6e0275df1f15b32661cf0d4b78d4d87ddec5e0c3c20f097433258"
dependencies = [
"jni-sys 0.4.1",
]
[[package]]
name = "jni-sys"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
dependencies = [
"jni-sys-macros",
]
[[package]]
name = "jni-sys-macros"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
dependencies = [
"quote",
"syn 2.0.117",
]
[[package]]
name = "jobserver"
@@ -8308,9 +8312,9 @@ dependencies = [
[[package]]
name = "malachite"
version = "0.4.18"
version = "0.4.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a6ecab92657eb234bfe98abd0b17920772c6b14ce69256950142e2eb36d000b"
checksum = "2fbdf9cb251732db30a7200ebb6ae5d22fe8e11397364416617d2c2cf0c51cb5"
dependencies = [
"malachite-base",
"malachite-nz",
@@ -8331,11 +8335,11 @@ dependencies = [
[[package]]
name = "malachite-bigint"
version = "0.2.0"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "17703a19c80bbdd0b7919f0f104f3b0597f7de4fc4e90a477c15366a5ba03faa"
checksum = "d149aaa2965d70381709d9df4c7ee1fc0de1c614a4efc2ee356f5e43d68749f8"
dependencies = [
"derive_more 0.99.20",
"derive_more 1.0.0",
"malachite",
"num-integer",
"num-traits",
@@ -8610,9 +8614,9 @@ dependencies = [
[[package]]
name = "moka"
version = "0.12.14"
version = "0.12.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85f8024e1c8e71c778968af91d43700ce1d11b219d127d79fb2934153b82b42b"
checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046"
dependencies = [
"async-lock",
"crossbeam-channel",
@@ -8826,7 +8830,7 @@ version = "0.5.0+25.2.9519653"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8c196769dd60fd4f363e11d948139556a344e79d451aeb2fa2fd040738ef7691"
dependencies = [
"jni-sys",
"jni-sys 0.3.1",
]
[[package]]
@@ -11663,7 +11667,7 @@ dependencies = [
"once_cell",
"ring 0.17.14",
"rustls-pki-types",
"rustls-webpki 0.103.9",
"rustls-webpki 0.103.10",
"subtle",
"zeroize",
]
@@ -11747,7 +11751,7 @@ dependencies = [
"rustls 0.23.35",
"rustls-native-certs 0.8.3",
"rustls-platform-verifier-android",
"rustls-webpki 0.103.9",
"rustls-webpki 0.103.10",
"security-framework 3.6.0",
"security-framework-sys",
"webpki-root-certs 1.0.6",
@@ -11795,9 +11799,9 @@ dependencies = [
[[package]]
name = "rustls-webpki"
version = "0.103.9"
version = "0.103.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53"
checksum = "df33b2b81ac578cabaf06b89b0631153a3f416b0a886e8a7a1707fb51abbd1ef"
dependencies = [
"aws-lc-rs",
"ring 0.17.14",
@@ -13877,12 +13881,12 @@ dependencies = [
[[package]]
name = "terminal_size"
version = "0.4.3"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60b8cb979cb11c32ce1603f8137b22262a9d131aaa5c37b5678025f22b8becd0"
checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874"
dependencies = [
"rustix 1.1.4",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -14468,9 +14472,9 @@ dependencies = [
[[package]]
name = "toml_parser"
version = "1.0.10+spec-1.1.0"
version = "1.1.0+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7df25b4befd31c4816df190124375d5a20c6b6921e2cad937316de3fccd63420"
checksum = "2334f11ee363607eb04df9b8fc8a13ca1715a72ba8662a26ac285c98aabb4011"
dependencies = [
"winnow 1.0.0",
]
@@ -15742,7 +15746,7 @@ dependencies = [
[[package]]
name = "windmill"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-nats",
@@ -15818,7 +15822,7 @@ dependencies = [
[[package]]
name = "windmill-alerting"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -15831,7 +15835,7 @@ dependencies = [
[[package]]
name = "windmill-api"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"argon2",
@@ -15972,7 +15976,7 @@ dependencies = [
[[package]]
name = "windmill-api-agent-workers"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -15995,7 +15999,7 @@ dependencies = [
[[package]]
name = "windmill-api-assets"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16008,7 +16012,7 @@ dependencies = [
[[package]]
name = "windmill-api-auth"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"axum 0.7.9",
@@ -16034,7 +16038,7 @@ dependencies = [
[[package]]
name = "windmill-api-client"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"reqwest 0.12.28",
"serde",
@@ -16044,7 +16048,7 @@ dependencies = [
[[package]]
name = "windmill-api-configs"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16061,7 +16065,7 @@ dependencies = [
[[package]]
name = "windmill-api-debug"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"base64 0.22.1",
@@ -16084,7 +16088,7 @@ dependencies = [
[[package]]
name = "windmill-api-embeddings"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"axum 0.7.9",
@@ -16107,7 +16111,7 @@ dependencies = [
[[package]]
name = "windmill-api-flow-conversations"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16123,7 +16127,7 @@ dependencies = [
[[package]]
name = "windmill-api-flows"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16143,7 +16147,7 @@ dependencies = [
[[package]]
name = "windmill-api-groups"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16163,7 +16167,7 @@ dependencies = [
[[package]]
name = "windmill-api-inputs"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16177,7 +16181,7 @@ dependencies = [
[[package]]
name = "windmill-api-integration-tests"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-nats",
@@ -16205,7 +16209,7 @@ dependencies = [
[[package]]
name = "windmill-api-jobs"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"axum 0.7.9",
@@ -16230,7 +16234,7 @@ dependencies = [
[[package]]
name = "windmill-api-npm-proxy"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"flate2",
@@ -16248,7 +16252,7 @@ dependencies = [
[[package]]
name = "windmill-api-openapi"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"axum 0.7.9",
@@ -16260,6 +16264,7 @@ dependencies = [
"serde_json",
"serde_yml",
"sqlx",
"tracing",
"url",
"windmill-api-auth",
"windmill-common",
@@ -16269,7 +16274,7 @@ dependencies = [
[[package]]
name = "windmill-api-schedule"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16289,7 +16294,7 @@ dependencies = [
[[package]]
name = "windmill-api-scripts"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16319,7 +16324,7 @@ dependencies = [
[[package]]
name = "windmill-api-settings"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"axum 0.7.9",
@@ -16346,7 +16351,7 @@ dependencies = [
[[package]]
name = "windmill-api-sse"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"lazy_static",
"serde",
@@ -16358,7 +16363,7 @@ dependencies = [
[[package]]
name = "windmill-api-users"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"argon2",
"axum 0.7.9",
@@ -16381,7 +16386,7 @@ dependencies = [
[[package]]
name = "windmill-api-workers"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16395,7 +16400,7 @@ dependencies = [
[[package]]
name = "windmill-api-workspaces"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"axum 0.7.9",
"chrono",
@@ -16426,7 +16431,7 @@ dependencies = [
[[package]]
name = "windmill-audit"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"chrono",
"lazy_static",
@@ -16440,7 +16445,7 @@ dependencies = [
[[package]]
name = "windmill-autoscaling"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"axum 0.7.9",
@@ -16459,7 +16464,7 @@ dependencies = [
[[package]]
name = "windmill-common"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"aes-gcm",
"anyhow",
@@ -16559,7 +16564,7 @@ dependencies = [
[[package]]
name = "windmill-dep-map"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"chrono",
"itertools 0.14.0",
@@ -16578,7 +16583,7 @@ dependencies = [
[[package]]
name = "windmill-git-sync"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"regex",
"serde",
@@ -16593,7 +16598,7 @@ dependencies = [
[[package]]
name = "windmill-indexer"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -16617,7 +16622,7 @@ dependencies = [
[[package]]
name = "windmill-jseval"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"futures",
@@ -16634,7 +16639,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"itertools 0.14.0",
"lazy_static",
@@ -16650,7 +16655,7 @@ dependencies = [
[[package]]
name = "windmill-mcp"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16671,7 +16676,7 @@ dependencies = [
[[package]]
name = "windmill-native-triggers"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16702,7 +16707,7 @@ dependencies = [
[[package]]
name = "windmill-oauth"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-oauth2",
@@ -16726,7 +16731,7 @@ dependencies = [
[[package]]
name = "windmill-object-store"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-stream",
@@ -16760,7 +16765,7 @@ dependencies = [
[[package]]
name = "windmill-operator"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"futures",
@@ -16778,7 +16783,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"convert_case 0.6.0",
"serde",
@@ -16787,7 +16792,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16799,7 +16804,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"serde_json",
@@ -16811,7 +16816,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"gosyn",
@@ -16823,7 +16828,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16835,7 +16840,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"serde_json",
@@ -16847,7 +16852,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -16858,7 +16863,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -16869,7 +16874,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -16881,7 +16886,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -16892,14 +16897,12 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-recursion",
"itertools 0.14.0",
"lazy_static",
"malachite",
"malachite-bigint",
"pep440_rs",
"phf 0.11.3",
"regex",
@@ -16916,7 +16919,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16930,7 +16933,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"convert_case 0.6.0",
@@ -16947,7 +16950,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16956,12 +16959,11 @@ dependencies = [
"serde",
"serde_json",
"windmill-parser",
"windmill-types",
]
[[package]]
name = "windmill-parser-sql-asset"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"serde",
@@ -16973,7 +16975,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16991,7 +16993,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -17007,7 +17009,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -17023,7 +17025,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"serde",
@@ -17034,7 +17036,7 @@ dependencies = [
[[package]]
name = "windmill-queue"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -17071,7 +17073,7 @@ dependencies = [
[[package]]
name = "windmill-runtime-nativets"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"const_format",
@@ -17109,7 +17111,7 @@ dependencies = [
[[package]]
name = "windmill-sql-datatype-parser-wasm"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"getrandom 0.3.4",
"wasm-bindgen",
@@ -17120,7 +17122,7 @@ dependencies = [
[[package]]
name = "windmill-store"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -17149,7 +17151,7 @@ dependencies = [
[[package]]
name = "windmill-test-utils"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"axum 0.7.9",
@@ -17172,7 +17174,7 @@ dependencies = [
[[package]]
name = "windmill-trigger"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17205,7 +17207,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-email"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17225,7 +17227,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-gcp"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17259,7 +17261,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-http"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17294,7 +17296,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-kafka"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17317,7 +17319,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-mqtt"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17341,7 +17343,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-nats"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-nats",
@@ -17365,7 +17367,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-postgres"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17400,7 +17402,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-sqs"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17428,7 +17430,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-websocket"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-trait",
@@ -17451,7 +17453,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"bitflags 2.9.4",
@@ -17465,11 +17467,12 @@ dependencies = [
"strum 0.27.2",
"tracing",
"uuid",
"windmill-parser",
]
[[package]]
name = "windmill-worker"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"anyhow",
"async-once-cell",
@@ -17576,7 +17579,7 @@ dependencies = [
[[package]]
name = "windmill-worker-volumes"
version = "1.662.0"
version = "1.663.0"
dependencies = [
"bytes",
"futures",
@@ -18191,12 +18194,12 @@ checksum = "a90e88e4667264a994d34e6d1ab2d26d398dcdca8b7f52bec8668957517fc7d8"
[[package]]
name = "winreg"
version = "0.50.0"
version = "0.55.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "524e57b2c537c0f9b1e69f1965311ec12182b4122e45035b1508cd24d2adadb1"
checksum = "cb5a765337c50e9ec252c2069be9bf91c7df47afb103b642ba3a53bf8101be97"
dependencies = [
"cfg-if",
"windows-sys 0.48.0",
"windows-sys 0.59.0",
]
[[package]]
backend/Cargo.toml
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.662.0"
version = "1.663.0"
authors.workspace = true
edition.workspace = true
@@ -82,7 +82,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal"]
[workspace.package]
version = "1.662.0"
version = "1.663.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
backend/migrations/20260304000000_raw_script_temp.down.sql
+2
View File
@@ -0,0 +1,2 @@
DROP INDEX IF EXISTS idx_raw_script_temp_created_at;
DROP TABLE IF EXISTS raw_script_temp;
backend/migrations/20260304000000_raw_script_temp.up.sql
+11
View File
@@ -0,0 +1,11 @@
-- Temporary storage for raw script content during CLI lock generation
-- Content is stored with hash as key, cleaned up after 1 week
CREATE TABLE raw_script_temp (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id),
hash CHAR(64) NOT NULL,
content TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (workspace_id, hash)
);
CREATE INDEX IF NOT EXISTS idx_raw_script_temp_created_at ON raw_script_temp (created_at);
backend/migrations/20260323000000_native_trigger_summary.down.sql
+1
View File
@@ -0,0 +1 @@
ALTER TABLE native_trigger DROP COLUMN IF EXISTS summary;
backend/migrations/20260323000000_native_trigger_summary.up.sql
+1
View File
@@ -0,0 +1 @@
ALTER TABLE native_trigger ADD COLUMN summary VARCHAR(1000);
backend/parsers/windmill-parser-py-imports/Cargo.toml
+9 -11
View File
@@ -13,21 +13,19 @@ regex-lite.workspace = true
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
regex.workspace = true
[dependencies]
windmill-parser.workspace = true
windmill-common.workspace = true
rustpython-parser.workspace = true
malachite.workspace = true
malachite-bigint.workspace = true
phf.workspace = true
itertools.workspace = true
serde_json.workspace = true
anyhow.workspace = true
lazy_static.workspace = true
sqlx.workspace = true
async-recursion.workspace = true
toml.workspace = true
serde.workspace = true
pep440_rs.workspace = true
tracing.workspace = true
[dependencies]
windmill-parser.workspace = true
rustpython-parser.workspace = true
phf.workspace = true
itertools.workspace = true
serde_json.workspace = true
anyhow.workspace = true
lazy_static.workspace = true
backend/parsers/windmill-parser-py-imports/src/lib.rs
+60 -20
View File
@@ -8,10 +8,14 @@
mod mapping;
#[cfg(not(target_arch = "wasm32"))]
use async_recursion::async_recursion;
use itertools::Itertools;
use lazy_static::lazy_static;
use std::{collections::HashMap, str::FromStr};
#[cfg(not(target_arch = "wasm32"))]
use std::str::FromStr;
#[cfg(not(target_arch = "wasm32"))]
use std::collections::HashMap;
use mapping::{FULL_IMPORTS_MAP, SHORT_IMPORTS_MAP};
#[cfg(not(target_arch = "wasm32"))]
@@ -24,7 +28,9 @@ use rustpython_parser::{
text_size::TextRange,
Parse,
};
#[cfg(not(target_arch = "wasm32"))]
use sqlx::{Pool, Postgres};
#[cfg(not(target_arch = "wasm32"))]
use windmill_common::{
error::{self, to_anyhow},
worker::{
@@ -46,10 +52,14 @@ fn replace_full_import(x: &str) -> Option<String> {
FULL_IMPORTS_MAP.get(x).map(|x| (*x).to_owned())
}
#[cfg(not(target_arch = "wasm32"))]
lazy_static! {
static ref RE: Regex = Regex::new(r"^\#\s?(\S+)\s*$").unwrap();
static ref PIN_RE: Regex = Regex::new(r"(?:\s*#\s*(pin|repin):\s*)(\S*)").unwrap();
static ref PKG_RE: Regex = Regex::new(r"^([^!=<>]+)(?:[!=<>]|$)").unwrap();
}
lazy_static! {
static ref PIN_RE: Regex = Regex::new(r"(?:\s*#\s*(pin|repin):\s*)(\S*)").unwrap();
// Regex to properly match main function definition at line start,
// capturing both sync and async variants
static ref DEF_MAIN_RE: Regex = Regex::new(r"(?m)^(async\s+)?def\s+main\s*\(").unwrap();
@@ -82,7 +92,7 @@ fn process_import(module: Option<String>, path: &str, level: usize) -> Vec<NImpo
}
}
pub fn parse_relative_imports(code: &str, path: &str) -> error::Result<Vec<String>> {
pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<String>> {
let nimports = parse_code_for_imports(code, path)?;
return Ok(nimports
.into_iter()
@@ -94,7 +104,7 @@ pub fn parse_relative_imports(code: &str, path: &str) -> error::Result<Vec<Strin
}
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
enum NImport {
pub enum NImport {
// Order matters! First we want to resolve all repins
// manually repinned requirement
@@ -134,6 +144,8 @@ enum NImport {
// Relative imports
Relative(String),
}
#[cfg(not(target_arch = "wasm32"))]
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
enum NImportResolved {
Repin { pin: ImportPin, key: String },
@@ -142,12 +154,12 @@ enum NImportResolved {
}
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
struct ImportPin {
pkg: String,
path: String,
pub struct ImportPin {
pub pkg: String,
pub path: String,
}
fn parse_code_for_imports(code: &str, path: &str) -> error::Result<Vec<NImport>> {
pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImport>> {
// Use regex to safely find the main function definition
let mut code = DEF_MAIN_RE
.split(code)
@@ -175,7 +187,7 @@ fn parse_code_for_imports(code: &str, path: &str) -> error::Result<Vec<NImport>>
let code_with_fake_main = format!("{}\n\ndef main(): pass", code);
let ast = Suite::parse(&code_with_fake_main, "main.py").map_err(|e| {
error::Error::ExecutionErr(format!("Error parsing code for imports: {}", e.to_string()))
anyhow::anyhow!("Error parsing code for imports: {}", e.to_string())
})?;
// Note: We're still using the original code for finding pins,
// as the TextRange values from the parsed AST would be based on code_with_fake_main
@@ -256,6 +268,7 @@ fn parse_code_for_imports(code: &str, path: &str) -> error::Result<Vec<NImport>>
return Ok(nimports);
}
#[cfg(not(target_arch = "wasm32"))]
pub async fn parse_python_imports(
code: &str,
w_id: &str,
@@ -264,6 +277,7 @@ pub async fn parse_python_imports(
version_specifiers: &mut Vec<pep440_rs::VersionSpecifier>,
locked_v: &mut Option<pep440_rs::Version>,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
temp_script_refs: &Option<HashMap<String, String>>,
) -> error::Result<(Vec<String>, Option<String>)> {
let mut compile_error_hint: Option<String> = None;
let mut imports = parse_python_imports_inner(
@@ -276,6 +290,7 @@ pub async fn parse_python_imports(
&mut None,
locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
)
.await?
.into_values()
@@ -313,6 +328,7 @@ pub async fn parse_python_imports(
Ok((imports, compile_error_hint))
}
#[cfg(not(target_arch = "wasm32"))]
fn extract_pkg_name(requirement: &str) -> String {
PKG_RE
.captures(requirement)
@@ -320,6 +336,7 @@ fn extract_pkg_name(requirement: &str) -> String {
.unwrap_or_default()
}
#[cfg(not(target_arch = "wasm32"))]
#[async_recursion]
async fn parse_python_imports_inner(
code: &str,
@@ -331,6 +348,7 @@ async fn parse_python_imports_inner(
path_where_annotated_pyv: &mut Option<String>,
locked_v: &mut Option<pep440_rs::Version>,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
temp_script_refs: &Option<HashMap<String, String>>,
) -> error::Result<HashMap<String, NImportResolved>> {
tracing::debug!("Parsing python imports for path: {}", path);
let PythonAnnotations { py310, py311, py312, py313, .. } = PythonAnnotations::parse(&code);
@@ -494,17 +512,37 @@ async fn parse_python_imports_inner(
for n in nimports.into_iter() {
let mut nested = match n {
NImport::Relative(rpath) => {
let code = sqlx::query_scalar!(
r#"
SELECT content FROM script WHERE path = $1 AND workspace_id = $2
AND archived = false ORDER BY created_at DESC LIMIT 1
"#,
&rpath,
w_id
)
.fetch_optional(db)
.await?
.unwrap_or_else(|| "".to_string());
// First try to get content from temp_script_refs cache if available
let code_from_cache = if let Some(hash) = temp_script_refs.as_ref().and_then(|dt| dt.get(&rpath)) {
tracing::debug!("Found relative import '{}' in temp_script_refs with hash '{}'", rpath, hash);
match windmill_common::cache::raw_script_temp::load(hash.clone(), db).await {
Ok(content) => Some(content),
Err(e) => {
tracing::warn!("temp_script_refs hash '{}' not found in cache: {}, falling back to deployed script", hash, e);
None
}
}
} else {
None
};
// Use cached content if available, otherwise fall back to deployed script
let code = match code_from_cache {
Some(content) => content,
None => {
sqlx::query_scalar!(
r#"
SELECT content FROM script WHERE path = $1 AND workspace_id = $2
AND archived = false ORDER BY created_at DESC LIMIT 1
"#,
&rpath,
w_id
)
.fetch_optional(db)
.await?
.unwrap_or_else(|| "".to_string())
}
};
if already_visited.contains(&rpath) {
vec![]
@@ -522,6 +560,7 @@ async fn parse_python_imports_inner(
path_where_annotated_pyv,
locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
)
.await?
.into_values()
@@ -646,6 +685,7 @@ async fn parse_python_imports_inner(
Ok(final_imports)
}
#[cfg(not(target_arch = "wasm32"))]
fn extract_nimports_from_content(
content: &str,
hm: &mut HashMap<String, NImportResolved>,
backend/parsers/windmill-parser-py-imports/tests/tests.rs
+3
View File
@@ -26,6 +26,7 @@ def main():
&mut vec![],
&mut None,
&None,
&None,
)
.await?;
// println!("{}", serde_json::to_string(&r)?);
@@ -67,6 +68,7 @@ def main():
&mut vec![],
&mut None,
&None,
&None,
)
.await?;
println!("{}", serde_json::to_string(&r)?);
@@ -98,6 +100,7 @@ def main():
&mut vec![],
&mut None,
&None,
&None,
)
.await?;
println!("{}", serde_json::to_string(&r)?);
backend/parsers/windmill-parser-sql/Cargo.toml
-1
View File
@@ -16,7 +16,6 @@ regex.workspace = true
[dependencies]
windmill-parser.workspace = true
windmill-types.workspace = true
anyhow.workspace = true
lazy_static.workspace = true
serde_json.workspace = true
backend/parsers/windmill-parser-sql/src/lib.rs
+1 -2
View File
@@ -15,7 +15,7 @@ use std::{
iter::Peekable,
str::CharIndices,
};
pub use windmill_parser::{Arg, MainArgSignature, ObjectType, Typ};
pub use windmill_parser::{s3_mode_extension, Arg, MainArgSignature, ObjectType, S3ModeFormat, Typ};
pub const SANITIZED_ENUM_STR: &str = "__sanitized_enum__";
pub const SANITIZED_RAW_STRING_STR: &str = "__sanitized_raw_string__";
@@ -143,7 +143,6 @@ pub fn parse_db_resource(code: &str) -> Option<String> {
cap.map(|x| x.get(1).map(|x| x.as_str().to_string()).unwrap())
}
pub use windmill_types::s3::{s3_mode_extension, S3ModeFormat};
pub struct S3ModeArgs {
pub prefix: Option<String>,
pub storage: Option<String>,
backend/parsers/windmill-parser-ts/src/lib.rs
+82
View File
@@ -117,6 +117,12 @@ impl Visit for ImportsFinder {
}
}
/// Parse TypeScript/JavaScript code and extract all import paths as raw strings.
///
/// Returns import paths exactly as written in the code (e.g., `"./module"`, `"../utils"`, `"lodash"`).
/// Does not resolve relative paths to absolute Windmill paths.
///
/// See also: [`parse_relative_imports`] for resolved absolute paths.
pub fn parse_expr_for_imports(code: &str, skip_type_only: bool) -> anyhow::Result<Vec<String>> {
let cm: Lrc<SourceMap> = Default::default();
let fm = cm.new_source_file(FileName::Custom("main.d.ts".into()).into(), code.into());
@@ -151,6 +157,82 @@ pub fn parse_expr_for_imports(code: &str, skip_type_only: bool) -> anyhow::Resul
Ok(imports)
}
/// Parse TypeScript/JavaScript code and extract relative imports resolved to absolute Windmill paths.
///
/// Takes the script's Windmill path (e.g., `"f/folder/script"`) and resolves relative imports
/// like `"./module"` or `"../utils"` to absolute paths like `"f/folder/module"` or `"f/utils"`.
///
/// Only returns relative imports (those starting with `./`, `../`, or `/`).
/// External package imports (e.g., `"lodash"`) are filtered out.
///
/// See also: [`parse_expr_for_imports`] for raw import strings without resolution.
///
/// # Arguments
/// * `code` - The TypeScript/JavaScript source code
/// * `path` - The Windmill path of the script (e.g., `"f/folder/script"`)
///
/// # Returns
/// A sorted, deduplicated list of resolved absolute Windmill paths.
///
/// # Examples
/// ```ignore
/// // Script at "f/folder/script" with: import { x } from "../utils"
/// // Returns: ["f/utils"]
/// ```
pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<String>> {
let imports = parse_expr_for_imports(code, false)?;
let script_dir = path.rsplit_once('/').map(|(dir, _)| dir).unwrap_or("");
let mut resolved: Vec<String> = imports
.into_iter()
.filter(|imp| is_relative_import(imp))
.map(|imp| {
// Remove .ts extension if present
let imp = imp.strip_suffix(".ts").unwrap_or(&imp);
if imp.starts_with("/") {
// Absolute path (e.g., /f/folder/script) - remove leading slash
imp[1..].to_string()
} else {
// Relative path (e.g., ./script or ../folder/script)
let combined = format!("{}/{}", script_dir, imp);
normalize_path(&combined)
}
})
.collect();
resolved.sort();
resolved.dedup();
Ok(resolved)
}
/// Check if an import path is a relative import (starts with `./`, `../`, or `/`)
fn is_relative_import(import_path: &str) -> bool {
import_path.starts_with("./")
|| import_path.starts_with("../")
|| import_path.starts_with("/")
}
/// Normalize a path by resolving `.` and `..` components
fn normalize_path(input_path: &str) -> String {
let parts: Vec<&str> = input_path.split('/').filter(|p| !p.is_empty()).collect();
let mut result: Vec<&str> = Vec::new();
for part in parts {
if part == "." {
continue;
} else if part == ".." {
if !result.is_empty() {
result.pop();
}
} else {
result.push(part);
}
}
result.join("/")
}
struct OutputFinder {
idents: HashSet<(String, String)>,
}
backend/parsers/windmill-parser-ts/tests/tests.rs
+82 -2
View File
@@ -2,7 +2,7 @@
mod tests {
use serde_json::json;
use windmill_parser::{Arg, MainArgSignature, ObjectProperty, ObjectType, Typ};
use windmill_parser_ts::{parse_deno_signature, parse_expr_for_imports};
use windmill_parser_ts::{parse_deno_signature, parse_expr_for_imports, parse_relative_imports};
#[test]
fn test_imports_basic() {
@@ -798,7 +798,7 @@ mod tests {
// Test case where there are exports but no preprocessor
let code = r#"
export { foo, bar } from "./utils";
export async function main(param: string) {
return param;
}
@@ -806,4 +806,84 @@ mod tests {
let sig = parse_deno_signature(code, false, false, None).unwrap();
assert_eq!(sig.has_preprocessor, Some(false));
}
// ==========================================================================
// Tests for parse_relative_imports
// ==========================================================================
#[test]
fn test_relative_imports_dot() {
let code = r#"
import { helper } from "./helper";
export async function main() { return helper(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert_eq!(result, vec!["f/folder/helper"]);
}
#[test]
fn test_relative_imports_double_dot() {
let code = r#"
import { utils } from "../utils/helper";
export async function main() { return utils(); }
"#;
let result = parse_relative_imports(code, "f/folder/subfolder/script").unwrap();
assert_eq!(result, vec!["f/folder/utils/helper"]);
}
#[test]
fn test_relative_imports_absolute_path() {
let code = r#"
import { shared } from "/f/shared/utils";
export async function main() { return shared(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert_eq!(result, vec!["f/shared/utils"]);
}
#[test]
fn test_relative_imports_mixed() {
let code = r#"
import { helper } from "./helper";
import { utils } from "../utils";
import { shared } from "/f/shared/lib";
import lodash from "lodash";
export async function main() { return helper() + utils() + shared(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
// Should only include relative imports, not external packages like lodash
assert_eq!(result, vec!["f/folder/helper", "f/shared/lib", "f/utils"]);
}
#[test]
fn test_relative_imports_with_ts_extension() {
let code = r#"
import { helper } from "./helper.ts";
export async function main() { return helper(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert_eq!(result, vec!["f/folder/helper"]);
}
#[test]
fn test_relative_imports_external_only() {
let code = r#"
import lodash from "lodash";
import { something } from "@scope/package";
export async function main() { return lodash.map([]); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert!(result.is_empty());
}
#[test]
fn test_relative_imports_deeply_nested() {
let code = r#"
import { a } from "../../a";
import { b } from "../../../b";
export async function main() { return a() + b(); }
"#;
let result = parse_relative_imports(code, "f/one/two/three/script").unwrap();
assert_eq!(result, vec!["f/b", "f/one/a"]);
}
}
backend/parsers/windmill-parser-wasm/Cargo.toml
+2
View File
@@ -40,6 +40,7 @@ java-parser = [ "dep:windmill-parser-java"]
ruby-parser = [ "dep:windmill-parser-ruby"]
wac-parser = [ "dep:windmill-parser-wac"]
asset-parser = [ "dep:windmill-parser-ts-asset", "dep:windmill-parser-py-asset", "dep:windmill-parser-sql-asset"]
py-imports-parser = [ "dep:windmill-parser-py-imports"]
[dependencies]
anyhow.workspace = true
@@ -61,6 +62,7 @@ windmill-parser-wac = { workspace = true, optional = true }
windmill-parser-ts-asset = { workspace = true, optional = true }
windmill-parser-py-asset = { workspace = true, optional = true }
windmill-parser-sql-asset = { workspace = true, optional = true }
windmill-parser-py-imports = { workspace = true, optional = true }
wasm-bindgen.workspace = true
serde_json.workspace = true
backend/parsers/windmill-parser-wasm/build.nu
+6
View File
@@ -67,6 +67,12 @@ const targets = [
features: "asset-parser",
env: "default",
},
{
ident: "py-imports",
desc: "Python imports"
features: "py-imports-parser",
env: "default",
},
# ^^^ Add new entry here ^^^
];
# NOTE: This is legacy command for building all, but it is not more used
backend/parsers/windmill-parser-wasm/dev.nu
+5 -2
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env nu
# Build in debug mode specified lang parser to wasm
# Build in debug mode specified lang parser to wasm
# and perform installation to frontend
def "main" [
lang: string # Example: nu
@@ -9,4 +9,7 @@ def "main" [
(
cd ../../../frontend; npm install ../backend/parsers/windmill-parser-wasm/pkg-($lang)
)
(
cd ../../../cli; bun install ../backend/parsers/windmill-parser-wasm/pkg-($lang)
)
}
backend/parsers/windmill-parser-wasm/publish-pkgs.sh
+3
View File
@@ -36,3 +36,6 @@ popd
pushd "pkg-asset" && npm publish ${args}
popd
pushd "pkg-py-imports" && npm publish ${args}
popd
backend/parsers/windmill-parser-wasm/src/lib.rs
+19
View File
@@ -38,6 +38,8 @@ pub fn parse_outputs(code: &str) -> String {
return serde_json::to_string(&r).unwrap();
}
/// Parse TypeScript imports and return raw import strings.
/// See [`parse_ts_relative_imports`] for resolved absolute paths.
#[cfg(feature = "ts-parser")]
#[wasm_bindgen]
pub fn parse_ts_imports(code: &str) -> String {
@@ -50,6 +52,15 @@ pub fn parse_ts_imports(code: &str) -> String {
return serde_json::to_string(&r).unwrap();
}
/// Parse TypeScript imports and return relative imports resolved to absolute Windmill paths.
/// Throws JS error on parse failure.
/// See [`parse_ts_imports`] for raw import strings.
#[cfg(feature = "ts-parser")]
#[wasm_bindgen]
pub fn parse_ts_relative_imports(code: &str, path: &str) -> Result<Vec<String>, String> {
windmill_parser_ts::parse_relative_imports(code, path).map_err(|e| e.to_string())
}
#[cfg(feature = "bash-parser")]
#[wasm_bindgen]
pub fn parse_bash(code: &str) -> String {
@@ -214,6 +225,14 @@ pub fn parse_assets_py(code: &str) -> String {
}
}
/// Parse Python imports and return relative imports resolved to absolute Windmill paths.
/// Throws JS error on parse failure.
#[cfg(feature = "py-imports-parser")]
#[wasm_bindgen]
pub fn parse_py_relative_imports(code: &str, path: &str) -> Result<Vec<String>, String> {
windmill_parser_py_imports::parse_relative_imports(code, path).map_err(|e| e.to_string())
}
#[cfg(feature = "ansible-parser")]
#[wasm_bindgen]
pub fn parse_assets_ansible(code: &str) -> String {
backend/parsers/windmill-parser/src/lib.rs
+17
View File
@@ -14,6 +14,23 @@ use serde_json::Value;
pub mod asset_parser;
/// S3 output format for SQL queries (moved here to avoid pulling sqlx into WASM via windmill-types)
#[derive(Clone, Copy, Debug)]
pub enum S3ModeFormat {
Json,
Csv,
Parquet,
}
/// Returns the file extension for the given S3 mode format
pub fn s3_mode_extension(format: S3ModeFormat) -> &'static str {
match format {
S3ModeFormat::Json => "json",
S3ModeFormat::Csv => "csv",
S3ModeFormat::Parquet => "parquet",
}
}
#[derive(Serialize, Debug, PartialEq, Default)]
pub struct MainArgSignature {
pub star_args: bool,
backend/src/main.rs
+1
View File
@@ -312,6 +312,7 @@ async fn cache_hub_scripts(file_path: Option<String>) -> anyhow::Result<()> {
"cache_init",
"",
&mut None,
&None,
)
.await
{
backend/src/monitor.rs
+7 -1
View File
@@ -98,7 +98,7 @@ use windmill_worker::{
CARGO_REGISTRIES, INSTANCE_PYTHON_VERSION, JAVA_HOME_DIR, JOB_DEFAULT_TIMEOUT, JOB_ISOLATION,
KEEP_JOB_DIR, MAVEN_REPOS, MAVEN_SETTINGS_XML, NO_DEFAULT_MAVEN, NPMRC, NPM_CONFIG_REGISTRY,
NSJAIL_AVAILABLE, NUGET_CONFIG, OTEL_TRACING_PROXY_SETTINGS, PIP_EXTRA_INDEX_URL,
PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, UV_INDEX_STRATEGY,
PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, UNSHARE_PATH, UV_INDEX_STRATEGY,
WORKSPACE_REGISTRIES,
};
@@ -1701,6 +1701,12 @@ pub async fn reload_job_isolation_setting(conn: &Connection) {
All jobs will fail until nsjail is installed or the setting is changed."
);
}
if value == JobIsolationLevel::Unshare && UNSHARE_PATH.is_none() {
tracing::error!(
"job_isolation is set to unshare but the unshare binary is not available on this worker. \
Jobs will run without isolation until unshare is installed or the setting is changed."
);
}
}
pub async fn reload_request_size(conn: &Connection) {
backend/tests/bun_jobs.rs
+2
View File
@@ -921,6 +921,7 @@ mod dedicated_worker_protocol {
"test-workspace",
"f/test/script",
LoaderMode::Node,
&None,
))
.expect("build_loader failed");
@@ -1299,6 +1300,7 @@ mod bun_builder_tests {
// Write build.js using the loader and builder constants directly
// Parameters are dummy values since tests don't use Windmill relative imports
let loader = RELATIVE_BUN_LOADER
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", "{}")
.replace("W_ID", "test-workspace")
.replace("BASE_INTERNAL_URL", "http://localhost:8000")
.replace("TOKEN", "test-token")
backend/tests/flow_engine_parity.rs
+1
View File
@@ -48,6 +48,7 @@ fn flow_module(id: &str, value: FlowModuleValue) -> FlowModule {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
}
}
backend/tests/nativets_dedicated.rs
+1
View File
@@ -32,6 +32,7 @@ mod prewarmed_isolate_tests {
"test-workspace",
"f/test/script",
LoaderMode::BrowserBundle,
&None,
)
.await
.expect("build_loader failed");
backend/tests/worker.rs
+8
View File
@@ -209,6 +209,7 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "b".to_string(),
@@ -255,6 +256,7 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
}],
modules_node: None,
}
@@ -275,6 +277,7 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
same_worker: false,
@@ -389,6 +392,7 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "b".to_string(),
@@ -444,6 +448,7 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "e".to_string(),
@@ -485,6 +490,7 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
modules_node: None,
@@ -505,6 +511,7 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "c".to_string(),
@@ -552,6 +559,7 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
same_worker: true,
backend/windmill-api-flows/src/flows.rs
+42 -12
View File
@@ -33,8 +33,10 @@ use sqlx::{FromRow, Postgres, Transaction};
use windmill_audit::audit_oss::{audit_log, AuditAuthorable};
use windmill_audit::ActionKind;
use windmill_common::assets::{clear_static_asset_usage, AssetUsageKind};
use windmill_common::flows::FlowModule;
use windmill_common::min_version::{
MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2,
MIN_VERSION_SUPPORTS_NODE_DEBOUNCING,
};
use windmill_common::runnable_settings::RunnableSettingsTrait;
use windmill_common::utils::query_elems_from_hub;
@@ -1576,30 +1578,54 @@ async fn archive_flow_by_path(
/// Validates that flow debouncing configuration is supported by all workers
/// Returns an error if debouncing is configured but workers are behind required version
async fn guard_flow_from_debounce_data(nf: &NewFlow) -> Result<()> {
if !MIN_VERSION_SUPPORTS_DEBOUNCING.met().await
&& !nf.parse_flow_value()?.debouncing_settings.is_default()
let flow_value = nf.parse_flow_value()?;
if !MIN_VERSION_SUPPORTS_DEBOUNCING.met().await && !flow_value.debouncing_settings.is_default()
{
tracing::warn!(
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
Err(Error::WorkersAreBehind { feature: "Debouncing".into(), min_version: "1.566.0".into() })
} else if !MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
&& !nf
.parse_flow_value()?
.debouncing_settings
.is_legacy_compatible()
return Err(Error::WorkersAreBehind {
feature: "Debouncing".into(),
min_version: "1.566.0".into(),
});
}
if !MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
&& !flow_value.debouncing_settings.is_legacy_compatible()
&& !*WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT
{
tracing::warn!(
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
Err(Error::WorkersAreBehind {
return Err(Error::WorkersAreBehind {
feature: "V2 Debouncing".into(),
min_version: "1.597.0".into(),
})
} else {
Ok(())
});
}
// Check node-level debouncing on all modules (including nested branches/loops)
let mut has_node_debouncing = false;
let check_result = FlowModule::traverse_modules(&flow_value.modules, &mut |m| {
if m.debouncing
.as_ref()
.is_some_and(|d| d.debounce_delay_s.is_some_and(|s| s > 0))
{
has_node_debouncing = true;
}
Ok(())
});
if let Err(e) = check_result {
tracing::warn!("Failed to traverse flow modules for debounce guard: {e}");
}
if has_node_debouncing && !MIN_VERSION_SUPPORTS_NODE_DEBOUNCING.met().await {
return Err(Error::WorkersAreBehind {
feature: "Flow node debouncing".into(),
min_version: "1.658.0".into(),
});
}
Ok(())
}
#[derive(Deserialize)]
@@ -1768,6 +1794,7 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "b".to_string(),
@@ -1801,6 +1828,7 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "c".to_string(),
@@ -1834,6 +1862,7 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
failure_module: Some(Box::new(FlowModule {
@@ -1866,6 +1895,7 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
})),
preprocessor_module: None,
same_worker: false,
backend/windmill-api-groups/src/folders.rs
+15 -9
View File
@@ -18,7 +18,6 @@ use regex::Regex;
use windmill_api_auth::{check_scopes, ApiAuthed, AuthCache, Tokened};
use windmill_audit::audit_oss::{audit_log, AuditAuthorable};
use windmill_audit::ActionKind;
use windmill_common::{error::Error, webhook::{WebhookMessage, WebhookShared}, workspaces::{check_user_against_rule, ProtectionRuleKind, RuleCheckResult}};
use windmill_common::DB;
use windmill_common::{
db::UserDB,
@@ -26,6 +25,11 @@ use windmill_common::{
users::username_to_permissioned_as,
utils::{not_found_if_none, paginate, Pagination},
};
use windmill_common::{
error::Error,
webhook::{WebhookMessage, WebhookShared},
workspaces::{check_user_against_rule, ProtectionRuleKind, RuleCheckResult},
};
use serde::{Deserialize, Serialize};
use sqlx::{FromRow, Postgres, Transaction};
@@ -716,13 +720,14 @@ async fn add_owner(
.await?;
validate_owner(&owner)?;
sqlx::query(&format!(
"UPDATE folder SET extra_perms = jsonb_set(extra_perms, '{{\"{owner}\"}}', to_jsonb($1), \
true) WHERE name = $2 AND workspace_id = $3 RETURNING extra_perms"
))
sqlx::query(
"UPDATE folder SET extra_perms = jsonb_set(extra_perms, array[$4]::text[], to_jsonb($1), \
true) WHERE name = $2 AND workspace_id = $3 RETURNING extra_perms",
)
.bind(true)
.bind(&name)
.bind(&w_id)
.bind(&owner)
.fetch_optional(&mut *tx)
.await?;
@@ -787,14 +792,15 @@ async fn remove_owner(
}
if let Some(write) = write {
let old_write = sqlx::query_scalar::<_, Option<bool>>(&format!(
"UPDATE folder SET extra_perms = jsonb_set(extra_perms, '{{\"{owner}\"}}', to_jsonb($1), \
true) FROM (SELECT (extra_perms->>'{owner}')::boolean as old_val FROM folder WHERE name = $2 AND workspace_id = $3) old \
let old_write = sqlx::query_scalar::<_, Option<bool>>(
"UPDATE folder SET extra_perms = jsonb_set(extra_perms, array[$4]::text[], to_jsonb($1), \
true) FROM (SELECT (extra_perms->>$4)::boolean as old_val FROM folder WHERE name = $2 AND workspace_id = $3) old \
WHERE name = $2 AND workspace_id = $3 RETURNING old.old_val"
))
)
.bind(write)
.bind(&name)
.bind(&w_id)
.bind(&owner)
.fetch_optional(&mut *tx)
.await?
.flatten();
backend/windmill-api-integration-tests/tests/native_triggers.rs
+2
View File
@@ -422,6 +422,7 @@ async fn test_delete_integration_full_cascade(db: Pool<Postgres>) -> anyhow::Res
"ext-1",
&trigger_config,
json!({"triggerType": "drive"}),
None,
)
.await?;
@@ -511,6 +512,7 @@ async fn test_cleanup_preserves_triggers(db: Pool<Postgres>) -> anyhow::Result<(
"ext-1",
&trigger_config,
json!({"triggerType": "drive"}),
None,
)
.await?;
backend/windmill-api-jobs/src/query.rs
+24 -28
View File
@@ -50,11 +50,11 @@ pub fn filter_list_queue_query(
.values
.iter()
.map(|v| {
let p = v.replace("*", "%").replace("'", "''");
let p = v.replace("*", "%");
if w.negated {
format!("v2_job_queue.worker NOT LIKE '{p}'")
format!("v2_job_queue.worker NOT LIKE {}", quote(&p))
} else {
format!("v2_job_queue.worker LIKE '{p}'")
format!("v2_job_queue.worker LIKE {}", quote(&p))
}
})
.collect();
@@ -77,11 +77,11 @@ pub fn filter_list_queue_query(
.values
.iter()
.map(|v| {
let e = v.replace("'", "''");
let p = format!("{}%", v);
if ps.negated {
format!("runnable_path NOT LIKE '{e}%'")
format!("runnable_path NOT LIKE {}", quote(&p))
} else {
format!("runnable_path LIKE '{e}%'")
format!("runnable_path LIKE {}", quote(&p))
}
})
.collect();
@@ -123,11 +123,11 @@ pub fn filter_list_queue_query(
.values
.iter()
.map(|v| {
let p = v.replace("*", "%").replace("'", "''");
let p = v.replace("*", "%");
if t.negated {
format!("v2_job.tag NOT LIKE '{p}'")
format!("v2_job.tag NOT LIKE {}", quote(&p))
} else {
format!("v2_job.tag LIKE '{p}'")
format!("v2_job.tag LIKE {}", quote(&p))
}
})
.collect();
@@ -287,14 +287,14 @@ pub fn filter_list_completed_query(
.values
.iter()
.map(|v| {
let p = v.replace("*", "%").replace("'", "''");
let p = v.replace("*", "%");
if label.negated {
format!(
"NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE '{p}')"
"NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE {})", quote(&p)
)
} else {
format!(
"EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE '{p}')"
"EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE {})", quote(&p)
)
}
})
@@ -308,14 +308,14 @@ pub fn filter_list_completed_query(
let clauses: Vec<_> = label
.values
.iter()
.map(|v| format!("NOT (result->'wm_labels' ? '{}')", v.replace("'", "''")))
.map(|v| format!("NOT (result->'wm_labels' ? {})", quote(v)))
.collect();
sqlb.and_where(format!("({})", clauses.join(" AND ")));
} else {
let clauses: Vec<_> = label
.values
.iter()
.map(|v| format!("result->'wm_labels' ? '{}'", v.replace("'", "''")))
.map(|v| format!("result->'wm_labels' ? {}", quote(v)))
.collect();
sqlb.and_where("result ? 'wm_labels'");
sqlb.and_where(format!("({})", clauses.join(" OR ")));
@@ -329,11 +329,11 @@ pub fn filter_list_completed_query(
.values
.iter()
.map(|v| {
let p = v.replace("*", "%").replace("'", "''");
let p = v.replace("*", "%");
if worker.negated {
format!("v2_job_completed.worker NOT LIKE '{p}'")
format!("v2_job_completed.worker NOT LIKE {}", quote(&p))
} else {
format!("v2_job_completed.worker LIKE '{p}'")
format!("v2_job_completed.worker LIKE {}", quote(&p))
}
})
.collect();
@@ -366,11 +366,11 @@ pub fn filter_list_completed_query(
.values
.iter()
.map(|v| {
let e = v.replace("'", "''");
let p = format!("{}%", v);
if ps.negated {
format!("runnable_path NOT LIKE '{e}%'")
format!("runnable_path NOT LIKE {}", quote(&p))
} else {
format!("runnable_path LIKE '{e}%'")
format!("runnable_path LIKE {}", quote(&p))
}
})
.collect();
@@ -400,11 +400,11 @@ pub fn filter_list_completed_query(
.values
.iter()
.map(|v| {
let p = v.replace("*", "%").replace("'", "''");
let p = v.replace("*", "%");
if t.negated {
format!("v2_job.tag NOT LIKE '{p}'")
format!("v2_job.tag NOT LIKE {}", quote(&p))
} else {
format!("v2_job.tag LIKE '{p}'")
format!("v2_job.tag LIKE {}", quote(&p))
}
})
.collect();
@@ -449,11 +449,7 @@ pub fn filter_list_completed_query(
}
if let Some(dt) = &lq.created_or_started_after {
let ts = dt.to_rfc3339();
sqlb.and_where(format!(
"(created_at >= '{}' OR started_at >= '{}')",
ts.replace("'", "''"),
ts.replace("'", "''")
));
sqlb.and_where("(created_at >= ? OR started_at >= ?)".bind(&ts).bind(&ts));
}
if let Some(dt) = &lq.created_before {
backend/windmill-api-openapi/Cargo.toml
+1
View File
@@ -23,4 +23,5 @@ serde.workspace = true
serde_json.workspace = true
serde_yml.workspace = true
sqlx.workspace = true
tracing.workspace = true
url.workspace = true
backend/windmill-api-openapi/src/lib.rs
+119 -7
View File
@@ -164,6 +164,7 @@ pub struct FuturePath {
summary: Option<String>,
description: Option<String>,
security_scheme: Option<SecurityScheme>,
args_schema: Option<Value>,
}
impl FuturePath {
@@ -174,8 +175,17 @@ impl FuturePath {
summary: Option<String>,
description: Option<String>,
security_scheme: Option<SecurityScheme>,
args_schema: Option<Value>,
) -> FuturePath {
FuturePath { route_path, kind, request_type, summary, description, security_scheme }
FuturePath {
route_path,
kind,
request_type,
summary,
description,
security_scheme,
args_schema,
}
}
}
@@ -397,7 +407,21 @@ fn generate_paths(
);
if method != Method::GET {
method_map.insert("requestBody", generate_default_request());
if let Some(schema) = &path.args_schema {
method_map.insert(
"requestBody",
serde_json::json!({
"required": true,
"content": {
"application/json": {
"schema": schema
}
}
}),
);
} else {
method_map.insert("requestBody", generate_default_request());
}
} else if is_webhook {
method_map.insert(
"parameters",
@@ -655,6 +679,76 @@ struct GenerateOpenAPI {
openapi_spec_format: Format,
}
fn clean_schema_for_openapi(schema: Value) -> Value {
if let Value::Object(mut obj) = schema {
obj.remove("$schema");
obj.remove("order");
Value::Object(obj)
} else {
schema
}
}
async fn get_runnable_schema(
db: &DB,
w_id: &str,
script_path: &str,
is_flow: bool,
) -> Option<Value> {
if is_flow {
let row = sqlx::query!(
r#"SELECT
f.schema AS "schema: serde_json::Value",
fv.value->>'preprocessor_module' IS NOT NULL AS "has_preprocessor: bool"
FROM flow f
LEFT JOIN flow_version fv ON fv.id = f.versions[array_length(f.versions, 1)]
AND fv.workspace_id = f.workspace_id
WHERE f.path = $1 AND f.workspace_id = $2 AND NOT f.archived"#,
script_path,
w_id,
)
.fetch_optional(db)
.await
.map_err(|e| {
tracing::warn!("Failed to fetch flow schema for {script_path}: {e}");
e
})
.ok()
.flatten()?;
if row.has_preprocessor.unwrap_or(false) {
return None;
}
row.schema.map(clean_schema_for_openapi)
} else {
let row = sqlx::query!(
r#"SELECT
schema AS "schema: serde_json::Value",
has_preprocessor
FROM script
WHERE path = $1 AND workspace_id = $2
AND NOT archived AND NOT deleted
ORDER BY created_at DESC
LIMIT 1"#,
script_path,
w_id,
)
.fetch_optional(db)
.await
.map_err(|e| {
tracing::warn!("Failed to fetch script schema for {script_path}: {e}");
e
})
.ok()
.flatten()?;
if row.has_preprocessor.unwrap_or(false) {
return None;
}
row.schema.map(clean_schema_for_openapi)
}
}
async fn http_routes_to_future_paths(
db: &DB,
user_db: UserDB,
@@ -691,6 +785,9 @@ async fn http_routes_to_future_paths(
description: Option<String>,
authentication_method: AuthenticationMethod,
authentication_resource_path: Option<String>,
script_path: String,
is_flow: bool,
wrap_body: bool,
}
http_routes = sqlx::query_as!(
@@ -704,7 +801,10 @@ async fn http_routes_to_future_paths(
summary,
description,
authentication_method AS "authentication_method: _",
authentication_resource_path
authentication_resource_path,
script_path,
is_flow,
wrap_body
FROM
http_trigger
WHERE
@@ -764,6 +864,12 @@ async fn http_routes_to_future_paths(
HttpMethod::Delete => Method::DELETE,
};
let args_schema = if !http_route.wrap_body {
get_runnable_schema(db, w_id, &http_route.script_path, http_route.is_flow).await
} else {
None
};
let future_path = FuturePath::new(
route_path,
Kind::HttpRoute(HttpRouteConfig::new(method)),
@@ -771,6 +877,7 @@ async fn http_routes_to_future_paths(
http_route.summary,
http_route.description,
auth_method,
args_schema,
);
openapi_future_paths.push(future_path);
@@ -780,6 +887,7 @@ async fn http_routes_to_future_paths(
}
async fn webhook_to_future_paths(
db: &DB,
pg_pool: &mut PgConnection,
webhook_filters: Option<&[WebhookFilter]>,
w_id: &str,
@@ -805,7 +913,7 @@ async fn webhook_to_future_paths(
}
}
#[derive(Debug, Deserialize, Clone, Hash)]
#[derive(Debug, Deserialize, Clone)]
struct MinifiedWebhook {
path: String,
description: Option<String>,
@@ -814,7 +922,7 @@ async fn webhook_to_future_paths(
let webhook_scripts = sqlx::query_as!(
MinifiedWebhook,
r#"SELECT
r#"SELECT
path,
summary,
description
@@ -833,7 +941,7 @@ async fn webhook_to_future_paths(
let webhook_flows = sqlx::query_as!(
MinifiedWebhook,
r#"SELECT
r#"SELECT
path,
summary,
description
@@ -853,6 +961,7 @@ async fn webhook_to_future_paths(
openapi_future_paths.reserve_exact(webhook_scripts.len() + webhook_flows.len());
for webhook in webhook_scripts {
let args_schema = get_runnable_schema(db, w_id, &webhook.path, false).await;
openapi_future_paths.push(FuturePath::new(
webhook.path,
Kind::Webhook(WebhookConfig::new(RunnableKind::Script)),
@@ -860,10 +969,12 @@ async fn webhook_to_future_paths(
webhook.summary,
webhook.description,
Some(SecurityScheme::BearerJwt),
args_schema,
));
}
for webhook in webhook_flows {
let args_schema = get_runnable_schema(db, w_id, &webhook.path, true).await;
openapi_future_paths.push(FuturePath::new(
webhook.path,
Kind::Webhook(WebhookConfig::new(RunnableKind::Flow)),
@@ -871,6 +982,7 @@ async fn webhook_to_future_paths(
webhook.summary,
webhook.description,
Some(SecurityScheme::BearerJwt),
args_schema,
));
}
}
@@ -898,7 +1010,7 @@ async fn generate_openapi_future_path(
http_routes_to_future_paths(db, user_db, authed, &mut tx, http_route_filters, w_id).await?;
openapi_future_paths
.append(&mut webhook_to_future_paths(&mut tx, webhook_filters, w_id).await?);
.append(&mut webhook_to_future_paths(db, &mut tx, webhook_filters, w_id).await?);
tx.commit().await?;
backend/windmill-api-scripts/src/scripts.rs
+142
View File
@@ -239,6 +239,9 @@ pub fn workspaced_service() -> Router {
"/history_update/h/:hash/p/*path",
post(update_script_history),
)
// Temporary raw script storage for CLI lock generation
.route("/raw_temp/store", post(store_raw_script_temp))
.route("/raw_temp/diff", post(diff_raw_scripts_with_deployed))
}
#[derive(Serialize, FromRow)]
@@ -1614,6 +1617,9 @@ struct RawScriptByPathQuery {
cache_key: Option<String>,
// used specifically for python to cache folders on import success to avoid extra db calls on package fetch
cache_folders: Option<bool>,
// If provided, load content from raw_script_temp table using this hash instead of deployed script.
// Used by CLI lock generation to resolve imports from not-yet-deployed scripts.
temp_script_hash: Option<String>,
}
struct StringWithLength(String);
@@ -1672,6 +1678,16 @@ async fn raw_script_by_path_internal(
) -> Result<String> {
let path = path.to_path();
check_scopes(&authed, || format!("scripts:read:{}", path))?;
// If temp_script_hash is provided, try loading from temp storage first.
// This is used by CLI lock generation to resolve imports from not-yet-deployed scripts.
// Falls back to the normal deployed script lookup if not found in temp storage.
if let Some(hash) = query.temp_script_hash {
if let Ok(content) = windmill_common::cache::raw_script_temp::load(hash, &db).await {
return Ok(content);
}
}
let cache_path = query
.cache_key
.map(|x| format!("{w_id}:{path}:{x}{}", if unpin { ":unpinned" } else { "" }));
@@ -2463,3 +2479,129 @@ async fn guard_script_from_debounce_data(ns: &NewScript) -> Result<()> {
Ok(())
}
}
// ============================================================================
// Temporary Raw Script Storage for CLI Lock Generation
// ============================================================================
/// Store raw script content temporarily for CLI lock generation.
async fn store_raw_script_temp(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(content): Json<String>,
) -> Result<Json<String>> {
check_scopes(&authed, || "scripts:write".to_string())?;
let hash = windmill_common::cache::raw_script_temp::compute_hash(&w_id, &content);
// Store to DB
sqlx::query!(
"INSERT INTO raw_script_temp (workspace_id, hash, content, created_at)
VALUES ($1, $2, $3, NOW())
ON CONFLICT (workspace_id, hash) DO UPDATE SET created_at = NOW()",
&w_id,
&hash,
&content
)
.execute(&db)
.await?;
// Clean up old entries (1 week TTL)
sqlx::query!(
"DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'"
)
.execute(&db)
.await?;
Ok(Json(hash))
}
/// Compare local script content hashes with deployed versions.
/// Receives a map of path → SHA256(content), returns paths where the hash
/// differs from the deployed script (or the script doesn't exist on remote).
/// Hash comparison is done entirely in Postgres to avoid transferring content.
#[derive(Deserialize)]
struct WorkspaceDepDiff {
path: String,
language: ScriptLang,
name: Option<String>,
hash: String,
}
#[derive(Deserialize)]
struct DiffRequest {
scripts: std::collections::HashMap<String, String>,
#[serde(default)]
workspace_deps: Vec<WorkspaceDepDiff>,
}
async fn diff_raw_scripts_with_deployed(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(req): Json<DiffRequest>,
) -> Result<Json<Vec<String>>> {
check_scopes(&authed, || "scripts:read".to_string())?;
let mut matching_set: std::collections::HashSet<String> = std::collections::HashSet::new();
let mut all_paths: Vec<String> = Vec::new();
// --- Scripts ---
if !req.scripts.is_empty() {
let paths: Vec<String> = req.scripts.keys().cloned().collect();
let hashes: Vec<String> = paths.iter().map(|p| req.scripts[p].clone()).collect();
let matching: Vec<String> = sqlx::query_scalar(
"SELECT local.path FROM \
unnest($1::text[], $2::text[]) AS local(path, hash) \
INNER JOIN LATERAL ( \
SELECT encode(sha256(convert_to(s.content, 'UTF8')), 'hex') AS deployed_hash \
FROM script s \
WHERE s.path = local.path AND s.workspace_id = $3 AND s.archived = false \
ORDER BY s.created_at DESC LIMIT 1 \
) deployed ON deployed.deployed_hash = local.hash"
)
.bind(&paths)
.bind(&hashes)
.bind(&w_id)
.fetch_all(&db)
.await?;
matching_set.extend(matching);
all_paths.extend(paths);
}
// --- Workspace dependencies ---
for dep in &req.workspace_deps {
let matching: Option<String> = sqlx::query_scalar(
"SELECT $1::text \
WHERE EXISTS ( \
SELECT 1 FROM workspace_dependencies wd \
WHERE wd.workspace_id = $2 AND wd.archived = false \
AND wd.language = $3::SCRIPT_LANG \
AND wd.name IS NOT DISTINCT FROM $4 \
AND encode(sha256(convert_to(wd.content, 'UTF8')), 'hex') = $5 \
)"
)
.bind(&dep.path)
.bind(&w_id)
.bind(dep.language.as_str())
.bind(&dep.name)
.bind(&dep.hash)
.fetch_optional(&db)
.await?;
if let Some(path) = matching {
matching_set.insert(path);
}
all_paths.push(dep.path.clone());
}
let mismatched: Vec<String> = all_paths
.into_iter()
.filter(|p| !matching_set.contains(p))
.collect();
Ok(Json(mismatched))
}
backend/windmill-api/openapi.yaml
+90 -1
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.662.0
version: 1.663.0
title: Windmill API
contact:
@@ -6988,6 +6988,83 @@ paths:
type: string
format: uuid
/w/{workspace}/scripts/raw_temp/store:
post:
summary: store raw script content temporarily for CLI lock generation
operationId: storeRawScriptTemp
tags:
- script
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
description: script content to store
required: true
content:
application/json:
schema:
type: string
responses:
"200":
description: hash of stored content
content:
application/json:
schema:
type: string
/w/{workspace}/scripts/raw_temp/diff:
post:
summary: diff local script hashes against deployed versions
operationId: diffRawScriptsWithDeployed
tags:
- script
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
description: scripts and workspace deps to diff against deployed versions
required: true
content:
application/json:
schema:
type: object
required:
- scripts
properties:
scripts:
description: map of script path to SHA256 content hash
type: object
additionalProperties:
type: string
workspace_deps:
description: workspace dependencies to diff
type: array
items:
type: object
required:
- path
- language
- hash
properties:
path:
description: CLI path (e.g. dependencies/package.json)
type: string
language:
$ref: "#/components/schemas/ScriptLang"
name:
description: named workspace dependency (null for default)
type: string
hash:
description: SHA256 content hash
type: string
responses:
"200":
description: list of paths that differ from deployed versions
content:
application/json:
schema:
type: array
items:
type: string
/w/{workspace}/jobs/list_selected_job_groups:
# We use post because sending a huge array as a query param can produce
# URLs that may be too long
@@ -24515,6 +24592,10 @@ components:
type: string
nullable: true
description: Error message if the trigger is in an error state
summary:
type: string
nullable: true
description: Short summary to be displayed when listed
required:
- external_id
- workspace_id
@@ -24549,6 +24630,10 @@ components:
type: string
nullable: true
description: Error message if the trigger is in an error state
summary:
type: string
nullable: true
description: Short summary to be displayed when listed
external_data:
type: object
description: Configuration data from the external service
@@ -24641,6 +24726,10 @@ components:
type: object
description: Service-specific configuration (e.g., event types, filters)
additionalProperties: true
summary:
type: string
nullable: true
description: Short summary to be displayed when listed
required:
- script_path
- is_flow
backend/windmill-api/src/apps.rs
+1
View File
@@ -2053,6 +2053,7 @@ async fn execute_component(
.triggerables_v2
.as_ref()
.ok_or_else(|| Error::BadRequest(format!("Policy is missing triggerables")))?;
let policy_triggerables = triggerables_v2
.get(path) // start with `path` in case we can avoid the next` format!`.
.or_else(|| triggerables_v2.get(&format!("{}:{}", payload.component, &path)))
backend/windmill-api/src/internal_db.rs
+37
View File
@@ -0,0 +1,37 @@
use crate::db::ApiAuthed;
use axum::{extract::Path, routing::post, Json, Router};
use serde::{Deserialize, Serialize};
use windmill_common::{
error::{Error, Result},
query_builders::try_expand_internal_db_query,
scripts::ScriptLang,
};
pub fn workspaced_service() -> Router {
Router::new().route("/expand_marker", post(expand_marker))
}
#[derive(Deserialize)]
struct ExpandMarkerRequest {
language: ScriptLang,
content: String,
}
#[derive(Serialize)]
struct ExpandMarkerResponse {
code: String,
}
async fn expand_marker(
_authed: ApiAuthed,
Path(_w_id): Path<String>,
Json(req): Json<ExpandMarkerRequest>,
) -> Result<Json<ExpandMarkerResponse>> {
match try_expand_internal_db_query(&req.content, &req.language) {
Some(Ok(expanded)) => Ok(Json(ExpandMarkerResponse { code: expanded.code })),
Some(Err(msg)) => Err(Error::BadRequest(msg)),
None => Err(Error::BadRequest(
"Content is not a WM_INTERNAL_DB marker".to_string(),
)),
}
}
backend/windmill-api/src/jobs.rs
+13 -4
View File
@@ -2027,10 +2027,7 @@ async fn count_completed_jobs_detail(
if let Some(tags) = query.tags {
sqlb.and_where_in(
"v2_job.tag",
&tags
.split(",")
.map(|t| format!("'{}'", t))
.collect::<Vec<_>>(),
&tags.split(",").map(|t| quote(t)).collect::<Vec<_>>(),
);
}
@@ -5074,6 +5071,10 @@ pub struct RunDependenciesRequest {
pub raw_workspace_dependencies: Option<RawWorkspaceDependencies>,
#[serde(default)]
pub raw_deps: Option<String>,
/// Map of script path -> content hash for resolving imports from temp storage.
/// Used by CLI to provide local script content during lock generation.
#[serde(default)]
pub temp_script_refs: Option<HashMap<String, String>>,
}
#[derive(Deserialize, Clone, Debug)]
@@ -5133,6 +5134,8 @@ async fn run_dependencies_job(
let mut hm = HashMap::new();
req.raw_workspace_dependencies
.map(|v| hm.insert("raw_workspace_dependencies".to_owned(), to_raw_value(&v)));
req.temp_script_refs
.map(|v| hm.insert("temp_script_refs".to_owned(), to_raw_value(&v)));
let (uuid, tx) = push(
&db,
@@ -5183,6 +5186,8 @@ pub struct RunFlowDependenciesRequest {
pub raw_workspace_dependencies: Option<RawWorkspaceDependencies>,
#[serde(default)]
pub raw_deps: Option<HashMap<String, String>>,
#[serde(default)]
pub temp_script_refs: Option<HashMap<String, String>>,
}
#[derive(Serialize)]
@@ -5226,6 +5231,10 @@ async fn run_flow_dependencies_job(
req.raw_workspace_dependencies
.map(|v| args_map.insert("raw_workspace_dependencies".to_string(), to_raw_value(&v)));
// Add temp_script_refs to args if present (for CLI local import resolution)
req.temp_script_refs
.map(|v| args_map.insert("temp_script_refs".to_string(), to_raw_value(&v)));
let (uuid, tx) = push(
&db,
PushIsolationLevel::IsolatedRoot(db.clone()),
backend/windmill-api/src/lib.rs
+2
View File
@@ -98,6 +98,7 @@ mod indexer_oss;
mod inkeep_ee;
mod inkeep_oss;
mod integration;
mod internal_db;
mod live_migrations;
#[cfg(all(feature = "private", feature = "parquet"))]
pub mod s3_proxy_ee;
@@ -554,6 +555,7 @@ pub async fn run_server(
.nest("/groups", groups::workspaced_service())
.nest("/groups_history", group_history::workspaced_service())
.nest("/inputs", windmill_api_inputs::workspaced_service())
.nest("/internal_db", internal_db::workspaced_service())
.nest("/job_metrics", job_metrics::workspaced_service())
.nest("/job_helpers", job_helpers_service)
.nest("/jobs", jobs::workspaced_service())
backend/windmill-common/src/cache.rs
+40 -6
View File
@@ -286,16 +286,17 @@ pub struct FlowData {
}
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct FlowNotes {
pub struct FlowExtras {
pub notes: Option<Box<RawValue>>,
pub groups: Option<Box<RawValue>>,
}
impl FlowData {
pub fn notes(&self) -> Option<FlowNotes> {
serde_json::from_str::<FlowNotes>(self.raw_flow.get())
pub fn extras(&self) -> Option<FlowExtras> {
serde_json::from_str::<FlowExtras>(self.raw_flow.get())
.map_err(|e| {
tracing::error!("Failed to parse notes into FlowNotes: {}", e);
error::Error::internal_err(format!("Failed to parse notes into FlowNotes: {}", e))
tracing::error!("Failed to parse flow extras: {}", e);
error::Error::internal_err(format!("Failed to parse flow extras: {}", e))
})
.ok()
}
@@ -991,6 +992,38 @@ pub mod workspace_dependencies {
}
}
/// Temporary raw script content cache for CLI lock generation.
pub mod raw_script_temp {
use super::*;
use crate::DB;
make_static! {
static ref CACHE: { String => String } in "raw_script_temp" <= 10000;
}
/// Compute hash for raw script content (includes workspace_id for isolation).
pub fn compute_hash(workspace_id: &str, content: &str) -> String {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(workspace_id.as_bytes());
hasher.update(content.as_bytes());
format!("{:x}", hasher.finalize())
}
/// Load content from cache, falling back to DB.
pub fn load(hash: String, db: &DB) -> impl Future<Output = error::Result<String>> + '_ {
CACHE.get_or_insert_async(hash.clone(), async move {
sqlx::query_scalar!(
"SELECT content FROM raw_script_temp WHERE hash = $1",
&hash
)
.fetch_optional(db)
.await?
.ok_or_else(|| error::Error::NotFound(format!("raw_script_temp hash: {}", hash)))
})
}
}
const _: () = {
impl Import for RawFlow {
fn import(src: &impl Storage) -> error::Result<Self> {
@@ -1183,7 +1216,8 @@ const _: () = {
((u8, ScriptHash), |x| format!("{:02x}-{:016x}", x.0, x.1.0)),
(FlowNodeId, |x| format!("{:016x}", x.0)),
(AppScriptId, |x| format!("{:016x}", x.0)),
((i64, String), |x| format!("{}-{}", x.1, x.0))
((i64, String), |x| format!("{}-{}", x.1, x.0)),
(String, |x| x.as_str())
}
#[cfg(feature = "scoped_cache")]
backend/windmill-common/src/lib.rs
+1
View File
@@ -77,6 +77,7 @@ pub mod oidc_oss;
#[cfg(feature = "private")]
pub mod otel_ee;
pub mod otel_oss;
pub mod query_builders;
pub mod queue;
pub mod result_stream;
pub mod runnable_settings;
backend/windmill-common/src/min_version.rs
+1
View File
@@ -5,6 +5,7 @@ use tokio::sync::RwLock;
// ============ Feature Definitions ============
pub const MIN_VERSION_SUPPORTS_NODE_DEBOUNCING: VC = vc(1, 658, 0, "Flow node debouncing");
pub const MIN_VERSION_SUPPORTS_TOKEN_HASH: VC = vc(1, 659, 0, "Token hash storage");
pub const MIN_VERSION_SUPPORTS_SYNC_JOBS_DEBOUNCING: VC = vc(1, 602, 0, "Sync jobs debouncing");
pub const MIN_VERSION_SUPPORTS_DEBOUNCING_V2: VC = vc(1, 597, 0, "Debouncing V2");
backend/windmill-common/src/query_builders.rs
+4465
View File
File diff suppressed because it is too large Load Diff
backend/windmill-common/src/tracing_init.rs
+5 -1
View File
@@ -68,7 +68,11 @@ pub fn initialize_tracing(
mode: &Mode,
environment: &str,
) -> (WorkerGuard, crate::otel_oss::OtelProvider) {
let style = std::env::var("RUST_LOG_STYLE").unwrap_or_else(|_| "auto".into());
let style = if std::env::var("NO_COLOR").is_ok() {
"never".into()
} else {
std::env::var("RUST_LOG_STYLE").unwrap_or_else(|_| "auto".into())
};
let rust_log_env = std::env::var("RUST_LOG");
let rust_log_stdout_env = std::env::var("RUST_LOG_STDOUT");
backend/windmill-dep-map/src/trigger_dependents.rs
+1 -1
View File
@@ -68,7 +68,7 @@ pub async fn trigger_dependents_to_recompute_dependencies(
let job_payload = match importer_kind.as_str() {
"script" => match sqlx::query_scalar!(
"SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false ORDER BY created_at DESC LIMIT 1",
"SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false AND archived = false ORDER BY created_at DESC LIMIT 1",
importer_path,
w_id
)
backend/windmill-native-triggers/src/handler.rs
+2
View File
@@ -181,6 +181,7 @@ async fn create_native_trigger<T: External>(
&external_id,
&config,
service_config,
data.summary.as_deref(),
)
.await?;
@@ -304,6 +305,7 @@ async fn update_native_trigger_handler<T: External>(
&external_id,
&config,
service_config,
data.summary.as_deref(),
)
.await?;
backend/windmill-native-triggers/src/lib.rs
+17 -7
View File
@@ -195,6 +195,7 @@ pub struct NativeTrigger {
pub error: Option<String>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub summary: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -209,6 +210,7 @@ pub struct NativeTriggerData<C> {
pub script_path: String,
pub is_flow: bool,
pub service_config: C,
pub summary: Option<String>,
}
#[derive(Debug, Clone, FromRow, Serialize, Deserialize)]
@@ -821,6 +823,7 @@ pub async fn store_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres>
external_id: &str,
config: &NativeTriggerConfig,
service_config: C,
summary: Option<&str>,
) -> Result<()> {
use windmill_common::auth::hash_token;
@@ -835,12 +838,13 @@ pub async fn store_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres>
script_path,
is_flow,
webhook_token_hash,
service_config
service_config,
summary
) VALUES (
$1, $2, $3, $4, $5, $6, $7
$1, $2, $3, $4, $5, $6, $7, $8
)
ON CONFLICT (external_id, workspace_id, service_name)
DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, error = NULL, updated_at = NOW()
DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, summary = $8, error = NULL, updated_at = NOW()
"#,
external_id,
workspace_id,
@@ -849,6 +853,7 @@ pub async fn store_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres>
config.is_flow,
webhook_token_hash,
sqlx::types::Json(service_config) as _,
summary,
)
.execute(db)
.await?;
@@ -863,6 +868,7 @@ pub async fn update_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres
external_id: &str,
config: &NativeTriggerConfig,
service_config: Option<&RawValue>,
summary: Option<&str>,
) -> Result<()> {
use windmill_common::auth::hash_token;
@@ -871,7 +877,7 @@ pub async fn update_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres
sqlx::query!(
r#"
UPDATE native_trigger
SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, error = NULL, updated_at = NOW()
SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, summary = $8, error = NULL, updated_at = NOW()
WHERE
workspace_id = $5
AND service_name = $6
@@ -884,6 +890,7 @@ pub async fn update_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres
workspace_id,
service_name as ServiceName,
external_id,
summary,
)
.execute(db)
.await?;
@@ -934,7 +941,8 @@ pub async fn get_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres>>(
service_config,
error,
created_at,
updated_at
updated_at,
summary
FROM
native_trigger
WHERE
@@ -972,7 +980,8 @@ pub async fn get_native_trigger_by_script<'c, E: sqlx::Executor<'c, Database = P
service_config,
error,
created_at,
updated_at
updated_at,
summary
FROM
native_trigger
WHERE
@@ -1018,7 +1027,8 @@ pub async fn list_native_triggers<'c, E: sqlx::Executor<'c, Database = Postgres>
nt.service_config,
nt.error,
nt.created_at,
nt.updated_at
nt.updated_at,
nt.summary
FROM
native_trigger nt
WHERE
backend/windmill-object-store/src/lib.rs
+106 -86
View File
@@ -65,8 +65,8 @@ pub mod object_store_reexports {
pub use object_store::memory::InMemory;
pub use object_store::path::Path;
pub use object_store::{
Attribute, Attributes, Error as ObjectStoreError, GetResult, ObjectStore,
PutMultipartOpts, PutPayload, PutResult, Result as ObjectStoreResult, WriteMultipart,
Attribute, Attributes, Error as ObjectStoreError, GetResult, ObjectStore, PutMultipartOpts,
PutPayload, PutResult, Result as ObjectStoreResult, WriteMultipart,
};
}
@@ -530,10 +530,7 @@ async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result<Arc<d
#[cfg(feature = "parquet")]
pub fn build_filesystem_client(root_path: &str) -> error::Result<Arc<dyn ObjectStore>> {
let store = object_store::local::LocalFileSystem::new_with_prefix(root_path).map_err(|e| {
error::Error::internal_err(format!(
"Error building filesystem object store: {:?}",
e
))
error::Error::internal_err(format!("Error building filesystem object store: {:?}", e))
})?;
Ok(Arc::new(store))
}
@@ -644,36 +641,45 @@ lazy_static::lazy_static! {
static ref S3_BUCKET_RESTRICTIONS: Option<HashMap<String, Vec<String>>> = {
parse_bucket_restrictions()
};
static ref AZ_ACCOUNT_NAME_RESTRICTIONS: Option<HashMap<String, Vec<String>>> = {
parse_az_account_name_restrictions()
};
}
fn parse_bucket_restrictions() -> Option<HashMap<String, Vec<String>>> {
let env_var = std::env::var("S3_BUCKETS_WORKSPACE_RESTRICTIONS").ok()?;
parse_bucket_restrictions_from_str(&env_var)
parse_restrictions_from_str(&env_var, "S3 bucket")
}
fn parse_bucket_restrictions_from_str(input: &str) -> Option<HashMap<String, Vec<String>>> {
fn parse_az_account_name_restrictions() -> Option<HashMap<String, Vec<String>>> {
let env_var = std::env::var("AZ_ACCOUNT_NAME_WORKSPACE_RESTRICTIONS").ok()?;
parse_restrictions_from_str(&env_var, "Azure account name")
}
fn parse_restrictions_from_str(input: &str, label: &str) -> Option<HashMap<String, Vec<String>>> {
if input.trim().is_empty() {
return None;
}
let mut restrictions = HashMap::new();
for bucket_rule in input.split(';') {
let bucket_rule = bucket_rule.trim();
if bucket_rule.is_empty() {
for rule in input.split(';') {
let rule = rule.trim();
if rule.is_empty() {
continue;
}
let parts: Vec<&str> = bucket_rule.splitn(2, ':').collect();
let parts: Vec<&str> = rule.splitn(2, ':').collect();
if parts.len() != 2 {
tracing::warn!(
"Invalid bucket restriction format: '{}'. Expected 'bucket:workspace1,workspace2'",
bucket_rule
"Invalid {} restriction format: '{}'. Expected 'name:workspace1,workspace2'",
label,
rule
);
continue;
}
let bucket_name = parts[0].trim().to_string();
let name = parts[0].trim().to_string();
let workspaces: Vec<String> = parts[1]
.split(',')
.map(|w| w.trim().to_string())
@@ -682,26 +688,33 @@ fn parse_bucket_restrictions_from_str(input: &str) -> Option<HashMap<String, Vec
if workspaces.is_empty() {
tracing::warn!(
"No workspaces specified for bucket '{}', skipping restriction",
bucket_name
"No workspaces specified for {} '{}', skipping restriction",
label,
name
);
continue;
}
restrictions.insert(bucket_name, workspaces);
restrictions.insert(name, workspaces);
}
if restrictions.is_empty() {
None
} else {
tracing::info!(
"S3 bucket restrictions loaded for {} buckets",
"{} restrictions loaded for {} entries",
label,
restrictions.len()
);
Some(restrictions)
}
}
#[cfg(test)]
fn parse_bucket_restrictions_from_str(input: &str) -> Option<HashMap<String, Vec<String>>> {
parse_restrictions_from_str(input, "S3 bucket")
}
pub fn check_bucket_workspace_restriction(
bucket_name: &str,
workspace_id: &str,
@@ -719,6 +732,23 @@ pub fn check_bucket_workspace_restriction(
Ok(())
}
pub fn check_az_account_name_workspace_restriction(
account_name: &str,
workspace_id: &str,
) -> error::Result<()> {
if let Some(ref restrictions) = *AZ_ACCOUNT_NAME_RESTRICTIONS {
if let Some(allowed_workspaces) = restrictions.get(account_name) {
if !allowed_workspaces.contains(&workspace_id.to_string()) {
return Err(error::Error::NotAuthorized(format!(
"Workspace '{}' is not authorized to access Azure account '{}'",
workspace_id, account_name
)));
}
}
}
Ok(())
}
pub const DEFAULT_STORAGE: &str = "_default_";
pub fn bundle(w_id: &str, hash: &str) -> String {
@@ -739,7 +769,8 @@ pub async fn upload_artifact_to_store(
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
let object_store: Option<()> = None;
Ok(
if &windmill_common::utils::MODE_AND_ADDONS.mode == &windmill_common::utils::Mode::Standalone
if &windmill_common::utils::MODE_AND_ADDONS.mode
== &windmill_common::utils::Mode::Standalone
&& object_store.is_none()
{
let path = format!("{}/{}", standalone_dir, path);
@@ -821,11 +852,11 @@ pub fn lfs_to_object_store_resource(
})?;
Ok(ObjectStoreResource::Gcs(gcs_resource))
}
LargeFileStorage::FilesystemStorage(fs) => Ok(ObjectStoreResource::Filesystem(
FilesystemSettings {
LargeFileStorage::FilesystemStorage(fs) => {
Ok(ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: fs.root_path.clone(),
},
)),
}))
}
}
}
@@ -995,13 +1026,9 @@ pub async fn convert_json_line_stream<E: Into<anyhow::Error>>(
drop(file);
let ctx = SessionContext::new();
ctx.register_json(
"my_table",
path_str,
NdJsonReadOptions::default(),
)
.await
.map_err(to_anyhow)?;
ctx.register_json("my_table", path_str, NdJsonReadOptions::default())
.await
.map_err(to_anyhow)?;
let df = ctx.sql("SELECT * FROM my_table").await.map_err(to_anyhow)?;
let schema = df.schema().clone().into();
@@ -1252,8 +1279,7 @@ mod tests {
advanced_permissions: None,
});
// resource_value is ignored for filesystem
let result =
lfs_to_object_store_resource(&lfs, serde_json::Value::Null).unwrap();
let result = lfs_to_object_store_resource(&lfs, serde_json::Value::Null).unwrap();
match result {
ObjectStoreResource::Filesystem(fs) => {
assert_eq!(fs.root_path, "/tmp/mydata");
@@ -1279,10 +1305,18 @@ mod tests {
port: None,
};
let result = duckdb_connection_settings_internal(s3).unwrap();
assert!(result.connection_settings_str.contains("SET s3_region='eu-west-1'"));
assert!(result.connection_settings_str.contains("SET s3_access_key_id='AKIA123'"));
assert!(result.connection_settings_str.contains("SET s3_secret_access_key='secret456'"));
assert!(result.connection_settings_str.contains("SET s3_url_style='path'"));
assert!(result
.connection_settings_str
.contains("SET s3_region='eu-west-1'"));
assert!(result
.connection_settings_str
.contains("SET s3_access_key_id='AKIA123'"));
assert!(result
.connection_settings_str
.contains("SET s3_secret_access_key='secret456'"));
assert!(result
.connection_settings_str
.contains("SET s3_url_style='path'"));
assert!(!result.connection_settings_str.contains("SET s3_use_ssl=0"));
assert_eq!(result.s3_bucket, Some("test-bucket".to_string()));
assert!(result.azure_container_path.is_none());
@@ -1304,7 +1338,9 @@ mod tests {
};
let result = duckdb_connection_settings_internal(s3).unwrap();
assert!(result.connection_settings_str.contains("SET s3_use_ssl=0"));
assert!(!result.connection_settings_str.contains("SET s3_url_style='path'"));
assert!(!result
.connection_settings_str
.contains("SET s3_url_style='path'"));
}
#[test]
@@ -1320,8 +1356,12 @@ mod tests {
federated_token_file: None,
});
let result = format_duckdb_connection_settings(resource).unwrap();
assert!(result.connection_settings_str.contains("AccountName=myaccount"));
assert!(result.connection_settings_str.contains("AccountKey=base64key=="));
assert!(result
.connection_settings_str
.contains("AccountName=myaccount"));
assert!(result
.connection_settings_str
.contains("AccountKey=base64key=="));
assert_eq!(
result.azure_container_path,
Some("az://mycontainer".to_string())
@@ -1337,7 +1377,10 @@ mod tests {
});
let result = format_duckdb_connection_settings(resource);
assert!(result.is_err());
assert!(result.unwrap_err().to_string().contains("GCS is not supported"));
assert!(result
.unwrap_err()
.to_string()
.contains("GCS is not supported"));
}
#[test]
@@ -1361,10 +1404,8 @@ mod tests {
use windmill_common::error::Error;
// NotFound
let err = object_store::Error::NotFound {
path: "test/path".into(),
source: "missing".into(),
};
let err =
object_store::Error::NotFound { path: "test/path".into(), source: "missing".into() };
let mapped = object_store_error_to_error(err);
assert!(matches!(mapped, Error::NotFound(_)));
@@ -1388,10 +1429,8 @@ mod tests {
assert!(matches!(mapped, Error::BadRequest(_)));
// Unauthenticated
let err = object_store::Error::Unauthenticated {
path: "obj".into(),
source: "no creds".into(),
};
let err =
object_store::Error::Unauthenticated { path: "obj".into(), source: "no creds".into() };
let mapped = object_store_error_to_error(err);
assert!(matches!(mapped, Error::NotAuthorized(_)));
}
@@ -1533,15 +1572,10 @@ mod tests {
.await
.unwrap();
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: root.to_string(),
});
let s3_obj = S3Object {
s3: "etag.txt".to_string(),
storage: None,
filename: None,
presigned: None,
};
let resource =
ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root.to_string() });
let s3_obj =
S3Object { s3: "etag.txt".to_string(), storage: None, filename: None, presigned: None };
let etag = get_etag_or_empty(&resource, s3_obj).await;
// LocalFileSystem should return an etag based on file metadata
@@ -1554,9 +1588,8 @@ mod tests {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().to_str().unwrap();
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: root.to_string(),
});
let resource =
ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root.to_string() });
let s3_obj = S3Object {
s3: "nonexistent.txt".to_string(),
storage: None,
@@ -1575,9 +1608,7 @@ mod tests {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().to_str().unwrap().to_string();
let settings = ObjectSettings::Filesystem(FilesystemSettings {
root_path: root,
});
let settings = ObjectSettings::Filesystem(FilesystemSettings { root_path: root });
let expirable = build_object_store_from_settings(settings, None)
.await
@@ -1585,10 +1616,7 @@ mod tests {
let data = bytes::Bytes::from("end to end via settings");
expirable
.store
.put(
&Path::from("e2e.txt"),
PutPayload::from(data.clone()),
)
.put(&Path::from("e2e.txt"), PutPayload::from(data.clone()))
.await
.unwrap();
let result = expirable
@@ -1610,9 +1638,7 @@ mod tests {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().to_str().unwrap().to_string();
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: root,
});
let resource = ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root });
let client = build_object_store_client(&resource).await.unwrap();
let data = bytes::Bytes::from("end to end via resource");
@@ -1634,7 +1660,10 @@ mod tests {
#[test]
fn test_bundle_path_format() {
assert_eq!(bundle("my_workspace", "abc123"), "script_bundle/my_workspace/abc123");
assert_eq!(
bundle("my_workspace", "abc123"),
"script_bundle/my_workspace/abc123"
);
}
#[test]
@@ -1646,8 +1675,7 @@ mod tests {
#[test]
fn test_parse_bucket_restrictions_single_bucket() {
let result =
parse_bucket_restrictions_from_str("my-bucket:workspace1,workspace2").unwrap();
let result = parse_bucket_restrictions_from_str("my-bucket:workspace1,workspace2").unwrap();
assert_eq!(result.len(), 1);
assert_eq!(
result.get("my-bucket").unwrap(),
@@ -1657,19 +1685,13 @@ mod tests {
#[test]
fn test_parse_bucket_restrictions_multiple_buckets() {
let result = parse_bucket_restrictions_from_str(
"bucket-a:ws1,ws2;bucket-b:ws3",
)
.unwrap();
let result = parse_bucket_restrictions_from_str("bucket-a:ws1,ws2;bucket-b:ws3").unwrap();
assert_eq!(result.len(), 2);
assert_eq!(
result.get("bucket-a").unwrap(),
&vec!["ws1".to_string(), "ws2".to_string()]
);
assert_eq!(
result.get("bucket-b").unwrap(),
&vec!["ws3".to_string()]
);
assert_eq!(result.get("bucket-b").unwrap(), &vec!["ws3".to_string()]);
}
#[test]
@@ -1681,16 +1703,14 @@ mod tests {
#[test]
fn test_parse_bucket_restrictions_invalid_format_skipped() {
// "no-colon" is invalid, only "valid:ws1" should be parsed
let result =
parse_bucket_restrictions_from_str("no-colon;valid:ws1").unwrap();
let result = parse_bucket_restrictions_from_str("no-colon;valid:ws1").unwrap();
assert_eq!(result.len(), 1);
assert!(result.contains_key("valid"));
}
#[test]
fn test_parse_bucket_restrictions_trailing_semicolons() {
let result =
parse_bucket_restrictions_from_str(";bucket:ws1;;").unwrap();
let result = parse_bucket_restrictions_from_str(";bucket:ws1;;").unwrap();
assert_eq!(result.len(), 1);
assert!(result.contains_key("bucket"));
}
backend/windmill-queue/src/jobs.rs
+37 -32
View File
@@ -3100,41 +3100,46 @@ impl PulledJobResult {
"Accumulated arguments from debounced jobs in batch"
);
let new_value = to_raw_value(&accumulated_arg);
// If the batch query returned no entries (e.g. CE where
// v2_job_debounce_batch is never populated), keep the
// original value unchanged instead of replacing it with [].
if !accumulated_arg.is_empty() {
let new_value = to_raw_value(&accumulated_arg);
let original_value = j
.args
.as_ref()
.and_then(|a| a.get(arg_name_to_accumulate))
.map(|v| v.get().to_string())
.unwrap_or_else(|| "null".to_string());
let original_value = j
.args
.as_ref()
.and_then(|a| a.get(arg_name_to_accumulate))
.map(|v| v.get().to_string())
.unwrap_or_else(|| "null".to_string());
append_logs(
&j_id,
&j.workspace_id,
format!(
"Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n",
&new_value
),
&(db.into()),
)
.await;
j.args
.get_or_insert(Json(Default::default()))
.as_mut()
.insert(arg_name_to_accumulate.to_owned(), new_value);
// Persist accumulated args to v2_job so that flow steps
// re-reading from the DB (via get_mini_pulled_job) see them
if let Some(ref args) = j.args {
sqlx::query!(
"UPDATE v2_job SET args = $2 WHERE id = $1",
j_id,
args as &Json<HashMap<String, Box<RawValue>>>,
append_logs(
&j_id,
&j.workspace_id,
format!(
"Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n",
&new_value
),
&(db.into()),
)
.execute(db)
.await?;
.await;
j.args
.get_or_insert(Json(Default::default()))
.as_mut()
.insert(arg_name_to_accumulate.to_owned(), new_value);
// Persist accumulated args to v2_job so that flow steps
// re-reading from the DB (via get_mini_pulled_job) see them
if let Some(ref args) = j.args {
sqlx::query!(
"UPDATE v2_job SET args = $2 WHERE id = $1",
j_id,
args as &Json<HashMap<String, Box<RawValue>>>,
)
.execute(db)
.await?;
}
}
}
backend/windmill-queue/tests/debounce_test.rs
+317
View File
@@ -4429,4 +4429,321 @@ mod debounce {
Ok(())
}
// =========================================================================
// Tests for maybe_debounce_flow_node (flow node debouncing)
// =========================================================================
/// Helper: insert a child job with a parent flow.
async fn insert_child_job_with_parent(
db: &Pool<Postgres>,
child_id: Uuid,
parent_id: Uuid,
workspace_id: &str,
) {
sqlx::query!(
"INSERT INTO v2_job (id, kind, tag, created_by, permissioned_as, permissioned_as_email, workspace_id, parent_job)
VALUES ($1, 'noop', 'deno', 'test-user', 'u/test-user', 'test@windmill.dev', $2, $3)",
child_id,
workspace_id,
parent_id,
)
.execute(db)
.await
.expect("insert v2_job (child)");
sqlx::query!(
"INSERT INTO v2_job_queue (id, workspace_id, scheduled_for, tag)
VALUES ($1, $2, now(), 'deno')",
child_id,
workspace_id,
)
.execute(db)
.await
.expect("insert v2_job_queue (child)");
sqlx::query!("INSERT INTO v2_job_runtime (id) VALUES ($1)", child_id)
.execute(db)
.await
.expect("insert v2_job_runtime (child)");
}
/// Test: First flow node job in a debounce batch should set scheduled_for
/// and create a debounce_key entry. The parent flow should remain in queue.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_first_job(db: Pool<Postgres>) -> anyhow::Result<()> {
let flow_id = Uuid::new_v4();
let child_id = Uuid::new_v4();
insert_flow_job(&db, flow_id, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: Some("test_flow_node_first".to_string()),
..Default::default()
};
let args_hm = empty_args();
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child_id,
flow_id,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
// Child job should still be in queue with delayed scheduled_for
assert!(is_queued(&db, &child_id).await, "child should be queued");
let sf = sqlx::query_scalar!(
"SELECT scheduled_for FROM v2_job_queue WHERE id = $1",
child_id
)
.fetch_one(&db)
.await?;
let diff = (sf - Utc::now()).num_seconds();
assert!(
diff >= 0 && diff <= 6,
"scheduled_for should be in the future (up to ~5s), got {diff}s"
);
// Parent flow should still be in queue
assert!(
is_queued(&db, &flow_id).await,
"parent flow should still be queued"
);
assert!(
!is_completed(&db, &flow_id).await,
"parent flow should not be completed"
);
// debounce_key should exist
let dk = get_debounce_key(&db, "test_flow_node_first").await;
assert!(dk.is_some(), "debounce_key entry should exist");
let (dk_job, dk_prev, dk_times) = dk.unwrap();
assert_eq!(dk_job, child_id);
assert!(dk_prev.is_none());
assert_eq!(dk_times, 0);
Ok(())
}
/// Test: Second flow node job with same key should cancel the first child and
/// complete the first parent flow with "Debounced by" result.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_second_cancels_first(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
// Flow 1 with child 1
let flow1 = Uuid::new_v4();
let child1 = Uuid::new_v4();
insert_flow_job(&db, flow1, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child1, flow1, "test-workspace").await;
// Flow 2 with child 2
let flow2 = Uuid::new_v4();
let child2 = Uuid::new_v4();
insert_flow_job(&db, flow2, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child2, flow2, "test-workspace").await;
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: Some("test_flow_node_cancel".to_string()),
..Default::default()
};
let args_hm = empty_args();
// Push child 1
{
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child1,
flow1,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
}
// Push child 2 — should cancel child 1 and complete flow 1
{
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child2,
flow2,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
}
// child1 should be completed (debounced/skipped)
assert!(
is_completed(&db, &child1).await,
"child1 should be completed"
);
assert!(
!is_queued(&db, &child1).await,
"child1 should not be in queue"
);
// flow1 (parent of child1) should also be completed
assert!(
is_completed(&db, &flow1).await,
"flow1 should be completed (debounced)"
);
assert!(
!is_queued(&db, &flow1).await,
"flow1 should not be in queue"
);
// Check flow1 result contains "Debounced by"
let result = sqlx::query_scalar!(
"SELECT result::text FROM v2_job_completed WHERE id = $1",
flow1
)
.fetch_one(&db)
.await?;
assert!(
result.as_ref().is_some_and(|r| r.contains("Debounced by")),
"flow1 result should contain 'Debounced by', got: {:?}",
result
);
// child2 should still be in queue (it's the winner)
assert!(
is_queued(&db, &child2).await,
"child2 should still be queued"
);
assert!(
!is_completed(&db, &child2).await,
"child2 should not be completed"
);
// flow2 should still be in queue
assert!(is_queued(&db, &flow2).await, "flow2 should still be queued");
// debounce_key should point to child2
let dk = get_debounce_key(&db, "test_flow_node_cancel").await;
assert!(dk.is_some());
let (dk_job, _, dk_times) = dk.unwrap();
assert_eq!(dk_job, child2);
assert_eq!(dk_times, 1);
Ok(())
}
/// Test: Default debounce key for flow nodes uses $workspace/flow/$path-$step_id.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_default_key(db: Pool<Postgres>) -> anyhow::Result<()> {
let flow_id = Uuid::new_v4();
let child_id = Uuid::new_v4();
insert_flow_job(&db, flow_id, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: None, // No custom key — use default
..Default::default()
};
let args_hm = empty_args();
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child_id,
flow_id,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
// Default key should be: test-workspace/flow/f/test/my_flow-step_a
let expected_key = "test-workspace/flow/f/test/my_flow-step_a";
let dk = get_debounce_key(&db, expected_key).await;
assert!(dk.is_some(), "debounce_key with default key should exist");
Ok(())
}
/// Test: Flow node debounce tracks debounced_times counter correctly.
/// Each debounce call increments the counter in the debounce_key table.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_counter_tracking(db: Pool<Postgres>) -> anyhow::Result<()> {
// No limits set so counter never resets
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: Some("test_flow_node_counter".to_string()),
..Default::default()
};
let args_hm = empty_args();
// Push 4 jobs. Each subsequent one increments debounced_times.
for _ in 0..4 {
let flow_id = Uuid::new_v4();
let child_id = Uuid::new_v4();
insert_flow_job(&db, flow_id, "test-workspace", "f/test/flow").await;
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child_id,
flow_id,
"f/test/flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
}
// After 4 jobs, debounced_times should be 3 (first job creates the entry with 0,
// subsequent 3 jobs each increment it)
let debounced_times = sqlx::query_scalar!(
"SELECT debounced_times FROM debounce_key WHERE key = $1",
"test_flow_node_counter"
)
.fetch_one(&db)
.await?;
assert_eq!(
debounced_times, 3,
"debounced_times should be 3 after 4 jobs"
);
Ok(())
}
}
backend/windmill-store/src/variables.rs
+3 -12
View File
@@ -133,21 +133,12 @@ async fn list_variables(
])
.left()
.join("account")
.on(&format!(
"variable.account = account.id AND account.workspace_id = '{}'",
w_id
))
.on("variable.account = account.id AND account.workspace_id = ?".bind(&w_id))
.left()
.join("resource")
.on(&format!(
"resource.path = variable.path AND resource.workspace_id = '{}'",
w_id
))
.on("resource.path = variable.path AND resource.workspace_id = ?".bind(&w_id))
.and_where("variable.workspace_id = ?".bind(&w_id))
.and_where(&format!(
"variable.path NOT LIKE 'u/' || '{}' || '/secret_arg/%'",
authed.username
))
.and_where("variable.path NOT LIKE 'u/' || ? || '/secret_arg/%'".bind(&authed.username))
.order_by("path", false)
.limit(per_page)
.offset(offset)
backend/windmill-types/Cargo.toml
+1
View File
@@ -9,6 +9,7 @@ name = "windmill_types"
path = "src/lib.rs"
[dependencies]
windmill-parser.workspace = true
serde.workspace = true
serde_json.workspace = true
chrono.workspace = true
backend/windmill-types/src/flows.rs
+3
View File
@@ -445,6 +445,8 @@ pub struct FlowModule {
pub apply_preprocessor: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub pass_flow_input_directly: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub debouncing: Option<DebouncingSettings>,
}
#[derive(Deserialize, Serialize, Debug, Clone)]
@@ -1117,6 +1119,7 @@ pub fn add_virtual_items_if_necessary(modules: &mut Vec<FlowModule>) {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
});
}
}
backend/windmill-types/src/s3.rs
+3 -14
View File
@@ -346,20 +346,9 @@ pub struct DuckdbConnectionSettingsQueryV2 {
pub storage: Option<String>,
}
#[derive(Clone, Copy, Debug)]
pub enum S3ModeFormat {
Json,
Csv,
Parquet,
}
pub fn s3_mode_extension(format: S3ModeFormat) -> &'static str {
match format {
S3ModeFormat::Json => "json",
S3ModeFormat::Csv => "csv",
S3ModeFormat::Parquet => "parquet",
}
}
// Re-export from windmill-parser to keep a single type definition
// (windmill-parser is WASM-compatible, windmill-types is not due to sqlx)
pub use windmill_parser::{s3_mode_extension, S3ModeFormat};
#[cfg(test)]
mod tests {
backend/windmill-worker/loader.bun.js
+13 -4
View File
@@ -1,8 +1,11 @@
// Injected by backend: maps normalized paths to temp storage hashes (or null)
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
const p = {
name: "windmill-relative-resolver",
async setup(build) {
const { writeFileSync, readFileSync, mkdirSync } = await import("fs");
const { dirname, resolve } = await import("node:path");
const { dirname, resolve, join } = await import("node:path");
const base_internal_url = "BASE_INTERNAL_URL".replace(
"localhost",
@@ -95,11 +98,17 @@ const p = {
: args.importer.replace(cdir + "/", "");
const isRelative = !args.path.startsWith("/");
const endExt = args.path.endsWith(".ts") ? "" : ".ts";
const pathNoExt = args.path.replace(/\.ts$/, "");
let endExt = args.path.endsWith(".ts") ? "" : ".ts";
const url = isRelative
// Lookup temp script hash
const normalized = (isRelative ? join(dirname(file_path), pathNoExt) : pathNoExt.slice(1)).replace(/\\/g, "/");
const hash = TEMP_SCRIPT_REFS?.[normalized];
const url = (isRelative
? `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${file_path}/../${args.path}${endExt}`
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`;
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`
) + (hash ? `?temp_script_hash=${hash}` : "");
const file = isRelative
? resolve("./" + file_path + "/../" + args.path + ".url")
: resolve("./" + args.path + ".url");
backend/windmill-worker/loader.bun.windows.js
+19 -4
View File
@@ -1,3 +1,6 @@
// Injected by backend: maps normalized paths to temp storage hashes (or null)
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
// Windows-specific bun loader that uses a virtual "windmill-url" namespace instead
// of writing .url files to disk. This avoids Windows path issues (backslashes in
// resolve(), 8.3 short filenames, drive letter prefixes). The virtual namespace
@@ -70,7 +73,13 @@ const p = {
const rawScriptPath = isAbsolute
? `${path}${endExt}`
: `${importerPath}/../${path}${endExt}`;
return { path: normalizePath(rawScriptPath), namespace: "windmill-url" };
const normalized = normalizePath(rawScriptPath);
// Look up temp script hash (keys are extensionless paths)
const lookupPath = normalized.replace(/\.ts$/, "");
const hash = TEMP_SCRIPT_REFS?.[lookupPath];
// Encode hash in the path so onLoad can extract it and append to fetch URL
const resolvedPath = hash ? `${normalized}?temp_script_hash=${hash}` : normalized;
return { path: resolvedPath, namespace: "windmill-url" };
}
build.onLoad({ filter: filterLoad }, async (args) => {
@@ -80,8 +89,13 @@ const p = {
// Load windmill scripts by fetching from the API
build.onLoad({ filter: /.*/, namespace: "windmill-url" }, async (args) => {
const path = args.path.replace(/^windmill-url:/, "");
const url = `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${path}`;
// Extract temp_script_hash if embedded in the path by resolveWindmillImport
const [scriptPath, queryString] = args.path.replace(/^windmill-url:/, "").split("?");
const hashParam = queryString?.startsWith("temp_script_hash=")
? queryString.replace("temp_script_hash=", "")
: undefined;
const url = `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${scriptPath}`
+ (hashParam ? `?temp_script_hash=${hashParam}` : "");
const req = await fetch(url, {
method: "GET",
headers: {
@@ -124,7 +138,8 @@ const p = {
// Resolve nested imports from within windmill-url modules
build.onResolve({ filter: /\.ts$/, namespace: "windmill-url" }, (args) => {
const importer = args.importer.replace(/^windmill-url:/, "");
// Strip any query string from the importer path before resolving
const importer = args.importer.replace(/^windmill-url:/, "").split("?")[0];
return resolveWindmillImport(importer, args.path);
});
},
backend/windmill-worker/src/bun_executor.rs
+28 -3
View File
@@ -206,6 +206,7 @@ pub async fn gen_bun_lockfile(
workspace_dependencies: &WorkspaceDependenciesPrefetched,
npm_mode: bool,
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
temp_script_refs: &Option<HashMap<String, String>>,
quiet: bool,
) -> Result<Option<String>> {
let common_bun_proc_envs: HashMap<String, String> = get_common_bun_proc_envs(None).await;
@@ -216,6 +217,11 @@ pub async fn gen_bun_lockfile(
gen_bunfig(job_dir, job_id, w_id, db).await?;
write_file(job_dir, "package.json", package_json_content.as_str())?;
} else {
let temp_refs_json = temp_script_refs
.as_ref()
.and_then(|m| serde_json::to_string(m).ok())
.unwrap_or_else(|| "null".to_string());
let loader = RELATIVE_BUN_LOADER
.replace("W_ID", w_id)
.replace("BASE_INTERNAL_URL", base_internal_url)
@@ -224,7 +230,8 @@ pub async fn gen_bun_lockfile(
"CURRENT_PATH",
&crate::common::use_flow_root_path(script_path),
)
.replace("RAW_GET_ENDPOINT", "raw");
.replace("RAW_GET_ENDPOINT", "raw")
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", &temp_refs_json);
write_file(
&job_dir,
@@ -615,9 +622,15 @@ pub async fn build_loader(
w_id: &str,
current_path: &str,
mode: LoaderMode,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<()> {
// Use forward slashes in JS strings to avoid backslash escape issues on Windows
let job_dir_js = job_dir.replace('\\', "/");
let temp_refs_json = temp_script_refs
.as_ref()
.and_then(|m| serde_json::to_string(m).ok())
.unwrap_or_else(|| "null".to_string());
let loader = RELATIVE_BUN_LOADER
.replace("W_ID", w_id)
.replace("BASE_INTERNAL_URL", base_internal_url)
@@ -626,7 +639,8 @@ pub async fn build_loader(
"CURRENT_PATH",
&crate::common::use_flow_root_path(current_path),
)
.replace("RAW_GET_ENDPOINT", "raw_unpinned");
.replace("RAW_GET_ENDPOINT", "raw_unpinned")
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", &temp_refs_json);
if mode == LoaderMode::Node {
write_file(
@@ -924,6 +938,7 @@ pub async fn prebundle_bun_script(
worker_name: &str,
token: &str,
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<()> {
let (local_path, remote_path) =
compute_bundle_local_and_remote_path(inner_content, lock, script_path, db, w_id).await;
@@ -950,6 +965,7 @@ pub async fn prebundle_bun_script(
} else {
LoaderMode::BunBundle
},
temp_script_refs,
)
.await?;
@@ -1271,6 +1287,7 @@ pub async fn handle_bun_job(
workspace_dependencies,
annotation.npm,
&mut Some(occupancy_metrics),
&None,
wac_replay_info.is_some(),
)
.await?;
@@ -1639,6 +1656,7 @@ try {{
} else {
LoaderMode::BunBundle
},
&None,
)
.await?;
@@ -1655,6 +1673,7 @@ try {{
} else {
LoaderMode::Bun
},
&None,
)
.await
} else {
@@ -2989,7 +3008,10 @@ pub fn build_nativets_env_code(
"const process = {{ env: {{}} }};\nconst BASE_URL = '{base_internal_url}';\nconst BASE_INTERNAL_URL = '{base_internal_url}';\nprocess.env['BASE_URL'] = BASE_URL;process.env['BASE_INTERNAL_URL'] = BASE_INTERNAL_URL;\n{}",
reserved_variables
.iter()
.map(|(k, v)| format!("process.env['{}'] = '{}';", k, v))
.map(|(k, v)| {
let escaped = v.replace('\\', "\\\\").replace('\'', "\\'").replace('\n', "\\n").replace('\r', "\\r");
format!("process.env['{}'] = '{}';", k, escaped)
})
.collect::<Vec<String>>()
.join("\n")
)
@@ -3358,6 +3380,7 @@ pub async fn start_worker(
w_id,
script_path,
LoaderMode::BrowserBundle,
&None,
)
.await?;
generate_bun_bundle(
@@ -3470,6 +3493,7 @@ pub async fn start_worker(
.await?,
annotation.npm,
&mut None,
&None,
false,
)
.await?;
@@ -3544,6 +3568,7 @@ pub async fn start_worker(
} else {
LoaderMode::Bun
},
&None,
)
.await?;
}
backend/windmill-worker/src/common.rs
+14 -3
View File
@@ -710,10 +710,13 @@ pub fn build_command_with_isolation(program: &str, args: &[&str]) -> Command {
cmd.args(args);
cmd
} else {
panic!(
"BUG: unshare isolation is enabled but UNSHARE_PATH is None. \
This should have been caught at worker startup."
tracing::error!(
"unshare isolation is enabled but UNSHARE_PATH is not available. \
Running job without isolation. Check Instance Settings > Job Isolation."
);
let mut cmd = Command::new(program);
cmd.args(args);
cmd
}
} else {
let mut cmd = Command::new(program);
@@ -1000,6 +1003,14 @@ pub(crate) async fn get_workspace_s3_resource_path(
windmill_object_store::check_bucket_workspace_restriction(bucket, workspace_id)?;
}
// Check Azure account name workspace restrictions
if let ObjectStoreResource::Azure(azure_resource) = &object_store_resource {
windmill_object_store::check_az_account_name_workspace_restriction(
&azure_resource.account_name,
workspace_id,
)?;
}
Ok(Some(object_store_resource))
}
backend/windmill-worker/src/python_executor.rs
+1
View File
@@ -1348,6 +1348,7 @@ async fn handle_python_deps(
&mut version_specifiers,
&mut locked_v,
&None,
&None, // temp_script_refs: only used during CLI lock generation
))
.await?;
backend/windmill-worker/src/worker.rs
+76 -17
View File
@@ -453,8 +453,13 @@ lazy_static::lazy_static! {
);
}
tracing::warn!(
"unshare test failed: {}. Flags: {}. Set ENABLE_UNSHARE_PID=true to fail on error.",
tracing::error!(
"unshare test command failed (exit code: {}). stderr: '{}'. flags: '{}'. \
Unshare isolation will NOT be available. \
If job_isolation is set to 'unshare' in Instance Settings, jobs will run without isolation. \
Common causes: user namespaces disabled (sysctl kernel.unprivileged_userns_clone=0), \
max_user_namespaces=0, or missing privileges (--mount-proc requires privileged mode).",
output.status,
stderr.trim(),
flags
);
@@ -476,9 +481,15 @@ lazy_static::lazy_static! {
}
if e.kind() == std::io::ErrorKind::NotFound {
tracing::debug!("unshare binary not found");
tracing::error!(
"unshare binary not found in PATH. Unshare isolation will NOT be available. \
Install the util-linux package to enable unshare isolation."
);
} else {
tracing::warn!("Failed to test unshare: {}", e);
tracing::error!(
"Failed to execute unshare test command: {}. Unshare isolation will NOT be available.",
e
);
}
None
}
@@ -499,23 +510,27 @@ lazy_static::lazy_static! {
},
Ok(output) => {
let stderr = String::from_utf8_lossy(&output.stderr);
tracing::warn!(
"nsjail test failed: {}. \
tracing::error!(
"nsjail test failed (exit code: {}). stderr: '{}'. path: '{}'. \
Nsjail sandboxing will NOT be available. \
nsjail should be included in all standard windmill images. \
Check that the nsjail binary is installed and working correctly.",
stderr.trim()
If job_isolation is set to 'nsjail_sandboxing' in Instance Settings, jobs will fail.",
output.status,
stderr.trim(),
nsjail_path
);
None
},
Err(e) => {
if e.kind() == std::io::ErrorKind::NotFound {
tracing::info!(
"nsjail not found at '{}'. Sandboxing will not be available.",
tracing::error!(
"nsjail not found at '{}'. Nsjail sandboxing will NOT be available. \
If using a custom image, ensure nsjail is installed.",
nsjail_path
);
} else {
tracing::warn!(
"Failed to test nsjail at '{}': {}.",
tracing::error!(
"Failed to execute nsjail test at '{}': {}. Nsjail sandboxing will NOT be available.",
nsjail_path,
e
);
@@ -950,6 +965,7 @@ async fn get_otel_tracing_proxy_envs(
TRACING_PROXY_CA_CERT_PATH.to_string(),
),
("CURL_CA_BUNDLE", TRACING_PROXY_CA_CERT_PATH.to_string()),
("GIT_SSL_CAINFO", TRACING_PROXY_CA_CERT_PATH.to_string()),
("DENO_CERT", TRACING_PROXY_CA_CERT_PATH.to_string()),
])
}
@@ -1569,10 +1585,27 @@ pub async fn run_worker(
);
}
// Force UNSHARE_PATH initialization now to fail-fast if unshare doesn't work
// This ensures we panic at startup rather than lazily when first accessed during job execution
if is_unshare_enabled() || *ENABLE_UNSHARE_PID || *FAVOR_UNSHARE_PID {
let _ = &*UNSHARE_PATH;
// Force UNSHARE_PATH and NSJAIL_AVAILABLE initialization now for clear startup logging
let _ = &*UNSHARE_PATH;
let _ = &*NSJAIL_AVAILABLE;
if (is_unshare_enabled() || *FAVOR_UNSHARE_PID) && UNSHARE_PATH.is_none() {
tracing::error!(
worker = %worker_name, hostname = %hostname,
"Worker is configured to use unshare isolation (FAVOR_UNSHARE_PID={}, job_isolation={:?}) \
but unshare is NOT available. Jobs will run without isolation. \
See errors above for the specific reason unshare initialization failed.",
*FAVOR_UNSHARE_PID,
JobIsolationLevel::from_u8(JOB_ISOLATION.load(std::sync::atomic::Ordering::Relaxed))
);
}
if is_sandboxing_enabled() && NSJAIL_AVAILABLE.is_none() {
tracing::error!(
worker = %worker_name, hostname = %hostname,
"Worker is configured to use nsjail sandboxing but nsjail is NOT available. \
Jobs requiring sandboxing will fail. \
See errors above for the specific reason nsjail initialization failed."
);
}
let start_time = Instant::now();
@@ -4177,6 +4210,29 @@ pub async fn run_language_executor(
modules: &Option<std::collections::HashMap<String, ScriptModule>>,
run_inline: bool,
) -> error::Result<Box<RawValue>> {
// Expand WM_INTERNAL_DB markers into real SQL before dispatching
let expanded_code: String;
let mut language = language;
let code = if let Some(ref lang) = language {
match windmill_common::query_builders::try_expand_internal_db_query(code, lang) {
Some(Ok(expanded)) => {
if let Some(lang_override) = expanded.language_override {
language = Some(lang_override);
}
expanded_code = expanded.code;
&expanded_code
}
Some(Err(e)) => {
return Err(Error::ExecutionErr(format!(
"Failed to expand WM_INTERNAL_DB marker: {}",
e
)));
}
None => code, // Not a marker, use original code
}
} else {
code
};
if let Some(modules) = modules {
#[cfg(feature = "python")]
let base_dir = if language == Some(ScriptLang::Python3) {
@@ -4438,7 +4494,10 @@ pub async fn run_language_executor(
"const process = {{ env: {{}} }};\nconst BASE_URL = '{base_internal_url}';\nconst BASE_INTERNAL_URL = '{base_internal_url}';\nprocess.env['BASE_URL'] = BASE_URL;process.env['BASE_INTERNAL_URL'] = BASE_INTERNAL_URL;\n{}",
reserved_variables
.iter()
.map(|(k, v)| format!("const {} = '{}';\nprocess.env['{}'] = '{}';\n", k, v, k, v))
.map(|(k, v)| {
let escaped = v.replace('\\', "\\\\").replace('\'', "\\'").replace('\n', "\\n").replace('\r', "\\r");
format!("const {} = '{}';\nprocess.env['{}'] = '{}';\n", k, escaped, k, escaped)
})
.collect::<Vec<String>>()
.join("\n"));
backend/windmill-worker/src/worker_flow.rs
+28
View File
@@ -3687,6 +3687,34 @@ async fn push_next_flow_job(
tracing::debug!(id = %flow_job.id, root_id = %job_root, "pushed next flow job: {uuid}");
// Apply flow node debouncing if configured. Skip parallel steps (for-loops, branchall)
// where len > 1 — debouncing those would cancel sibling sub-jobs within the same run.
#[cfg(feature = "private")]
if len == 1 {
if let Some(ref debouncing) = module.debouncing {
if debouncing.debounce_delay_s.is_some_and(|d| d > 0) {
let debounce_args = if let Ok(ref v) = nargs {
windmill_queue::PushArgs::from(v.as_ref())
} else {
windmill_queue::PushArgs::from(&*EHM)
};
let flow_path = flow_job.runnable_path().to_string();
windmill_queue::jobs_ee::maybe_debounce_flow_node(
debouncing,
uuid,
flow_job.id,
&flow_path,
&module.id,
&flow_job.workspace_id,
&debounce_args,
&mut inner_tx,
&db,
)
.await?;
}
}
}
if value_with_parallel.type_ == "forloopflow"
&& value_with_parallel.parallel.unwrap_or(false)
{
backend/windmill-worker/src/worker_lockfiles.rs
+58 -8
View File
@@ -139,6 +139,13 @@ pub async fn handle_dependency_job(
.map(|x| x.get("triggered_by_relative_import").is_some())
.unwrap_or_default();
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
let temp_script_refs: Option<HashMap<String, String>> = job
.args
.as_ref()
.and_then(|x| x.get("temp_script_refs"))
.and_then(|v| serde_json::from_str(v.get()).ok());
let content = capture_dependency_job(
&job.id,
job.script_lang.as_ref().map(|v| Ok(v)).unwrap_or_else(|| {
@@ -164,6 +171,7 @@ pub async fn handle_dependency_job(
script_path,
None,
"script",
&temp_script_refs,
)
.await;
@@ -210,6 +218,7 @@ pub async fn handle_dependency_job(
script_path,
None,
"script",
&None,
)
.await
{
@@ -368,6 +377,13 @@ pub async fn handle_flow_dependency_job(
.map(|x| x.get("triggered_by_relative_import").is_some())
.unwrap_or_default();
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
let temp_script_refs: Option<HashMap<String, String>> = job
.args
.as_ref()
.and_then(|x| x.get("temp_script_refs"))
.and_then(|v| serde_json::from_str(v.get()).ok());
let version = if skip_flow_update {
None
} else {
@@ -411,13 +427,13 @@ pub async fn handle_flow_dependency_job(
// `JobKind::FlowDependencies` job store either:
// - A saved flow version `id` in the `script_hash` column.
// - Preview raw flow in the `queue` or `job` table.
let (mut flow, notes) = match job.runnable_id {
let (mut flow, extras) = match job.runnable_id {
Some(ScriptHash(id)) => {
let flow = cache::flow::fetch_version(db, id).await?;
(flow.value().clone(), flow.notes())
(flow.value().clone(), flow.extras())
}
_ => match preview_data {
Some(RawData::Flow(data)) => (data.value().clone(), data.notes()),
Some(RawData::Flow(data)) => (data.value().clone(), data.extras()),
_ => return Err(Error::internal_err("expected script hash")),
},
};
@@ -467,6 +483,7 @@ pub async fn handle_flow_dependency_job(
&mut dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
&temp_script_refs,
)
.await?;
@@ -511,18 +528,22 @@ pub async fn handle_flow_dependency_job(
}
#[derive(Debug, Clone, Serialize)]
struct FlowValueWithNotes<'a> {
struct FlowValueWithExtras<'a> {
#[serde(flatten)]
value: &'a FlowValue,
#[serde(skip_serializing_if = "Option::is_none")]
notes: Option<Box<RawValue>>, // TODO: Make this a Vec<FlowNote>
notes: Option<Box<RawValue>>,
#[serde(skip_serializing_if = "Option::is_none")]
groups: Option<Box<RawValue>>,
}
let new_flow_value = Json(
serde_json::value::to_raw_value(&FlowValueWithNotes {
serde_json::value::to_raw_value(&FlowValueWithExtras {
value: &flow,
notes: notes.and_then(|n| n.notes).map(|n| n.into()),
notes: extras.as_ref().and_then(|e| e.notes.clone()),
groups: extras.as_ref().and_then(|e| e.groups.clone()),
})
.map_err(to_anyhow)?,
);
@@ -681,6 +702,7 @@ async fn lock_flow_value<'c>(
dependency_map: &mut ScopedDependencyMap,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
triggered_by_relative_import: bool,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<(
FlowValue,
sqlx::Transaction<'c, sqlx::Postgres>,
@@ -711,6 +733,7 @@ async fn lock_flow_value<'c>(
dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?;
@@ -742,6 +765,7 @@ async fn lock_flow_value<'c>(
dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?;
@@ -779,6 +803,7 @@ async fn lock_flow_value<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?;
@@ -817,6 +842,7 @@ async fn lock_modules<'c>(
dependency_map: &mut ScopedDependencyMap, // (modules to replace old seq (even unmmodified ones), new transaction, modified ids) )
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
triggered_by_relative_import: bool,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<(
Vec<FlowModule>,
sqlx::Transaction<'c, sqlx::Postgres>,
@@ -872,6 +898,7 @@ async fn lock_modules<'c>(
dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
e.value = FlowModuleValue::ForloopFlow {
@@ -911,6 +938,7 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
nmodified_ids.extend(inner_modified_ids);
@@ -942,6 +970,7 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
e.value = FlowModuleValue::WhileloopFlow {
@@ -978,6 +1007,7 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
nmodified_ids.extend(inner_modified_ids);
@@ -1008,6 +1038,7 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
errors.extend(ninner_errors);
@@ -1078,6 +1109,7 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
@@ -1179,6 +1211,7 @@ async fn lock_modules<'c>(
job_path,
Some(&e.id),
"flow",
&temp_script_refs,
)
.await;
//
@@ -1586,6 +1619,7 @@ async fn lock_modules_app(
dependency_map: &mut ScopedDependencyMap,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
triggered_by_relative_import: bool,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<Value> {
match value {
Value::Object(mut m) => {
@@ -1696,6 +1730,7 @@ async fn lock_modules_app(
&job.runnable_path(),
container_id.as_deref(),
"app",
temp_script_refs,
)
.await;
match new_lock {
@@ -1773,6 +1808,7 @@ async fn lock_modules_app(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?,
);
@@ -1801,6 +1837,7 @@ async fn lock_modules_app(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?,
);
@@ -1852,6 +1889,13 @@ pub async fn handle_app_dependency_job(
.map(|x| x.get("triggered_by_relative_import").is_some())
.unwrap_or_default();
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
let temp_script_refs: Option<HashMap<String, String>> = job
.args
.as_ref()
.and_then(|x| x.get("temp_script_refs"))
.and_then(|v| serde_json::from_str(v.get()).ok());
sqlx::query!(
"DELETE FROM workspace_runnable_dependencies WHERE app_path = $1 AND workspace_id = $2",
job_path,
@@ -1899,6 +1943,7 @@ pub async fn handle_app_dependency_job(
&mut dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
&temp_script_refs,
)
.await?;
@@ -2398,6 +2443,8 @@ async fn capture_dependency_job(
base_path: &str,
step_id: Option<&str>,
runnable_type: &str, // "script", "flow", or "app"
// Map of script path -> content hash for resolving imports from temp storage (CLI).
temp_script_refs: &Option<HashMap<String, String>>,
) -> error::Result<String> {
// Check if we can skip relocking:
// - Must be triggered by relative import
@@ -2456,12 +2503,13 @@ async fn capture_dependency_job(
let (mut version_specifiers, mut locked_v) = (vec![], None);
let reqs = windmill_parser_py_imports::parse_python_imports(
job_raw_code,
&w_id,
w_id,
script_path,
&db,
&mut version_specifiers,
&mut locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
)
.await?
.0
@@ -2585,6 +2633,7 @@ async fn capture_dependency_job(
&workspace_dependencies,
windmill_common::worker::TypeScriptAnnotations::parse(job_raw_code).npm,
&mut Some(occupancy_metrics),
temp_script_refs,
false,
)
.await?
@@ -2602,6 +2651,7 @@ async fn capture_dependency_job(
worker_name,
&token,
&mut Some(occupancy_metrics),
temp_script_refs,
)
.await?;
}
backend/windmill-worker/src/worker_utils.rs
+4 -4
View File
@@ -190,9 +190,9 @@ pub async fn insert_ping(
let vcpus = get_vcpus();
let memory = get_memory();
let job_isolation = if crate::is_sandboxing_enabled() {
let job_isolation = if crate::is_sandboxing_enabled() && crate::NSJAIL_AVAILABLE.is_some() {
Some("nsjail".to_string())
} else if crate::is_unshare_enabled() {
} else if crate::is_unshare_enabled() && crate::UNSHARE_PATH.is_some() {
Some("unshare".to_string())
} else {
Some("none".to_string())
@@ -265,9 +265,9 @@ pub async fn update_worker_ping_from_job(
let occupancy_rate_5m = occupancy.as_ref().and_then(|x| x.occupancy_rate_5m);
let occupancy_rate_30m = occupancy.as_ref().and_then(|x| x.occupancy_rate_30m);
let job_isolation = if crate::is_sandboxing_enabled() {
let job_isolation = if crate::is_sandboxing_enabled() && crate::NSJAIL_AVAILABLE.is_some() {
Some("nsjail".to_string())
} else if crate::is_unshare_enabled() {
} else if crate::is_unshare_enabled() && crate::UNSHARE_PATH.is_some() {
Some("unshare".to_string())
} else {
Some("none".to_string())
benchmarks/lib.ts
+1 -1
View File
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
export const VERSION = "v1.662.0";
export const VERSION = "v1.663.0";
export async function login(email: string, password: string): Promise<string> {
return await windmill.UserService.login({
cli/build-npm.ts
+1
View File
@@ -12,6 +12,7 @@ const parserPackages = [
"windmill-parser-wasm-yaml", "windmill-parser-wasm-csharp",
"windmill-parser-wasm-nu", "windmill-parser-wasm-java",
"windmill-parser-wasm-ruby",
"windmill-parser-wasm-py-imports",
];
const parserExternals = parserPackages.flatMap(p => ["--external", p]);

Some files were not shown because too many files have changed in this diff Show More