Compare commits

..
Author SHA1 Message Date
Guilhem LemouelandClaude Opus 4.6 632c8868bc feat(cli): add localhost reverse proxy to wmill dev for Claude Desktop preview
Adds a reverse proxy to `wmill dev` that serves the Windmill UI on localhost,
enabling Claude Desktop/Preview to open dev pages. Each connected dev page can
watch a specific file via the `path` URL param and `setWatch` WebSocket message.

Key changes:
- CLI: single-port proxy (default :3100) that forwards HTTP to remote Windmill,
  handles /ws_dev locally for dev file changes, and proxies /ws/* to remote
- CLI: per-client watch filtering so multiple tabs can watch different files
- CLI: flow broadcasts now include a `path` field
- Frontend: Dev.svelte connects to same-origin /ws_dev when no port param,
  sends setWatch on connect, filters messages client-side as safety net
- CLI init: generates .claude/skills/dev-preview/SKILL.md and .claude/launch.json

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 22:45:37 +01:00
135 changed files with 1212 additions and 11011 deletions
-59
View File
@@ -1,59 +0,0 @@
---
name: commit
description: Create a git commit with conventional commit format. MUST use anytime you want to commit changes.
---
# Git Commit Skill
Create a focused, single-line commit following conventional commit conventions.
## Instructions
1. **Analyze changes**: Run `git status` and `git diff` to understand what was modified
2. **Stage only modified files**: Add files individually by name. NEVER use `git add -A` or `git add .`
3. **Write commit message**: Follow the conventional commit format as a single line
## Conventional Commit Format
```
<type>: <description>
```
### Types
- `feat`: New feature or capability
- `fix`: Bug fix
- `refactor`: Code change that neither fixes a bug nor adds a feature
- `docs`: Documentation only changes
- `style`: Formatting, missing semicolons, etc (no code change)
- `test`: Adding or correcting tests
- `chore`: Maintenance tasks, dependency updates, etc
- `perf`: Performance improvement
### Rules
- Message MUST be a single line (no multi-line messages)
- Description should be lowercase, imperative mood ("add" not "added")
- No period at the end
- Keep under 72 characters total
### Examples
```
feat: add token usage tracking for AI providers
fix: resolve null pointer in job executor
refactor: extract common validation logic
docs: update API endpoint documentation
chore: upgrade sqlx to 0.7
```
## Execution Steps
1. Run `git status` to see all changes
2. Run `git diff` to understand the changes in detail
3. Run `git log --oneline -5` to see recent commit style
4. Stage ONLY the modified/relevant files: `git add <file1> <file2> ...`
5. Create the commit with conventional format:
```bash
git commit -m "<type>: <description>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
```
6. Run `git status` to verify the commit succeeded
-97
View File
@@ -1,97 +0,0 @@
---
name: local-review
description: Code review a pull request for bugs and CLAUDE.md compliance. MUST use when asked to review code.
---
# Local Code Review Skill
Review a pull request for real bugs and CLAUDE.md compliance violations. This review targets HIGH SIGNAL issues only.
## Review Philosophy
- **Only flag issues you are certain about.** If you are not sure an issue is real, do not flag it. False positives erode trust and waste reviewer time.
- Think like a senior engineer doing a final review — flag things that would cause incidents, not things that are merely imperfect.
## What to Flag
- Code that won't compile or parse (syntax errors, type errors, missing imports)
- Code that will definitely produce wrong results regardless of inputs
- Clear, unambiguous CLAUDE.md violations (quote the exact rule being violated)
- Security issues in introduced code (injection, auth bypass, data exposure)
- Incorrect logic that will fail in production
## What NOT to Flag
- Code style or quality concerns
- Potential issues that depend on specific inputs or runtime state
- Subjective suggestions or improvements
- Pre-existing issues not introduced by this PR
- Pedantic nitpicks a senior engineer wouldn't flag
- Issues a linter or type checker will catch
- General quality concerns unless explicitly prohibited in CLAUDE.md
- Issues silenced via lint ignore comments
## Execution Steps
1. **Determine the PR scope**:
- If an argument is provided, use it as the PR number or branch
- Otherwise, detect from the current branch vs main
- Run `gh pr view` if a PR exists, or use `git diff main...HEAD`
2. **Find relevant CLAUDE.md files**:
- Read the root `CLAUDE.md`
- Check for CLAUDE.md files in directories containing changed files
3. **Get the diff and metadata**:
- `gh pr diff` or `git diff main...HEAD` for the full diff
- `gh pr view` or `git log main..HEAD --oneline` for context
4. **Read changed files** where the diff alone is insufficient to understand context
5. **Review for**:
- CLAUDE.md compliance — check each rule against the changed code
- Bugs and logic errors — will this code work correctly?
- Security issues — injection, auth, data exposure in new code
6. **Self-validate each finding**: Before reporting, ask yourself:
- "Is this definitely a real issue, not a false positive?"
- "Would a senior engineer flag this in review?"
- If the answer to either is no, discard the finding
7. **Output findings** to the terminal (default) or post as PR comments (with `--comment` flag)
## Output Format
```
## Code review
Found N issues:
1. <description> (<reason: CLAUDE.md adherence | bug | security>)
<file_path:line_number>
2. <description> (<reason>)
<file_path:line_number>
```
If no issues are found:
```
## Code review
No issues found. Checked for bugs and CLAUDE.md compliance.
```
## Posting Comments (--comment flag)
If the user passes `--comment`, post findings as inline PR comments using:
```bash
gh pr review --comment --body "<summary>"
```
Or for inline comments on specific lines:
```bash
gh api repos/{owner}/{repo}/pulls/{pr}/reviews -f body="<summary>" -f event="COMMENT" -f comments="[...]"
```
-777
View File
@@ -1,777 +0,0 @@
# Skill: Adding Native Trigger Services
This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications.
## Architecture Overview
The native trigger system consists of:
1. **Database Layer** - PostgreSQL tables and enum types
2. **Backend Rust Implementation** - Core trait, handlers, and service modules in the `windmill-native-triggers` crate
3. **Frontend Svelte Components** - Configuration forms and UI components
### Key Files
| Component | Path |
|-----------|------|
| Core module with `External` trait | `backend/windmill-native-triggers/src/lib.rs` |
| Generic CRUD handlers | `backend/windmill-native-triggers/src/handler.rs` |
| Background sync logic | `backend/windmill-native-triggers/src/sync.rs` |
| OAuth/workspace integration | `backend/windmill-native-triggers/src/workspace_integrations.rs` |
| Re-export shim (windmill-api) | `backend/windmill-api/src/native_triggers/mod.rs` |
| TriggerKind enum | `backend/windmill-common/src/triggers.rs` |
| JobTriggerKind enum | `backend/windmill-common/src/jobs.rs` |
| Frontend service registry | `frontend/src/lib/components/triggers/native/utils.ts` |
| Frontend trigger utilities | `frontend/src/lib/components/triggers/utils.ts` |
| Trigger badges (icons + counts) | `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte` |
| Workspace integrations UI | `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte` |
| OAuth config form component | `frontend/src/lib/components/workspaceSettings/OAuthClientConfig.svelte` |
| OpenAPI spec | `backend/windmill-api/openapi.yaml` |
| Reference: Nextcloud module | `backend/windmill-native-triggers/src/nextcloud/` |
| Reference: Google module | `backend/windmill-native-triggers/src/google/` |
### Crate Structure
The native trigger code lives in the `windmill-native-triggers` crate (`backend/windmill-native-triggers/`). The `windmill-api` crate re-exports everything via a shim:
```rust
// backend/windmill-api/src/native_triggers/mod.rs
pub use windmill_native_triggers::*;
```
All new service modules go in `backend/windmill-native-triggers/src/`.
---
## Core Concepts
### The `External` Trait
Every native trigger service implements the `External` trait defined in `lib.rs`:
```rust
#[async_trait]
pub trait External: Send + Sync + 'static {
// Associated types:
type ServiceConfig: Debug + DeserializeOwned + Serialize + Send + Sync;
type TriggerData: Debug + Serialize + Send + Sync;
type OAuthData: DeserializeOwned + Serialize + Clone + Send + Sync;
type CreateResponse: DeserializeOwned + Send + Sync;
// Constants:
const SUPPORT_WEBHOOK: bool;
const SERVICE_NAME: ServiceName;
const DISPLAY_NAME: &'static str;
const TOKEN_ENDPOINT: &'static str;
const REFRESH_ENDPOINT: &'static str;
const AUTH_ENDPOINT: &'static str;
// Required methods:
async fn create(&self, w_id, oauth_data, webhook_token, data, db, tx) -> Result<Self::CreateResponse>;
async fn update(&self, w_id, oauth_data, external_id, webhook_token, data, db, tx) -> Result<serde_json::Value>;
async fn get(&self, w_id, oauth_data, external_id, db, tx) -> Result<Self::TriggerData>;
async fn delete(&self, w_id, oauth_data, external_id, db, tx) -> Result<()>;
async fn exists(&self, w_id, oauth_data, external_id, db, tx) -> Result<bool>;
async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors);
fn external_id_and_metadata_from_response(&self, resp) -> (String, Option<serde_json::Value>);
// Methods with defaults:
async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned>;
fn service_config_from_create_response(&self, data, resp) -> Option<serde_json::Value>;
fn additional_routes(&self) -> axum::Router;
async fn http_client_request<T, B>(&self, url, method, workspace_id, tx, db, headers, body) -> Result<T>;
}
```
Key design points:
- **`update()` returns `serde_json::Value`** - the resolved service_config to store. Each service is responsible for building the final config.
- **`maintain_triggers()`** - periodic background maintenance. Each service implements its own strategy (Nextcloud: reconcile with external state; Google: renew expiring channels).
- **No `list_all()` in the trait** - services that need it (Nextcloud) implement it privately; services that don't (Google) use different maintenance strategies.
- **No `get_external_id_from_trigger_data()` or `extract_service_config_from_trigger_data()`** - removed in favor of the `maintain_triggers` pattern.
### Create Lifecycle: Two Paths
The `create_native_trigger` handler in `handler.rs` supports two creation flows, controlled by `service_config_from_create_response()`:
**Path A: Short (Google pattern)** - `service_config_from_create_response()` returns `Some(config)`:
1. `create()` registers on external service
2. `external_id_and_metadata_from_response()` extracts the ID
3. `service_config_from_create_response()` builds the config directly from input data + response metadata
4. Stores trigger in DB -- done, no extra round-trip
Use this when the external_id is known before the create call (e.g., Google generates the channel_id as a UUID upfront and includes it in the webhook URL).
**Path B: Long (Nextcloud pattern)** - `service_config_from_create_response()` returns `None` (default):
1. `create()` registers on external service (webhook URL has no external_id yet)
2. `external_id_and_metadata_from_response()` extracts the ID
3. `update()` is called to fix the webhook URL with the now-known external_id
4. `update()` returns the resolved service_config
5. Stores trigger in DB
Use this when the external_id is assigned by the remote service and the webhook URL needs to be corrected after creation.
### OAuth Token Storage (Three-Table Pattern)
OAuth tokens are stored across three tables, NOT in `workspace_integrations.oauth_data` directly:
| Table | What's Stored |
|-------|---------------|
| `workspace_integrations` | `oauth_data` JSON with `base_url`, `client_id`, `client_secret`, `instance_shared` flag; `resource_path` pointing to the variable |
| `variable` | Encrypted `access_token` (at the path stored in `resource_path`), linked to `account` via `account` column |
| `account` | `refresh_token`, keyed by `workspace_id` + `client` (service name) + `is_workspace_integration = true` |
The `decrypt_oauth_data()` function in `lib.rs` assembles these into a unified struct:
```rust
pub struct OAuthConfig {
pub base_url: String,
pub access_token: String, // decrypted from variable
pub refresh_token: Option<String>, // from account table
pub client_id: String, // from oauth_data or instance settings
pub client_secret: String, // from oauth_data or instance settings
}
```
Instance-level sharing: when `oauth_data.instance_shared == true`, `client_id` and `client_secret` are read from global settings instead of workspace_integrations.
### URL Resolution
The `resolve_endpoint()` helper handles both absolute and relative OAuth URLs:
```rust
pub fn resolve_endpoint(base_url: &str, endpoint: &str) -> String {
if endpoint.starts_with("http://") || endpoint.starts_with("https://") {
endpoint.to_string() // Google: absolute URLs
} else {
format!("{}{}", base_url, endpoint) // Nextcloud: relative paths
}
}
```
### ServiceName Methods
`ServiceName` is the central registry enum. Each variant must implement these match arms:
| Method | Purpose |
|--------|---------|
| `as_str()` | Lowercase identifier (e.g., `"google"`) |
| `as_trigger_kind()` | Maps to `TriggerKind` enum |
| `as_job_trigger_kind()` | Maps to `JobTriggerKind` enum |
| `token_endpoint()` | OAuth token endpoint (relative or absolute) |
| `auth_endpoint()` | OAuth authorization endpoint |
| `oauth_scopes()` | Space-separated OAuth scopes |
| `resource_type()` | Resource type for token storage (e.g., `"gworkspace"`) |
| `extra_auth_params()` | Extra OAuth params (e.g., Google needs `access_type=offline`, `prompt=consent`) |
| `integration_service()` | Maps to the workspace integration service (usually `*self`) |
| `TryFrom<String>` | Parse from string |
| `Display` | Delegates to `as_str()` |
---
## Step-by-Step Implementation Guide
### Step 1: Database Migration
Create a new migration file: `backend/migrations/YYYYMMDDHHMMSS_newservice_trigger.up.sql`
```sql
-- Add the service to the native_trigger_service enum
ALTER TYPE native_trigger_service ADD VALUE IF NOT EXISTS 'newservice';
-- Add to TRIGGER_KIND enum (used for trigger tracking)
ALTER TYPE TRIGGER_KIND ADD VALUE IF NOT EXISTS 'newservice';
-- Add to job_trigger_kind enum (used for job tracking)
ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'newservice';
```
Also create the corresponding down migration.
### Step 2: Update windmill-common Enums
#### `backend/windmill-common/src/triggers.rs`
Add variant to `TriggerKind` enum, and update `to_key()` and `fmt()` implementations.
#### `backend/windmill-common/src/jobs.rs`
Add variant to `JobTriggerKind` enum and update the `Display` implementation.
### Step 3: Backend Service Module
Create a new directory: `backend/windmill-native-triggers/src/newservice/`
#### `mod.rs` - Type Definitions
```rust
use serde::{Deserialize, Serialize};
pub mod external;
// pub mod routes; // Only if you need additional service-specific routes
/// OAuth data deserialized from the three-table pattern.
/// The actual structure is built by decrypt_oauth_data() from variable + account + workspace_integrations.
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct NewServiceOAuthData {
pub base_url: String, // from workspace_integrations.oauth_data
pub access_token: String, // decrypted from variable table
pub refresh_token: Option<String>, // from account table
// Note: client_id and client_secret are in OAuthConfig, not here
// unless the service needs them at runtime for API calls
}
/// Configuration provided by user when creating/updating a trigger.
/// Stored as JSON in native_trigger.service_config.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct NewServiceConfig {
// Service-specific configuration fields
pub folder_path: String,
pub file_filter: Option<String>,
}
/// Data retrieved from the external service about a trigger.
/// Returned by the get() method and shown in the UI.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct NewServiceTriggerData {
pub folder_path: String,
pub file_filter: Option<String>,
// Fields that shouldn't affect service_config comparison should use #[serde(skip_serializing)]
}
/// Response from external service when creating a trigger/webhook.
#[derive(Debug, Deserialize)]
pub struct CreateTriggerResponse {
pub id: String,
}
/// Handler struct (stateless, used for routing)
#[derive(Copy, Clone)]
pub struct NewService;
```
#### `external.rs` - External Trait Implementation
```rust
use async_trait::async_trait;
use reqwest::Method;
use sqlx::PgConnection;
use std::collections::HashMap;
use windmill_common::{
error::{Error, Result},
BASE_URL, DB,
};
use crate::{
generate_webhook_service_url, External, NativeTrigger, NativeTriggerData, ServiceName,
sync::{SyncError, TriggerSyncInfo},
};
use super::{NewService, NewServiceConfig, NewServiceOAuthData, NewServiceTriggerData, CreateTriggerResponse};
#[async_trait]
impl External for NewService {
type ServiceConfig = NewServiceConfig;
type TriggerData = NewServiceTriggerData;
type OAuthData = NewServiceOAuthData;
type CreateResponse = CreateTriggerResponse;
const SERVICE_NAME: ServiceName = ServiceName::NewService;
const DISPLAY_NAME: &'static str = "New Service";
const SUPPORT_WEBHOOK: bool = true;
const TOKEN_ENDPOINT: &'static str = "/oauth/token";
const REFRESH_ENDPOINT: &'static str = "/oauth/token";
const AUTH_ENDPOINT: &'static str = "/oauth/authorize";
async fn create(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
webhook_token: &str,
data: &NativeTriggerData<Self::ServiceConfig>,
db: &DB,
tx: &mut PgConnection,
) -> Result<Self::CreateResponse> {
let base_url = &*BASE_URL.read().await;
// external_id is None during create (we get it from the response)
let webhook_url = generate_webhook_service_url(
base_url, w_id, &data.script_path, data.is_flow,
None, Self::SERVICE_NAME, webhook_token,
);
let url = format!("{}/api/webhooks/create", oauth_data.base_url);
let payload = serde_json::json!({
"callback_url": webhook_url,
"folder_path": data.service_config.folder_path,
});
let response: CreateTriggerResponse = self
.http_client_request(&url, Method::POST, w_id, tx, db, None, Some(&payload))
.await?;
Ok(response)
}
/// Update returns the resolved service_config as JSON.
/// For services using the update+get pattern, call self.get() and serialize.
async fn update(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
webhook_token: &str,
data: &NativeTriggerData<Self::ServiceConfig>,
db: &DB,
tx: &mut PgConnection,
) -> Result<serde_json::Value> {
let base_url = &*BASE_URL.read().await;
let webhook_url = generate_webhook_service_url(
base_url, w_id, &data.script_path, data.is_flow,
Some(external_id), Self::SERVICE_NAME, webhook_token,
);
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
let payload = serde_json::json!({
"callback_url": webhook_url,
"folder_path": data.service_config.folder_path,
});
let _: serde_json::Value = self
.http_client_request(&url, Method::PUT, w_id, tx, db, None, Some(&payload))
.await?;
// Fetch back the updated state to get the resolved config
let trigger_data = self.get(w_id, oauth_data, external_id, db, tx).await?;
serde_json::to_value(&trigger_data)
.map_err(|e| Error::InternalErr(format!("Failed to serialize trigger data: {}", e)))
}
async fn get(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
db: &DB,
tx: &mut PgConnection,
) -> Result<Self::TriggerData> {
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
self.http_client_request::<_, ()>(&url, Method::GET, w_id, tx, db, None, None).await
}
async fn delete(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
db: &DB,
tx: &mut PgConnection,
) -> Result<()> {
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
let _: serde_json::Value = self
.http_client_request::<_, ()>(&url, Method::DELETE, w_id, tx, db, None, None)
.await
.or_else(|e| match &e {
Error::InternalErr(msg) if msg.contains("404") => Ok(serde_json::Value::Null),
_ => Err(e),
})?;
Ok(())
}
async fn exists(
&self,
w_id: &str,
oauth_data: &Self::OAuthData,
external_id: &str,
db: &DB,
tx: &mut PgConnection,
) -> Result<bool> {
match self.get(w_id, oauth_data, external_id, db, tx).await {
Ok(_) => Ok(true),
Err(Error::NotFound(_)) => Ok(false),
Err(e) => Err(e),
}
}
/// Background maintenance. Choose the right pattern for your service:
/// - For services with queryable external state: use reconcile_with_external_state()
/// - For channel-based services with expiration: implement renewal logic
async fn maintain_triggers(
&self,
db: &DB,
workspace_id: &str,
triggers: &[NativeTrigger],
oauth_data: &Self::OAuthData,
synced: &mut Vec<TriggerSyncInfo>,
errors: &mut Vec<SyncError>,
) {
// Option A: Reconcile with external state (Nextcloud pattern)
// Fetch all triggers from external service and compare with DB
let external_triggers = match self.list_all(workspace_id, oauth_data, db).await {
Ok(triggers) => triggers,
Err(e) => {
errors.push(SyncError {
resource_path: format!("workspace:{}", workspace_id),
error_message: format!("Failed to list triggers: {}", e),
error_type: "api_error".to_string(),
});
return;
}
};
// Convert to (external_id, config_json) pairs
let external_pairs: Vec<(String, serde_json::Value)> = external_triggers
.into_iter()
.map(|t| (t.id.clone(), serde_json::to_value(&t).unwrap_or_default()))
.collect();
crate::sync::reconcile_with_external_state(
db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
).await;
}
fn external_id_and_metadata_from_response(
&self,
resp: &Self::CreateResponse,
) -> (String, Option<serde_json::Value>) {
(resp.id.clone(), None)
}
// service_config_from_create_response: NOT overridden (returns None).
// This means the handler uses the update+get pattern after create.
// Override and return Some(...) to skip the update+get cycle (Google pattern).
}
impl NewService {
/// Private helper to list all triggers from the external service.
async fn list_all(
&self,
w_id: &str,
oauth_data: &<Self as External>::OAuthData,
db: &DB,
) -> Result<Vec<<Self as External>::TriggerData>> {
// Implementation depends on the external service's API
todo!()
}
}
```
### Step 4: Update lib.rs Registry
In `backend/windmill-native-triggers/src/lib.rs`:
```rust
// Service modules - add new services here:
#[cfg(feature = "native_trigger")]
pub mod newservice; // <-- Add this
// ServiceName enum - add variant:
pub enum ServiceName {
Nextcloud,
Google,
NewService, // <-- Add this
}
// Then add match arms in ALL ServiceName methods:
// as_str(), as_trigger_kind(), as_job_trigger_kind(), token_endpoint(),
// auth_endpoint(), oauth_scopes(), resource_type(), extra_auth_params(),
// integration_service(), TryFrom<String>, Display
```
### Step 5: Update handler.rs Routes
In `backend/windmill-native-triggers/src/handler.rs`:
```rust
pub fn generate_native_trigger_routers() -> Router {
// ...
#[cfg(feature = "native_trigger")]
{
use crate::newservice::NewService;
return router
.nest("/nextcloud", service_routes(NextCloud))
.nest("/google", service_routes(Google))
.nest("/newservice", service_routes(NewService)); // <-- Add this
}
// ...
}
```
### Step 6: Update sync.rs
In `backend/windmill-native-triggers/src/sync.rs`:
```rust
pub async fn sync_all_triggers(db: &DB) -> Result<BackgroundSyncResult> {
// ...
#[cfg(feature = "native_trigger")]
{
use crate::newservice::NewService;
// ... existing service syncs ...
// New service sync
let (service_name, result) = sync_service_triggers(db, NewService).await;
total_synced += result.synced_triggers.len();
total_errors += result.errors.len();
service_results.insert(service_name, result);
}
// ...
}
```
### Step 7: Frontend Service Registry
In `frontend/src/lib/components/triggers/native/utils.ts`:
Add to `NATIVE_TRIGGER_SERVICES`, `getTriggerIconName()`, and `getServiceIcon()`.
### Step 8: Frontend Trigger Form Component
Create: `frontend/src/lib/components/triggers/native/services/newservice/NewServiceTriggerForm.svelte`
### Step 9: Frontend Icon Component
Create: `frontend/src/lib/components/icons/NewServiceIcon.svelte`
### Step 10: Update NativeTriggerEditor
Check `frontend/src/lib/components/triggers/native/NativeTriggerEditor.svelte` to ensure it dynamically loads form components based on service name.
### Step 11: Workspace Integration UI
Add your service to the `supportedServices` map in `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte`:
```typescript
const supportedServices: Record<string, ServiceConfig> = {
// ... existing services ...
newservice: {
name: 'newservice',
displayName: 'New Service',
description: 'Connect to New Service for triggers',
icon: NewServiceIcon,
docsUrl: 'https://www.windmill.dev/docs/integrations/newservice',
requiresBaseUrl: false, // false for cloud services, true for self-hosted
setupInstructions: [
'Step 1: Create an OAuth app on the service',
'Step 2: Configure the redirect URI shown below',
'Step 3: Enter the client credentials below'
]
}
}
```
### Step 12: Update `frontend/src/lib/components/triggers/utils.ts`
Update ALL of these maps/functions:
1. `triggerIconMap` - import and add icon
2. `triggerDisplayNamesMap` - add display name
3. `triggerTypeOrder` in `sortTriggers()` - add type
4. `getLightConfig()` - add case for your service
5. `getTriggerLabel()` - add case for your service
6. `jobTriggerKinds` - add to array
7. `countPropertyMap` - add count property
8. `triggerSaveFunctions` - add save function
### Step 13: Update TriggersBadge Component
In `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte`:
1. Import the icon
2. Add to `baseConfig` with `countKey` (the dynamic `availableNativeServices` loop does NOT set `countKey`)
3. Add to the `allTypes` array
### Step 14: Update TriggersWrapper.svelte
In `frontend/src/lib/components/triggers/TriggersWrapper.svelte`:
Add a `{:else if selectedTrigger.type === 'yourservice'}` case that renders `<NativeTriggersPanel service="yourservice" ...>` with the same props pattern as the existing native trigger cases (e.g., `nextcloud`).
### Step 15: Update AddTriggersButton.svelte
In `frontend/src/lib/components/triggers/AddTriggersButton.svelte`:
1. Add `yourserviceAvailable` state variable
2. Add `setYourserviceState()` async function using `isServiceAvailable('yourservice', $workspaceStore!)`
3. Call it at module level
4. Add a dropdown entry to `addTriggerItems` with `hidden: !yourserviceAvailable`
### Step 16: Update TriggersEditor.svelte Delete Handling
In `frontend/src/lib/components/triggers/TriggersEditor.svelte`:
Add your service to the `nativeTriggerServices` map in `deleteDeployedTrigger()`. Native triggers use `NativeTriggerService.deleteNativeTrigger({ workspace, serviceName, externalId })` instead of the standard `path`-based delete.
### Step 17: Update OpenAPI Spec and Regenerate Types
Add to `JobTriggerKind` enum in `backend/windmill-api/openapi.yaml`, then:
```bash
cd frontend && npm run generate-backend-client
```
---
## Special Patterns
### Unified Service with `trigger_type` (Google Pattern)
When a single service handles multiple trigger types (e.g., Google Drive + Calendar share OAuth and API patterns), use a single `ServiceName` variant with a discriminator field:
```rust
pub enum GoogleTriggerType { Drive, Calendar }
pub struct GoogleServiceConfig {
pub trigger_type: GoogleTriggerType,
// Drive-specific fields (only used when trigger_type = Drive)
pub resource_id: Option<String>,
pub resource_name: Option<String>,
// Calendar-specific fields (only used when trigger_type = Calendar)
pub calendar_id: Option<String>,
pub calendar_name: Option<String>,
// Metadata set after creation
pub google_resource_id: Option<String>,
pub expiration: Option<String>,
}
```
Branch in trait methods based on `trigger_type`. Frontend uses a `ToggleButtonGroup` to switch between types. This keeps the codebase simpler (one service, one OAuth flow, one set of routes).
See `backend/windmill-native-triggers/src/google/` for the reference implementation.
### Skipping update+get After Create (Google Pattern)
Override `service_config_from_create_response()` to return `Some(config)` when the external_id is known before the create call:
```rust
fn service_config_from_create_response(
&self,
data: &NativeTriggerData<Self::ServiceConfig>,
resp: &Self::CreateResponse,
) -> Option<serde_json::Value> {
// Clone input config, add metadata from response
let mut config = data.service_config.clone();
config.google_resource_id = Some(resp.resource_id.clone());
config.expiration = Some(resp.expiration.clone());
Some(serde_json::to_value(&config).unwrap())
}
```
### Services with Absolute OAuth Endpoints (Google)
Unlike self-hosted services where OAuth endpoints are relative paths appended to `base_url`, services like Google have absolute URLs:
```rust
// Nextcloud: relative paths
ServiceName::Nextcloud => "/apps/oauth2/api/v1/token",
// Google: absolute URLs
ServiceName::Google => "https://oauth2.googleapis.com/token",
```
The `resolve_endpoint()` function handles both. For services with absolute endpoints:
- `base_url` can be empty
- `requiresBaseUrl: false` in the frontend workspace integration config
- Add `extra_auth_params()` if needed (Google requires `access_type=offline` and `prompt=consent`)
### Channel-Based Push Notifications with Renewal (Google Pattern)
For services using expiring watch channels instead of persistent webhooks:
1. Store expiration in `service_config` (as part of `ServiceConfig`)
2. In `maintain_triggers()`, implement renewal logic instead of using `reconcile_with_external_state()`:
```rust
async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors) {
for trigger in triggers {
if should_renew_channel(trigger) {
self.renew_channel(db, trigger, oauth_data).await;
}
}
}
```
3. Renewal: best-effort stop old channel, create new one with same external_id, update service_config with new expiration
4. Google example: Drive channels expire in 24h (renew when <1h left), Calendar channels expire in 7 days (renew when <1 day left)
### reconcile_with_external_state (Nextcloud Pattern)
The reusable function in `sync.rs` compares external triggers with DB state:
- Triggers missing externally: sets error "Trigger no longer exists on external service"
- Triggers present externally: clears errors, updates service_config if it differs
Usage in `maintain_triggers()`:
```rust
let external_pairs: Vec<(String, serde_json::Value)> = /* fetch from external */;
crate::sync::reconcile_with_external_state(
db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
).await;
```
### Webhook Payload Processing
Override `prepare_webhook()` to parse service-specific payloads into script/flow args:
```rust
async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned> {
let mut args = HashMap::new();
args.insert("event_type".to_string(), Box::new(headers.get("x-event-type").cloned()) as _);
args.insert("payload".to_string(), Box::new(serde_json::from_str::<serde_json::Value>(&body)?) as _);
Ok(PushArgsOwned { extra: None, args })
}
```
Then register in `prepare_native_trigger_args()` in `lib.rs`:
```rust
pub async fn prepare_native_trigger_args(service_name, db, w_id, headers, body) -> Result<Option<PushArgsOwned>> {
match service_name {
ServiceName::Google => { /* ... */ Ok(Some(args)) }
ServiceName::NewService => { /* ... */ Ok(Some(args)) }
ServiceName::Nextcloud => Ok(None), // Uses default body parsing
}
}
```
### Instance-Level OAuth Credentials
When `workspace_integrations.oauth_data.instance_shared == true`, `decrypt_oauth_data()` reads `client_id` and `client_secret` from instance-level global settings instead of workspace-level. This allows admins to share OAuth app credentials across workspaces.
The frontend handles this via the `generate_instance_connect_url` endpoint in `workspace_integrations.rs`.
---
## Testing Checklist
- [ ] Database migration runs successfully
- [ ] `cargo check -p windmill-native-triggers --features native_trigger` passes
- [ ] `npx svelte-check --threshold error` passes (in frontend/)
- [ ] Service appears in workspace integrations list
- [ ] OAuth flow completes successfully
- [ ] Can create a new trigger
- [ ] Can view trigger details
- [ ] Can update trigger configuration
- [ ] Can delete trigger
- [ ] Webhook receives and processes payloads
- [ ] Background sync works correctly (reconciliation or channel renewal)
- [ ] Error handling works (expired tokens, service unavailable)
---
## Reference Implementations
### Nextcloud (Self-Hosted, Update+Get Pattern)
| File | Purpose |
|------|---------|
| `nextcloud/mod.rs` | Types: NextCloudOAuthData, NextcloudServiceConfig, NextCloudTriggerData |
| `nextcloud/external.rs` | External trait: uses update+get pattern, reconcile_with_external_state for sync |
| `nextcloud/routes.rs` | Additional route: `GET /events` |
Key patterns: relative OAuth endpoints, base_url required, list_all + reconcile for sync, update returns JSON from get().
### Google (Cloud, Unified Service, Short Create)
| File | Purpose |
|------|---------|
| `google/mod.rs` | Types: GoogleServiceConfig with trigger_type discriminator, GoogleTriggerType enum |
| `google/external.rs` | External trait: overrides service_config_from_create_response, channel renewal for sync |
| `google/routes.rs` | Additional routes: `GET /calendars`, `GET /drive/files`, `GET /drive/shared_drives` |
Key patterns: absolute OAuth endpoints, empty base_url, trigger_type for Drive/Calendar, expiring watch channels with renewal, service_config_from_create_response skips update+get, get() reconstructs data from stored service_config (no external "get channel" API).
-109
View File
@@ -1,109 +0,0 @@
---
name: pr
description: Open a draft pull request on GitHub. MUST use when you want to create/open a PR.
---
# Pull Request Skill
Create a draft pull request with a clear title and explicit description of changes.
## Instructions
1. **Analyze branch changes**: Understand all commits since diverging from main
2. **Push to remote**: Ensure all commits are pushed
3. **Create draft PR**: Always open as draft for review before merging
## PR Title Format
Follow conventional commit format for the PR title:
```
<type>: <description>
```
### Types
- `feat`: New feature or capability
- `fix`: Bug fix
- `refactor`: Code restructuring
- `docs`: Documentation changes
- `chore`: Maintenance tasks
- `perf`: Performance improvements
### Title Rules
- Keep under 70 characters
- Use lowercase, imperative mood
- No period at the end
- If `*_ee.rs` files were modified, prefix with `[ee]`: `[ee] <type>: <description>`
## PR Body Format
The body MUST be explicit about what changed. Structure:
```markdown
## Summary
<Clear description of what this PR does and why>
## Changes
- <Specific change 1>
- <Specific change 2>
- <Specific change 3>
## Test plan
- [ ] <How to verify change 1>
- [ ] <How to verify change 2>
---
Generated with [Claude Code](https://claude.com/claude-code)
```
## Execution Steps
1. Run `git status` to check for uncommitted changes
2. Run `git log main..HEAD --oneline` to see all commits in this branch
3. Run `git diff main...HEAD` to see the full diff against main
4. Check if remote branch exists and is up to date:
```bash
git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
```
5. Push to remote if needed: `git push -u origin HEAD`
6. Create draft PR using gh CLI:
```bash
gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
## Summary
<description>
## Changes
- <change 1>
- <change 2>
## Test plan
- [ ] <test 1>
- [ ] <test 2>
---
Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"
```
7. Return the PR URL to the user
## EE Companion PR (when `*_ee.rs` files were modified)
The `*_ee.rs` files in the windmill repo are **symlinks** to `windmill-ee-private` — changes won't appear in `git diff` of the windmill repo. Instead, check the EE repo for uncommitted or unpushed changes.
Follow the full EE PR workflow in `docs/enterprise.md`. The key PR-specific details:
1. Find the EE repo/worktree: see "Finding the EE Repo" in `docs/enterprise.md`
2. Check for changes: `git -C <ee-path> status --short`
- If there are no changes in the EE repo, skip this entire section
3. Follow steps 15 from the "EE PR Workflow" in `docs/enterprise.md`
4. Create the companion PR (title does NOT get the `[ee]` prefix):
```bash
gh pr create --draft --repo windmill-labs/windmill-ee-private --title "<type>: <description>" --body "$(cat <<'EOF'
Companion PR for windmill-labs/windmill#<PR_NUMBER>
---
Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"
```
5. Commit `ee-repo-ref.txt` and push the updated windmill branch
-38
View File
@@ -1,38 +0,0 @@
---
name: refine
description: End-of-session reflection. Reviews friction encountered during the session and proposes updates to docs/ to capture lessons learned.
---
# Refine Skill
Reflect on the current session and update documentation with lessons learned.
## Instructions
1. **Identify friction**: Review what happened in this session:
- Run `git diff main...HEAD --stat` to see what files were touched
- Think about: what was slow, what failed, what required multiple attempts, what information was missing or hard to find
2. **Read current docs**: Read the docs that were relevant to this session:
- `docs/validation.md`
- `docs/enterprise.md`
- `docs/autonomous-mode.md`
- Any skills that were invoked
3. **Propose updates**: For each piece of friction, decide if it warrants a doc update:
- **Missing knowledge**: Information you had to discover that should be documented
- **Wrong guidance**: Instructions that led you astray
- **Missing validation rule**: A check that should be in the validation matrix
- **New pattern**: A codebase pattern worth capturing for next time
4. **Apply updates**: Edit the relevant `docs/` files. Keep changes minimal and specific — add only what would have saved time this session.
5. **Report**: Summarize what was added/changed and why.
## Rules
- Only add knowledge confirmed by this session — no speculative additions
- Keep docs concise — add a line or two, not a paragraph
- If a whole new doc is needed, create it in `docs/` and add a pointer in `CLAUDE.md`
- Don't update skills unless a coding pattern was genuinely wrong
- Don't add things Claude already knows — only Windmill-specific knowledge
-107
View File
@@ -1,107 +0,0 @@
---
name: rust-backend
description: Rust coding guidelines for the Windmill backend. MUST use when writing or modifying Rust code in the backend directory.
---
# Windmill Rust Patterns
Apply these Windmill-specific patterns when writing Rust code in `backend/`.
## Error Handling
Use `Error` from `windmill_common::error`. Return `Result<T, Error>` or `JsonResult<T>`:
```rust
use windmill_common::error::{Error, Result};
pub async fn get_job(db: &DB, id: Uuid) -> Result<Job> {
sqlx::query_as!(Job, "SELECT id, workspace_id FROM v2_job WHERE id = $1", id)
.fetch_optional(db)
.await?
.ok_or_else(|| Error::NotFound("job not found".to_string()))?;
}
```
Never panic in library code. Reserve `.unwrap()` for compile-time guarantees.
## SQLx Patterns
**Never use `SELECT *`** — always list columns explicitly. Critical for backwards compatibility when workers lag behind API version:
```rust
// Correct
sqlx::query_as!(Job, "SELECT id, workspace_id, path FROM v2_job WHERE id = $1", id)
// Wrong — breaks when columns are added
sqlx::query_as!(Job, "SELECT * FROM v2_job WHERE id = $1", id)
```
Use batch operations to avoid N+1:
```rust
// Preferred — single query with IN clause
sqlx::query!("SELECT ... WHERE id = ANY($1)", &ids[..]).fetch_all(db).await?
```
Use transactions for multi-step operations. Parameterize all queries.
## JSON Handling
Prefer `Box<serde_json::value::RawValue>` over `serde_json::Value` when storing/passing JSON without inspection:
```rust
pub struct Job {
pub args: Option<Box<serde_json::value::RawValue>>,
}
```
Only use `serde_json::Value` when you need to inspect or modify the JSON.
## Serde Optimizations
```rust
#[derive(Serialize, Deserialize)]
pub struct Job {
#[serde(skip_serializing_if = "Option::is_none")]
pub parent_job: Option<Uuid>,
#[serde(skip_serializing_if = "Vec::is_empty")]
pub tags: Vec<String>,
#[serde(default)]
pub priority: i32,
}
```
## Async & Concurrency
Never block the async runtime. Use `spawn_blocking` for CPU-intensive work:
```rust
let result = tokio::task::spawn_blocking(move || expensive_computation(&data)).await?;
```
**Mutex selection**: Prefer `std::sync::Mutex` (or `parking_lot::Mutex`) for data protection. Only use `tokio::sync::Mutex` when holding locks across `.await` points.
Use `tokio::sync::mpsc` (bounded) for channels. Avoid `std::thread::sleep` in async contexts.
## Module Structure & Visibility
- Use `pub(crate)` instead of `pub` when possible
- Place new code in the appropriate crate based on functionality
- API endpoints go in `windmill-api/src/` organized by domain
- Shared functionality goes in `windmill-common/src/`
## Code Navigation
Always use rust-analyzer LSP for go-to-definition, find-references, and type info. Do not guess at module paths.
## Axum Handlers
Destructure extractors directly in function signatures:
```rust
async fn process_job(
Extension(db): Extension<DB>,
Path((workspace, job_id)): Path<(String, Uuid)>,
Query(pagination): Query<Pagination>,
) -> Result<Json<Job>> { ... }
```
-80
View File
@@ -1,80 +0,0 @@
---
name: svelte-frontend
description: Svelte coding guidelines for the Windmill frontend. MUST use when writing or modifying code in the frontend directory.
---
# Windmill Svelte Patterns
Apply these Windmill-specific patterns when writing Svelte code in `frontend/`. For general Svelte 5 syntax (runes, snippets, event handling), use the Svelte MCP server.
## Windmill UI Components (MUST use)
Always use Windmill's design-system components. Never use raw HTML elements.
### Buttons — `<Button>`
```svelte
<script>
import { Button } from '$lib/components/common'
import { ChevronLeft } from 'lucide-svelte'
</script>
<Button variant="default" onclick={handleClick}>Label</Button>
<Button startIcon={{ icon: ChevronLeft }} iconOnly onclick={prev} />
```
Props: `variant?: 'accent' | 'accent-secondary' | 'default' | 'subtle'`, `unifiedSize?: 'sm' | 'md' | 'lg'`, `startIcon?: { icon: SvelteComponent }`, `iconOnly?: boolean`, `disabled?: boolean`
### Text inputs — `<TextInput>`
```svelte
<script>
import { TextInput } from '$lib/components/common'
</script>
<TextInput bind:value={val} placeholder="Enter value" />
```
Props: `value?: string | number` (bindable), `placeholder?: string`, `disabled?: boolean`, `error?: string | boolean`, `size?: 'sm' | 'md' | 'lg'`
### Selects — `<Select>`
```svelte
<script>
import Select from '$lib/components/select/Select.svelte'
</script>
<Select items={[{ label: 'Jan', value: 1 }]} bind:value={selected} />
```
Props: `items?: Array<{ label?: string; value: any }>`, `value` (bindable), `placeholder?: string`, `clearable?: boolean`, `size?: 'sm' | 'md' | 'lg'`
### Icons — `lucide-svelte`
Never write inline SVGs. Import from `lucide-svelte`:
```svelte
<script>
import { ChevronLeft, X } from 'lucide-svelte'
</script>
<ChevronLeft size={16} />
```
## Form Components
Form components (TextInput, Toggle, Select, etc.) should use the unified size system when placed together.
## Styling
- Use Tailwind CSS for all styling — no custom CSS
- Use Windmill's theming classes for colors/surfaces (see `frontend/brand-guidelines.md`)
- Read component props JSDoc before using them
## Svelte MCP Server
Use the Svelte MCP tools when working on Svelte code:
1. **list-sections**: Call first to discover available docs
2. **get-documentation**: Fetch relevant sections based on use_cases
3. **svelte-autofixer**: MUST use on all Svelte code before finalizing — keep calling until no issues
4. **playground-link**: Only after user confirms and code was NOT written to project files
+4 -4
View File
@@ -13,10 +13,10 @@ on:
jobs:
check-membership:
if: |
(github.event_name == 'issue_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
(github.event_name == 'pull_request_review_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
(github.event_name == 'pull_request_review' && startsWith(github.event.review.body, '/ai') && !startsWith(github.event.review.body, '/ai-fast')) ||
(github.event_name == 'issues' && startsWith(github.event.issue.body, '/ai') && !startsWith(github.event.issue.body, '/ai-fast'))
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '/ai')) ||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/ai')) ||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '/ai')) ||
(github.event_name == 'issues' && contains(github.event.issue.body, '/ai'))
uses: ./.github/workflows/check-org-membership.yml
secrets:
access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
@@ -1,12 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "25ac66a1022c41267df199a95f532b0f778c25fbe0f7a7f9734c1f7e536ed6ce"
}
@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO raw_script_temp (workspace_id, hash, content, created_at)\n VALUES ($1, $2, $3, NOW())\n ON CONFLICT (workspace_id, hash) DO UPDATE SET created_at = NOW()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Bpchar",
"Text"
]
},
"nullable": []
},
"hash": "2d523cd0d5b7107b15846b885fa40af492d4c8a8871cef972980150f319fe6ff"
}
@@ -1,28 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT parent_job, flow_step_id FROM v2_job WHERE id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "flow_step_id",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
true,
true
]
},
"hash": "32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n path,\n summary,\n description\n FROM\n flow\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"query": "SELECT \n path,\n summary,\n description\n FROM\n flow\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"describe": {
"columns": [
{
@@ -31,5 +31,5 @@
false
]
},
"hash": "4615b37cb848f9589622426d291c721e532b230c527deb701e24605c7027e38b"
"hash": "33367c42e87e78ae987c0966dc4d445c5eff75b2e2843ffd7a46b03cbaea9ae8"
}
@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n f.schema AS \"schema: serde_json::Value\",\n fv.value->>'preprocessor_module' IS NOT NULL AS \"has_preprocessor: bool\"\n FROM flow f\n LEFT JOIN flow_version fv ON fv.id = f.versions[array_length(f.versions, 1)]\n AND fv.workspace_id = f.workspace_id\n WHERE f.path = $1 AND f.workspace_id = $2 AND NOT f.archived",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "schema: serde_json::Value",
"type_info": "Json"
},
{
"ordinal": 1,
"name": "has_preprocessor: bool",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
true,
null
]
},
"hash": "372ec62fad93831b00d44f6c52a148b6bfa6008e49b2a80bec06c76d9432ec77"
}
@@ -1,27 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH step_index AS (\n SELECT idx::text AS idx\n FROM v2_job_status,\n jsonb_array_elements(flow_status->'modules') WITH ORDINALITY arr(elem, idx)\n WHERE id = $1\n AND elem->>'id' = $5\n LIMIT 1\n ), completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n $3::text::jsonb,\n CASE WHEN si.idx IS NOT NULL\n THEN jsonb_set(\n s.flow_status,\n ARRAY['modules', (si.idx::int - 1)::text],\n $6::jsonb\n )\n ELSE s.flow_status\n END,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n LEFT JOIN step_index si ON true\n WHERE q.id = $1\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id = $1\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n VALUES ($4, $1, $2)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "x",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Text",
"Text",
"Text",
"Jsonb"
]
},
"nullable": [
null
]
},
"hash": "4461fe84370e7f07b4423ea5e71b913dd6ee9c655f9ec7087cc0fec9b9c3099a"
}
@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO v2_job (id, kind, tag, created_by, permissioned_as, permissioned_as_email, workspace_id, parent_job)\n VALUES ($1, 'noop', 'deno', 'test-user', 'u/test-user', 'test@windmill.dev', $2, $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Varchar",
"Uuid"
]
},
"nullable": []
},
"hash": "4538bea4159677d8e653159d7d01649cae08e6ffef668a7cbac49b312bf30766"
}
@@ -1,35 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "first_started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "job_id_to_debounce",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618"
}
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT debounced_times FROM debounce_key WHERE key = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false
]
},
"hash": "6009cf60c43608bb1c7924dbd0a9cdc01986d787eb24f0fffe4550d939851e22"
}
@@ -1,35 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "first_started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "job_id_to_debounce",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073"
}
@@ -1,35 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "debounced_times",
"type_info": "Int4"
},
{
"ordinal": 1,
"name": "first_started_at",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "job_id_to_debounce",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid",
"Varchar"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b"
}
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT content FROM raw_script_temp WHERE hash = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "content",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Bpchar"
]
},
"nullable": [
false
]
},
"hash": "88ec0ddcc86fb67089b551ccbce7b932800e33cd462a651f7e6716929ee9b6f2"
}
@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n schema AS \"schema: serde_json::Value\",\n has_preprocessor\n FROM script\n WHERE path = $1 AND workspace_id = $2\n AND NOT archived AND NOT deleted\n ORDER BY created_at DESC\n LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "schema: serde_json::Value",
"type_info": "Json"
},
{
"ordinal": 1,
"name": "has_preprocessor",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
true,
true
]
},
"hash": "909a095d679ee7b8266fc93fe0c598e4b6583daa2b0d7c593fc72ff9b8d1d06b"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n route_path,\n http_method AS \"http_method: _\",\n request_type AS \"request_type: _\",\n workspaced_route,\n summary,\n description,\n authentication_method AS \"authentication_method: _\",\n authentication_resource_path,\n script_path,\n is_flow,\n wrap_body\n FROM\n http_trigger\n WHERE\n path ~ ANY($1) AND\n route_path ~ ANY($2) AND\n workspace_id = $3\n ",
"query": "\n SELECT\n route_path,\n http_method AS \"http_method: _\",\n request_type AS \"request_type: _\",\n workspaced_route,\n summary,\n description,\n authentication_method AS \"authentication_method: _\",\n authentication_resource_path\n FROM\n http_trigger\n WHERE\n path ~ ANY($1) AND\n route_path ~ ANY($2) AND\n workspace_id = $3\n ",
"describe": {
"columns": [
{
@@ -80,21 +80,6 @@
"ordinal": 7,
"name": "authentication_resource_path",
"type_info": "Varchar"
},
{
"ordinal": 8,
"name": "script_path",
"type_info": "Varchar"
},
{
"ordinal": 9,
"name": "is_flow",
"type_info": "Bool"
},
{
"ordinal": 10,
"name": "wrap_body",
"type_info": "Bool"
}
],
"parameters": {
@@ -112,11 +97,8 @@
true,
true,
false,
true,
false,
false,
false
true
]
},
"hash": "1cb21a66ffc89ebe53fd8f58690eec4cf11cb4b7816738202b474e8d3ffef427"
"hash": "9360d00990822f153ff09c7905ae3180f07d02f38ac12d07a5664d93f160e7ee"
}
@@ -1,46 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH RECURSIVE chain AS (\n SELECT\n j.id,\n j.parent_job,\n j.flow_step_id,\n 1 AS depth\n FROM v2_job j\n WHERE j.id = $1\n UNION ALL\n SELECT\n pj.id,\n pj.parent_job,\n pj.flow_step_id,\n c.depth + 1\n FROM chain c\n JOIN v2_job pj ON pj.id = c.parent_job\n WHERE c.parent_job IS NOT NULL\n )\n SELECT\n c.id,\n c.parent_job,\n c.flow_step_id,\n EXISTS(SELECT 1 FROM v2_job_queue q WHERE q.id = c.parent_job) AS \"parent_in_queue!\",\n EXISTS(\n SELECT 1 FROM v2_job sib\n WHERE sib.parent_job = c.parent_job\n AND sib.id != c.id\n AND sib.id IN (SELECT sq.id FROM v2_job_queue sq)\n ) AS \"has_other_active_siblings!\"\n FROM chain c\n WHERE c.depth >= 1\n ORDER BY c.depth ASC\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 2,
"name": "flow_step_id",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "parent_in_queue!",
"type_info": "Bool"
},
{
"ordinal": 4,
"name": "has_other_active_siblings!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
null,
null,
null,
null,
null
]
},
"hash": "950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb"
}
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT scheduled_for FROM v2_job_queue WHERE id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "scheduled_for",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
false
]
},
"hash": "a9c2019c7bafb172dfb353870238c02673113b806bc3c7a71354c34255ecd31c"
}
@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_status SET flow_status = (\n SELECT jsonb_set(\n flow_status,\n ARRAY['modules', (idx - 1)::text],\n $2::jsonb\n )\n FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n LIMIT 1\n ) WHERE id = $1 AND (\n SELECT COUNT(*) FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n ) > 0",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Uuid",
"Jsonb",
"Text"
]
},
"nullable": []
},
"hash": "b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c"
}
@@ -1,40 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n WITH RECURSIVE chain AS (\n SELECT\n j.id,\n j.parent_job,\n j.flow_step_id,\n 1 AS depth\n FROM v2_job j\n WHERE j.id = $1\n UNION ALL\n SELECT\n pj.id,\n pj.parent_job,\n pj.flow_step_id,\n c.depth + 1\n FROM chain c\n JOIN v2_job pj ON pj.id = c.parent_job\n WHERE c.parent_job IS NOT NULL\n )\n SELECT\n c.id,\n c.parent_job,\n c.flow_step_id,\n EXISTS(SELECT 1 FROM v2_job_queue q WHERE q.id = c.parent_job) AS \"parent_in_queue!\"\n FROM chain c\n WHERE c.depth >= 1\n ORDER BY c.depth ASC\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
},
{
"ordinal": 1,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 2,
"name": "flow_step_id",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "parent_in_queue!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
null,
null,
null,
null
]
},
"hash": "c79501b30fc28a3ae761579d05f2296e848554e62a00ff7c164109bf3d97f44f"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT content FROM script WHERE path = $1 AND workspace_id = $2\n AND archived = false ORDER BY created_at DESC LIMIT 1\n ",
"query": "\n SELECT content FROM script WHERE path = $1 AND workspace_id = $2\n AND archived = false ORDER BY created_at DESC LIMIT 1\n ",
"describe": {
"columns": [
{
@@ -19,5 +19,5 @@
false
]
},
"hash": "96f6163a164b9ffb4ec52372d7fea158db101f54f0908551b5a4f5e6655e122b"
"hash": "c7cae4cf872fce0a989cf89aa35929218a9d459ee1c2b36a28b110e9741ab623"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n path,\n summary,\n description\n FROM\n script\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"query": "SELECT \n path,\n summary,\n description\n FROM\n script\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
"describe": {
"columns": [
{
@@ -31,5 +31,5 @@
false
]
},
"hash": "69b44efb0144fececccafc8a77d040649bb17e239e5afbc27b915efc4c95d54e"
"hash": "dc36b46b9eb80cb7c92fa72519d117eda99a6f482a073ccd36a6431ef689a3fd"
}
@@ -1,23 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS(\n SELECT 1 FROM v2_job\n WHERE parent_job = $1 AND id != $2\n AND id IN (SELECT id FROM v2_job_queue)\n ) as has",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "has",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Uuid",
"Uuid"
]
},
"nullable": [
null
]
},
"hash": "ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2"
}
@@ -1,15 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE v2_job_queue SET scheduled_for = $1 WHERE id = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Timestamptz",
"Uuid"
]
},
"nullable": []
},
"hash": "f9eabfab66ae102c8a61ae5e6349b5b167ff7939828a747ad09bb06c9d6c6d8d"
}
+3 -1
View File
@@ -16898,6 +16898,8 @@ dependencies = [
"async-recursion",
"itertools 0.14.0",
"lazy_static",
"malachite",
"malachite-bigint",
"pep440_rs",
"phf 0.11.3",
"regex",
@@ -16954,6 +16956,7 @@ dependencies = [
"serde",
"serde_json",
"windmill-parser",
"windmill-types",
]
[[package]]
@@ -17462,7 +17465,6 @@ dependencies = [
"strum 0.27.2",
"tracing",
"uuid",
"windmill-parser",
]
[[package]]
+1 -1
View File
@@ -1 +1 @@
c04f3851c03758662e4936ff4b6e71bc56dbae7e
563877bf1c8b4184f638bab51be89b1c0aec6dad
@@ -1,2 +0,0 @@
DROP INDEX IF EXISTS idx_raw_script_temp_created_at;
DROP TABLE IF EXISTS raw_script_temp;
@@ -1,11 +0,0 @@
-- Temporary storage for raw script content during CLI lock generation
-- Content is stored with hash as key, cleaned up after 1 week
CREATE TABLE raw_script_temp (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id),
hash CHAR(64) NOT NULL,
content TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (workspace_id, hash)
);
CREATE INDEX IF NOT EXISTS idx_raw_script_temp_created_at ON raw_script_temp (created_at);
@@ -13,19 +13,21 @@ regex-lite.workspace = true
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
regex.workspace = true
[dependencies]
windmill-parser.workspace = true
windmill-common.workspace = true
rustpython-parser.workspace = true
malachite.workspace = true
malachite-bigint.workspace = true
phf.workspace = true
itertools.workspace = true
serde_json.workspace = true
anyhow.workspace = true
lazy_static.workspace = true
sqlx.workspace = true
async-recursion.workspace = true
toml.workspace = true
serde.workspace = true
pep440_rs.workspace = true
tracing.workspace = true
[dependencies]
windmill-parser.workspace = true
rustpython-parser.workspace = true
phf.workspace = true
itertools.workspace = true
serde_json.workspace = true
anyhow.workspace = true
lazy_static.workspace = true
@@ -8,14 +8,10 @@
mod mapping;
#[cfg(not(target_arch = "wasm32"))]
use async_recursion::async_recursion;
use itertools::Itertools;
use lazy_static::lazy_static;
#[cfg(not(target_arch = "wasm32"))]
use std::str::FromStr;
#[cfg(not(target_arch = "wasm32"))]
use std::collections::HashMap;
use std::{collections::HashMap, str::FromStr};
use mapping::{FULL_IMPORTS_MAP, SHORT_IMPORTS_MAP};
#[cfg(not(target_arch = "wasm32"))]
@@ -28,9 +24,7 @@ use rustpython_parser::{
text_size::TextRange,
Parse,
};
#[cfg(not(target_arch = "wasm32"))]
use sqlx::{Pool, Postgres};
#[cfg(not(target_arch = "wasm32"))]
use windmill_common::{
error::{self, to_anyhow},
worker::{
@@ -52,14 +46,10 @@ fn replace_full_import(x: &str) -> Option<String> {
FULL_IMPORTS_MAP.get(x).map(|x| (*x).to_owned())
}
#[cfg(not(target_arch = "wasm32"))]
lazy_static! {
static ref RE: Regex = Regex::new(r"^\#\s?(\S+)\s*$").unwrap();
static ref PKG_RE: Regex = Regex::new(r"^([^!=<>]+)(?:[!=<>]|$)").unwrap();
}
lazy_static! {
static ref PIN_RE: Regex = Regex::new(r"(?:\s*#\s*(pin|repin):\s*)(\S*)").unwrap();
static ref PKG_RE: Regex = Regex::new(r"^([^!=<>]+)(?:[!=<>]|$)").unwrap();
// Regex to properly match main function definition at line start,
// capturing both sync and async variants
static ref DEF_MAIN_RE: Regex = Regex::new(r"(?m)^(async\s+)?def\s+main\s*\(").unwrap();
@@ -92,7 +82,7 @@ fn process_import(module: Option<String>, path: &str, level: usize) -> Vec<NImpo
}
}
pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<String>> {
pub fn parse_relative_imports(code: &str, path: &str) -> error::Result<Vec<String>> {
let nimports = parse_code_for_imports(code, path)?;
return Ok(nimports
.into_iter()
@@ -104,7 +94,7 @@ pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<Stri
}
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum NImport {
enum NImport {
// Order matters! First we want to resolve all repins
// manually repinned requirement
@@ -144,8 +134,6 @@ pub enum NImport {
// Relative imports
Relative(String),
}
#[cfg(not(target_arch = "wasm32"))]
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
enum NImportResolved {
Repin { pin: ImportPin, key: String },
@@ -154,12 +142,12 @@ enum NImportResolved {
}
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct ImportPin {
pub pkg: String,
pub path: String,
struct ImportPin {
pkg: String,
path: String,
}
pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImport>> {
fn parse_code_for_imports(code: &str, path: &str) -> error::Result<Vec<NImport>> {
// Use regex to safely find the main function definition
let mut code = DEF_MAIN_RE
.split(code)
@@ -187,7 +175,7 @@ pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImp
let code_with_fake_main = format!("{}\n\ndef main(): pass", code);
let ast = Suite::parse(&code_with_fake_main, "main.py").map_err(|e| {
anyhow::anyhow!("Error parsing code for imports: {}", e.to_string())
error::Error::ExecutionErr(format!("Error parsing code for imports: {}", e.to_string()))
})?;
// Note: We're still using the original code for finding pins,
// as the TextRange values from the parsed AST would be based on code_with_fake_main
@@ -268,7 +256,6 @@ pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImp
return Ok(nimports);
}
#[cfg(not(target_arch = "wasm32"))]
pub async fn parse_python_imports(
code: &str,
w_id: &str,
@@ -277,7 +264,6 @@ pub async fn parse_python_imports(
version_specifiers: &mut Vec<pep440_rs::VersionSpecifier>,
locked_v: &mut Option<pep440_rs::Version>,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
temp_script_refs: &Option<HashMap<String, String>>,
) -> error::Result<(Vec<String>, Option<String>)> {
let mut compile_error_hint: Option<String> = None;
let mut imports = parse_python_imports_inner(
@@ -290,7 +276,6 @@ pub async fn parse_python_imports(
&mut None,
locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
)
.await?
.into_values()
@@ -328,7 +313,6 @@ pub async fn parse_python_imports(
Ok((imports, compile_error_hint))
}
#[cfg(not(target_arch = "wasm32"))]
fn extract_pkg_name(requirement: &str) -> String {
PKG_RE
.captures(requirement)
@@ -336,7 +320,6 @@ fn extract_pkg_name(requirement: &str) -> String {
.unwrap_or_default()
}
#[cfg(not(target_arch = "wasm32"))]
#[async_recursion]
async fn parse_python_imports_inner(
code: &str,
@@ -348,7 +331,6 @@ async fn parse_python_imports_inner(
path_where_annotated_pyv: &mut Option<String>,
locked_v: &mut Option<pep440_rs::Version>,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
temp_script_refs: &Option<HashMap<String, String>>,
) -> error::Result<HashMap<String, NImportResolved>> {
tracing::debug!("Parsing python imports for path: {}", path);
let PythonAnnotations { py310, py311, py312, py313, .. } = PythonAnnotations::parse(&code);
@@ -512,37 +494,17 @@ async fn parse_python_imports_inner(
for n in nimports.into_iter() {
let mut nested = match n {
NImport::Relative(rpath) => {
// First try to get content from temp_script_refs cache if available
let code_from_cache = if let Some(hash) = temp_script_refs.as_ref().and_then(|dt| dt.get(&rpath)) {
tracing::debug!("Found relative import '{}' in temp_script_refs with hash '{}'", rpath, hash);
match windmill_common::cache::raw_script_temp::load(hash.clone(), db).await {
Ok(content) => Some(content),
Err(e) => {
tracing::warn!("temp_script_refs hash '{}' not found in cache: {}, falling back to deployed script", hash, e);
None
}
}
} else {
None
};
// Use cached content if available, otherwise fall back to deployed script
let code = match code_from_cache {
Some(content) => content,
None => {
sqlx::query_scalar!(
r#"
SELECT content FROM script WHERE path = $1 AND workspace_id = $2
AND archived = false ORDER BY created_at DESC LIMIT 1
"#,
&rpath,
w_id
)
.fetch_optional(db)
.await?
.unwrap_or_else(|| "".to_string())
}
};
let code = sqlx::query_scalar!(
r#"
SELECT content FROM script WHERE path = $1 AND workspace_id = $2
AND archived = false ORDER BY created_at DESC LIMIT 1
"#,
&rpath,
w_id
)
.fetch_optional(db)
.await?
.unwrap_or_else(|| "".to_string());
if already_visited.contains(&rpath) {
vec![]
@@ -560,7 +522,6 @@ async fn parse_python_imports_inner(
path_where_annotated_pyv,
locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
)
.await?
.into_values()
@@ -685,7 +646,6 @@ async fn parse_python_imports_inner(
Ok(final_imports)
}
#[cfg(not(target_arch = "wasm32"))]
fn extract_nimports_from_content(
content: &str,
hm: &mut HashMap<String, NImportResolved>,
@@ -26,7 +26,6 @@ def main():
&mut vec![],
&mut None,
&None,
&None,
)
.await?;
// println!("{}", serde_json::to_string(&r)?);
@@ -68,7 +67,6 @@ def main():
&mut vec![],
&mut None,
&None,
&None,
)
.await?;
println!("{}", serde_json::to_string(&r)?);
@@ -100,7 +98,6 @@ def main():
&mut vec![],
&mut None,
&None,
&None,
)
.await?;
println!("{}", serde_json::to_string(&r)?);
@@ -16,6 +16,7 @@ regex.workspace = true
[dependencies]
windmill-parser.workspace = true
windmill-types.workspace = true
anyhow.workspace = true
lazy_static.workspace = true
serde_json.workspace = true
@@ -15,7 +15,7 @@ use std::{
iter::Peekable,
str::CharIndices,
};
pub use windmill_parser::{s3_mode_extension, Arg, MainArgSignature, ObjectType, S3ModeFormat, Typ};
pub use windmill_parser::{Arg, MainArgSignature, ObjectType, Typ};
pub const SANITIZED_ENUM_STR: &str = "__sanitized_enum__";
pub const SANITIZED_RAW_STRING_STR: &str = "__sanitized_raw_string__";
@@ -143,6 +143,7 @@ pub fn parse_db_resource(code: &str) -> Option<String> {
cap.map(|x| x.get(1).map(|x| x.as_str().to_string()).unwrap())
}
pub use windmill_types::s3::{s3_mode_extension, S3ModeFormat};
pub struct S3ModeArgs {
pub prefix: Option<String>,
pub storage: Option<String>,
@@ -117,12 +117,6 @@ impl Visit for ImportsFinder {
}
}
/// Parse TypeScript/JavaScript code and extract all import paths as raw strings.
///
/// Returns import paths exactly as written in the code (e.g., `"./module"`, `"../utils"`, `"lodash"`).
/// Does not resolve relative paths to absolute Windmill paths.
///
/// See also: [`parse_relative_imports`] for resolved absolute paths.
pub fn parse_expr_for_imports(code: &str, skip_type_only: bool) -> anyhow::Result<Vec<String>> {
let cm: Lrc<SourceMap> = Default::default();
let fm = cm.new_source_file(FileName::Custom("main.d.ts".into()).into(), code.into());
@@ -157,82 +151,6 @@ pub fn parse_expr_for_imports(code: &str, skip_type_only: bool) -> anyhow::Resul
Ok(imports)
}
/// Parse TypeScript/JavaScript code and extract relative imports resolved to absolute Windmill paths.
///
/// Takes the script's Windmill path (e.g., `"f/folder/script"`) and resolves relative imports
/// like `"./module"` or `"../utils"` to absolute paths like `"f/folder/module"` or `"f/utils"`.
///
/// Only returns relative imports (those starting with `./`, `../`, or `/`).
/// External package imports (e.g., `"lodash"`) are filtered out.
///
/// See also: [`parse_expr_for_imports`] for raw import strings without resolution.
///
/// # Arguments
/// * `code` - The TypeScript/JavaScript source code
/// * `path` - The Windmill path of the script (e.g., `"f/folder/script"`)
///
/// # Returns
/// A sorted, deduplicated list of resolved absolute Windmill paths.
///
/// # Examples
/// ```ignore
/// // Script at "f/folder/script" with: import { x } from "../utils"
/// // Returns: ["f/utils"]
/// ```
pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<String>> {
let imports = parse_expr_for_imports(code, false)?;
let script_dir = path.rsplit_once('/').map(|(dir, _)| dir).unwrap_or("");
let mut resolved: Vec<String> = imports
.into_iter()
.filter(|imp| is_relative_import(imp))
.map(|imp| {
// Remove .ts extension if present
let imp = imp.strip_suffix(".ts").unwrap_or(&imp);
if imp.starts_with("/") {
// Absolute path (e.g., /f/folder/script) - remove leading slash
imp[1..].to_string()
} else {
// Relative path (e.g., ./script or ../folder/script)
let combined = format!("{}/{}", script_dir, imp);
normalize_path(&combined)
}
})
.collect();
resolved.sort();
resolved.dedup();
Ok(resolved)
}
/// Check if an import path is a relative import (starts with `./`, `../`, or `/`)
fn is_relative_import(import_path: &str) -> bool {
import_path.starts_with("./")
|| import_path.starts_with("../")
|| import_path.starts_with("/")
}
/// Normalize a path by resolving `.` and `..` components
fn normalize_path(input_path: &str) -> String {
let parts: Vec<&str> = input_path.split('/').filter(|p| !p.is_empty()).collect();
let mut result: Vec<&str> = Vec::new();
for part in parts {
if part == "." {
continue;
} else if part == ".." {
if !result.is_empty() {
result.pop();
}
} else {
result.push(part);
}
}
result.join("/")
}
struct OutputFinder {
idents: HashSet<(String, String)>,
}
@@ -2,7 +2,7 @@
mod tests {
use serde_json::json;
use windmill_parser::{Arg, MainArgSignature, ObjectProperty, ObjectType, Typ};
use windmill_parser_ts::{parse_deno_signature, parse_expr_for_imports, parse_relative_imports};
use windmill_parser_ts::{parse_deno_signature, parse_expr_for_imports};
#[test]
fn test_imports_basic() {
@@ -798,7 +798,7 @@ mod tests {
// Test case where there are exports but no preprocessor
let code = r#"
export { foo, bar } from "./utils";
export async function main(param: string) {
return param;
}
@@ -806,84 +806,4 @@ mod tests {
let sig = parse_deno_signature(code, false, false, None).unwrap();
assert_eq!(sig.has_preprocessor, Some(false));
}
// ==========================================================================
// Tests for parse_relative_imports
// ==========================================================================
#[test]
fn test_relative_imports_dot() {
let code = r#"
import { helper } from "./helper";
export async function main() { return helper(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert_eq!(result, vec!["f/folder/helper"]);
}
#[test]
fn test_relative_imports_double_dot() {
let code = r#"
import { utils } from "../utils/helper";
export async function main() { return utils(); }
"#;
let result = parse_relative_imports(code, "f/folder/subfolder/script").unwrap();
assert_eq!(result, vec!["f/folder/utils/helper"]);
}
#[test]
fn test_relative_imports_absolute_path() {
let code = r#"
import { shared } from "/f/shared/utils";
export async function main() { return shared(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert_eq!(result, vec!["f/shared/utils"]);
}
#[test]
fn test_relative_imports_mixed() {
let code = r#"
import { helper } from "./helper";
import { utils } from "../utils";
import { shared } from "/f/shared/lib";
import lodash from "lodash";
export async function main() { return helper() + utils() + shared(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
// Should only include relative imports, not external packages like lodash
assert_eq!(result, vec!["f/folder/helper", "f/shared/lib", "f/utils"]);
}
#[test]
fn test_relative_imports_with_ts_extension() {
let code = r#"
import { helper } from "./helper.ts";
export async function main() { return helper(); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert_eq!(result, vec!["f/folder/helper"]);
}
#[test]
fn test_relative_imports_external_only() {
let code = r#"
import lodash from "lodash";
import { something } from "@scope/package";
export async function main() { return lodash.map([]); }
"#;
let result = parse_relative_imports(code, "f/folder/script").unwrap();
assert!(result.is_empty());
}
#[test]
fn test_relative_imports_deeply_nested() {
let code = r#"
import { a } from "../../a";
import { b } from "../../../b";
export async function main() { return a() + b(); }
"#;
let result = parse_relative_imports(code, "f/one/two/three/script").unwrap();
assert_eq!(result, vec!["f/b", "f/one/a"]);
}
}
@@ -40,7 +40,6 @@ java-parser = [ "dep:windmill-parser-java"]
ruby-parser = [ "dep:windmill-parser-ruby"]
wac-parser = [ "dep:windmill-parser-wac"]
asset-parser = [ "dep:windmill-parser-ts-asset", "dep:windmill-parser-py-asset", "dep:windmill-parser-sql-asset"]
py-imports-parser = [ "dep:windmill-parser-py-imports"]
[dependencies]
anyhow.workspace = true
@@ -62,7 +61,6 @@ windmill-parser-wac = { workspace = true, optional = true }
windmill-parser-ts-asset = { workspace = true, optional = true }
windmill-parser-py-asset = { workspace = true, optional = true }
windmill-parser-sql-asset = { workspace = true, optional = true }
windmill-parser-py-imports = { workspace = true, optional = true }
wasm-bindgen.workspace = true
serde_json.workspace = true
@@ -67,12 +67,6 @@ const targets = [
features: "asset-parser",
env: "default",
},
{
ident: "py-imports",
desc: "Python imports"
features: "py-imports-parser",
env: "default",
},
# ^^^ Add new entry here ^^^
];
# NOTE: This is legacy command for building all, but it is not more used
+2 -5
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env nu
# Build in debug mode specified lang parser to wasm
# Build in debug mode specified lang parser to wasm
# and perform installation to frontend
def "main" [
lang: string # Example: nu
@@ -9,7 +9,4 @@ def "main" [
(
cd ../../../frontend; npm install ../backend/parsers/windmill-parser-wasm/pkg-($lang)
)
(
cd ../../../cli; bun install ../backend/parsers/windmill-parser-wasm/pkg-($lang)
)
}
@@ -36,6 +36,3 @@ popd
pushd "pkg-asset" && npm publish ${args}
popd
pushd "pkg-py-imports" && npm publish ${args}
popd
@@ -38,8 +38,6 @@ pub fn parse_outputs(code: &str) -> String {
return serde_json::to_string(&r).unwrap();
}
/// Parse TypeScript imports and return raw import strings.
/// See [`parse_ts_relative_imports`] for resolved absolute paths.
#[cfg(feature = "ts-parser")]
#[wasm_bindgen]
pub fn parse_ts_imports(code: &str) -> String {
@@ -52,15 +50,6 @@ pub fn parse_ts_imports(code: &str) -> String {
return serde_json::to_string(&r).unwrap();
}
/// Parse TypeScript imports and return relative imports resolved to absolute Windmill paths.
/// Throws JS error on parse failure.
/// See [`parse_ts_imports`] for raw import strings.
#[cfg(feature = "ts-parser")]
#[wasm_bindgen]
pub fn parse_ts_relative_imports(code: &str, path: &str) -> Result<Vec<String>, String> {
windmill_parser_ts::parse_relative_imports(code, path).map_err(|e| e.to_string())
}
#[cfg(feature = "bash-parser")]
#[wasm_bindgen]
pub fn parse_bash(code: &str) -> String {
@@ -225,14 +214,6 @@ pub fn parse_assets_py(code: &str) -> String {
}
}
/// Parse Python imports and return relative imports resolved to absolute Windmill paths.
/// Throws JS error on parse failure.
#[cfg(feature = "py-imports-parser")]
#[wasm_bindgen]
pub fn parse_py_relative_imports(code: &str, path: &str) -> Result<Vec<String>, String> {
windmill_parser_py_imports::parse_relative_imports(code, path).map_err(|e| e.to_string())
}
#[cfg(feature = "ansible-parser")]
#[wasm_bindgen]
pub fn parse_assets_ansible(code: &str) -> String {
@@ -14,23 +14,6 @@ use serde_json::Value;
pub mod asset_parser;
/// S3 output format for SQL queries (moved here to avoid pulling sqlx into WASM via windmill-types)
#[derive(Clone, Copy, Debug)]
pub enum S3ModeFormat {
Json,
Csv,
Parquet,
}
/// Returns the file extension for the given S3 mode format
pub fn s3_mode_extension(format: S3ModeFormat) -> &'static str {
match format {
S3ModeFormat::Json => "json",
S3ModeFormat::Csv => "csv",
S3ModeFormat::Parquet => "parquet",
}
}
#[derive(Serialize, Debug, PartialEq, Default)]
pub struct MainArgSignature {
pub star_args: bool,
-1
View File
@@ -312,7 +312,6 @@ async fn cache_hub_scripts(file_path: Option<String>) -> anyhow::Result<()> {
"cache_init",
"",
&mut None,
&None,
)
.await
{
+1 -7
View File
@@ -98,7 +98,7 @@ use windmill_worker::{
CARGO_REGISTRIES, INSTANCE_PYTHON_VERSION, JAVA_HOME_DIR, JOB_DEFAULT_TIMEOUT, JOB_ISOLATION,
KEEP_JOB_DIR, MAVEN_REPOS, MAVEN_SETTINGS_XML, NO_DEFAULT_MAVEN, NPMRC, NPM_CONFIG_REGISTRY,
NSJAIL_AVAILABLE, NUGET_CONFIG, OTEL_TRACING_PROXY_SETTINGS, PIP_EXTRA_INDEX_URL,
PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, UNSHARE_PATH, UV_INDEX_STRATEGY,
PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, UV_INDEX_STRATEGY,
WORKSPACE_REGISTRIES,
};
@@ -1701,12 +1701,6 @@ pub async fn reload_job_isolation_setting(conn: &Connection) {
All jobs will fail until nsjail is installed or the setting is changed."
);
}
if value == JobIsolationLevel::Unshare && UNSHARE_PATH.is_none() {
tracing::error!(
"job_isolation is set to unshare but the unshare binary is not available on this worker. \
Jobs will run without isolation until unshare is installed or the setting is changed."
);
}
}
pub async fn reload_request_size(conn: &Connection) {
-2
View File
@@ -921,7 +921,6 @@ mod dedicated_worker_protocol {
"test-workspace",
"f/test/script",
LoaderMode::Node,
&None,
))
.expect("build_loader failed");
@@ -1300,7 +1299,6 @@ mod bun_builder_tests {
// Write build.js using the loader and builder constants directly
// Parameters are dummy values since tests don't use Windmill relative imports
let loader = RELATIVE_BUN_LOADER
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", "{}")
.replace("W_ID", "test-workspace")
.replace("BASE_INTERNAL_URL", "http://localhost:8000")
.replace("TOKEN", "test-token")
-1
View File
@@ -48,7 +48,6 @@ fn flow_module(id: &str, value: FlowModuleValue) -> FlowModule {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
}
}
-1
View File
@@ -32,7 +32,6 @@ mod prewarmed_isolate_tests {
"test-workspace",
"f/test/script",
LoaderMode::BrowserBundle,
&None,
)
.await
.expect("build_loader failed");
-8
View File
@@ -209,7 +209,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "b".to_string(),
@@ -256,7 +255,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
}],
modules_node: None,
}
@@ -277,7 +275,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
same_worker: false,
@@ -392,7 +389,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "b".to_string(),
@@ -448,7 +444,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "e".to_string(),
@@ -490,7 +485,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
modules_node: None,
@@ -511,7 +505,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "c".to_string(),
@@ -559,7 +552,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
same_worker: true,
+11 -41
View File
@@ -33,10 +33,8 @@ use sqlx::{FromRow, Postgres, Transaction};
use windmill_audit::audit_oss::{audit_log, AuditAuthorable};
use windmill_audit::ActionKind;
use windmill_common::assets::{clear_static_asset_usage, AssetUsageKind};
use windmill_common::flows::FlowModule;
use windmill_common::min_version::{
MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2,
MIN_VERSION_SUPPORTS_NODE_DEBOUNCING,
};
use windmill_common::runnable_settings::RunnableSettingsTrait;
use windmill_common::utils::query_elems_from_hub;
@@ -1578,54 +1576,30 @@ async fn archive_flow_by_path(
/// Validates that flow debouncing configuration is supported by all workers
/// Returns an error if debouncing is configured but workers are behind required version
async fn guard_flow_from_debounce_data(nf: &NewFlow) -> Result<()> {
let flow_value = nf.parse_flow_value()?;
if !MIN_VERSION_SUPPORTS_DEBOUNCING.met().await && !flow_value.debouncing_settings.is_default()
if !MIN_VERSION_SUPPORTS_DEBOUNCING.met().await
&& !nf.parse_flow_value()?.debouncing_settings.is_default()
{
tracing::warn!(
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
return Err(Error::WorkersAreBehind {
feature: "Debouncing".into(),
min_version: "1.566.0".into(),
});
}
if !MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
&& !flow_value.debouncing_settings.is_legacy_compatible()
Err(Error::WorkersAreBehind { feature: "Debouncing".into(), min_version: "1.566.0".into() })
} else if !MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
&& !nf
.parse_flow_value()?
.debouncing_settings
.is_legacy_compatible()
&& !*WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT
{
tracing::warn!(
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
);
return Err(Error::WorkersAreBehind {
Err(Error::WorkersAreBehind {
feature: "V2 Debouncing".into(),
min_version: "1.597.0".into(),
});
}
// Check node-level debouncing on all modules (including nested branches/loops)
let mut has_node_debouncing = false;
let check_result = FlowModule::traverse_modules(&flow_value.modules, &mut |m| {
if m.debouncing
.as_ref()
.is_some_and(|d| d.debounce_delay_s.is_some_and(|s| s > 0))
{
has_node_debouncing = true;
}
})
} else {
Ok(())
});
if let Err(e) = check_result {
tracing::warn!("Failed to traverse flow modules for debounce guard: {e}");
}
if has_node_debouncing && !MIN_VERSION_SUPPORTS_NODE_DEBOUNCING.met().await {
return Err(Error::WorkersAreBehind {
feature: "Flow node debouncing".into(),
min_version: "1.658.0".into(),
});
}
Ok(())
}
#[derive(Deserialize)]
@@ -1794,7 +1768,6 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "b".to_string(),
@@ -1828,7 +1801,6 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
FlowModule {
id: "c".to_string(),
@@ -1862,7 +1834,6 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
},
],
failure_module: Some(Box::new(FlowModule {
@@ -1895,7 +1866,6 @@ mod tests {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
})),
preprocessor_module: None,
same_worker: false,
-1
View File
@@ -23,5 +23,4 @@ serde.workspace = true
serde_json.workspace = true
serde_yml.workspace = true
sqlx.workspace = true
tracing.workspace = true
url.workspace = true
+7 -119
View File
@@ -164,7 +164,6 @@ pub struct FuturePath {
summary: Option<String>,
description: Option<String>,
security_scheme: Option<SecurityScheme>,
args_schema: Option<Value>,
}
impl FuturePath {
@@ -175,17 +174,8 @@ impl FuturePath {
summary: Option<String>,
description: Option<String>,
security_scheme: Option<SecurityScheme>,
args_schema: Option<Value>,
) -> FuturePath {
FuturePath {
route_path,
kind,
request_type,
summary,
description,
security_scheme,
args_schema,
}
FuturePath { route_path, kind, request_type, summary, description, security_scheme }
}
}
@@ -407,21 +397,7 @@ fn generate_paths(
);
if method != Method::GET {
if let Some(schema) = &path.args_schema {
method_map.insert(
"requestBody",
serde_json::json!({
"required": true,
"content": {
"application/json": {
"schema": schema
}
}
}),
);
} else {
method_map.insert("requestBody", generate_default_request());
}
method_map.insert("requestBody", generate_default_request());
} else if is_webhook {
method_map.insert(
"parameters",
@@ -679,76 +655,6 @@ struct GenerateOpenAPI {
openapi_spec_format: Format,
}
fn clean_schema_for_openapi(schema: Value) -> Value {
if let Value::Object(mut obj) = schema {
obj.remove("$schema");
obj.remove("order");
Value::Object(obj)
} else {
schema
}
}
async fn get_runnable_schema(
db: &DB,
w_id: &str,
script_path: &str,
is_flow: bool,
) -> Option<Value> {
if is_flow {
let row = sqlx::query!(
r#"SELECT
f.schema AS "schema: serde_json::Value",
fv.value->>'preprocessor_module' IS NOT NULL AS "has_preprocessor: bool"
FROM flow f
LEFT JOIN flow_version fv ON fv.id = f.versions[array_length(f.versions, 1)]
AND fv.workspace_id = f.workspace_id
WHERE f.path = $1 AND f.workspace_id = $2 AND NOT f.archived"#,
script_path,
w_id,
)
.fetch_optional(db)
.await
.map_err(|e| {
tracing::warn!("Failed to fetch flow schema for {script_path}: {e}");
e
})
.ok()
.flatten()?;
if row.has_preprocessor.unwrap_or(false) {
return None;
}
row.schema.map(clean_schema_for_openapi)
} else {
let row = sqlx::query!(
r#"SELECT
schema AS "schema: serde_json::Value",
has_preprocessor
FROM script
WHERE path = $1 AND workspace_id = $2
AND NOT archived AND NOT deleted
ORDER BY created_at DESC
LIMIT 1"#,
script_path,
w_id,
)
.fetch_optional(db)
.await
.map_err(|e| {
tracing::warn!("Failed to fetch script schema for {script_path}: {e}");
e
})
.ok()
.flatten()?;
if row.has_preprocessor.unwrap_or(false) {
return None;
}
row.schema.map(clean_schema_for_openapi)
}
}
async fn http_routes_to_future_paths(
db: &DB,
user_db: UserDB,
@@ -785,9 +691,6 @@ async fn http_routes_to_future_paths(
description: Option<String>,
authentication_method: AuthenticationMethod,
authentication_resource_path: Option<String>,
script_path: String,
is_flow: bool,
wrap_body: bool,
}
http_routes = sqlx::query_as!(
@@ -801,10 +704,7 @@ async fn http_routes_to_future_paths(
summary,
description,
authentication_method AS "authentication_method: _",
authentication_resource_path,
script_path,
is_flow,
wrap_body
authentication_resource_path
FROM
http_trigger
WHERE
@@ -864,12 +764,6 @@ async fn http_routes_to_future_paths(
HttpMethod::Delete => Method::DELETE,
};
let args_schema = if !http_route.wrap_body {
get_runnable_schema(db, w_id, &http_route.script_path, http_route.is_flow).await
} else {
None
};
let future_path = FuturePath::new(
route_path,
Kind::HttpRoute(HttpRouteConfig::new(method)),
@@ -877,7 +771,6 @@ async fn http_routes_to_future_paths(
http_route.summary,
http_route.description,
auth_method,
args_schema,
);
openapi_future_paths.push(future_path);
@@ -887,7 +780,6 @@ async fn http_routes_to_future_paths(
}
async fn webhook_to_future_paths(
db: &DB,
pg_pool: &mut PgConnection,
webhook_filters: Option<&[WebhookFilter]>,
w_id: &str,
@@ -913,7 +805,7 @@ async fn webhook_to_future_paths(
}
}
#[derive(Debug, Deserialize, Clone)]
#[derive(Debug, Deserialize, Clone, Hash)]
struct MinifiedWebhook {
path: String,
description: Option<String>,
@@ -922,7 +814,7 @@ async fn webhook_to_future_paths(
let webhook_scripts = sqlx::query_as!(
MinifiedWebhook,
r#"SELECT
r#"SELECT
path,
summary,
description
@@ -941,7 +833,7 @@ async fn webhook_to_future_paths(
let webhook_flows = sqlx::query_as!(
MinifiedWebhook,
r#"SELECT
r#"SELECT
path,
summary,
description
@@ -961,7 +853,6 @@ async fn webhook_to_future_paths(
openapi_future_paths.reserve_exact(webhook_scripts.len() + webhook_flows.len());
for webhook in webhook_scripts {
let args_schema = get_runnable_schema(db, w_id, &webhook.path, false).await;
openapi_future_paths.push(FuturePath::new(
webhook.path,
Kind::Webhook(WebhookConfig::new(RunnableKind::Script)),
@@ -969,12 +860,10 @@ async fn webhook_to_future_paths(
webhook.summary,
webhook.description,
Some(SecurityScheme::BearerJwt),
args_schema,
));
}
for webhook in webhook_flows {
let args_schema = get_runnable_schema(db, w_id, &webhook.path, true).await;
openapi_future_paths.push(FuturePath::new(
webhook.path,
Kind::Webhook(WebhookConfig::new(RunnableKind::Flow)),
@@ -982,7 +871,6 @@ async fn webhook_to_future_paths(
webhook.summary,
webhook.description,
Some(SecurityScheme::BearerJwt),
args_schema,
));
}
}
@@ -1010,7 +898,7 @@ async fn generate_openapi_future_path(
http_routes_to_future_paths(db, user_db, authed, &mut tx, http_route_filters, w_id).await?;
openapi_future_paths
.append(&mut webhook_to_future_paths(db, &mut tx, webhook_filters, w_id).await?);
.append(&mut webhook_to_future_paths(&mut tx, webhook_filters, w_id).await?);
tx.commit().await?;
-142
View File
@@ -239,9 +239,6 @@ pub fn workspaced_service() -> Router {
"/history_update/h/:hash/p/*path",
post(update_script_history),
)
// Temporary raw script storage for CLI lock generation
.route("/raw_temp/store", post(store_raw_script_temp))
.route("/raw_temp/diff", post(diff_raw_scripts_with_deployed))
}
#[derive(Serialize, FromRow)]
@@ -1617,9 +1614,6 @@ struct RawScriptByPathQuery {
cache_key: Option<String>,
// used specifically for python to cache folders on import success to avoid extra db calls on package fetch
cache_folders: Option<bool>,
// If provided, load content from raw_script_temp table using this hash instead of deployed script.
// Used by CLI lock generation to resolve imports from not-yet-deployed scripts.
temp_script_hash: Option<String>,
}
struct StringWithLength(String);
@@ -1678,16 +1672,6 @@ async fn raw_script_by_path_internal(
) -> Result<String> {
let path = path.to_path();
check_scopes(&authed, || format!("scripts:read:{}", path))?;
// If temp_script_hash is provided, try loading from temp storage first.
// This is used by CLI lock generation to resolve imports from not-yet-deployed scripts.
// Falls back to the normal deployed script lookup if not found in temp storage.
if let Some(hash) = query.temp_script_hash {
if let Ok(content) = windmill_common::cache::raw_script_temp::load(hash, &db).await {
return Ok(content);
}
}
let cache_path = query
.cache_key
.map(|x| format!("{w_id}:{path}:{x}{}", if unpin { ":unpinned" } else { "" }));
@@ -2479,129 +2463,3 @@ async fn guard_script_from_debounce_data(ns: &NewScript) -> Result<()> {
Ok(())
}
}
// ============================================================================
// Temporary Raw Script Storage for CLI Lock Generation
// ============================================================================
/// Store raw script content temporarily for CLI lock generation.
async fn store_raw_script_temp(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(content): Json<String>,
) -> Result<Json<String>> {
check_scopes(&authed, || "scripts:write".to_string())?;
let hash = windmill_common::cache::raw_script_temp::compute_hash(&w_id, &content);
// Store to DB
sqlx::query!(
"INSERT INTO raw_script_temp (workspace_id, hash, content, created_at)
VALUES ($1, $2, $3, NOW())
ON CONFLICT (workspace_id, hash) DO UPDATE SET created_at = NOW()",
&w_id,
&hash,
&content
)
.execute(&db)
.await?;
// Clean up old entries (1 week TTL)
sqlx::query!(
"DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'"
)
.execute(&db)
.await?;
Ok(Json(hash))
}
/// Compare local script content hashes with deployed versions.
/// Receives a map of path → SHA256(content), returns paths where the hash
/// differs from the deployed script (or the script doesn't exist on remote).
/// Hash comparison is done entirely in Postgres to avoid transferring content.
#[derive(Deserialize)]
struct WorkspaceDepDiff {
path: String,
language: ScriptLang,
name: Option<String>,
hash: String,
}
#[derive(Deserialize)]
struct DiffRequest {
scripts: std::collections::HashMap<String, String>,
#[serde(default)]
workspace_deps: Vec<WorkspaceDepDiff>,
}
async fn diff_raw_scripts_with_deployed(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(req): Json<DiffRequest>,
) -> Result<Json<Vec<String>>> {
check_scopes(&authed, || "scripts:read".to_string())?;
let mut matching_set: std::collections::HashSet<String> = std::collections::HashSet::new();
let mut all_paths: Vec<String> = Vec::new();
// --- Scripts ---
if !req.scripts.is_empty() {
let paths: Vec<String> = req.scripts.keys().cloned().collect();
let hashes: Vec<String> = paths.iter().map(|p| req.scripts[p].clone()).collect();
let matching: Vec<String> = sqlx::query_scalar(
"SELECT local.path FROM \
unnest($1::text[], $2::text[]) AS local(path, hash) \
INNER JOIN LATERAL ( \
SELECT encode(sha256(convert_to(s.content, 'UTF8')), 'hex') AS deployed_hash \
FROM script s \
WHERE s.path = local.path AND s.workspace_id = $3 AND s.archived = false \
ORDER BY s.created_at DESC LIMIT 1 \
) deployed ON deployed.deployed_hash = local.hash"
)
.bind(&paths)
.bind(&hashes)
.bind(&w_id)
.fetch_all(&db)
.await?;
matching_set.extend(matching);
all_paths.extend(paths);
}
// --- Workspace dependencies ---
for dep in &req.workspace_deps {
let matching: Option<String> = sqlx::query_scalar(
"SELECT $1::text \
WHERE EXISTS ( \
SELECT 1 FROM workspace_dependencies wd \
WHERE wd.workspace_id = $2 AND wd.archived = false \
AND wd.language = $3::SCRIPT_LANG \
AND wd.name IS NOT DISTINCT FROM $4 \
AND encode(sha256(convert_to(wd.content, 'UTF8')), 'hex') = $5 \
)"
)
.bind(&dep.path)
.bind(&w_id)
.bind(dep.language.as_str())
.bind(&dep.name)
.bind(&dep.hash)
.fetch_optional(&db)
.await?;
if let Some(path) = matching {
matching_set.insert(path);
}
all_paths.push(dep.path.clone());
}
let mismatched: Vec<String> = all_paths
.into_iter()
.filter(|p| !matching_set.contains(p))
.collect();
Ok(Json(mismatched))
}
-77
View File
@@ -6988,83 +6988,6 @@ paths:
type: string
format: uuid
/w/{workspace}/scripts/raw_temp/store:
post:
summary: store raw script content temporarily for CLI lock generation
operationId: storeRawScriptTemp
tags:
- script
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
description: script content to store
required: true
content:
application/json:
schema:
type: string
responses:
"200":
description: hash of stored content
content:
application/json:
schema:
type: string
/w/{workspace}/scripts/raw_temp/diff:
post:
summary: diff local script hashes against deployed versions
operationId: diffRawScriptsWithDeployed
tags:
- script
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
description: scripts and workspace deps to diff against deployed versions
required: true
content:
application/json:
schema:
type: object
required:
- scripts
properties:
scripts:
description: map of script path to SHA256 content hash
type: object
additionalProperties:
type: string
workspace_deps:
description: workspace dependencies to diff
type: array
items:
type: object
required:
- path
- language
- hash
properties:
path:
description: CLI path (e.g. dependencies/package.json)
type: string
language:
$ref: "#/components/schemas/ScriptLang"
name:
description: named workspace dependency (null for default)
type: string
hash:
description: SHA256 content hash
type: string
responses:
"200":
description: list of paths that differ from deployed versions
content:
application/json:
schema:
type: array
items:
type: string
/w/{workspace}/jobs/list_selected_job_groups:
# We use post because sending a huge array as a query param can produce
# URLs that may be too long
-1
View File
@@ -2053,7 +2053,6 @@ async fn execute_component(
.triggerables_v2
.as_ref()
.ok_or_else(|| Error::BadRequest(format!("Policy is missing triggerables")))?;
let policy_triggerables = triggerables_v2
.get(path) // start with `path` in case we can avoid the next` format!`.
.or_else(|| triggerables_v2.get(&format!("{}:{}", payload.component, &path)))
-37
View File
@@ -1,37 +0,0 @@
use crate::db::ApiAuthed;
use axum::{extract::Path, routing::post, Json, Router};
use serde::{Deserialize, Serialize};
use windmill_common::{
error::{Error, Result},
query_builders::try_expand_internal_db_query,
scripts::ScriptLang,
};
pub fn workspaced_service() -> Router {
Router::new().route("/expand_marker", post(expand_marker))
}
#[derive(Deserialize)]
struct ExpandMarkerRequest {
language: ScriptLang,
content: String,
}
#[derive(Serialize)]
struct ExpandMarkerResponse {
code: String,
}
async fn expand_marker(
_authed: ApiAuthed,
Path(_w_id): Path<String>,
Json(req): Json<ExpandMarkerRequest>,
) -> Result<Json<ExpandMarkerResponse>> {
match try_expand_internal_db_query(&req.content, &req.language) {
Some(Ok(expanded)) => Ok(Json(ExpandMarkerResponse { code: expanded.code })),
Some(Err(msg)) => Err(Error::BadRequest(msg)),
None => Err(Error::BadRequest(
"Content is not a WM_INTERNAL_DB marker".to_string(),
)),
}
}
-12
View File
@@ -5074,10 +5074,6 @@ pub struct RunDependenciesRequest {
pub raw_workspace_dependencies: Option<RawWorkspaceDependencies>,
#[serde(default)]
pub raw_deps: Option<String>,
/// Map of script path -> content hash for resolving imports from temp storage.
/// Used by CLI to provide local script content during lock generation.
#[serde(default)]
pub temp_script_refs: Option<HashMap<String, String>>,
}
#[derive(Deserialize, Clone, Debug)]
@@ -5137,8 +5133,6 @@ async fn run_dependencies_job(
let mut hm = HashMap::new();
req.raw_workspace_dependencies
.map(|v| hm.insert("raw_workspace_dependencies".to_owned(), to_raw_value(&v)));
req.temp_script_refs
.map(|v| hm.insert("temp_script_refs".to_owned(), to_raw_value(&v)));
let (uuid, tx) = push(
&db,
@@ -5189,8 +5183,6 @@ pub struct RunFlowDependenciesRequest {
pub raw_workspace_dependencies: Option<RawWorkspaceDependencies>,
#[serde(default)]
pub raw_deps: Option<HashMap<String, String>>,
#[serde(default)]
pub temp_script_refs: Option<HashMap<String, String>>,
}
#[derive(Serialize)]
@@ -5234,10 +5226,6 @@ async fn run_flow_dependencies_job(
req.raw_workspace_dependencies
.map(|v| args_map.insert("raw_workspace_dependencies".to_string(), to_raw_value(&v)));
// Add temp_script_refs to args if present (for CLI local import resolution)
req.temp_script_refs
.map(|v| args_map.insert("temp_script_refs".to_string(), to_raw_value(&v)));
let (uuid, tx) = push(
&db,
PushIsolationLevel::IsolatedRoot(db.clone()),
-2
View File
@@ -98,7 +98,6 @@ mod indexer_oss;
mod inkeep_ee;
mod inkeep_oss;
mod integration;
mod internal_db;
mod live_migrations;
#[cfg(all(feature = "private", feature = "parquet"))]
pub mod s3_proxy_ee;
@@ -555,7 +554,6 @@ pub async fn run_server(
.nest("/groups", groups::workspaced_service())
.nest("/groups_history", group_history::workspaced_service())
.nest("/inputs", windmill_api_inputs::workspaced_service())
.nest("/internal_db", internal_db::workspaced_service())
.nest("/job_metrics", job_metrics::workspaced_service())
.nest("/job_helpers", job_helpers_service)
.nest("/jobs", jobs::workspaced_service())
+1 -34
View File
@@ -991,38 +991,6 @@ pub mod workspace_dependencies {
}
}
/// Temporary raw script content cache for CLI lock generation.
pub mod raw_script_temp {
use super::*;
use crate::DB;
make_static! {
static ref CACHE: { String => String } in "raw_script_temp" <= 10000;
}
/// Compute hash for raw script content (includes workspace_id for isolation).
pub fn compute_hash(workspace_id: &str, content: &str) -> String {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(workspace_id.as_bytes());
hasher.update(content.as_bytes());
format!("{:x}", hasher.finalize())
}
/// Load content from cache, falling back to DB.
pub fn load(hash: String, db: &DB) -> impl Future<Output = error::Result<String>> + '_ {
CACHE.get_or_insert_async(hash.clone(), async move {
sqlx::query_scalar!(
"SELECT content FROM raw_script_temp WHERE hash = $1",
&hash
)
.fetch_optional(db)
.await?
.ok_or_else(|| error::Error::NotFound(format!("raw_script_temp hash: {}", hash)))
})
}
}
const _: () = {
impl Import for RawFlow {
fn import(src: &impl Storage) -> error::Result<Self> {
@@ -1215,8 +1183,7 @@ const _: () = {
((u8, ScriptHash), |x| format!("{:02x}-{:016x}", x.0, x.1.0)),
(FlowNodeId, |x| format!("{:016x}", x.0)),
(AppScriptId, |x| format!("{:016x}", x.0)),
((i64, String), |x| format!("{}-{}", x.1, x.0)),
(String, |x| x.as_str())
((i64, String), |x| format!("{}-{}", x.1, x.0))
}
#[cfg(feature = "scoped_cache")]
-1
View File
@@ -77,7 +77,6 @@ pub mod oidc_oss;
#[cfg(feature = "private")]
pub mod otel_ee;
pub mod otel_oss;
pub mod query_builders;
pub mod queue;
pub mod result_stream;
pub mod runnable_settings;
@@ -5,7 +5,6 @@ use tokio::sync::RwLock;
// ============ Feature Definitions ============
pub const MIN_VERSION_SUPPORTS_NODE_DEBOUNCING: VC = vc(1, 658, 0, "Flow node debouncing");
pub const MIN_VERSION_SUPPORTS_TOKEN_HASH: VC = vc(1, 659, 0, "Token hash storage");
pub const MIN_VERSION_SUPPORTS_SYNC_JOBS_DEBOUNCING: VC = vc(1, 602, 0, "Sync jobs debouncing");
pub const MIN_VERSION_SUPPORTS_DEBOUNCING_V2: VC = vc(1, 597, 0, "Debouncing V2");
File diff suppressed because it is too large Load Diff
+1 -5
View File
@@ -68,11 +68,7 @@ pub fn initialize_tracing(
mode: &Mode,
environment: &str,
) -> (WorkerGuard, crate::otel_oss::OtelProvider) {
let style = if std::env::var("NO_COLOR").is_ok() {
"never".into()
} else {
std::env::var("RUST_LOG_STYLE").unwrap_or_else(|_| "auto".into())
};
let style = std::env::var("RUST_LOG_STYLE").unwrap_or_else(|_| "auto".into());
let rust_log_env = std::env::var("RUST_LOG");
let rust_log_stdout_env = std::env::var("RUST_LOG_STDOUT");
+86 -106
View File
@@ -65,8 +65,8 @@ pub mod object_store_reexports {
pub use object_store::memory::InMemory;
pub use object_store::path::Path;
pub use object_store::{
Attribute, Attributes, Error as ObjectStoreError, GetResult, ObjectStore, PutMultipartOpts,
PutPayload, PutResult, Result as ObjectStoreResult, WriteMultipart,
Attribute, Attributes, Error as ObjectStoreError, GetResult, ObjectStore,
PutMultipartOpts, PutPayload, PutResult, Result as ObjectStoreResult, WriteMultipart,
};
}
@@ -530,7 +530,10 @@ async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result<Arc<d
#[cfg(feature = "parquet")]
pub fn build_filesystem_client(root_path: &str) -> error::Result<Arc<dyn ObjectStore>> {
let store = object_store::local::LocalFileSystem::new_with_prefix(root_path).map_err(|e| {
error::Error::internal_err(format!("Error building filesystem object store: {:?}", e))
error::Error::internal_err(format!(
"Error building filesystem object store: {:?}",
e
))
})?;
Ok(Arc::new(store))
}
@@ -641,45 +644,36 @@ lazy_static::lazy_static! {
static ref S3_BUCKET_RESTRICTIONS: Option<HashMap<String, Vec<String>>> = {
parse_bucket_restrictions()
};
static ref AZ_ACCOUNT_NAME_RESTRICTIONS: Option<HashMap<String, Vec<String>>> = {
parse_az_account_name_restrictions()
};
}
fn parse_bucket_restrictions() -> Option<HashMap<String, Vec<String>>> {
let env_var = std::env::var("S3_BUCKETS_WORKSPACE_RESTRICTIONS").ok()?;
parse_restrictions_from_str(&env_var, "S3 bucket")
parse_bucket_restrictions_from_str(&env_var)
}
fn parse_az_account_name_restrictions() -> Option<HashMap<String, Vec<String>>> {
let env_var = std::env::var("AZ_ACCOUNT_NAME_WORKSPACE_RESTRICTIONS").ok()?;
parse_restrictions_from_str(&env_var, "Azure account name")
}
fn parse_restrictions_from_str(input: &str, label: &str) -> Option<HashMap<String, Vec<String>>> {
fn parse_bucket_restrictions_from_str(input: &str) -> Option<HashMap<String, Vec<String>>> {
if input.trim().is_empty() {
return None;
}
let mut restrictions = HashMap::new();
for rule in input.split(';') {
let rule = rule.trim();
if rule.is_empty() {
for bucket_rule in input.split(';') {
let bucket_rule = bucket_rule.trim();
if bucket_rule.is_empty() {
continue;
}
let parts: Vec<&str> = rule.splitn(2, ':').collect();
let parts: Vec<&str> = bucket_rule.splitn(2, ':').collect();
if parts.len() != 2 {
tracing::warn!(
"Invalid {} restriction format: '{}'. Expected 'name:workspace1,workspace2'",
label,
rule
"Invalid bucket restriction format: '{}'. Expected 'bucket:workspace1,workspace2'",
bucket_rule
);
continue;
}
let name = parts[0].trim().to_string();
let bucket_name = parts[0].trim().to_string();
let workspaces: Vec<String> = parts[1]
.split(',')
.map(|w| w.trim().to_string())
@@ -688,33 +682,26 @@ fn parse_restrictions_from_str(input: &str, label: &str) -> Option<HashMap<Strin
if workspaces.is_empty() {
tracing::warn!(
"No workspaces specified for {} '{}', skipping restriction",
label,
name
"No workspaces specified for bucket '{}', skipping restriction",
bucket_name
);
continue;
}
restrictions.insert(name, workspaces);
restrictions.insert(bucket_name, workspaces);
}
if restrictions.is_empty() {
None
} else {
tracing::info!(
"{} restrictions loaded for {} entries",
label,
"S3 bucket restrictions loaded for {} buckets",
restrictions.len()
);
Some(restrictions)
}
}
#[cfg(test)]
fn parse_bucket_restrictions_from_str(input: &str) -> Option<HashMap<String, Vec<String>>> {
parse_restrictions_from_str(input, "S3 bucket")
}
pub fn check_bucket_workspace_restriction(
bucket_name: &str,
workspace_id: &str,
@@ -732,23 +719,6 @@ pub fn check_bucket_workspace_restriction(
Ok(())
}
pub fn check_az_account_name_workspace_restriction(
account_name: &str,
workspace_id: &str,
) -> error::Result<()> {
if let Some(ref restrictions) = *AZ_ACCOUNT_NAME_RESTRICTIONS {
if let Some(allowed_workspaces) = restrictions.get(account_name) {
if !allowed_workspaces.contains(&workspace_id.to_string()) {
return Err(error::Error::NotAuthorized(format!(
"Workspace '{}' is not authorized to access Azure account '{}'",
workspace_id, account_name
)));
}
}
}
Ok(())
}
pub const DEFAULT_STORAGE: &str = "_default_";
pub fn bundle(w_id: &str, hash: &str) -> String {
@@ -769,8 +739,7 @@ pub async fn upload_artifact_to_store(
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
let object_store: Option<()> = None;
Ok(
if &windmill_common::utils::MODE_AND_ADDONS.mode
== &windmill_common::utils::Mode::Standalone
if &windmill_common::utils::MODE_AND_ADDONS.mode == &windmill_common::utils::Mode::Standalone
&& object_store.is_none()
{
let path = format!("{}/{}", standalone_dir, path);
@@ -852,11 +821,11 @@ pub fn lfs_to_object_store_resource(
})?;
Ok(ObjectStoreResource::Gcs(gcs_resource))
}
LargeFileStorage::FilesystemStorage(fs) => {
Ok(ObjectStoreResource::Filesystem(FilesystemSettings {
LargeFileStorage::FilesystemStorage(fs) => Ok(ObjectStoreResource::Filesystem(
FilesystemSettings {
root_path: fs.root_path.clone(),
}))
}
},
)),
}
}
@@ -1026,9 +995,13 @@ pub async fn convert_json_line_stream<E: Into<anyhow::Error>>(
drop(file);
let ctx = SessionContext::new();
ctx.register_json("my_table", path_str, NdJsonReadOptions::default())
.await
.map_err(to_anyhow)?;
ctx.register_json(
"my_table",
path_str,
NdJsonReadOptions::default(),
)
.await
.map_err(to_anyhow)?;
let df = ctx.sql("SELECT * FROM my_table").await.map_err(to_anyhow)?;
let schema = df.schema().clone().into();
@@ -1279,7 +1252,8 @@ mod tests {
advanced_permissions: None,
});
// resource_value is ignored for filesystem
let result = lfs_to_object_store_resource(&lfs, serde_json::Value::Null).unwrap();
let result =
lfs_to_object_store_resource(&lfs, serde_json::Value::Null).unwrap();
match result {
ObjectStoreResource::Filesystem(fs) => {
assert_eq!(fs.root_path, "/tmp/mydata");
@@ -1305,18 +1279,10 @@ mod tests {
port: None,
};
let result = duckdb_connection_settings_internal(s3).unwrap();
assert!(result
.connection_settings_str
.contains("SET s3_region='eu-west-1'"));
assert!(result
.connection_settings_str
.contains("SET s3_access_key_id='AKIA123'"));
assert!(result
.connection_settings_str
.contains("SET s3_secret_access_key='secret456'"));
assert!(result
.connection_settings_str
.contains("SET s3_url_style='path'"));
assert!(result.connection_settings_str.contains("SET s3_region='eu-west-1'"));
assert!(result.connection_settings_str.contains("SET s3_access_key_id='AKIA123'"));
assert!(result.connection_settings_str.contains("SET s3_secret_access_key='secret456'"));
assert!(result.connection_settings_str.contains("SET s3_url_style='path'"));
assert!(!result.connection_settings_str.contains("SET s3_use_ssl=0"));
assert_eq!(result.s3_bucket, Some("test-bucket".to_string()));
assert!(result.azure_container_path.is_none());
@@ -1338,9 +1304,7 @@ mod tests {
};
let result = duckdb_connection_settings_internal(s3).unwrap();
assert!(result.connection_settings_str.contains("SET s3_use_ssl=0"));
assert!(!result
.connection_settings_str
.contains("SET s3_url_style='path'"));
assert!(!result.connection_settings_str.contains("SET s3_url_style='path'"));
}
#[test]
@@ -1356,12 +1320,8 @@ mod tests {
federated_token_file: None,
});
let result = format_duckdb_connection_settings(resource).unwrap();
assert!(result
.connection_settings_str
.contains("AccountName=myaccount"));
assert!(result
.connection_settings_str
.contains("AccountKey=base64key=="));
assert!(result.connection_settings_str.contains("AccountName=myaccount"));
assert!(result.connection_settings_str.contains("AccountKey=base64key=="));
assert_eq!(
result.azure_container_path,
Some("az://mycontainer".to_string())
@@ -1377,10 +1337,7 @@ mod tests {
});
let result = format_duckdb_connection_settings(resource);
assert!(result.is_err());
assert!(result
.unwrap_err()
.to_string()
.contains("GCS is not supported"));
assert!(result.unwrap_err().to_string().contains("GCS is not supported"));
}
#[test]
@@ -1404,8 +1361,10 @@ mod tests {
use windmill_common::error::Error;
// NotFound
let err =
object_store::Error::NotFound { path: "test/path".into(), source: "missing".into() };
let err = object_store::Error::NotFound {
path: "test/path".into(),
source: "missing".into(),
};
let mapped = object_store_error_to_error(err);
assert!(matches!(mapped, Error::NotFound(_)));
@@ -1429,8 +1388,10 @@ mod tests {
assert!(matches!(mapped, Error::BadRequest(_)));
// Unauthenticated
let err =
object_store::Error::Unauthenticated { path: "obj".into(), source: "no creds".into() };
let err = object_store::Error::Unauthenticated {
path: "obj".into(),
source: "no creds".into(),
};
let mapped = object_store_error_to_error(err);
assert!(matches!(mapped, Error::NotAuthorized(_)));
}
@@ -1572,10 +1533,15 @@ mod tests {
.await
.unwrap();
let resource =
ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root.to_string() });
let s3_obj =
S3Object { s3: "etag.txt".to_string(), storage: None, filename: None, presigned: None };
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: root.to_string(),
});
let s3_obj = S3Object {
s3: "etag.txt".to_string(),
storage: None,
filename: None,
presigned: None,
};
let etag = get_etag_or_empty(&resource, s3_obj).await;
// LocalFileSystem should return an etag based on file metadata
@@ -1588,8 +1554,9 @@ mod tests {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().to_str().unwrap();
let resource =
ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root.to_string() });
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: root.to_string(),
});
let s3_obj = S3Object {
s3: "nonexistent.txt".to_string(),
storage: None,
@@ -1608,7 +1575,9 @@ mod tests {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().to_str().unwrap().to_string();
let settings = ObjectSettings::Filesystem(FilesystemSettings { root_path: root });
let settings = ObjectSettings::Filesystem(FilesystemSettings {
root_path: root,
});
let expirable = build_object_store_from_settings(settings, None)
.await
@@ -1616,7 +1585,10 @@ mod tests {
let data = bytes::Bytes::from("end to end via settings");
expirable
.store
.put(&Path::from("e2e.txt"), PutPayload::from(data.clone()))
.put(
&Path::from("e2e.txt"),
PutPayload::from(data.clone()),
)
.await
.unwrap();
let result = expirable
@@ -1638,7 +1610,9 @@ mod tests {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().to_str().unwrap().to_string();
let resource = ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root });
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: root,
});
let client = build_object_store_client(&resource).await.unwrap();
let data = bytes::Bytes::from("end to end via resource");
@@ -1660,10 +1634,7 @@ mod tests {
#[test]
fn test_bundle_path_format() {
assert_eq!(
bundle("my_workspace", "abc123"),
"script_bundle/my_workspace/abc123"
);
assert_eq!(bundle("my_workspace", "abc123"), "script_bundle/my_workspace/abc123");
}
#[test]
@@ -1675,7 +1646,8 @@ mod tests {
#[test]
fn test_parse_bucket_restrictions_single_bucket() {
let result = parse_bucket_restrictions_from_str("my-bucket:workspace1,workspace2").unwrap();
let result =
parse_bucket_restrictions_from_str("my-bucket:workspace1,workspace2").unwrap();
assert_eq!(result.len(), 1);
assert_eq!(
result.get("my-bucket").unwrap(),
@@ -1685,13 +1657,19 @@ mod tests {
#[test]
fn test_parse_bucket_restrictions_multiple_buckets() {
let result = parse_bucket_restrictions_from_str("bucket-a:ws1,ws2;bucket-b:ws3").unwrap();
let result = parse_bucket_restrictions_from_str(
"bucket-a:ws1,ws2;bucket-b:ws3",
)
.unwrap();
assert_eq!(result.len(), 2);
assert_eq!(
result.get("bucket-a").unwrap(),
&vec!["ws1".to_string(), "ws2".to_string()]
);
assert_eq!(result.get("bucket-b").unwrap(), &vec!["ws3".to_string()]);
assert_eq!(
result.get("bucket-b").unwrap(),
&vec!["ws3".to_string()]
);
}
#[test]
@@ -1703,14 +1681,16 @@ mod tests {
#[test]
fn test_parse_bucket_restrictions_invalid_format_skipped() {
// "no-colon" is invalid, only "valid:ws1" should be parsed
let result = parse_bucket_restrictions_from_str("no-colon;valid:ws1").unwrap();
let result =
parse_bucket_restrictions_from_str("no-colon;valid:ws1").unwrap();
assert_eq!(result.len(), 1);
assert!(result.contains_key("valid"));
}
#[test]
fn test_parse_bucket_restrictions_trailing_semicolons() {
let result = parse_bucket_restrictions_from_str(";bucket:ws1;;").unwrap();
let result =
parse_bucket_restrictions_from_str(";bucket:ws1;;").unwrap();
assert_eq!(result.len(), 1);
assert!(result.contains_key("bucket"));
}
+32 -37
View File
@@ -3100,46 +3100,41 @@ impl PulledJobResult {
"Accumulated arguments from debounced jobs in batch"
);
// If the batch query returned no entries (e.g. CE where
// v2_job_debounce_batch is never populated), keep the
// original value unchanged instead of replacing it with [].
if !accumulated_arg.is_empty() {
let new_value = to_raw_value(&accumulated_arg);
let new_value = to_raw_value(&accumulated_arg);
let original_value = j
.args
.as_ref()
.and_then(|a| a.get(arg_name_to_accumulate))
.map(|v| v.get().to_string())
.unwrap_or_else(|| "null".to_string());
let original_value = j
.args
.as_ref()
.and_then(|a| a.get(arg_name_to_accumulate))
.map(|v| v.get().to_string())
.unwrap_or_else(|| "null".to_string());
append_logs(
&j_id,
&j.workspace_id,
format!(
"Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n",
&new_value
),
&(db.into()),
append_logs(
&j_id,
&j.workspace_id,
format!(
"Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n",
&new_value
),
&(db.into()),
)
.await;
j.args
.get_or_insert(Json(Default::default()))
.as_mut()
.insert(arg_name_to_accumulate.to_owned(), new_value);
// Persist accumulated args to v2_job so that flow steps
// re-reading from the DB (via get_mini_pulled_job) see them
if let Some(ref args) = j.args {
sqlx::query!(
"UPDATE v2_job SET args = $2 WHERE id = $1",
j_id,
args as &Json<HashMap<String, Box<RawValue>>>,
)
.await;
j.args
.get_or_insert(Json(Default::default()))
.as_mut()
.insert(arg_name_to_accumulate.to_owned(), new_value);
// Persist accumulated args to v2_job so that flow steps
// re-reading from the DB (via get_mini_pulled_job) see them
if let Some(ref args) = j.args {
sqlx::query!(
"UPDATE v2_job SET args = $2 WHERE id = $1",
j_id,
args as &Json<HashMap<String, Box<RawValue>>>,
)
.execute(db)
.await?;
}
.execute(db)
.await?;
}
}
@@ -4429,321 +4429,4 @@ mod debounce {
Ok(())
}
// =========================================================================
// Tests for maybe_debounce_flow_node (flow node debouncing)
// =========================================================================
/// Helper: insert a child job with a parent flow.
async fn insert_child_job_with_parent(
db: &Pool<Postgres>,
child_id: Uuid,
parent_id: Uuid,
workspace_id: &str,
) {
sqlx::query!(
"INSERT INTO v2_job (id, kind, tag, created_by, permissioned_as, permissioned_as_email, workspace_id, parent_job)
VALUES ($1, 'noop', 'deno', 'test-user', 'u/test-user', 'test@windmill.dev', $2, $3)",
child_id,
workspace_id,
parent_id,
)
.execute(db)
.await
.expect("insert v2_job (child)");
sqlx::query!(
"INSERT INTO v2_job_queue (id, workspace_id, scheduled_for, tag)
VALUES ($1, $2, now(), 'deno')",
child_id,
workspace_id,
)
.execute(db)
.await
.expect("insert v2_job_queue (child)");
sqlx::query!("INSERT INTO v2_job_runtime (id) VALUES ($1)", child_id)
.execute(db)
.await
.expect("insert v2_job_runtime (child)");
}
/// Test: First flow node job in a debounce batch should set scheduled_for
/// and create a debounce_key entry. The parent flow should remain in queue.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_first_job(db: Pool<Postgres>) -> anyhow::Result<()> {
let flow_id = Uuid::new_v4();
let child_id = Uuid::new_v4();
insert_flow_job(&db, flow_id, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: Some("test_flow_node_first".to_string()),
..Default::default()
};
let args_hm = empty_args();
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child_id,
flow_id,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
// Child job should still be in queue with delayed scheduled_for
assert!(is_queued(&db, &child_id).await, "child should be queued");
let sf = sqlx::query_scalar!(
"SELECT scheduled_for FROM v2_job_queue WHERE id = $1",
child_id
)
.fetch_one(&db)
.await?;
let diff = (sf - Utc::now()).num_seconds();
assert!(
diff >= 0 && diff <= 6,
"scheduled_for should be in the future (up to ~5s), got {diff}s"
);
// Parent flow should still be in queue
assert!(
is_queued(&db, &flow_id).await,
"parent flow should still be queued"
);
assert!(
!is_completed(&db, &flow_id).await,
"parent flow should not be completed"
);
// debounce_key should exist
let dk = get_debounce_key(&db, "test_flow_node_first").await;
assert!(dk.is_some(), "debounce_key entry should exist");
let (dk_job, dk_prev, dk_times) = dk.unwrap();
assert_eq!(dk_job, child_id);
assert!(dk_prev.is_none());
assert_eq!(dk_times, 0);
Ok(())
}
/// Test: Second flow node job with same key should cancel the first child and
/// complete the first parent flow with "Debounced by" result.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_second_cancels_first(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
// Flow 1 with child 1
let flow1 = Uuid::new_v4();
let child1 = Uuid::new_v4();
insert_flow_job(&db, flow1, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child1, flow1, "test-workspace").await;
// Flow 2 with child 2
let flow2 = Uuid::new_v4();
let child2 = Uuid::new_v4();
insert_flow_job(&db, flow2, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child2, flow2, "test-workspace").await;
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: Some("test_flow_node_cancel".to_string()),
..Default::default()
};
let args_hm = empty_args();
// Push child 1
{
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child1,
flow1,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
}
// Push child 2 — should cancel child 1 and complete flow 1
{
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child2,
flow2,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
}
// child1 should be completed (debounced/skipped)
assert!(
is_completed(&db, &child1).await,
"child1 should be completed"
);
assert!(
!is_queued(&db, &child1).await,
"child1 should not be in queue"
);
// flow1 (parent of child1) should also be completed
assert!(
is_completed(&db, &flow1).await,
"flow1 should be completed (debounced)"
);
assert!(
!is_queued(&db, &flow1).await,
"flow1 should not be in queue"
);
// Check flow1 result contains "Debounced by"
let result = sqlx::query_scalar!(
"SELECT result::text FROM v2_job_completed WHERE id = $1",
flow1
)
.fetch_one(&db)
.await?;
assert!(
result.as_ref().is_some_and(|r| r.contains("Debounced by")),
"flow1 result should contain 'Debounced by', got: {:?}",
result
);
// child2 should still be in queue (it's the winner)
assert!(
is_queued(&db, &child2).await,
"child2 should still be queued"
);
assert!(
!is_completed(&db, &child2).await,
"child2 should not be completed"
);
// flow2 should still be in queue
assert!(is_queued(&db, &flow2).await, "flow2 should still be queued");
// debounce_key should point to child2
let dk = get_debounce_key(&db, "test_flow_node_cancel").await;
assert!(dk.is_some());
let (dk_job, _, dk_times) = dk.unwrap();
assert_eq!(dk_job, child2);
assert_eq!(dk_times, 1);
Ok(())
}
/// Test: Default debounce key for flow nodes uses $workspace/flow/$path-$step_id.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_default_key(db: Pool<Postgres>) -> anyhow::Result<()> {
let flow_id = Uuid::new_v4();
let child_id = Uuid::new_v4();
insert_flow_job(&db, flow_id, "test-workspace", "f/test/my_flow").await;
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: None, // No custom key — use default
..Default::default()
};
let args_hm = empty_args();
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child_id,
flow_id,
"f/test/my_flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
// Default key should be: test-workspace/flow/f/test/my_flow-step_a
let expected_key = "test-workspace/flow/f/test/my_flow-step_a";
let dk = get_debounce_key(&db, expected_key).await;
assert!(dk.is_some(), "debounce_key with default key should exist");
Ok(())
}
/// Test: Flow node debounce tracks debounced_times counter correctly.
/// Each debounce call increments the counter in the debounce_key table.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_flow_node_debounce_counter_tracking(db: Pool<Postgres>) -> anyhow::Result<()> {
// No limits set so counter never resets
let settings = DebouncingSettings {
debounce_delay_s: Some(5),
debounce_key: Some("test_flow_node_counter".to_string()),
..Default::default()
};
let args_hm = empty_args();
// Push 4 jobs. Each subsequent one increments debounced_times.
for _ in 0..4 {
let flow_id = Uuid::new_v4();
let child_id = Uuid::new_v4();
insert_flow_job(&db, flow_id, "test-workspace", "f/test/flow").await;
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
let args = PushArgs::from(&args_hm);
let mut tx = db.begin().await?;
windmill_queue::jobs_ee::maybe_debounce_flow_node(
&settings,
child_id,
flow_id,
"f/test/flow",
"step_a",
"test-workspace",
&args,
&mut tx,
&db,
)
.await?;
tx.commit().await?;
}
// After 4 jobs, debounced_times should be 3 (first job creates the entry with 0,
// subsequent 3 jobs each increment it)
let debounced_times = sqlx::query_scalar!(
"SELECT debounced_times FROM debounce_key WHERE key = $1",
"test_flow_node_counter"
)
.fetch_one(&db)
.await?;
assert_eq!(
debounced_times, 3,
"debounced_times should be 3 after 4 jobs"
);
Ok(())
}
}
-1
View File
@@ -9,7 +9,6 @@ name = "windmill_types"
path = "src/lib.rs"
[dependencies]
windmill-parser.workspace = true
serde.workspace = true
serde_json.workspace = true
chrono.workspace = true
-3
View File
@@ -445,8 +445,6 @@ pub struct FlowModule {
pub apply_preprocessor: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub pass_flow_input_directly: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub debouncing: Option<DebouncingSettings>,
}
#[derive(Deserialize, Serialize, Debug, Clone)]
@@ -1119,7 +1117,6 @@ pub fn add_virtual_items_if_necessary(modules: &mut Vec<FlowModule>) {
skip_if: None,
apply_preprocessor: None,
pass_flow_input_directly: None,
debouncing: None,
});
}
}
+14 -3
View File
@@ -346,9 +346,20 @@ pub struct DuckdbConnectionSettingsQueryV2 {
pub storage: Option<String>,
}
// Re-export from windmill-parser to keep a single type definition
// (windmill-parser is WASM-compatible, windmill-types is not due to sqlx)
pub use windmill_parser::{s3_mode_extension, S3ModeFormat};
#[derive(Clone, Copy, Debug)]
pub enum S3ModeFormat {
Json,
Csv,
Parquet,
}
pub fn s3_mode_extension(format: S3ModeFormat) -> &'static str {
match format {
S3ModeFormat::Json => "json",
S3ModeFormat::Csv => "csv",
S3ModeFormat::Parquet => "parquet",
}
}
#[cfg(test)]
mod tests {
+4 -13
View File
@@ -1,11 +1,8 @@
// Injected by backend: maps normalized paths to temp storage hashes (or null)
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
const p = {
name: "windmill-relative-resolver",
async setup(build) {
const { writeFileSync, readFileSync, mkdirSync } = await import("fs");
const { dirname, resolve, join } = await import("node:path");
const { dirname, resolve } = await import("node:path");
const base_internal_url = "BASE_INTERNAL_URL".replace(
"localhost",
@@ -98,17 +95,11 @@ const p = {
: args.importer.replace(cdir + "/", "");
const isRelative = !args.path.startsWith("/");
const endExt = args.path.endsWith(".ts") ? "" : ".ts";
const pathNoExt = args.path.replace(/\.ts$/, "");
// Lookup temp script hash
const normalized = (isRelative ? join(dirname(file_path), pathNoExt) : pathNoExt.slice(1)).replace(/\\/g, "/");
const hash = TEMP_SCRIPT_REFS?.[normalized];
const url = (isRelative
let endExt = args.path.endsWith(".ts") ? "" : ".ts";
const url = isRelative
? `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${file_path}/../${args.path}${endExt}`
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`
) + (hash ? `?temp_script_hash=${hash}` : "");
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`;
const file = isRelative
? resolve("./" + file_path + "/../" + args.path + ".url")
: resolve("./" + args.path + ".url");
+4 -19
View File
@@ -1,6 +1,3 @@
// Injected by backend: maps normalized paths to temp storage hashes (or null)
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
// Windows-specific bun loader that uses a virtual "windmill-url" namespace instead
// of writing .url files to disk. This avoids Windows path issues (backslashes in
// resolve(), 8.3 short filenames, drive letter prefixes). The virtual namespace
@@ -73,13 +70,7 @@ const p = {
const rawScriptPath = isAbsolute
? `${path}${endExt}`
: `${importerPath}/../${path}${endExt}`;
const normalized = normalizePath(rawScriptPath);
// Look up temp script hash (keys are extensionless paths)
const lookupPath = normalized.replace(/\.ts$/, "");
const hash = TEMP_SCRIPT_REFS?.[lookupPath];
// Encode hash in the path so onLoad can extract it and append to fetch URL
const resolvedPath = hash ? `${normalized}?temp_script_hash=${hash}` : normalized;
return { path: resolvedPath, namespace: "windmill-url" };
return { path: normalizePath(rawScriptPath), namespace: "windmill-url" };
}
build.onLoad({ filter: filterLoad }, async (args) => {
@@ -89,13 +80,8 @@ const p = {
// Load windmill scripts by fetching from the API
build.onLoad({ filter: /.*/, namespace: "windmill-url" }, async (args) => {
// Extract temp_script_hash if embedded in the path by resolveWindmillImport
const [scriptPath, queryString] = args.path.replace(/^windmill-url:/, "").split("?");
const hashParam = queryString?.startsWith("temp_script_hash=")
? queryString.replace("temp_script_hash=", "")
: undefined;
const url = `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${scriptPath}`
+ (hashParam ? `?temp_script_hash=${hashParam}` : "");
const path = args.path.replace(/^windmill-url:/, "");
const url = `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${path}`;
const req = await fetch(url, {
method: "GET",
headers: {
@@ -138,8 +124,7 @@ const p = {
// Resolve nested imports from within windmill-url modules
build.onResolve({ filter: /\.ts$/, namespace: "windmill-url" }, (args) => {
// Strip any query string from the importer path before resolving
const importer = args.importer.replace(/^windmill-url:/, "").split("?")[0];
const importer = args.importer.replace(/^windmill-url:/, "");
return resolveWindmillImport(importer, args.path);
});
},
+2 -24
View File
@@ -206,7 +206,6 @@ pub async fn gen_bun_lockfile(
workspace_dependencies: &WorkspaceDependenciesPrefetched,
npm_mode: bool,
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
temp_script_refs: &Option<HashMap<String, String>>,
quiet: bool,
) -> Result<Option<String>> {
let common_bun_proc_envs: HashMap<String, String> = get_common_bun_proc_envs(None).await;
@@ -217,11 +216,6 @@ pub async fn gen_bun_lockfile(
gen_bunfig(job_dir, job_id, w_id, db).await?;
write_file(job_dir, "package.json", package_json_content.as_str())?;
} else {
let temp_refs_json = temp_script_refs
.as_ref()
.and_then(|m| serde_json::to_string(m).ok())
.unwrap_or_else(|| "null".to_string());
let loader = RELATIVE_BUN_LOADER
.replace("W_ID", w_id)
.replace("BASE_INTERNAL_URL", base_internal_url)
@@ -230,8 +224,7 @@ pub async fn gen_bun_lockfile(
"CURRENT_PATH",
&crate::common::use_flow_root_path(script_path),
)
.replace("RAW_GET_ENDPOINT", "raw")
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", &temp_refs_json);
.replace("RAW_GET_ENDPOINT", "raw");
write_file(
&job_dir,
@@ -622,15 +615,9 @@ pub async fn build_loader(
w_id: &str,
current_path: &str,
mode: LoaderMode,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<()> {
// Use forward slashes in JS strings to avoid backslash escape issues on Windows
let job_dir_js = job_dir.replace('\\', "/");
let temp_refs_json = temp_script_refs
.as_ref()
.and_then(|m| serde_json::to_string(m).ok())
.unwrap_or_else(|| "null".to_string());
let loader = RELATIVE_BUN_LOADER
.replace("W_ID", w_id)
.replace("BASE_INTERNAL_URL", base_internal_url)
@@ -639,8 +626,7 @@ pub async fn build_loader(
"CURRENT_PATH",
&crate::common::use_flow_root_path(current_path),
)
.replace("RAW_GET_ENDPOINT", "raw_unpinned")
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", &temp_refs_json);
.replace("RAW_GET_ENDPOINT", "raw_unpinned");
if mode == LoaderMode::Node {
write_file(
@@ -938,7 +924,6 @@ pub async fn prebundle_bun_script(
worker_name: &str,
token: &str,
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<()> {
let (local_path, remote_path) =
compute_bundle_local_and_remote_path(inner_content, lock, script_path, db, w_id).await;
@@ -965,7 +950,6 @@ pub async fn prebundle_bun_script(
} else {
LoaderMode::BunBundle
},
temp_script_refs,
)
.await?;
@@ -1287,7 +1271,6 @@ pub async fn handle_bun_job(
workspace_dependencies,
annotation.npm,
&mut Some(occupancy_metrics),
&None,
wac_replay_info.is_some(),
)
.await?;
@@ -1656,7 +1639,6 @@ try {{
} else {
LoaderMode::BunBundle
},
&None,
)
.await?;
@@ -1673,7 +1655,6 @@ try {{
} else {
LoaderMode::Bun
},
&None,
)
.await
} else {
@@ -3377,7 +3358,6 @@ pub async fn start_worker(
w_id,
script_path,
LoaderMode::BrowserBundle,
&None,
)
.await?;
generate_bun_bundle(
@@ -3490,7 +3470,6 @@ pub async fn start_worker(
.await?,
annotation.npm,
&mut None,
&None,
false,
)
.await?;
@@ -3565,7 +3544,6 @@ pub async fn start_worker(
} else {
LoaderMode::Bun
},
&None,
)
.await?;
}
+3 -14
View File
@@ -710,13 +710,10 @@ pub fn build_command_with_isolation(program: &str, args: &[&str]) -> Command {
cmd.args(args);
cmd
} else {
tracing::error!(
"unshare isolation is enabled but UNSHARE_PATH is not available. \
Running job without isolation. Check Instance Settings > Job Isolation."
panic!(
"BUG: unshare isolation is enabled but UNSHARE_PATH is None. \
This should have been caught at worker startup."
);
let mut cmd = Command::new(program);
cmd.args(args);
cmd
}
} else {
let mut cmd = Command::new(program);
@@ -1003,14 +1000,6 @@ pub(crate) async fn get_workspace_s3_resource_path(
windmill_object_store::check_bucket_workspace_restriction(bucket, workspace_id)?;
}
// Check Azure account name workspace restrictions
if let ObjectStoreResource::Azure(azure_resource) = &object_store_resource {
windmill_object_store::check_az_account_name_workspace_restriction(
&azure_resource.account_name,
workspace_id,
)?;
}
Ok(Some(object_store_resource))
}
@@ -1348,7 +1348,6 @@ async fn handle_python_deps(
&mut version_specifiers,
&mut locked_v,
&None,
&None, // temp_script_refs: only used during CLI lock generation
))
.await?;
+16 -71
View File
@@ -453,13 +453,8 @@ lazy_static::lazy_static! {
);
}
tracing::error!(
"unshare test command failed (exit code: {}). stderr: '{}'. flags: '{}'. \
Unshare isolation will NOT be available. \
If job_isolation is set to 'unshare' in Instance Settings, jobs will run without isolation. \
Common causes: user namespaces disabled (sysctl kernel.unprivileged_userns_clone=0), \
max_user_namespaces=0, or missing privileges (--mount-proc requires privileged mode).",
output.status,
tracing::warn!(
"unshare test failed: {}. Flags: {}. Set ENABLE_UNSHARE_PID=true to fail on error.",
stderr.trim(),
flags
);
@@ -481,15 +476,9 @@ lazy_static::lazy_static! {
}
if e.kind() == std::io::ErrorKind::NotFound {
tracing::error!(
"unshare binary not found in PATH. Unshare isolation will NOT be available. \
Install the util-linux package to enable unshare isolation."
);
tracing::debug!("unshare binary not found");
} else {
tracing::error!(
"Failed to execute unshare test command: {}. Unshare isolation will NOT be available.",
e
);
tracing::warn!("Failed to test unshare: {}", e);
}
None
}
@@ -510,27 +499,23 @@ lazy_static::lazy_static! {
},
Ok(output) => {
let stderr = String::from_utf8_lossy(&output.stderr);
tracing::error!(
"nsjail test failed (exit code: {}). stderr: '{}'. path: '{}'. \
Nsjail sandboxing will NOT be available. \
tracing::warn!(
"nsjail test failed: {}. \
nsjail should be included in all standard windmill images. \
If job_isolation is set to 'nsjail_sandboxing' in Instance Settings, jobs will fail.",
output.status,
stderr.trim(),
nsjail_path
Check that the nsjail binary is installed and working correctly.",
stderr.trim()
);
None
},
Err(e) => {
if e.kind() == std::io::ErrorKind::NotFound {
tracing::error!(
"nsjail not found at '{}'. Nsjail sandboxing will NOT be available. \
If using a custom image, ensure nsjail is installed.",
tracing::info!(
"nsjail not found at '{}'. Sandboxing will not be available.",
nsjail_path
);
} else {
tracing::error!(
"Failed to execute nsjail test at '{}': {}. Nsjail sandboxing will NOT be available.",
tracing::warn!(
"Failed to test nsjail at '{}': {}.",
nsjail_path,
e
);
@@ -1584,27 +1569,10 @@ pub async fn run_worker(
);
}
// Force UNSHARE_PATH and NSJAIL_AVAILABLE initialization now for clear startup logging
let _ = &*UNSHARE_PATH;
let _ = &*NSJAIL_AVAILABLE;
if (is_unshare_enabled() || *FAVOR_UNSHARE_PID) && UNSHARE_PATH.is_none() {
tracing::error!(
worker = %worker_name, hostname = %hostname,
"Worker is configured to use unshare isolation (FAVOR_UNSHARE_PID={}, job_isolation={:?}) \
but unshare is NOT available. Jobs will run without isolation. \
See errors above for the specific reason unshare initialization failed.",
*FAVOR_UNSHARE_PID,
JobIsolationLevel::from_u8(JOB_ISOLATION.load(std::sync::atomic::Ordering::Relaxed))
);
}
if is_sandboxing_enabled() && NSJAIL_AVAILABLE.is_none() {
tracing::error!(
worker = %worker_name, hostname = %hostname,
"Worker is configured to use nsjail sandboxing but nsjail is NOT available. \
Jobs requiring sandboxing will fail. \
See errors above for the specific reason nsjail initialization failed."
);
// Force UNSHARE_PATH initialization now to fail-fast if unshare doesn't work
// This ensures we panic at startup rather than lazily when first accessed during job execution
if is_unshare_enabled() || *ENABLE_UNSHARE_PID || *FAVOR_UNSHARE_PID {
let _ = &*UNSHARE_PATH;
}
let start_time = Instant::now();
@@ -4209,29 +4177,6 @@ pub async fn run_language_executor(
modules: &Option<std::collections::HashMap<String, ScriptModule>>,
run_inline: bool,
) -> error::Result<Box<RawValue>> {
// Expand WM_INTERNAL_DB markers into real SQL before dispatching
let expanded_code: String;
let mut language = language;
let code = if let Some(ref lang) = language {
match windmill_common::query_builders::try_expand_internal_db_query(code, lang) {
Some(Ok(expanded)) => {
if let Some(lang_override) = expanded.language_override {
language = Some(lang_override);
}
expanded_code = expanded.code;
&expanded_code
}
Some(Err(e)) => {
return Err(Error::ExecutionErr(format!(
"Failed to expand WM_INTERNAL_DB marker: {}",
e
)));
}
None => code, // Not a marker, use original code
}
} else {
code
};
if let Some(modules) = modules {
#[cfg(feature = "python")]
let base_dir = if language == Some(ScriptLang::Python3) {
@@ -3687,34 +3687,6 @@ async fn push_next_flow_job(
tracing::debug!(id = %flow_job.id, root_id = %job_root, "pushed next flow job: {uuid}");
// Apply flow node debouncing if configured. Skip parallel steps (for-loops, branchall)
// where len > 1 — debouncing those would cancel sibling sub-jobs within the same run.
#[cfg(feature = "private")]
if len == 1 {
if let Some(ref debouncing) = module.debouncing {
if debouncing.debounce_delay_s.is_some_and(|d| d > 0) {
let debounce_args = if let Ok(ref v) = nargs {
windmill_queue::PushArgs::from(v.as_ref())
} else {
windmill_queue::PushArgs::from(&*EHM)
};
let flow_path = flow_job.runnable_path().to_string();
windmill_queue::jobs_ee::maybe_debounce_flow_node(
debouncing,
uuid,
flow_job.id,
&flow_path,
&module.id,
&flow_job.workspace_id,
&debounce_args,
&mut inner_tx,
&db,
)
.await?;
}
}
}
if value_with_parallel.type_ == "forloopflow"
&& value_with_parallel.parallel.unwrap_or(false)
{
@@ -139,13 +139,6 @@ pub async fn handle_dependency_job(
.map(|x| x.get("triggered_by_relative_import").is_some())
.unwrap_or_default();
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
let temp_script_refs: Option<HashMap<String, String>> = job
.args
.as_ref()
.and_then(|x| x.get("temp_script_refs"))
.and_then(|v| serde_json::from_str(v.get()).ok());
let content = capture_dependency_job(
&job.id,
job.script_lang.as_ref().map(|v| Ok(v)).unwrap_or_else(|| {
@@ -171,7 +164,6 @@ pub async fn handle_dependency_job(
script_path,
None,
"script",
&temp_script_refs,
)
.await;
@@ -218,7 +210,6 @@ pub async fn handle_dependency_job(
script_path,
None,
"script",
&None,
)
.await
{
@@ -377,13 +368,6 @@ pub async fn handle_flow_dependency_job(
.map(|x| x.get("triggered_by_relative_import").is_some())
.unwrap_or_default();
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
let temp_script_refs: Option<HashMap<String, String>> = job
.args
.as_ref()
.and_then(|x| x.get("temp_script_refs"))
.and_then(|v| serde_json::from_str(v.get()).ok());
let version = if skip_flow_update {
None
} else {
@@ -483,7 +467,6 @@ pub async fn handle_flow_dependency_job(
&mut dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
&temp_script_refs,
)
.await?;
@@ -698,7 +681,6 @@ async fn lock_flow_value<'c>(
dependency_map: &mut ScopedDependencyMap,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
triggered_by_relative_import: bool,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<(
FlowValue,
sqlx::Transaction<'c, sqlx::Postgres>,
@@ -729,7 +711,6 @@ async fn lock_flow_value<'c>(
dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?;
@@ -761,7 +742,6 @@ async fn lock_flow_value<'c>(
dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?;
@@ -799,7 +779,6 @@ async fn lock_flow_value<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?;
@@ -838,7 +817,6 @@ async fn lock_modules<'c>(
dependency_map: &mut ScopedDependencyMap, // (modules to replace old seq (even unmmodified ones), new transaction, modified ids) )
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
triggered_by_relative_import: bool,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<(
Vec<FlowModule>,
sqlx::Transaction<'c, sqlx::Postgres>,
@@ -894,7 +872,6 @@ async fn lock_modules<'c>(
dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
e.value = FlowModuleValue::ForloopFlow {
@@ -934,7 +911,6 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
nmodified_ids.extend(inner_modified_ids);
@@ -966,7 +942,6 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
e.value = FlowModuleValue::WhileloopFlow {
@@ -1003,7 +978,6 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
nmodified_ids.extend(inner_modified_ids);
@@ -1034,7 +1008,6 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
errors.extend(ninner_errors);
@@ -1105,7 +1078,6 @@ async fn lock_modules<'c>(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
))
.await?;
@@ -1207,7 +1179,6 @@ async fn lock_modules<'c>(
job_path,
Some(&e.id),
"flow",
&temp_script_refs,
)
.await;
//
@@ -1615,7 +1586,6 @@ async fn lock_modules_app(
dependency_map: &mut ScopedDependencyMap,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
triggered_by_relative_import: bool,
temp_script_refs: &Option<HashMap<String, String>>,
) -> Result<Value> {
match value {
Value::Object(mut m) => {
@@ -1726,7 +1696,6 @@ async fn lock_modules_app(
&job.runnable_path(),
container_id.as_deref(),
"app",
temp_script_refs,
)
.await;
match new_lock {
@@ -1804,7 +1773,6 @@ async fn lock_modules_app(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?,
);
@@ -1833,7 +1801,6 @@ async fn lock_modules_app(
dependency_map,
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
)
.await?,
);
@@ -1885,13 +1852,6 @@ pub async fn handle_app_dependency_job(
.map(|x| x.get("triggered_by_relative_import").is_some())
.unwrap_or_default();
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
let temp_script_refs: Option<HashMap<String, String>> = job
.args
.as_ref()
.and_then(|x| x.get("temp_script_refs"))
.and_then(|v| serde_json::from_str(v.get()).ok());
sqlx::query!(
"DELETE FROM workspace_runnable_dependencies WHERE app_path = $1 AND workspace_id = $2",
job_path,
@@ -1939,7 +1899,6 @@ pub async fn handle_app_dependency_job(
&mut dependency_map,
&raw_workspace_dependencies_o,
triggered_by_relative_import,
&temp_script_refs,
)
.await?;
@@ -2439,8 +2398,6 @@ async fn capture_dependency_job(
base_path: &str,
step_id: Option<&str>,
runnable_type: &str, // "script", "flow", or "app"
// Map of script path -> content hash for resolving imports from temp storage (CLI).
temp_script_refs: &Option<HashMap<String, String>>,
) -> error::Result<String> {
// Check if we can skip relocking:
// - Must be triggered by relative import
@@ -2499,13 +2456,12 @@ async fn capture_dependency_job(
let (mut version_specifiers, mut locked_v) = (vec![], None);
let reqs = windmill_parser_py_imports::parse_python_imports(
job_raw_code,
w_id,
&w_id,
script_path,
&db,
&mut version_specifiers,
&mut locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
)
.await?
.0
@@ -2629,7 +2585,6 @@ async fn capture_dependency_job(
&workspace_dependencies,
windmill_common::worker::TypeScriptAnnotations::parse(job_raw_code).npm,
&mut Some(occupancy_metrics),
temp_script_refs,
false,
)
.await?
@@ -2647,7 +2602,6 @@ async fn capture_dependency_job(
worker_name,
&token,
&mut Some(occupancy_metrics),
temp_script_refs,
)
.await?;
}
+4 -4
View File
@@ -190,9 +190,9 @@ pub async fn insert_ping(
let vcpus = get_vcpus();
let memory = get_memory();
let job_isolation = if crate::is_sandboxing_enabled() && crate::NSJAIL_AVAILABLE.is_some() {
let job_isolation = if crate::is_sandboxing_enabled() {
Some("nsjail".to_string())
} else if crate::is_unshare_enabled() && crate::UNSHARE_PATH.is_some() {
} else if crate::is_unshare_enabled() {
Some("unshare".to_string())
} else {
Some("none".to_string())
@@ -265,9 +265,9 @@ pub async fn update_worker_ping_from_job(
let occupancy_rate_5m = occupancy.as_ref().and_then(|x| x.occupancy_rate_5m);
let occupancy_rate_30m = occupancy.as_ref().and_then(|x| x.occupancy_rate_30m);
let job_isolation = if crate::is_sandboxing_enabled() && crate::NSJAIL_AVAILABLE.is_some() {
let job_isolation = if crate::is_sandboxing_enabled() {
Some("nsjail".to_string())
} else if crate::is_unshare_enabled() && crate::UNSHARE_PATH.is_some() {
} else if crate::is_unshare_enabled() {
Some("unshare".to_string())
} else {
Some("none".to_string())
-1
View File
@@ -12,7 +12,6 @@ const parserPackages = [
"windmill-parser-wasm-yaml", "windmill-parser-wasm-csharp",
"windmill-parser-wasm-nu", "windmill-parser-wasm-java",
"windmill-parser-wasm-ruby",
"windmill-parser-wasm-py-imports",
];
const parserExternals = parserPackages.flatMap(p => ["--external", p]);
+2 -5
View File
@@ -24,11 +24,10 @@
"windmill-parser-wasm-nu": "*",
"windmill-parser-wasm-php": "*",
"windmill-parser-wasm-py": "*",
"windmill-parser-wasm-py-imports": "*",
"windmill-parser-wasm-regex": "*",
"windmill-parser-wasm-ruby": "*",
"windmill-parser-wasm-rust": "*",
"windmill-parser-wasm-ts": "^1.659.1",
"windmill-parser-wasm-ts": "*",
"windmill-parser-wasm-yaml": "*",
"windmill-yaml-validator": "1.1.1",
"ws": "8.18.0",
@@ -295,15 +294,13 @@
"windmill-parser-wasm-py": ["windmill-parser-wasm-py@1.628.3", "", {}, "sha512-TlluqknZpg8cZ+A3m6JFLPseY2PpKtDsxdj26fAnCUzKPtse8TxQR+n0dwC80rfW5TwdWSulvNGRDgcNuf7CTw=="],
"windmill-parser-wasm-py-imports": ["windmill-parser-wasm-py-imports@1.659.1", "", {}, "sha512-nfnf04WBRf8f/mNIwdvggYOgz3erxrFGjKqULYBH+bKFMlKA6V7eB19m6CXOBkq9rjTp0ZFG+rgsR+Us7JEkyQ=="],
"windmill-parser-wasm-regex": ["windmill-parser-wasm-regex@1.639.0", "", {}, "sha512-qvYM4sYxB6M0xrqwBljS2fWqOMk6rp++60TRltJnzZDzVaWQrKjTGwNMmfepGAIWy1OGVKp0SCVERhe2P+O6tQ=="],
"windmill-parser-wasm-ruby": ["windmill-parser-wasm-ruby@1.526.1", "", {}, "sha512-rMBQA8s21wmL2kA5ztRs/ZgVA3ckxe9/NLjxl3iQPL0CX6DlvfaUH0O+AnhpXXDMyBs1Y1SZIhcnbnvsHZ3R8g=="],
"windmill-parser-wasm-rust": ["windmill-parser-wasm-rust@1.558.1", "", {}, "sha512-21S7lm1KF8zO1187rbq14hzPHII2RdM2+D44MoAh1F6VoaScj+Puq0z5B1O/hwn/95R/a9jBlL2D8jbkXtlD1A=="],
"windmill-parser-wasm-ts": ["windmill-parser-wasm-ts@1.659.1", "", {}, "sha512-EmXMzOmazC5r29UZh+1TVF9g/N2X51pqK11qDL6xWGeWTIIonhfOZ5nWdGvKQMDUR650fGxehImZzW2v9hNy+w=="],
"windmill-parser-wasm-ts": ["windmill-parser-wasm-ts@1.623.1", "", {}, "sha512-FBwi/zXxjhZcCvi04oFdNivazru1ynIqSbafHSArfaaBWesBO3nye9UO/WXUlWZm5a7BExbU+3R/eVJrGaornw=="],
"windmill-parser-wasm-yaml": ["windmill-parser-wasm-yaml@1.593.0", "", {}, "sha512-Gyx4aR2jsJYuDrD3mCNTmz7LWOQQXPw5yKNCC1xRgUOPfjsD/tINAFfsBLwVOSmlQQcFZO+wHm4KtDtXOcnGVw=="],
+1 -2
View File
@@ -32,11 +32,10 @@
"windmill-parser-wasm-nu": "*",
"windmill-parser-wasm-php": "*",
"windmill-parser-wasm-py": "*",
"windmill-parser-wasm-py-imports": "*",
"windmill-parser-wasm-regex": "*",
"windmill-parser-wasm-ruby": "*",
"windmill-parser-wasm-rust": "*",
"windmill-parser-wasm-ts": "^1.659.1",
"windmill-parser-wasm-ts": "*",
"windmill-parser-wasm-yaml": "*",
"windmill-yaml-validator": "1.1.1",
"ws": "8.18.0",
+41 -107
View File
@@ -7,7 +7,6 @@ import { yamlParseFile } from "../../utils/yaml.ts";
import { stringify as yamlStringify } from "yaml";
import { GlobalOptions } from "../../types.ts";
import {
readLockfile,
checkifMetadataUptodate,
blueColor,
clearGlobalLock,
@@ -42,8 +41,6 @@ import { mergeConfigWithConfigFile, SyncOptions } from "../../core/conf.ts";
import { resolveWorkspace } from "../../core/context.ts";
import { requireLogin } from "../../core/auth.ts";
import { getNonDottedPaths } from "../../utils/resource_folders.ts";
import { extractRelativeImports } from "../../utils/relative_imports.ts";
import { DoubleLinkedDependencyTree } from "../../utils/dependency_tree.ts";
const TOP_HASH = "__app_hash";
export const APP_BACKEND_FOLDER = "backend";
@@ -116,9 +113,7 @@ export async function generateAppLocksInternal(
defaultTs?: "bun" | "deno";
},
justUpdateMetadataLock?: boolean,
noStaleMessage?: boolean,
legacyBehaviour?: boolean,
tree?: DoubleLinkedDependencyTree
noStaleMessage?: boolean
): Promise<string | AppLocksResult | void> {
if (appFolder.endsWith(SEP)) {
appFolder = appFolder.substring(0, appFolder.length - 1);
@@ -130,6 +125,9 @@ export async function generateAppLocksInternal(
log.info(`Generating locks for app ${appFolder} at ${remote_path}`);
}
const rawWorkspaceDependencies: Record<string, string> =
await getRawWorkspaceDependencies();
// Read the app file first to filter workspace dependencies
const appFilePath = path.join(
appFolder,
@@ -137,80 +135,35 @@ export async function generateAppLocksInternal(
);
const appFile = (await yamlParseFile(appFilePath)) as AppFile;
// Filter workspace dependencies based on inline scripts' languages and annotations
const appValue = rawApp ? (appFile as RawAppFile).runnables : (appFile as NormalAppFile).value;
const folderNormalized = appFolder.replaceAll(SEP, "/");
const filteredDeps = await filterWorkspaceDependenciesForApp(
appValue,
rawWorkspaceDependencies,
appFolder
);
let filteredDeps: Record<string, string> = {};
const conf = await readLockfile();
let hashes = await generateAppHash(
filteredDeps,
appFolder,
rawApp,
opts.defaultTs
);
// New behaviour: tree-based dependency tracking
if (!legacyBehaviour && tree) {
if (dryRun) {
const hashes = await generateAppHash({}, appFolder, rawApp, opts.defaultTs);
const isDirectlyStale = !(await checkifMetadataUptodate(appFolder, hashes[TOP_HASH], conf, TOP_HASH));
// For raw apps in new format, runnables are in separate files under backend/
let treeAppValue = structuredClone(appValue);
if (rawApp) {
const runnablesPath = path.join(appFolder, APP_BACKEND_FOLDER);
const runnablesFromFiles = await loadRunnablesFromBackend(runnablesPath);
if (Object.keys(runnablesFromFiles).length > 0) {
treeAppValue = runnablesFromFiles;
}
}
// First pass: add inline scripts as separate nodes, then add app node importing them
const inlineScriptPaths: string[] = [];
await traverseAndProcessInlineScripts(treeAppValue, async (inlineScript, context) => {
if (!inlineScript.content || !inlineScript.language) {
return inlineScript;
}
let content = inlineScript.content;
// Resolve !inline references
if (typeof content === "string" && content.startsWith("!inline ")) {
const filePath = appFolder + SEP + content.replace("!inline ", "");
try {
content = await readFile(filePath, "utf-8");
} catch {
return inlineScript;
}
}
const treePath = folderNormalized + "/" + context.path.join("/");
const language = inlineScript.language as ScriptLanguage;
const imports = await extractRelativeImports(content, treePath, language);
await tree.addNode(treePath, content, language, "", imports, "inline_script", folderNormalized, appFolder, false);
inlineScriptPaths.push(treePath);
return inlineScript;
});
await tree.addNode(folderNormalized, "", "bun", "", inlineScriptPaths, "app", folderNormalized, appFolder, isDirectlyStale, rawApp);
return;
}
// Second pass: get mismatched workspace deps from tree
// TODO: pass raw workspace deps more precisely to every inline script lock generation call
// (currently we pass the union of all mismatched deps filtered for the whole app)
filteredDeps = await filterWorkspaceDependenciesForApp(appValue, tree.getMismatchedWorkspaceDeps(), appFolder);
} else {
// Legacy behaviour
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
filteredDeps = await filterWorkspaceDependenciesForApp(appValue, rawWorkspaceDependencies, appFolder);
const hashes = await generateAppHash(filteredDeps, appFolder, rawApp, opts.defaultTs);
const isDirectlyStale = !(await checkifMetadataUptodate(appFolder, hashes[TOP_HASH], conf, TOP_HASH));
if (!isDirectlyStale) {
if (!noStaleMessage) {
log.info(
colors.green(`App ${remote_path} metadata is up-to-date, skipping`)
);
}
return;
} else if (dryRun) {
return remote_path;
const conf = await import("../../utils/metadata.ts").then((m) =>
m.readLockfile()
);
if (
await checkifMetadataUptodate(appFolder, hashes[TOP_HASH], conf, TOP_HASH)
) {
if (!noStaleMessage) {
log.info(
colors.green(`App ${remote_path} metadata is up-to-date, skipping`)
);
}
return;
} else if (dryRun) {
return remote_path;
}
if (Object.keys(filteredDeps).length > 0 && !noStaleMessage) {
@@ -226,8 +179,6 @@ export async function generateAppLocksInternal(
let updatedScripts: string[] = [];
if (!justUpdateMetadataLock) {
const hashes = await generateAppHash(filteredDeps, appFolder, rawApp, opts.defaultTs);
const changedScripts = [];
// Find hashes that do not correspond to previous hashes
for (const [scriptPath, hash] of Object.entries(hashes)) {
@@ -239,13 +190,7 @@ export async function generateAppLocksInternal(
}
}
// Get temp_script_refs from tree for relative import resolution
const tempScriptRefs = tree?.getTempScriptRefs(folderNormalized);
// In tree mode, the tree already verified this app is stale (possibly via dependency change).
// Per-script hashes only detect content changes, not transitive dependency changes,
// so we must regenerate locks for all inline scripts regardless.
if (changedScripts.length > 0 || (tree && !legacyBehaviour)) {
if (changedScripts.length > 0) {
if (!noStaleMessage) {
log.info(
`Recomputing locks of ${changedScripts.join(", ")} in ${appFolder}`
@@ -274,8 +219,7 @@ export async function generateAppLocksInternal(
appFolder,
filteredDeps,
opts.defaultTs,
noStaleMessage,
tempScriptRefs
noStaleMessage
);
// Note: updateRawAppRunnables now writes each runnable to its own file
} else {
@@ -292,8 +236,7 @@ export async function generateAppLocksInternal(
appFolder,
filteredDeps,
opts.defaultTs,
noStaleMessage,
tempScriptRefs
noStaleMessage
);
normalAppFile.value = result.value;
updatedScripts = result.updatedScripts;
@@ -309,16 +252,15 @@ export async function generateAppLocksInternal(
}
}
// Non-legacy mode excludes workspace deps from hash (tracked via tree instead)
const depsForHash = (tree && !legacyBehaviour) ? {} : filteredDeps;
const finalHashes = await generateAppHash(
depsForHash,
// Regenerate hashes after updates
hashes = await generateAppHash(
filteredDeps,
appFolder,
rawApp,
opts.defaultTs
);
await clearGlobalLock(appFolder);
for (const [scriptPath, hash] of Object.entries(finalHashes)) {
for (const [scriptPath, hash] of Object.entries(hashes)) {
await updateMetadataGlobalLock(appFolder, hash, scriptPath);
}
if (!noStaleMessage) {
@@ -424,8 +366,7 @@ async function updateRawAppRunnables(
appFolder: string,
rawDeps?: Record<string, string>,
defaultTs: "bun" | "deno" = "bun",
noStaleMessage?: boolean,
tempScriptRefs?: Record<string, string>
noStaleMessage?: boolean
): Promise<string[]> {
const updatedRunnables: string[] = [];
const runnablesFolder = path.join(appFolder, APP_BACKEND_FOLDER);
@@ -505,8 +446,7 @@ async function updateRawAppRunnables(
content,
language,
`${remotePath}/${runnableId}`,
rawDeps,
tempScriptRefs
rawDeps
);
// Determine file extension for this language
@@ -573,8 +513,7 @@ async function updateAppInlineScripts(
appFolder: string,
rawDeps?: Record<string, string>,
defaultTs: "bun" | "deno" = "bun",
noStaleMessage?: boolean,
tempScriptRefs?: Record<string, string>
noStaleMessage?: boolean
): Promise<{ value: any; updatedScripts: string[] }> {
const pathAssigner = newPathAssigner(defaultTs, { skipInlineScriptSuffix: getNonDottedPaths() });
const updatedScripts: string[] = [];
@@ -622,8 +561,7 @@ async function updateAppInlineScripts(
content,
language,
scriptPath,
rawDeps,
tempScriptRefs
rawDeps
);
}
// Determine file extension for this language (following extractInlineScriptsForApps pattern)
@@ -688,8 +626,7 @@ async function generateInlineScriptLock(
content: string,
language: string,
scriptPath: string,
rawWorkspaceDependencies: Record<string, string> | undefined,
tempScriptRefs?: Record<string, string>
rawWorkspaceDependencies: Record<string, string> | undefined
): Promise<string> {
// Filter workspace dependencies to only include those matching this script's language and annotations
const filteredDeps = rawWorkspaceDependencies
@@ -720,9 +657,6 @@ async function generateInlineScriptLock(
? filteredDeps
: null,
entrypoint: scriptPath,
...(tempScriptRefs && Object.keys(tempScriptRefs).length > 0
? { temp_script_refs: tempScriptRefs }
: {}),
}),
}
);
+190 -31
View File
@@ -6,6 +6,7 @@ import { WebSocket, WebSocketServer } from "ws";
import * as getPort from "get-port";
import * as http from "node:http";
import * as https from "node:https";
import * as open from "open";
import { readFile, realpath } from "node:fs/promises";
import { watch } from "node:fs";
@@ -32,7 +33,9 @@ import { listSyncCodebases } from "../../utils/codebase.ts";
import { createPreviewLocalScriptReader } from "../../utils/local_path_scripts.ts";
const PORT = 3001;
async function dev(opts: GlobalOptions & SyncOptions) {
const PROXY_PORT = 3100;
async function dev(opts: GlobalOptions & SyncOptions & { proxyPort?: number }) {
opts = await mergeConfigWithConfigFile(opts);
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
@@ -106,12 +109,14 @@ async function dev(opts: GlobalOptions & SyncOptions) {
codebases,
});
await replaceAllPathScriptsWithLocal(localFlow.value, localScriptReader, log);
const wmFlowPath = localPath.replace(/\.flow\/$/, "").replace(/\/$/, "");
currentLastEdit = {
type: "flow",
flow: localFlow,
uriPath: localPath,
path: wmFlowPath,
};
log.info("Updated " + localPath);
log.info("Updated " + wmFlowPath);
broadcastChanges(currentLastEdit);
} else if (typ == "script") {
const content = await readFile(cpath, "utf-8");
@@ -153,38 +158,49 @@ async function dev(opts: GlobalOptions & SyncOptions) {
type: "flow";
flow: OpenFlow;
uriPath: string;
path: string;
};
const connectedClients: Set<WebSocket> = new Set();
// Map each connected client to its optional watchPath filter
const clientWatchPaths: Map<WebSocket, string | undefined> = new Map();
// Function to send a message to all connected clients
function getEditPath(lastEdit: LastEditScript | LastEditFlow): string {
return lastEdit.path;
}
function normalizePath(p: string): string {
return p.replace(/\.flow\/?$/, "").replace(/\/$/, "");
}
// Send file changes to clients, filtered by their watchPath
function broadcastChanges(lastEdit: LastEditScript | LastEditFlow) {
for (const client of connectedClients.values()) {
client.send(JSON.stringify(lastEdit));
const editPath = normalizePath(getEditPath(lastEdit));
const msg = JSON.stringify(lastEdit);
for (const [client, watchPath] of clientWatchPaths.entries()) {
if (watchPath === undefined || normalizePath(watchPath) === editPath) {
client.send(msg);
}
}
}
async function startApp() {
const server = http.createServer((_req, res) => {
res.writeHead(200);
res.end();
});
const wss = new WebSocketServer({ server });
// WebSocket server event listeners
function setupDevWs(wss: WebSocketServer) {
wss.on("connection", (ws: WebSocket) => {
connectedClients.add(ws);
console.log("New client connected");
clientWatchPaths.set(ws, undefined);
console.log("New dev client connected");
ws.on("open", () => {
if (currentLastEdit) {
broadcastChanges(currentLastEdit);
// Send the current state to the new client
const watchPath = clientWatchPaths.get(ws);
if (watchPath === undefined || normalizePath(watchPath) === normalizePath(getEditPath(currentLastEdit))) {
ws.send(JSON.stringify(currentLastEdit));
}
}
});
ws.on("close", () => {
connectedClients.delete(ws);
console.log("Client disconnected");
clientWatchPaths.delete(ws);
console.log("Dev client disconnected");
});
ws.on("message", (message: WebSocket.RawData) => {
@@ -198,37 +214,176 @@ async function dev(opts: GlobalOptions & SyncOptions) {
if (data.type === "load") {
loadPaths([data.path]);
} else if (data.type === "setWatch") {
const path = data.path as string;
clientWatchPaths.set(ws, path);
console.log(`Client watching: ${path}`);
ws.send(JSON.stringify({ type: "watchSet", path }));
// Send current state for the watched path if available
if (currentLastEdit && normalizePath(getEditPath(currentLastEdit)) === normalizePath(path)) {
ws.send(JSON.stringify(currentLastEdit));
}
}
});
});
}
// Start the server
const port = await getPort.default({ port: 3001 });
const url =
async function startLegacyServer(): Promise<number> {
const server = http.createServer((_req, res) => {
res.writeHead(200);
res.end();
});
const wss = new WebSocketServer({ server });
setupDevWs(wss);
const port = await getPort.default({ port: PORT });
return new Promise((resolve) => {
server.listen(port, () => {
console.log(`Legacy dev server listening on port ${port}`);
resolve(port);
});
});
}
async function startProxyServer(remoteUrl: string, proxyPort: number, wsPort: number) {
const remote = new URL(remoteUrl);
const isHttps = remote.protocol === "https:";
const remoteHost = remote.hostname;
const remotePort = remote.port ? parseInt(remote.port) : (isHttps ? 443 : 80);
const httpModule = isHttps ? https : http;
// Dev WebSocket server (handles /ws_dev path)
const devWss = new WebSocketServer({ noServer: true });
setupDevWs(devWss);
// Separate WSS for proxied WebSocket connections (not tracked as dev clients)
const proxyWss = new WebSocketServer({ noServer: true });
const proxyServer = http.createServer((clientReq, clientRes) => {
const proxyOpts: http.RequestOptions = {
hostname: remoteHost,
port: remotePort,
path: clientReq.url,
method: clientReq.method,
headers: {
...clientReq.headers,
host: remote.host,
},
};
const proxyReq = httpModule.request(proxyOpts, (proxyRes) => {
// Rewrite Set-Cookie domain to localhost
const setCookie = proxyRes.headers["set-cookie"];
if (setCookie) {
proxyRes.headers["set-cookie"] = setCookie.map((cookie) =>
cookie.replace(/domain=[^;]+/gi, "domain=localhost")
);
}
clientRes.writeHead(proxyRes.statusCode ?? 502, proxyRes.headers);
proxyRes.pipe(clientRes, { end: true });
});
proxyReq.on("error", (err) => {
console.error("Proxy error:", err.message);
clientRes.writeHead(502);
clientRes.end("Bad Gateway");
});
clientReq.pipe(proxyReq, { end: true });
});
// Handle WebSocket upgrades
proxyServer.on("upgrade", (req, socket, head) => {
const pathname = req.url?.split("?")[0] ?? "";
if (pathname === "/ws_dev") {
// Handle locally: dev file-change WebSocket
devWss.handleUpgrade(req, socket, head, (ws) => {
devWss.emit("connection", ws, req);
});
return;
}
// Proxy all other WebSocket paths to remote
if (pathname.startsWith("/ws/") || pathname.startsWith("/ws_mp/") || pathname.startsWith("/ws_debug/")) {
const wsProtocol = isHttps ? "wss" : "ws";
const remoteWsUrl = `${wsProtocol}://${remote.host}${req.url}`;
const remoteWs = new WebSocket(remoteWsUrl, {
headers: {
...req.headers,
host: remote.host,
},
});
remoteWs.on("open", () => {
proxyWss.handleUpgrade(req, socket, head, (clientWs) => {
clientWs.on("message", (data) => {
if (remoteWs.readyState === WebSocket.OPEN) {
remoteWs.send(data);
}
});
remoteWs.on("message", (data) => {
if (clientWs.readyState === WebSocket.OPEN) {
clientWs.send(data);
}
});
clientWs.on("close", () => remoteWs.close());
remoteWs.on("close", () => clientWs.close());
});
});
remoteWs.on("error", (err) => {
console.error("WebSocket proxy error:", err.message);
socket.destroy();
});
return;
}
// Unknown WS path — destroy
socket.destroy();
});
return new Promise<void>((resolve) => {
proxyServer.listen(proxyPort, () => {
console.log(`Dev proxy listening on http://localhost:${proxyPort}`);
resolve();
});
});
}
async function startApp() {
const wsPort = await startLegacyServer();
const proxyPort = opts.proxyPort ?? PROXY_PORT;
await startProxyServer(workspace.remote, proxyPort, wsPort);
const legacyUrl =
`${workspace.remote}dev?workspace=${workspace.workspaceId}&local=true&wm_token=${workspace.token}` +
(port === PORT ? "" : `&port=${port}`);
(wsPort === PORT ? "" : `&port=${wsPort}`);
const proxyUrl =
`http://localhost:${proxyPort}/dev?workspace=${workspace.workspaceId}&local=true&wm_token=${workspace.token}`;
console.log(`\nLegacy dev URL: ${legacyUrl}`);
console.log(`Proxy dev URL: ${proxyUrl}`);
console.log(`\nTo watch a specific file: ${proxyUrl}&path=<wm_path>`);
console.log(`Go to ${url}`);
try {
open.openApp(open.apps.browser, { arguments: [url] }).catch((error) => {
open.openApp(open.apps.browser, { arguments: [legacyUrl] }).catch((error) => {
console.error(
`Failed to open browser, please navigate to ${url}, error: ${error}`
`Failed to open browser, please navigate to ${legacyUrl}, error: ${error}`
);
});
console.log("Opened browser for you");
} catch (error) {
console.error(
`Failed to open browser, please navigate to ${url}, ${error}`
`Failed to open browser, please navigate to ${legacyUrl}, ${error}`
);
}
console.log(
"Dev server will automatically point to the last script edited locally"
);
server.listen(port, () => {
console.log(`Server listening on port ${port}`);
});
}
await Promise.all([startApp(), watchChanges()]);
@@ -241,6 +396,10 @@ const command = new Command()
"--includes <pattern...:string>",
"Filter paths givena glob pattern or path"
)
.option(
"--proxy-port <port:number>",
"Port for the localhost reverse proxy (default: 3100)"
)
.action(dev as any);
export default command;
-6
View File
@@ -437,7 +437,6 @@ async function preview(
export async function generateLocks(
opts: GlobalOptions & {
yes?: boolean;
dryRun?: boolean;
} & SyncOptions,
folder: string | undefined
) {
@@ -488,10 +487,6 @@ export async function generateLocks(
}
if (hasAny) {
if (opts.dryRun) {
log.info(colors.gray("Dry run complete."));
return;
}
if (
!opts.yes &&
!(await Confirm.prompt({
@@ -597,7 +592,6 @@ const command = new Command()
)
.arguments("[flow:file]")
.option("--yes", "Skip confirmation prompt")
.option("--dry-run", "Perform a dry run without making changes")
.option(
"-i --includes <patterns:file[]>",
"Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string)"
+33 -114
View File
@@ -29,9 +29,10 @@ import { FlowFile } from "./flow.ts";
import { FlowValue } from "../../../gen/types.gen.ts";
import { replaceInlineScripts } from "../../../windmill-utils-internal/src/inline-scripts/replacer.ts";
import { workspaceDependenciesLanguages } from "../../utils/script_common.ts";
import { extractNameFromFolder, getFolderSuffix, getNonDottedPaths } from "../../utils/resource_folders.ts";
import { extractRelativeImports } from "../../utils/relative_imports.ts";
import { DoubleLinkedDependencyTree } from "../../utils/dependency_tree.ts";
import {
extractNameFromFolder,
getNonDottedPaths,
} from "../../utils/resource_folders.ts";
const TOP_HASH = "__flow_hash";
async function generateFlowHash(
@@ -69,9 +70,7 @@ export async function generateFlowLockInternal(
defaultTs?: "bun" | "deno";
},
justUpdateMetadataLock?: boolean,
noStaleMessage?: boolean,
legacyBehaviour?: boolean,
tree?: DoubleLinkedDependencyTree
noStaleMessage?: boolean
): Promise<string | FlowLocksResult | void> {
if (folder.endsWith(SEP)) {
folder = folder.substring(0, folder.length - 1);
@@ -81,67 +80,33 @@ export async function generateFlowLockInternal(
log.info(`Generating lock for flow ${folder} at ${remote_path}`);
}
// Always get out-of-sync workspace dependencies
const rawWorkspaceDependencies: Record<string, string> =
await getRawWorkspaceDependencies();
const flowValue = (await yamlParseFile(
folder! + SEP + "flow.yaml"
)) as FlowFile;
const folderNormalized = folder.replaceAll(SEP, "/");
const inlineScriptsForTree = extractInlineScriptsForFlows(
structuredClone(flowValue.value.modules),
{},
SEP,
// Filter workspace dependencies based on inline scripts' languages and annotations
const filteredDeps = await filterWorkspaceDependenciesForFlow(flowValue.value as FlowValue, rawWorkspaceDependencies, folder);
let hashes = await generateFlowHash(
filteredDeps,
folder,
opts.defaultTs
).filter(s => !s.is_lock);
);
let filteredDeps: Record<string, string> = {};
const conf = await readLockfile();
if (!legacyBehaviour && tree) {
if (dryRun) {
const inlineScriptPaths: string[] = [];
for (const script of inlineScriptsForTree) {
let content = script.content;
if (content.startsWith("!inline ")) {
const filePath = folder + SEP + content.replace("!inline ", "");
try {
content = await readFile(filePath, "utf-8");
} catch {
continue;
}
}
const treePath = folderNormalized + "/" + path.basename(script.path, path.extname(script.path));
const language = script.language as ScriptLanguage;
const imports = await extractRelativeImports(content, treePath, language);
await tree.addNode(treePath, content, language, "", imports, "inline_script", folderNormalized, folder, false);
inlineScriptPaths.push(treePath);
}
const hashes = await generateFlowHash({}, folder, opts.defaultTs);
const isDirectlyStale = !(await checkifMetadataUptodate(folder, hashes[TOP_HASH], conf, TOP_HASH));
await tree.addNode(folderNormalized, "", "bun", "", inlineScriptPaths, "flow", folderNormalized, folder, isDirectlyStale);
return;
}
// Second pass: get mismatched workspace deps from tree
filteredDeps = await filterWorkspaceDependenciesForFlow(flowValue.value as FlowValue, tree.getMismatchedWorkspaceDeps(), folder);
} else {
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
filteredDeps = await filterWorkspaceDependenciesForFlow(flowValue.value as FlowValue, rawWorkspaceDependencies, folder);
const hashes = await generateFlowHash(filteredDeps, folder, opts.defaultTs);
const isDirectlyStale = !(await checkifMetadataUptodate(folder, hashes[TOP_HASH], conf, TOP_HASH));
if (!isDirectlyStale) {
if (!noStaleMessage) {
log.info(
colors.green(`Flow ${remote_path} metadata is up-to-date, skipping`)
);
}
return;
} else if (dryRun) {
return remote_path;
if (await checkifMetadataUptodate(folder, hashes[TOP_HASH], conf, TOP_HASH)) {
if (!noStaleMessage) {
log.info(
colors.green(`Flow ${remote_path} metadata is up-to-date, skipping`)
);
}
return;
} else if (dryRun) {
return remote_path;
}
if (Object.keys(filteredDeps).length > 0 && !noStaleMessage) {
@@ -157,23 +122,7 @@ export async function generateFlowLockInternal(
let changedScripts: string[] = [];
// Build mapping from on-disk file names (hash keys like "a.py") to tree paths
// (like "folder/a.inline_script"). The tree uses extractInlineScriptsForFlows without
// a path assigner, so paths always have .inline_script suffix, but on-disk files
// may not (non-dotted mode).
const fileToTreePath = new Map<string, string>();
for (const script of inlineScriptsForTree) {
const c = script.content;
if (c.startsWith("!inline ")) {
const fileName = c.replace("!inline ", "");
const treePath = folderNormalized + "/" + path.basename(script.path, path.extname(script.path));
fileToTreePath.set(fileName, treePath);
}
}
if (!justUpdateMetadataLock) {
const hashes = await generateFlowHash(filteredDeps, folder, opts.defaultTs);
//find hashes that do not correspond to previous hashes
for (const [path, hash] of Object.entries(hashes)) {
if (path == TOP_HASH) {
@@ -188,39 +137,27 @@ export async function generateFlowLockInternal(
log.info(`Recomputing locks of ${changedScripts.join(", ")} in ${folder}`);
}
const fileReader = async (path: string) => await readFile(folder + SEP + path, "utf-8");
// In tree mode, use the tree's staleness info (which includes transitive dependency changes)
// to determine which scripts need relocking, instead of only content-changed ones.
const locksToRemove = (tree && !legacyBehaviour)
? Object.keys(hashes).filter(k => {
if (k === TOP_HASH) return false;
const treePath = fileToTreePath.get(k)
?? (folderNormalized + "/" + path.basename(k, path.extname(k)));
return tree.isStale(treePath);
})
: changedScripts;
await replaceInlineScripts(
flowValue.value.modules,
fileReader,
log,
folder + SEP!,
SEP,
locksToRemove
changedScripts
);
if (flowValue.value.failure_module) {
await replaceInlineScripts([flowValue.value.failure_module], fileReader, log, folder + SEP!, SEP, locksToRemove);
await replaceInlineScripts([flowValue.value.failure_module], fileReader, log, folder + SEP!, SEP, changedScripts);
}
if (flowValue.value.preprocessor_module) {
await replaceInlineScripts([flowValue.value.preprocessor_module], fileReader, log, folder + SEP!, SEP, locksToRemove);
await replaceInlineScripts([flowValue.value.preprocessor_module], fileReader, log, folder + SEP!, SEP, changedScripts);
}
//removeChangedLocks
const tempScriptRefs = tree?.getTempScriptRefs(folderNormalized);
flowValue.value = await updateFlow(
workspace,
flowValue.value,
remote_path,
filteredDeps,
tempScriptRefs
filteredDeps
);
const lockAssigner = newPathAssigner(opts.defaultTs ?? "bun", {
@@ -250,15 +187,13 @@ export async function generateFlowLockInternal(
);
}
// Non-legacy mode excludes workspace deps from hash (tracked via tree instead)
const depsForHash = (tree && !legacyBehaviour) ? {} : filteredDeps;
const finalHashes = await generateFlowHash(
depsForHash,
hashes = await generateFlowHash(
filteredDeps,
folder,
opts.defaultTs
);
await clearGlobalLock(folder);
for (const [path, hash] of Object.entries(finalHashes)) {
for (const [path, hash] of Object.entries(hashes)) {
await updateMetadataGlobalLock(folder, hash, path);
}
if (!noStaleMessage) {
@@ -266,16 +201,7 @@ export async function generateFlowLockInternal(
}
// Return the list of updated scripts (extract just the filename from the path)
// In tree mode, use the same staleness-aware list we used for lock removal
const relocked = (tree && !legacyBehaviour)
? Object.keys(finalHashes).filter(k => {
if (k === TOP_HASH) return false;
const treePath = fileToTreePath.get(k)
?? (folderNormalized + "/" + path.basename(k, path.extname(k)));
return tree.isStale(treePath);
})
: changedScripts;
const updatedScripts = relocked.map(p => {
const updatedScripts = changedScripts.map(p => {
const parts = p.split(SEP);
return parts[parts.length - 1].replace(/\.[^.]+$/, ""); // Remove extension
});
@@ -313,8 +239,7 @@ export async function updateFlow(
workspace: Workspace,
flow_value: FlowValue,
remotePath: string,
rawWorkspaceDependencies: Record<string, string>,
tempScriptRefs?: Record<string, string>
rawWorkspaceDependencies: Record<string, string>
): Promise<FlowValue | undefined> {
let rawResponse;
@@ -339,9 +264,6 @@ export async function updateFlow(
path: remotePath,
use_local_lockfiles: true,
raw_workspace_dependencies: rawWorkspaceDependencies,
...(tempScriptRefs && Object.keys(tempScriptRefs).length > 0
? { temp_script_refs: tempScriptRefs }
: {}),
}),
}
);
@@ -360,9 +282,6 @@ export async function updateFlow(
body: JSON.stringify({
flow_value,
path: remotePath,
...(tempScriptRefs && Object.keys(tempScriptRefs).length > 0
? { temp_script_refs: tempScriptRefs }
: {}),
}),
}
);
@@ -10,8 +10,6 @@ import * as log from "../../core/log.ts";
import {
generateScriptMetadataInternal,
getRawWorkspaceDependencies,
readLockfile,
checkifMetadataUptodate,
} from "../../utils/metadata.ts";
import { generateFlowLockInternal, FlowLocksResult } from "../flow/flow_metadata.ts";
import { generateAppLocksInternal, getAppFolders, AppLocksResult } from "../app/app_metadata.ts";
@@ -21,20 +19,14 @@ import {
ignoreF,
} from "../sync/sync.ts";
import { exts } from "../script/script.ts";
import { isFolderResourcePathAnyFormat, isScriptModulePath, isModuleEntryPoint } from "../../utils/resource_folders.ts";
import { isFlowPath, isAppPath, isRawAppPath, isScriptModulePath, isModuleEntryPoint } from "../../utils/resource_folders.ts";
import { listSyncCodebases } from "../../utils/codebase.ts";
import {
DoubleLinkedDependencyTree,
uploadScripts,
ItemType,
} from "../../utils/dependency_tree.ts";
interface StaleItem {
type: ItemType;
type: "script" | "flow" | "app";
path: string;
folder: string;
isRawApp?: boolean;
staleReason?: string;
}
async function generateMetadata(
@@ -46,7 +38,6 @@ async function generateMetadata(
skipScripts?: boolean;
skipFlows?: boolean;
skipApps?: boolean;
strictFolderBoundaries?: boolean;
} & SyncOptions,
folder?: string
) {
@@ -58,10 +49,12 @@ async function generateMetadata(
await requireLogin(opts);
opts = await mergeConfigWithConfigFile(opts);
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(false);
const rawWorkspaceDependencies = await getRawWorkspaceDependencies();
const codebases = await listSyncCodebases(opts);
const ignore = await ignoreF(opts);
const staleItems: StaleItem[] = [];
// --schema-only implies skipping flows and apps (they only have locks, no schemas)
const skipScripts = opts.skipScripts ?? false;
const skipFlows = opts.skipFlows ?? opts.schemaOnly ?? false;
@@ -77,11 +70,7 @@ async function generateMetadata(
return;
}
log.info(`Checking ${checking.join(", ")}...`);
// Build dependency tree for relative import tracking
const tree = new DoubleLinkedDependencyTree();
tree.setWorkspaceDeps(rawWorkspaceDependencies);
log.info(colors.gray(`Checking ${checking.join(", ")}...`));
// === Collect stale scripts ===
if (!skipScripts) {
@@ -92,7 +81,9 @@ async function generateMetadata(
return (
(!isD && !exts.some((ext) => p.endsWith(ext))) ||
ignore(p, isD) ||
isFolderResourcePathAnyFormat(p) ||
isFlowPath(p) ||
isAppPath(p) ||
isRawAppPath(p) ||
(isScriptModulePath(p) && !isModuleEntryPoint(p))
);
},
@@ -101,18 +92,19 @@ async function generateMetadata(
);
for (const e of Object.keys(scriptElems)) {
await generateScriptMetadataInternal(
const candidate = await generateScriptMetadataInternal(
e,
workspace,
opts,
true, // dryRun - populate tree
true, // dryRun
true, // noStaleMessage
rawWorkspaceDependencies,
codebases,
false,
false, // legacyBehaviour
tree
false
);
if (candidate) {
staleItems.push({ type: "script", path: candidate, folder: e });
}
}
}
@@ -134,17 +126,18 @@ async function generateMetadata(
)
).map((x) => x.substring(0, x.lastIndexOf(SEP)));
for (const flowFolder of flowElems) {
await generateFlowLockInternal(
flowFolder,
true, // dryRun - populate tree
for (const folder of flowElems) {
const candidate = await generateFlowLockInternal(
folder,
true, // dryRun
workspace,
opts,
false,
true, // noStaleMessage
false, // legacyBehaviour
tree
true // noStaleMessage
);
if (candidate) {
staleItems.push({ type: "flow", path: candidate, folder });
}
}
}
@@ -168,74 +161,33 @@ async function generateMetadata(
const appFolders = getAppFolders(elems, "app.yaml");
for (const appFolder of rawAppFolders) {
await generateAppLocksInternal(
const candidate = await generateAppLocksInternal(
appFolder,
true, // rawApp
true, // dryRun - populate tree
true, // dryRun
workspace,
opts,
false,
true, // noStaleMessage
false, // legacyBehaviour
tree
true // noStaleMessage
);
if (candidate) {
staleItems.push({ type: "app", path: candidate, folder: appFolder, isRawApp: true });
}
}
for (const appFolder of appFolders) {
await generateAppLocksInternal(
const candidate = await generateAppLocksInternal(
appFolder,
false, // rawApp
true, // dryRun - populate tree
true, // dryRun
workspace,
opts,
false,
true, // noStaleMessage
false, // legacyBehaviour
tree
true // noStaleMessage
);
}
}
// === Propagate staleness through imports ===
tree.propagateStaleness();
// Upload stale scripts to temp storage so the backend can resolve relative imports.
// If this fails (e.g. backend is older and doesn't have /raw_temp endpoints),
// degrade gracefully: locks will be generated using deployed script content only.
try {
await uploadScripts(tree, workspace);
} catch (e) {
log.warn(colors.yellow(
`Failed to upload scripts to temp storage (backend may be too old): ${e}. ` +
`Locks will be generated using deployed script versions only — locally modified ` +
`relative imports may not be reflected.`
));
}
// === Populate staleItems from tree ===
const staleItems: StaleItem[] = [];
const seenFolders = new Set<string>();
for (const p of tree.allPaths()) {
const staleReason = tree.getStaleReason(p);
if (!staleReason) continue;
const itemType = tree.getItemType(p)!;
const itemFolder = tree.getFolder(p)!;
if (itemType === "dependencies") {
staleItems.push({ type: itemType, path: p, folder: itemFolder, staleReason });
} else if (itemType === "inline_script") {
// Inline scripts are not listed separately — their parent flow/app is stale via propagation
continue;
} else if (itemType === "script") {
const originalPath = tree.getOriginalPath(p)!;
staleItems.push({ type: itemType, path: originalPath, folder: itemFolder, staleReason });
} else if (!seenFolders.has(itemFolder)) {
// Flows/Apps: one entry per folder (dedupe multiple inline scripts)
seenFolders.add(itemFolder);
const originalPath = tree.getOriginalPath(p)!;
staleItems.push({ type: itemType, path: originalPath, folder: itemFolder, isRawApp: tree.getIsRawApp(p), staleReason });
if (candidate) {
staleItems.push({ type: "app", path: candidate, folder: appFolder, isRawApp: false });
}
}
}
@@ -248,54 +200,11 @@ async function generateMetadata(
if (folder.endsWith("/")) {
folder = folder.substring(0, folder.length - 1);
}
// Strip file extension if user passed a specific file path (e.g. f/test/script.ts)
const folderNoExt = folder.replace(/\.[^/.]+$/, "");
// Check if an item is inside the specified folder
const isInsideFolder = (item: StaleItem) => {
// Normalize item.folder for comparison (Windows file paths use backslashes)
filteredItems = staleItems.filter((item) => {
const normalizedFolder = item.folder.replaceAll("\\", "/");
const normalizedPath = item.path.replaceAll("\\", "/");
return normalizedFolder === folder || normalizedFolder.startsWith(folder + "/")
|| normalizedPath === folder || normalizedPath === folderNoExt;
};
const isPathInFolder = (p: string) => p.startsWith(folder + "/") || p === folder || p === folderNoExt;
// Check if a tree path or any of its transitive deps is inside the folder
const touchesFolder = (treePath: string) => {
if (isPathInFolder(treePath)) return true;
let found = false;
tree.traverseTransitive(treePath, (importPath) => {
if (isPathInFolder(importPath)) {
found = true;
return true; // stop early
}
});
return found;
};
const isRelevant = (item: StaleItem) => {
if (isInsideFolder(item)) return true;
if (item.type === "dependencies") return true;
const treePath = (item.type === "script"
? item.path.replace(/\.[^/.]+$/, "")
: item.folder).replaceAll("\\", "/");
return touchesFolder(treePath);
};
if (opts.strictFolderBoundaries) {
// Strict mode: only items inside the folder
filteredItems = staleItems.filter(isInsideFolder);
// Warn about stale items outside the folder that would be included by default
const excludedStale = staleItems.filter((item) => !isInsideFolder(item) && isRelevant(item) && item.type !== "dependencies");
for (const item of excludedStale) {
const normalizedPath = item.path.replaceAll("\\", "/");
log.warn(colors.yellow(
`Warning: ${normalizedPath} depends on something inside "${folder}" but is outside it — skipped due to --strict-folder-boundaries. Next generate-metadata will not detect it as stale.`
));
}
} else {
// Default: include items inside the folder and any stale importers that transitively depend on it
filteredItems = staleItems.filter(isRelevant);
}
return normalizedFolder === folder || normalizedFolder.startsWith(folder + "/");
});
}
// === Show stale items and confirm ===
@@ -308,24 +217,28 @@ async function generateMetadata(
const scripts = filteredItems.filter((i) => i.type === "script");
const flows = filteredItems.filter((i) => i.type === "flow");
const apps = filteredItems.filter((i) => i.type === "app");
const deps = filteredItems.filter((i) => i.type === "dependencies");
log.info("");
log.info(`Found ${colors.bold(String(filteredItems.length))} item(s) with stale metadata:`);
log.info(`Found ${filteredItems.length} item(s) with stale metadata:`);
const printItems = (label: string, items: StaleItem[]) => {
if (items.length === 0) return;
log.info(` ${label} (${items.length}):`);
for (const item of items) {
const reason = item.staleReason ? colors.dim(colors.white(`${item.staleReason}`)) : "";
log.info(` ~ ${item.path}` + reason);
if (scripts.length > 0) {
log.info(colors.gray(` Scripts (${scripts.length}):`));
for (const item of scripts) {
log.info(colors.yellow(` ${item.path}`));
}
};
printItems("Workspace dependencies", deps);
printItems("Scripts", scripts);
printItems("Flows", flows);
printItems("Apps", apps);
}
if (flows.length > 0) {
log.info(colors.gray(` Flows (${flows.length}):`));
for (const item of flows) {
log.info(colors.yellow(` ${item.path}`));
}
}
if (apps.length > 0) {
log.info(colors.gray(` Apps (${apps.length}):`));
for (const item of apps) {
log.info(colors.yellow(` ${item.path}`));
}
}
if (opts.dryRun) {
return;
@@ -346,30 +259,28 @@ async function generateMetadata(
log.info("");
// === Process all stale items with progress counter ===
const mismatchedWorkspaceDeps = tree.getMismatchedWorkspaceDeps();
const total = filteredItems.length - deps.length;
const total = filteredItems.length;
const maxWidth = `[${total}/${total}]`.length;
let current = 0;
const formatProgress = (n: number) => {
return colors.dim(colors.white(`[${n}/${total}]`.padEnd(maxWidth, " ")));
const bracket = `[${n}/${total}]`;
return colors.gray(bracket.padEnd(maxWidth, " "));
};
// Process scripts
for (const item of scripts) {
current++;
log.info(`${formatProgress(current)} script ${item.path}`);
log.info(`${formatProgress(current)} script ${colors.cyan(item.path)}`);
await generateScriptMetadataInternal(
item.path, // originalPath with extension
item.folder,
workspace,
opts,
false, // dryRun
true, // noStaleMessage
mismatchedWorkspaceDeps,
true, // noStaleMessage - we handle output
rawWorkspaceDependencies,
codebases,
false,
false, // legacyBehaviour
tree
false
);
}
@@ -377,49 +288,38 @@ async function generateMetadata(
for (const item of flows) {
current++;
const result = await generateFlowLockInternal(
item.folder.replaceAll("/", SEP),
item.folder,
false, // dryRun
workspace,
opts,
false,
true, // noStaleMessage
false, // legacyBehaviour
tree
);
const flowResult = result as FlowLocksResult | undefined;
const scriptsInfo = flowResult?.updatedScripts?.length
? colors.dim(colors.white(`: ${flowResult.updatedScripts.join(", ")}`))
true // noStaleMessage - we handle output
) as FlowLocksResult | void;
const scriptsInfo = result?.updatedScripts?.length
? `: ${colors.gray(result.updatedScripts.join(", "))}`
: "";
log.info(`${formatProgress(current)} flow ${item.path}${scriptsInfo}`);
log.info(`${formatProgress(current)} flow ${colors.cyan(item.path)}${scriptsInfo}`);
}
// Process apps
for (const item of apps) {
current++;
const result = await generateAppLocksInternal(
item.folder.replaceAll("/", SEP),
item.folder,
item.isRawApp!, // rawApp
false, // dryRun
workspace,
opts,
false,
true, // noStaleMessage
false, // legacyBehaviour
tree
);
const appResult = result as AppLocksResult | undefined;
const scriptsInfo = appResult?.updatedScripts?.length
? colors.dim(colors.white(`: ${appResult.updatedScripts.join(", ")}`))
true // noStaleMessage - we handle output
) as AppLocksResult | void;
const scriptsInfo = result?.updatedScripts?.length
? `: ${colors.gray(result.updatedScripts.join(", "))}`
: "";
log.info(`${formatProgress(current)} app ${item.path}${scriptsInfo}`);
log.info(`${formatProgress(current)} app ${colors.cyan(item.path)}${scriptsInfo}`);
}
// Persist all stale workspace dep hashes (not just filtered — deps are global, not folder-scoped)
const allStaleDeps = staleItems.filter((i) => i.type === "dependencies");
await tree.persistDepsHashes(allStaleDeps.map((d) => d.path));
log.info("");
log.info(`Done. Updated ${colors.bold(String(total))} item(s).`);
log.info(colors.green(`Done. Updated ${total} item(s).`));
}
const command = new Command()
@@ -432,7 +332,6 @@ const command = new Command()
.option("--skip-scripts", "Skip processing scripts")
.option("--skip-flows", "Skip processing flows")
.option("--skip-apps", "Skip processing apps")
.option("--strict-folder-boundaries", "Only update items inside the specified folder (requires folder argument)")
.option(
"-i --includes <patterns:file[]>",
"Comma separated patterns to specify which files to include"
+24
View File
@@ -346,6 +346,30 @@ async function initAction(opts: InitOptions) {
log.warn(`Could not create skills: ${skillError}`);
}
}
// Create .claude/launch.json for Claude Preview dev server
try {
const launchJsonPath = ".claude/launch.json";
const launchJson = {
version: "0.0.1",
configurations: [
{
name: "windmill-dev",
runtimeExecutable: "wmill",
runtimeArgs: ["dev"],
port: 3100,
},
],
};
await writeFile(launchJsonPath, JSON.stringify(launchJson, null, 2) + "\n", "utf-8");
log.info(colors.green("Created .claude/launch.json"));
} catch (launchError) {
if (launchError instanceof Error) {
log.warn(`Could not create launch.json: ${launchError.message}`);
} else {
log.warn(`Could not create launch.json: ${launchError}`);
}
}
} catch (error) {
if (error instanceof Error) {
log.warn(`Could not create guidance files: ${error.message}`);
+3 -11
View File
@@ -1,7 +1,7 @@
import { GlobalOptions } from "../../types.ts";
import { requireLogin } from "../../core/auth.ts";
import { resolveWorkspace, validatePath } from "../../core/context.ts";
import { readFile, writeFile, stat, mkdir } from "node:fs/promises";
import { readFile, writeFile, stat } from "node:fs/promises";
import { Buffer } from "node:buffer";
import { colors } from "@cliffy/ansi/colors";
import { Command } from "@cliffy/command";
@@ -130,7 +130,7 @@ async function push(opts: PushOptions, filePath: string) {
[],
undefined,
opts,
await getRawWorkspaceDependencies(true),
await getRawWorkspaceDependencies(),
codebases
);
log.info(colors.bold.underline.green(`Script ${filePath} pushed`));
@@ -1078,11 +1078,6 @@ async function bootstrap(
return;
}
// normalize language aliases
if (language === ("python" as any)) {
language = "python3";
}
const scriptInitialCode = scriptBootstrapCode[language];
if (scriptInitialCode === undefined) {
throw new Error("Language unknown");
@@ -1123,9 +1118,6 @@ async function bootstrap(
yamlOptions
);
const parentDir = path.dirname(scriptCodeFileFullPath);
await mkdir(parentDir, { recursive: true });
await writeFile(scriptCodeFileFullPath, scriptInitialCode, {
flag: 'wx', encoding: 'utf-8',
});
@@ -1169,7 +1161,7 @@ export async function generateMetadata(
opts = await mergeConfigWithConfigFile(opts);
const codebases = await listSyncCodebases(opts);
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
const rawWorkspaceDependencies = await getRawWorkspaceDependencies();
if (scriptPath) {
// read script metadata file
await generateScriptMetadataInternal(
+5 -5
View File
@@ -2280,7 +2280,7 @@ export async function pull(
const tracker: ChangeTracker = await buildTracker(changes);
const rawWorkspaceDependencies: Record<string, string> =
await getRawWorkspaceDependencies(true);
await getRawWorkspaceDependencies();
for (const change of tracker.scripts) {
await generateScriptMetadataInternal(
@@ -2611,7 +2611,7 @@ export async function push(
false, // els1 (local) is not the remote source
);
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
const rawWorkspaceDependencies = await getRawWorkspaceDependencies();
const tracker: ChangeTracker = await buildTracker(changes);
@@ -2657,7 +2657,7 @@ export async function push(
true,
);
if (stale) {
staleFlows.push(stale as string);
staleFlows.push(stale);
}
}
@@ -2682,7 +2682,7 @@ export async function push(
true,
);
if (stale) {
staleApps.push(stale as string);
staleApps.push(stale);
}
}
@@ -2697,7 +2697,7 @@ export async function push(
true,
);
if (stale) {
staleApps.push(stale as string);
staleApps.push(stale);
}
}
File diff suppressed because one or more lines are too long
-373
View File
@@ -1,373 +0,0 @@
/**
* Double-linked dependency tree for tracking script imports and propagating staleness.
*/
import { Workspace } from "../commands/workspace/workspace.ts";
import * as wmill from "../../gen/services.gen.ts";
import type { ScriptLang } from "../../gen/types.gen.ts";
import { ScriptLanguage } from "./script_common.ts";
import {
filterWorkspaceDependencies,
generateScriptHash,
checkifMetadataUptodate,
workspaceDependenciesPathToLanguageAndFilename,
updateMetadataGlobalLock,
} from "./metadata.ts";
import { generateHash } from "./utils.ts";
/**
* Diff local scripts against deployed versions, upload only those that differ.
* Only uploaded (mismatched) scripts get contentHash set, so flatten() returns
* temp_script_refs only for scripts the backend can't resolve from deployed versions.
*/
export async function uploadScripts(
tree: DoubleLinkedDependencyTree,
workspace: Workspace
): Promise<void> {
// Split into scripts vs workspace deps and compute SHA256(content) for each
const scriptHashes: Record<string, string> = {};
const workspaceDeps: { path: string; language: ScriptLang; name?: string; hash: string }[] = [];
for (const path of tree.allPaths()) {
const content = tree.getContent(path);
const itemType = tree.getItemType(path);
if (itemType === "dependencies") {
// Empty string is valid for workspace deps (means "no deps") — only skip undefined
if (content === undefined) continue;
const info = workspaceDependenciesPathToLanguageAndFilename(path);
if (info) {
const hash = await generateHash(content);
workspaceDeps.push({ path, language: info.language as ScriptLang, name: info.name, hash });
}
} else if (itemType === "script") {
if (!content) continue;
const hash = await generateHash(content);
scriptHashes[path] = hash;
}
// Skip inline_script, flow, app — they don't need temp storage uploads
}
if (Object.keys(scriptHashes).length === 0 && workspaceDeps.length === 0) return;
// Single batch query: find which scripts/deps differ from deployed versions
const mismatched = await wmill.diffRawScriptsWithDeployed({
workspace: workspace.workspaceId,
requestBody: {
scripts: scriptHashes,
workspace_deps: workspaceDeps,
},
});
// Upload only mismatched scripts to temp storage
for (const path of mismatched) {
const content = tree.getContent(path);
const itemType = tree.getItemType(path);
if (itemType === "dependencies") {
// Workspace deps don't need temp storage — just mark as mismatched.
// Empty string is valid (means the dep file was emptied locally).
if (content !== undefined) {
tree.setContentHash(path, "mismatched");
}
} else if (content) {
const hash = await wmill.storeRawScriptTemp({
workspace: workspace.workspaceId,
requestBody: content,
});
tree.setContentHash(path, hash);
}
}
}
export type ItemType = "script" | "inline_script" | "flow" | "app" | "dependencies";
interface DependencyNode {
content: string;
stalenessHash: string; // Hash for staleness detection (includes deps, content, metadata)
contentHash?: string; // Hash for temp storage lookup (content only)
language: ScriptLanguage;
metadata: string;
imports: Set<string>;
importedBy: Set<string>;
staleReason?: string;
// Item metadata for generate-metadata command
itemType: ItemType;
folder: string; // Folder path (for flows/apps) or remote path (for scripts)
originalPath: string; // Original path passed to handler (with extension for scripts)
isRawApp?: boolean; // Only set for apps
isDirectlyStale: boolean; // True if this item's content changed (vs transitively stale)
}
export class DoubleLinkedDependencyTree {
private nodes: Map<string, DependencyNode> = new Map();
private workspaceDeps: Record<string, string> = {};
setWorkspaceDeps(deps: Record<string, string>): void {
this.workspaceDeps = deps;
}
async addNode(
path: string,
content: string,
language: ScriptLanguage,
metadata: string,
imports: string[],
itemType: ItemType,
folder: string,
originalPath: string,
isDirectlyStale: boolean,
isRawApp?: boolean
): Promise<void> {
const hasWorkspaceDeps = itemType === "script" || itemType === "inline_script";
const filteredDeps = hasWorkspaceDeps
? filterWorkspaceDependencies(this.workspaceDeps, content, language)
: {};
const stalenessHash = await generateScriptHash({}, content, metadata);
if (!this.nodes.has(path)) {
this.nodes.set(path, {
content: "", stalenessHash: "", language: "deno", metadata: "",
imports: new Set(), importedBy: new Set(),
itemType: "script", folder: "", originalPath: "", isDirectlyStale: false,
});
}
const node = this.nodes.get(path)!;
node.content = content;
node.stalenessHash = stalenessHash;
node.language = language;
node.metadata = metadata;
node.itemType = itemType;
node.folder = folder;
node.originalPath = originalPath;
node.isDirectlyStale = isDirectlyStale;
node.isRawApp = isRawApp;
// Create nodes for referenced workspace deps with content and language.
const filteredDepsPaths = Object.keys(filteredDeps);
for (const depsPath of filteredDepsPaths) {
if (!this.nodes.has(depsPath)) {
const depsInfo = workspaceDependenciesPathToLanguageAndFilename(depsPath);
const contentHash = await generateHash(filteredDeps[depsPath] + depsPath);
const isUpToDate = await checkifMetadataUptodate(depsPath, contentHash, undefined);
this.nodes.set(depsPath, {
content: filteredDeps[depsPath],
stalenessHash: "", language: depsInfo?.language ?? "deno", metadata: "",
imports: new Set(), importedBy: new Set(),
itemType: "dependencies", folder: "", originalPath: depsPath,
isDirectlyStale: !isUpToDate,
});
}
}
const allImports = [...imports, ...filteredDepsPaths];
for (const importPath of allImports) {
node.imports.add(importPath);
if (!this.nodes.has(importPath)) {
this.nodes.set(importPath, {
content: "", stalenessHash: "", language: "deno", metadata: "",
imports: new Set(), importedBy: new Set(),
itemType: "script", folder: "", originalPath: "", isDirectlyStale: false,
});
}
this.nodes.get(importPath)!.importedBy.add(path);
}
}
getContent(path: string): string | undefined {
return this.nodes.get(path)?.content;
}
getStalenessHash(path: string): string | undefined {
return this.nodes.get(path)?.stalenessHash;
}
getContentHash(path: string): string | undefined {
return this.nodes.get(path)?.contentHash;
}
setContentHash(path: string, hash: string): void {
const node = this.nodes.get(path);
if (node) {
node.contentHash = hash;
}
}
getLanguage(path: string): ScriptLanguage | undefined {
return this.nodes.get(path)?.language;
}
getMetadata(path: string): string | undefined {
return this.nodes.get(path)?.metadata;
}
getStaleReason(path: string): string | undefined {
return this.nodes.get(path)?.staleReason;
}
getItemType(path: string): ItemType | undefined {
return this.nodes.get(path)?.itemType;
}
getFolder(path: string): string | undefined {
return this.nodes.get(path)?.folder;
}
getIsRawApp(path: string): boolean | undefined {
return this.nodes.get(path)?.isRawApp;
}
getIsDirectlyStale(path: string): boolean {
return this.nodes.get(path)?.isDirectlyStale ?? false;
}
getOriginalPath(path: string): string | undefined {
return this.nodes.get(path)?.originalPath;
}
getImports(path: string): Set<string> | undefined {
return this.nodes.get(path)?.imports;
}
/**
* Returns true if this node has been marked stale (directly or transitively).
*/
isStale(path: string): boolean {
return this.nodes.get(path)?.staleReason !== undefined;
}
/**
* Mutates the tree by removing all nodes that are not stale.
* Uses BFS on reverse graph (importedBy) to find all stale scripts.
* Starts from nodes with isDirectlyStale=true.
*/
propagateStaleness(): void {
// Collect directly stale nodes
const directlyStale = new Set<string>();
for (const [path, node] of this.nodes.entries()) {
if (node.isDirectlyStale) {
directlyStale.add(path);
node.staleReason = "content changed";
}
}
const allStale = new Set(directlyStale);
const queue = [...directlyStale];
const visited = new Set<string>();
while (queue.length > 0) {
const scriptPath = queue.shift()!;
if (visited.has(scriptPath)) continue;
visited.add(scriptPath);
const node = this.nodes.get(scriptPath);
if (!node) continue;
for (const importer of node.importedBy) {
if (!allStale.has(importer)) {
allStale.add(importer);
queue.push(importer);
// Set reason for transitively stale scripts
const importerNode = this.nodes.get(importer);
if (importerNode) importerNode.staleReason = `depends on ${scriptPath}`;
}
}
}
}
/**
* Walks all transitive imports for a node, calling the callback for each.
* Callback may return true to stop traversing that branch.
*/
traverseTransitive(scriptPath: string, callback: (importPath: string, node: DependencyNode) => boolean | void): void {
const queue = [scriptPath];
const visited = new Set<string>();
while (queue.length > 0) {
const current = queue.shift()!;
if (visited.has(current)) continue;
visited.add(current);
const node = this.nodes.get(current);
if (!node) continue;
for (const importPath of node.imports) {
const importNode = this.nodes.get(importPath);
if (importNode) {
const stop = callback(importPath, importNode);
if (!stop) {
queue.push(importPath);
}
}
}
}
}
allPaths(): IterableIterator<string> {
return this.nodes.keys();
}
/**
* Returns paths of all stale nodes (those with a staleReason).
*/
*stalePaths(): IterableIterator<string> {
for (const [path, node] of this.nodes.entries()) {
if (node.staleReason) {
yield path;
}
}
}
has(path: string): boolean {
return this.nodes.has(path);
}
/**
* Returns workspace deps that were uploaded as mismatched with remote.
* These need to be passed as raw_workspace_dependencies in job args
* so the backend uses local content instead of deployed.
*/
getMismatchedWorkspaceDeps(): Record<string, string> {
const result: Record<string, string> = {};
for (const [path, node] of this.nodes.entries()) {
if (node.itemType === "dependencies" && node.contentHash && node.content !== undefined) {
result[path] = node.content;
}
}
return result;
}
/**
* Returns path contentHash for all transitive imports that have been uploaded.
* Must be called after uploadScripts() has populated contentHash values.
*/
getTempScriptRefs(scriptPath: string): Record<string, string> {
const result: Record<string, string> = {};
this.traverseTransitive(scriptPath, (_path, node) => {
if (node.contentHash) {
result[_path] = node.contentHash;
}
});
return result;
}
/**
* Persist workspace dep hashes to wmill-lock.yaml so getRawWorkspaceDependencies
* considers them up-to-date on the next run.
*/
async persistDepsHashes(depsPaths: string[]): Promise<void> {
for (const path of depsPaths) {
const node = this.nodes.get(path);
if (node?.itemType === "dependencies" && node.content !== undefined) {
const hash = await generateHash(node.content + path);
await updateMetadataGlobalLock(path, hash);
}
}
}
get size(): number {
return this.nodes.size;
}
}
+34 -70
View File
@@ -26,13 +26,11 @@ import { generateHash, readInlinePathSync, getHeaders } from "./utils.ts";
import { SyncCodebase } from "./codebase.ts";
import { argSigToJsonSchemaType } from "../../windmill-utils-internal/src/parse/parse-schema.ts";
import { getIsWin } from "./utils.ts";
import { extractRelativeImports } from "./relative_imports.ts";
import { DoubleLinkedDependencyTree } from "./dependency_tree.ts";
const _require = createRequire(import.meta.url);
const _parserCache = new Map<string, Promise<any>>();
export function loadParser(pkgName: string): Promise<any> {
function loadParser(pkgName: string): Promise<any> {
let p = _parserCache.get(pkgName);
if (!p) {
p = (async () => {
@@ -56,7 +54,7 @@ export class LockfileGenerationError extends Error {
}
export async function getRawWorkspaceDependencies(legacyBehaviour: boolean): Promise<Record<string, string>> {
export async function getRawWorkspaceDependencies(): Promise<Record<string, string>> {
const rawWorkspaceDeps: Record<string, string> = {};
try {
@@ -70,13 +68,11 @@ export async function getRawWorkspaceDependencies(legacyBehaviour: boolean): Pro
// Find matching language
for (const lang of workspaceDependenciesLanguages) {
if (entry.name.endsWith(lang.filename)) {
if (legacyBehaviour) {
const contentHash = await generateHash(content + filePath);
const isUpToDate = await checkifMetadataUptodate(filePath, contentHash, undefined);
if (!isUpToDate) {
rawWorkspaceDeps[filePath] = content;
}
} else {
// Check if out of sync
const contentHash = await generateHash(content + filePath);
const isUpToDate = await checkifMetadataUptodate(filePath, contentHash, undefined);
if (!isUpToDate) {
rawWorkspaceDeps[filePath] = content;
}
break;
@@ -190,9 +186,7 @@ export async function generateScriptMetadataInternal(
noStaleMessage: boolean,
rawWorkspaceDependencies: Record<string, string>,
codebases: SyncCodebase[],
justUpdateMetadataLock?: boolean,
legacyBehaviour?: boolean,
tree?: DoubleLinkedDependencyTree
justUpdateMetadataLock?: boolean
): Promise<string | undefined> {
// Detect folder layout: my_script__mod/script.ts
const isFolderLayout = isModuleEntryPoint(scriptPath);
@@ -228,15 +222,13 @@ export async function generateScriptMetadataInternal(
const hasModules = existsSync(moduleFolderPath) && statSync(moduleFolderPath).isDirectory();
// In non-legacy mode, workspace deps are tracked via the tree — exclude from hash
const depsForHash = (!legacyBehaviour && tree) ? {} : filteredRawWorkspaceDependencies;
let hash = await generateScriptHash(depsForHash, scriptContent, metadataContent);
let hash = await generateScriptHash(filteredRawWorkspaceDependencies, scriptContent, metadataContent);
// Compute per-module hashes for stale detection (like flow inline scripts)
let moduleHashes: Record<string, string> = {};
if (hasModules) {
moduleHashes = await computeModuleHashes(
moduleFolderPath, opts.defaultTs, (!legacyBehaviour && tree) ? {} : rawWorkspaceDependencies, isFolderLayout
moduleFolderPath, opts.defaultTs, rawWorkspaceDependencies, isFolderLayout
);
}
const hasModuleHashes = Object.keys(moduleHashes).length > 0;
@@ -251,43 +243,27 @@ export async function generateScriptMetadataInternal(
}
const conf = await readLockfile();
// Use checkHash (includes module hashes) so module changes are detected as stale
const isDirectlyStale = !(await checkifMetadataUptodate(remotePath, checkHash, conf, checkSubpath));
// New behaviour: tree-based dependency tracking
if (!legacyBehaviour && tree) {
if (dryRun) {
// First pass: populate tree with script and its imports
const imports = await extractRelativeImports(scriptContent, remotePath, language);
await tree.addNode(remotePath, scriptContent, language, metadataContent, imports, "script", remotePath, scriptPath, isDirectlyStale);
return;
if (await checkifMetadataUptodate(remotePath, checkHash, conf, checkSubpath)) {
if (!noStaleMessage) {
log.info(
colors.green(`Script ${remotePath} metadata is up-to-date, skipping`)
);
}
// Second pass: proceed to generate (caller verified this script is stale via tree)
} else {
// Legacy behaviour: use existing staleness check
if (await checkifMetadataUptodate(remotePath, checkHash, conf, checkSubpath)) {
if (!noStaleMessage) {
log.info(
colors.green(`Script ${remotePath} metadata is up-to-date, skipping`)
);
}
return;
} else if (dryRun) {
let detail = `${remotePath} (${language})`;
if (hasModuleHashes) {
const changed: string[] = [];
for (const [modulePath, moduleHash] of Object.entries(moduleHashes)) {
if (!(await checkifMetadataUptodate(remotePath, moduleHash, conf, modulePath))) {
changed.push(modulePath);
}
}
if (changed.length > 0) {
detail += ` [changed modules: ${changed.join(", ")}]`;
return;
} else if (dryRun) {
let detail = `${remotePath} (${language})`;
if (hasModuleHashes) {
const changed: string[] = [];
for (const [modulePath, moduleHash] of Object.entries(moduleHashes)) {
if (!(await checkifMetadataUptodate(remotePath, moduleHash, conf, modulePath))) {
changed.push(modulePath);
}
}
return detail;
if (changed.length > 0) {
detail += ` [changed modules: ${changed.join(", ")}]`;
}
}
return detail;
}
if (!justUpdateMetadataLock && !noStaleMessage) {
@@ -312,7 +288,6 @@ export async function generateScriptMetadataInternal(
const hasCodebase = findCodebase(scriptPath, codebases) != undefined;
if (!hasCodebase) {
const tempScriptRefs = tree?.getTempScriptRefs(remotePath);
const lockPathOverride = isFolderLayout
? path.dirname(scriptPath) + "/script.lock"
: undefined;
@@ -323,7 +298,6 @@ export async function generateScriptMetadataInternal(
remotePath,
metadataParsedContent,
filteredRawWorkspaceDependencies,
tempScriptRefs,
lockPathOverride,
);
} else {
@@ -384,7 +358,7 @@ export async function generateScriptMetadataInternal(
const metadataContentUsedForHash = newMetadataContent;
hash = await generateScriptHash(
depsForHash,
filteredRawWorkspaceDependencies,
scriptContent,
metadataContentUsedForHash
);
@@ -537,7 +511,6 @@ export async function computeLockCacheKey(
scriptContent: string,
language: ScriptLanguage,
rawWorkspaceDependencies: Record<string, string>,
tempScriptRefs?: Record<string, string>
): Promise<string> {
const annotation = extractWorkspaceDepsAnnotation(scriptContent, language);
const annotationStr = annotation
@@ -545,10 +518,7 @@ export async function computeLockCacheKey(
: "none";
const sortedDepsKeys = Object.keys(rawWorkspaceDependencies).sort();
const depsStr = sortedDepsKeys.map((k) => `${k}=${rawWorkspaceDependencies[k]}`).join(";");
const tempRefsStr = tempScriptRefs
? Object.keys(tempScriptRefs).sort().map((k) => `${k}=${tempScriptRefs[k]}`).join(";")
: "";
return await generateHash(`${language}|${annotationStr}|${depsStr}|${tempRefsStr}`);
return await generateHash(`${language}|${annotationStr}|${depsStr}`);
}
const lockCache = new Map<string, string>();
@@ -563,15 +533,13 @@ async function fetchScriptLock(
language: ScriptLanguage,
remotePath: string,
rawWorkspaceDependencies: Record<string, string>,
tempScriptRefs?: Record<string, string>
): Promise<string> {
const hasRawDeps = Object.keys(rawWorkspaceDependencies).length > 0;
const hasTempRefs = tempScriptRefs && Object.keys(tempScriptRefs).length > 0;
const cacheKey = (hasRawDeps || hasTempRefs)
? await computeLockCacheKey(scriptContent, language, rawWorkspaceDependencies, tempScriptRefs)
const cacheKey = hasRawDeps
? await computeLockCacheKey(scriptContent, language, rawWorkspaceDependencies)
: undefined;
if (cacheKey && lockCache.has(cacheKey)) {
log.debug(`Using cached lockfile for ${remotePath}`);
log.info(`Using cached lockfile for ${remotePath}`);
return lockCache.get(cacheKey)!;
}
@@ -596,8 +564,6 @@ async function fetchScriptLock(
raw_workspace_dependencies: Object.keys(rawWorkspaceDependencies).length > 0
? rawWorkspaceDependencies : null,
entrypoint: remotePath,
temp_script_refs: tempScriptRefs && Object.keys(tempScriptRefs).length > 0
? tempScriptRefs : null,
}),
}
);
@@ -638,7 +604,6 @@ async function updateScriptLock(
remotePath: string,
metadataContent: Record<string, any>,
rawWorkspaceDependencies: Record<string, string>,
tempScriptRefs?: Record<string, string>,
lockPathOverride?: string,
): Promise<void> {
if (
@@ -656,7 +621,7 @@ async function updateScriptLock(
if (Object.keys(rawWorkspaceDependencies).length > 0) {
const dependencyPaths = Object.keys(rawWorkspaceDependencies).join(', ');
log.debug(`Generating script lock for ${remotePath} with raw workspace dependencies: ${dependencyPaths}`);
log.info(`Generating script lock for ${remotePath} with raw workspace dependencies: ${dependencyPaths}`);
}
const lock = await fetchScriptLock(
@@ -665,7 +630,6 @@ async function updateScriptLock(
language,
remotePath,
rawWorkspaceDependencies,
tempScriptRefs
);
const lockPath = lockPathOverride ?? remotePath + ".script.lock";
@@ -728,7 +692,7 @@ async function updateModuleLocks(
const moduleContent = readFileSync(fullPath, "utf-8");
const moduleRemotePath = scriptRemotePath + "/" + relPath;
log.debug(`Generating lock for module ${relPath}`);
log.info(colors.gray(`Generating lock for module ${relPath}`));
try {
const lock = await fetchScriptLock(
-39
View File
@@ -1,39 +0,0 @@
/**
* Relative Imports Utilities for CLI
*
* Provides functions to parse relative imports from TypeScript/Python scripts using WASM.
*/
import { ScriptLanguage } from "./script_common.ts";
import { loadParser } from "./metadata.ts";
import * as log from "../core/log.ts";
/**
* Extract relative imports from script content based on language.
* Returns resolved absolute Windmill paths (e.g., "f/folder/helper").
*/
export async function extractRelativeImports(
code: string,
scriptPath: string,
language: ScriptLanguage
): Promise<string[]> {
try {
switch (language) {
case "bun":
case "nativets":
case "deno": {
const { parse_ts_relative_imports } = await loadParser("windmill-parser-wasm-ts");
return parse_ts_relative_imports(code, scriptPath);
}
case "python3": {
const { parse_py_relative_imports } = await loadParser("windmill-parser-wasm-py-imports");
return parse_py_relative_imports(code, scriptPath);
}
default:
return [];
}
} catch (e) {
log.warn(`Failed to parse relative imports for ${scriptPath}: ${e}. Dependency tracking for relative imports will be disabled.`);
return [];
}
}
-20
View File
@@ -189,26 +189,6 @@ export function isFolderResourcePath(p: string): boolean {
return isFlowPath(p) || isAppPath(p) || isRawAppPath(p);
}
/**
* Check if a path is inside a folder-based resource, checking BOTH dotted (.flow, .app, .raw_app)
* and non-dotted (__flow, __app, __raw_app) formats regardless of the global nonDottedPaths setting.
* Use this instead of isFolderResourcePath when the config may not yet be loaded or when
* you need to handle mixed-format workspaces (e.g. generate-metadata scanning all files).
*/
export function isFolderResourcePathAnyFormat(p: string): boolean {
const n = normalizeSep(p);
for (const suffixes of [DOTTED_SUFFIXES, NON_DOTTED_SUFFIXES]) {
if (
n.includes(suffixes.flow + "/") ||
n.includes(suffixes.app + "/") ||
n.includes(suffixes.raw_app + "/")
) {
return true;
}
}
return false;
}
/**
* Detect the resource type from a path, if any
*/
+2 -27
View File
@@ -63,11 +63,11 @@ export class CargoBackend {
// Determine default features based on environment
// CI mode: minimal features (zip only)
// Local mode with license key: full features (zip, private, enterprise, license, python)
// Local mode with license key: full features (zip, private, enterprise, license)
// Local mode without license key: zip only (EE features reject API calls without valid license)
const isCI = process.env["CI_MINIMAL_FEATURES"] === "true";
const hasLicenseKey = !!process.env["EE_LICENSE_KEY"];
const defaultFeatures = isCI ? ["zip"] : (hasLicenseKey ? ["zip", "private", "enterprise", "license", "python"] : ["zip", "python"]);
const defaultFeatures = isCI ? ["zip"] : (hasLicenseKey ? ["zip", "private", "enterprise", "license"] : ["zip"]);
// Parse additional features from environment variable
const envFeatures = process.env["TEST_FEATURES"]?.split(",").filter(f => f.trim()) || [];
@@ -328,8 +328,6 @@ export class CargoBackend {
SQLX_OFFLINE: "true",
// Disable embedding to speed up startup
DISABLE_EMBEDDING: "true",
// Skip worker version check for workspace deps (workers need time to report version)
WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES: "1",
// Create default admin user
CREATE_SUPERADMIN_IF_NOT_EXISTS: "1",
SUPERADMIN_EMAIL: this.config.username,
@@ -710,7 +708,6 @@ export class CargoBackend {
this.deleteAll("resources"),
this.deleteAll("variables"),
this.deleteAll("folders"),
this.deleteAllWorkspaceDeps(),
]);
console.log("Workspace reset complete");
@@ -738,28 +735,6 @@ export class CargoBackend {
// Ignore listing failures
}
}
private async deleteAllWorkspaceDeps(): Promise<void> {
try {
const listResponse = await this.apiRequest(`/api/w/${this.config.workspace}/workspace_dependencies/list`);
if (!listResponse.ok) return;
const items = await listResponse.json() as { language: string; name?: string }[];
for (const item of items) {
try {
const nameParam = item.name ? `?name=${encodeURIComponent(item.name)}` : "";
await this.apiRequest(
`/api/w/${this.config.workspace}/workspace_dependencies/delete/${item.language}${nameParam}`,
{ method: "POST" }
);
} catch {
// Ignore individual deletion failures
}
}
} catch {
// Ignore failures
}
}
}
// Global backend instance
-420
View File
@@ -1,420 +0,0 @@
/**
* Relative Imports Tests
*
* E2E tests for the `generate-metadata` command with relative imports:
* - Lock files correctly include transitive dependencies
* - Staleness propagates through import chains
* - Various import patterns handled correctly
*/
import { expect, test } from "bun:test";
import { writeFile, readFile, mkdir } from "node:fs/promises";
import { withTestBackend } from "./test_backend.ts";
// TODO: re-enable Python tests on CI if python feature is included by default
const isCI = process.env["CI_MINIMAL_FEATURES"] === "true";
const defaultMetadata = `summary: "Test"
schema:
type: object
properties: {}
lock: ""
`;
// =============================================================================
// Test 1: TS basic import with npm dependency propagation
// =============================================================================
test("TS: imported script's npm dep appears in importer's lock", { timeout: 60000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/test`, { recursive: true });
const scriptA = `import { helper } from "./script_b.ts";
export async function main() { return helper(); }
`;
const scriptB = `import _ from "lodash";
export function helper() { return _.VERSION; }
`;
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
const result = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
expect(result.code, `generate-metadata failed:\nSTDOUT: ${result.stdout}\nSTDERR: ${result.stderr}`).toBe(0);
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
expect(lockB).toContain("lodash");
expect(lockA).toContain("lodash");
});
});
// =============================================================================
// Test 2: TS chained imports - dependency propagates through chain
// =============================================================================
test("TS: chained imports propagate npm deps through entire chain", { timeout: 60000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/test`, { recursive: true });
const scriptA = `import { utilB } from "./script_b.ts";
export async function main() { return utilB(); }
`;
const scriptB = `import { utilC } from "./script_c.ts";
export function utilB() { return utilC() + " B"; }
`;
const scriptC = `import _ from "lodash";
export function utilC() { return _.VERSION; }
`;
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_c.ts`, scriptC);
await writeFile(`${tempDir}/f/test/script_c.script.yaml`, defaultMetadata);
const result = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
expect(result.code).toBe(0);
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
const lockC = await readFile(`${tempDir}/f/test/script_c.script.lock`, "utf-8").catch(() => "");
expect(lockC).toContain("lodash");
expect(lockB).toContain("lodash");
expect(lockA).toContain("lodash");
});
});
// =============================================================================
// Test 3: TS circular imports - completes without hanging, locks generated
// =============================================================================
test("TS: circular imports handled gracefully with correct locks", { timeout: 60000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/test`, { recursive: true });
// Circular: A imports B, B imports A, B has npm dep
const scriptA = `import { funcB } from "./script_b.ts";
export function funcA() { return "A"; }
export async function main() { return funcA() + funcB(); }
`;
const scriptB = `import { funcA } from "./script_a.ts";
import _ from "lodash";
export function funcB() { return _.VERSION + funcA(); }
`;
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
const result = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
expect(result.code).toBe(0);
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
expect(lockB).toContain("lodash");
expect(lockA).toContain("lodash");
});
});
// =============================================================================
// Test 4: Python basic import with pip dependency propagation
// =============================================================================
test.skipIf(isCI)("Python: imported script's pip dep appears in importer's lock", { timeout: 60000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/test`, { recursive: true });
const mainPy = `from f.test.helper import helper_func
def main():
return helper_func()
`;
const helperPy = `import requests
def helper_func():
return requests.__version__
`;
await writeFile(`${tempDir}/f/test/main.py`, mainPy);
await writeFile(`${tempDir}/f/test/main.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/helper.py`, helperPy);
await writeFile(`${tempDir}/f/test/helper.script.yaml`, defaultMetadata);
const result = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
if (result.code !== 0) {
console.log("STDOUT:", result.stdout);
console.log("STDERR:", result.stderr);
}
expect(result.code).toBe(0);
const lockMain = await readFile(`${tempDir}/f/test/main.script.lock`, "utf-8").catch(() => "");
const lockHelper = await readFile(`${tempDir}/f/test/helper.script.lock`, "utf-8").catch(() => "");
expect(lockHelper).toContain("requests");
expect(lockMain).toContain("requests");
});
});
// =============================================================================
// Test 5: Diamond dependency - A imports B and C, both import D
// =============================================================================
test.skipIf(isCI)("Python: diamond dependency pattern propagates correctly", { timeout: 60000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/test`, { recursive: true });
// Diamond: A -> B, A -> C, B -> D, C -> D
const scriptA = `from f.test.script_b import func_b
from f.test.script_c import func_c
def main():
return func_b() + func_c()
`;
const scriptB = `from f.test.script_d import func_d
def func_b():
return "B" + func_d()
`;
const scriptC = `from f.test.script_d import func_d
def func_c():
return "C" + func_d()
`;
const scriptD = `import requests
def func_d():
return requests.__version__
`;
await writeFile(`${tempDir}/f/test/script_a.py`, scriptA);
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_b.py`, scriptB);
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_c.py`, scriptC);
await writeFile(`${tempDir}/f/test/script_c.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_d.py`, scriptD);
await writeFile(`${tempDir}/f/test/script_d.script.yaml`, defaultMetadata);
const result = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
expect(result.code).toBe(0);
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
const lockC = await readFile(`${tempDir}/f/test/script_c.script.lock`, "utf-8").catch(() => "");
const lockD = await readFile(`${tempDir}/f/test/script_d.script.lock`, "utf-8").catch(() => "");
expect(lockD).toContain("requests");
expect(lockB).toContain("requests");
expect(lockC).toContain("requests");
expect(lockA).toContain("requests");
});
});
// =============================================================================
// Test 6: Script isolation - unrelated script not marked stale
// =============================================================================
test("Script isolation: unrelated script not affected by changes", { timeout: 120000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/test`, { recursive: true });
// A imports B, C is isolated
const scriptA = `import { helper } from "./script_b.ts";
export async function main() { return helper(); }
`;
const scriptB = `export function helper() { return "B"; }
`;
const scriptC = `export async function main() { return "isolated"; }
`;
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/script_c.ts`, scriptC);
await writeFile(`${tempDir}/f/test/script_c.script.yaml`, defaultMetadata);
// Generate initial metadata
const initial = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
expect(initial.code).toBe(0);
// Verify all up to date
const check1 = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes", "--dry-run"],
tempDir
);
expect(check1.stdout).toContain("All metadata up-to-date");
// Change script_b
await writeFile(`${tempDir}/f/test/script_b.ts`,
`export function helper() { return "B changed"; }
`);
// script_a and script_b should be stale, script_c should NOT be mentioned
const check2 = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes", "--dry-run"],
tempDir
);
expect(check2.code).toBe(0);
expect(check2.stdout).toContain("script_b");
expect(check2.stdout).toContain("script_a");
expect(check2.stdout).not.toMatch(/script_c/);
});
});
// =============================================================================
// Test 7: Python relative imports with dot syntax
// =============================================================================
test.skipIf(isCI)("Python: relative imports with dot syntax work correctly", { timeout: 60000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/mymodule`, { recursive: true });
// Using relative import syntax
const mainPy = `from .helper import helper_func
def main():
return helper_func()
`;
const helperPy = `import requests
def helper_func():
return requests.__version__
`;
await writeFile(`${tempDir}/f/mymodule/main.py`, mainPy);
await writeFile(`${tempDir}/f/mymodule/main.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/mymodule/helper.py`, helperPy);
await writeFile(`${tempDir}/f/mymodule/helper.script.yaml`, defaultMetadata);
const result = await backend.runCLICommand(
["generate-metadata", "-i", "f/mymodule/*", "--yes"],
tempDir
);
expect(result.code).toBe(0);
const lockMain = await readFile(`${tempDir}/f/mymodule/main.script.lock`, "utf-8").catch(() => "");
const lockHelper = await readFile(`${tempDir}/f/mymodule/helper.script.lock`, "utf-8").catch(() => "");
expect(lockHelper).toContain("requests");
expect(lockMain).toContain("requests");
});
});
// =============================================================================
// Test 8: Adding new import updates importer's lock
// =============================================================================
test.skipIf(isCI)("Python: adding new import updates importer's lock correctly", { timeout: 120000 }, async () => {
await withTestBackend(async (backend, tempDir) => {
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
excludes: []`);
await mkdir(`${tempDir}/f/test`, { recursive: true });
// Initial: main imports helper, helper has no external deps
const mainPy = `from f.test.helper import helper_func
def main():
return helper_func()
`;
const helperPyInitial = `def helper_func():
return "no deps"
`;
await writeFile(`${tempDir}/f/test/main.py`, mainPy);
await writeFile(`${tempDir}/f/test/main.script.yaml`, defaultMetadata);
await writeFile(`${tempDir}/f/test/helper.py`, helperPyInitial);
await writeFile(`${tempDir}/f/test/helper.script.yaml`, defaultMetadata);
// Generate initial locks
const initial = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
expect(initial.code).toBe(0);
// Add new script with pip dep
const utilsPy = `import requests
def get_version():
return requests.__version__
`;
await writeFile(`${tempDir}/f/test/utils.py`, utilsPy);
await writeFile(`${tempDir}/f/test/utils.script.yaml`, defaultMetadata);
// Modify helper to import utils
const helperPyWithImport = `from f.test.utils import get_version
def helper_func():
return get_version()
`;
await writeFile(`${tempDir}/f/test/helper.py`, helperPyWithImport);
// Regenerate - main should now have requests
const afterAdd = await backend.runCLICommand(
["generate-metadata", "-i", "f/test/*", "--yes"],
tempDir
);
expect(afterAdd.code).toBe(0);
const lockUtils = await readFile(`${tempDir}/f/test/utils.script.lock`, "utf-8").catch(() => "");
const lockHelper = await readFile(`${tempDir}/f/test/helper.script.lock`, "utf-8").catch(() => "");
const lockMain = await readFile(`${tempDir}/f/test/main.script.lock`, "utf-8").catch(() => "");
expect(lockUtils).toContain("requests");
expect(lockHelper).toContain("requests");
expect(lockMain).toContain("requests");
});
});

Some files were not shown because too many files have changed in this diff Show More