mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-16 00:02:29 +00:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
632c8868bc |
@@ -1,59 +0,0 @@
|
||||
---
|
||||
name: commit
|
||||
description: Create a git commit with conventional commit format. MUST use anytime you want to commit changes.
|
||||
---
|
||||
|
||||
# Git Commit Skill
|
||||
|
||||
Create a focused, single-line commit following conventional commit conventions.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. **Analyze changes**: Run `git status` and `git diff` to understand what was modified
|
||||
2. **Stage only modified files**: Add files individually by name. NEVER use `git add -A` or `git add .`
|
||||
3. **Write commit message**: Follow the conventional commit format as a single line
|
||||
|
||||
## Conventional Commit Format
|
||||
|
||||
```
|
||||
<type>: <description>
|
||||
```
|
||||
|
||||
### Types
|
||||
- `feat`: New feature or capability
|
||||
- `fix`: Bug fix
|
||||
- `refactor`: Code change that neither fixes a bug nor adds a feature
|
||||
- `docs`: Documentation only changes
|
||||
- `style`: Formatting, missing semicolons, etc (no code change)
|
||||
- `test`: Adding or correcting tests
|
||||
- `chore`: Maintenance tasks, dependency updates, etc
|
||||
- `perf`: Performance improvement
|
||||
|
||||
### Rules
|
||||
- Message MUST be a single line (no multi-line messages)
|
||||
- Description should be lowercase, imperative mood ("add" not "added")
|
||||
- No period at the end
|
||||
- Keep under 72 characters total
|
||||
|
||||
### Examples
|
||||
```
|
||||
feat: add token usage tracking for AI providers
|
||||
fix: resolve null pointer in job executor
|
||||
refactor: extract common validation logic
|
||||
docs: update API endpoint documentation
|
||||
chore: upgrade sqlx to 0.7
|
||||
```
|
||||
|
||||
## Execution Steps
|
||||
|
||||
1. Run `git status` to see all changes
|
||||
2. Run `git diff` to understand the changes in detail
|
||||
3. Run `git log --oneline -5` to see recent commit style
|
||||
4. Stage ONLY the modified/relevant files: `git add <file1> <file2> ...`
|
||||
5. Create the commit with conventional format:
|
||||
```bash
|
||||
git commit -m "<type>: <description>
|
||||
|
||||
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
|
||||
```
|
||||
6. Run `git status` to verify the commit succeeded
|
||||
@@ -1,97 +0,0 @@
|
||||
---
|
||||
name: local-review
|
||||
description: Code review a pull request for bugs and CLAUDE.md compliance. MUST use when asked to review code.
|
||||
---
|
||||
|
||||
# Local Code Review Skill
|
||||
|
||||
Review a pull request for real bugs and CLAUDE.md compliance violations. This review targets HIGH SIGNAL issues only.
|
||||
|
||||
## Review Philosophy
|
||||
|
||||
- **Only flag issues you are certain about.** If you are not sure an issue is real, do not flag it. False positives erode trust and waste reviewer time.
|
||||
- Think like a senior engineer doing a final review — flag things that would cause incidents, not things that are merely imperfect.
|
||||
|
||||
## What to Flag
|
||||
|
||||
- Code that won't compile or parse (syntax errors, type errors, missing imports)
|
||||
- Code that will definitely produce wrong results regardless of inputs
|
||||
- Clear, unambiguous CLAUDE.md violations (quote the exact rule being violated)
|
||||
- Security issues in introduced code (injection, auth bypass, data exposure)
|
||||
- Incorrect logic that will fail in production
|
||||
|
||||
## What NOT to Flag
|
||||
|
||||
- Code style or quality concerns
|
||||
- Potential issues that depend on specific inputs or runtime state
|
||||
- Subjective suggestions or improvements
|
||||
- Pre-existing issues not introduced by this PR
|
||||
- Pedantic nitpicks a senior engineer wouldn't flag
|
||||
- Issues a linter or type checker will catch
|
||||
- General quality concerns unless explicitly prohibited in CLAUDE.md
|
||||
- Issues silenced via lint ignore comments
|
||||
|
||||
## Execution Steps
|
||||
|
||||
1. **Determine the PR scope**:
|
||||
- If an argument is provided, use it as the PR number or branch
|
||||
- Otherwise, detect from the current branch vs main
|
||||
- Run `gh pr view` if a PR exists, or use `git diff main...HEAD`
|
||||
|
||||
2. **Find relevant CLAUDE.md files**:
|
||||
- Read the root `CLAUDE.md`
|
||||
- Check for CLAUDE.md files in directories containing changed files
|
||||
|
||||
3. **Get the diff and metadata**:
|
||||
- `gh pr diff` or `git diff main...HEAD` for the full diff
|
||||
- `gh pr view` or `git log main..HEAD --oneline` for context
|
||||
|
||||
4. **Read changed files** where the diff alone is insufficient to understand context
|
||||
|
||||
5. **Review for**:
|
||||
- CLAUDE.md compliance — check each rule against the changed code
|
||||
- Bugs and logic errors — will this code work correctly?
|
||||
- Security issues — injection, auth, data exposure in new code
|
||||
|
||||
6. **Self-validate each finding**: Before reporting, ask yourself:
|
||||
- "Is this definitely a real issue, not a false positive?"
|
||||
- "Would a senior engineer flag this in review?"
|
||||
- If the answer to either is no, discard the finding
|
||||
|
||||
7. **Output findings** to the terminal (default) or post as PR comments (with `--comment` flag)
|
||||
|
||||
## Output Format
|
||||
|
||||
```
|
||||
## Code review
|
||||
|
||||
Found N issues:
|
||||
|
||||
1. <description> (<reason: CLAUDE.md adherence | bug | security>)
|
||||
<file_path:line_number>
|
||||
|
||||
2. <description> (<reason>)
|
||||
<file_path:line_number>
|
||||
```
|
||||
|
||||
If no issues are found:
|
||||
|
||||
```
|
||||
## Code review
|
||||
|
||||
No issues found. Checked for bugs and CLAUDE.md compliance.
|
||||
```
|
||||
|
||||
## Posting Comments (--comment flag)
|
||||
|
||||
If the user passes `--comment`, post findings as inline PR comments using:
|
||||
|
||||
```bash
|
||||
gh pr review --comment --body "<summary>"
|
||||
```
|
||||
|
||||
Or for inline comments on specific lines:
|
||||
|
||||
```bash
|
||||
gh api repos/{owner}/{repo}/pulls/{pr}/reviews -f body="<summary>" -f event="COMMENT" -f comments="[...]"
|
||||
```
|
||||
@@ -1,777 +0,0 @@
|
||||
# Skill: Adding Native Trigger Services
|
||||
|
||||
This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The native trigger system consists of:
|
||||
|
||||
1. **Database Layer** - PostgreSQL tables and enum types
|
||||
2. **Backend Rust Implementation** - Core trait, handlers, and service modules in the `windmill-native-triggers` crate
|
||||
3. **Frontend Svelte Components** - Configuration forms and UI components
|
||||
|
||||
### Key Files
|
||||
|
||||
| Component | Path |
|
||||
|-----------|------|
|
||||
| Core module with `External` trait | `backend/windmill-native-triggers/src/lib.rs` |
|
||||
| Generic CRUD handlers | `backend/windmill-native-triggers/src/handler.rs` |
|
||||
| Background sync logic | `backend/windmill-native-triggers/src/sync.rs` |
|
||||
| OAuth/workspace integration | `backend/windmill-native-triggers/src/workspace_integrations.rs` |
|
||||
| Re-export shim (windmill-api) | `backend/windmill-api/src/native_triggers/mod.rs` |
|
||||
| TriggerKind enum | `backend/windmill-common/src/triggers.rs` |
|
||||
| JobTriggerKind enum | `backend/windmill-common/src/jobs.rs` |
|
||||
| Frontend service registry | `frontend/src/lib/components/triggers/native/utils.ts` |
|
||||
| Frontend trigger utilities | `frontend/src/lib/components/triggers/utils.ts` |
|
||||
| Trigger badges (icons + counts) | `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte` |
|
||||
| Workspace integrations UI | `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte` |
|
||||
| OAuth config form component | `frontend/src/lib/components/workspaceSettings/OAuthClientConfig.svelte` |
|
||||
| OpenAPI spec | `backend/windmill-api/openapi.yaml` |
|
||||
| Reference: Nextcloud module | `backend/windmill-native-triggers/src/nextcloud/` |
|
||||
| Reference: Google module | `backend/windmill-native-triggers/src/google/` |
|
||||
|
||||
### Crate Structure
|
||||
|
||||
The native trigger code lives in the `windmill-native-triggers` crate (`backend/windmill-native-triggers/`). The `windmill-api` crate re-exports everything via a shim:
|
||||
|
||||
```rust
|
||||
// backend/windmill-api/src/native_triggers/mod.rs
|
||||
pub use windmill_native_triggers::*;
|
||||
```
|
||||
|
||||
All new service modules go in `backend/windmill-native-triggers/src/`.
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### The `External` Trait
|
||||
|
||||
Every native trigger service implements the `External` trait defined in `lib.rs`:
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait External: Send + Sync + 'static {
|
||||
// Associated types:
|
||||
type ServiceConfig: Debug + DeserializeOwned + Serialize + Send + Sync;
|
||||
type TriggerData: Debug + Serialize + Send + Sync;
|
||||
type OAuthData: DeserializeOwned + Serialize + Clone + Send + Sync;
|
||||
type CreateResponse: DeserializeOwned + Send + Sync;
|
||||
|
||||
// Constants:
|
||||
const SUPPORT_WEBHOOK: bool;
|
||||
const SERVICE_NAME: ServiceName;
|
||||
const DISPLAY_NAME: &'static str;
|
||||
const TOKEN_ENDPOINT: &'static str;
|
||||
const REFRESH_ENDPOINT: &'static str;
|
||||
const AUTH_ENDPOINT: &'static str;
|
||||
|
||||
// Required methods:
|
||||
async fn create(&self, w_id, oauth_data, webhook_token, data, db, tx) -> Result<Self::CreateResponse>;
|
||||
async fn update(&self, w_id, oauth_data, external_id, webhook_token, data, db, tx) -> Result<serde_json::Value>;
|
||||
async fn get(&self, w_id, oauth_data, external_id, db, tx) -> Result<Self::TriggerData>;
|
||||
async fn delete(&self, w_id, oauth_data, external_id, db, tx) -> Result<()>;
|
||||
async fn exists(&self, w_id, oauth_data, external_id, db, tx) -> Result<bool>;
|
||||
async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors);
|
||||
fn external_id_and_metadata_from_response(&self, resp) -> (String, Option<serde_json::Value>);
|
||||
|
||||
// Methods with defaults:
|
||||
async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned>;
|
||||
fn service_config_from_create_response(&self, data, resp) -> Option<serde_json::Value>;
|
||||
fn additional_routes(&self) -> axum::Router;
|
||||
async fn http_client_request<T, B>(&self, url, method, workspace_id, tx, db, headers, body) -> Result<T>;
|
||||
}
|
||||
```
|
||||
|
||||
Key design points:
|
||||
- **`update()` returns `serde_json::Value`** - the resolved service_config to store. Each service is responsible for building the final config.
|
||||
- **`maintain_triggers()`** - periodic background maintenance. Each service implements its own strategy (Nextcloud: reconcile with external state; Google: renew expiring channels).
|
||||
- **No `list_all()` in the trait** - services that need it (Nextcloud) implement it privately; services that don't (Google) use different maintenance strategies.
|
||||
- **No `get_external_id_from_trigger_data()` or `extract_service_config_from_trigger_data()`** - removed in favor of the `maintain_triggers` pattern.
|
||||
|
||||
### Create Lifecycle: Two Paths
|
||||
|
||||
The `create_native_trigger` handler in `handler.rs` supports two creation flows, controlled by `service_config_from_create_response()`:
|
||||
|
||||
**Path A: Short (Google pattern)** - `service_config_from_create_response()` returns `Some(config)`:
|
||||
1. `create()` registers on external service
|
||||
2. `external_id_and_metadata_from_response()` extracts the ID
|
||||
3. `service_config_from_create_response()` builds the config directly from input data + response metadata
|
||||
4. Stores trigger in DB -- done, no extra round-trip
|
||||
|
||||
Use this when the external_id is known before the create call (e.g., Google generates the channel_id as a UUID upfront and includes it in the webhook URL).
|
||||
|
||||
**Path B: Long (Nextcloud pattern)** - `service_config_from_create_response()` returns `None` (default):
|
||||
1. `create()` registers on external service (webhook URL has no external_id yet)
|
||||
2. `external_id_and_metadata_from_response()` extracts the ID
|
||||
3. `update()` is called to fix the webhook URL with the now-known external_id
|
||||
4. `update()` returns the resolved service_config
|
||||
5. Stores trigger in DB
|
||||
|
||||
Use this when the external_id is assigned by the remote service and the webhook URL needs to be corrected after creation.
|
||||
|
||||
### OAuth Token Storage (Three-Table Pattern)
|
||||
|
||||
OAuth tokens are stored across three tables, NOT in `workspace_integrations.oauth_data` directly:
|
||||
|
||||
| Table | What's Stored |
|
||||
|-------|---------------|
|
||||
| `workspace_integrations` | `oauth_data` JSON with `base_url`, `client_id`, `client_secret`, `instance_shared` flag; `resource_path` pointing to the variable |
|
||||
| `variable` | Encrypted `access_token` (at the path stored in `resource_path`), linked to `account` via `account` column |
|
||||
| `account` | `refresh_token`, keyed by `workspace_id` + `client` (service name) + `is_workspace_integration = true` |
|
||||
|
||||
The `decrypt_oauth_data()` function in `lib.rs` assembles these into a unified struct:
|
||||
```rust
|
||||
pub struct OAuthConfig {
|
||||
pub base_url: String,
|
||||
pub access_token: String, // decrypted from variable
|
||||
pub refresh_token: Option<String>, // from account table
|
||||
pub client_id: String, // from oauth_data or instance settings
|
||||
pub client_secret: String, // from oauth_data or instance settings
|
||||
}
|
||||
```
|
||||
|
||||
Instance-level sharing: when `oauth_data.instance_shared == true`, `client_id` and `client_secret` are read from global settings instead of workspace_integrations.
|
||||
|
||||
### URL Resolution
|
||||
|
||||
The `resolve_endpoint()` helper handles both absolute and relative OAuth URLs:
|
||||
|
||||
```rust
|
||||
pub fn resolve_endpoint(base_url: &str, endpoint: &str) -> String {
|
||||
if endpoint.starts_with("http://") || endpoint.starts_with("https://") {
|
||||
endpoint.to_string() // Google: absolute URLs
|
||||
} else {
|
||||
format!("{}{}", base_url, endpoint) // Nextcloud: relative paths
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### ServiceName Methods
|
||||
|
||||
`ServiceName` is the central registry enum. Each variant must implement these match arms:
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `as_str()` | Lowercase identifier (e.g., `"google"`) |
|
||||
| `as_trigger_kind()` | Maps to `TriggerKind` enum |
|
||||
| `as_job_trigger_kind()` | Maps to `JobTriggerKind` enum |
|
||||
| `token_endpoint()` | OAuth token endpoint (relative or absolute) |
|
||||
| `auth_endpoint()` | OAuth authorization endpoint |
|
||||
| `oauth_scopes()` | Space-separated OAuth scopes |
|
||||
| `resource_type()` | Resource type for token storage (e.g., `"gworkspace"`) |
|
||||
| `extra_auth_params()` | Extra OAuth params (e.g., Google needs `access_type=offline`, `prompt=consent`) |
|
||||
| `integration_service()` | Maps to the workspace integration service (usually `*self`) |
|
||||
| `TryFrom<String>` | Parse from string |
|
||||
| `Display` | Delegates to `as_str()` |
|
||||
|
||||
---
|
||||
|
||||
## Step-by-Step Implementation Guide
|
||||
|
||||
### Step 1: Database Migration
|
||||
|
||||
Create a new migration file: `backend/migrations/YYYYMMDDHHMMSS_newservice_trigger.up.sql`
|
||||
|
||||
```sql
|
||||
-- Add the service to the native_trigger_service enum
|
||||
ALTER TYPE native_trigger_service ADD VALUE IF NOT EXISTS 'newservice';
|
||||
|
||||
-- Add to TRIGGER_KIND enum (used for trigger tracking)
|
||||
ALTER TYPE TRIGGER_KIND ADD VALUE IF NOT EXISTS 'newservice';
|
||||
|
||||
-- Add to job_trigger_kind enum (used for job tracking)
|
||||
ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'newservice';
|
||||
```
|
||||
|
||||
Also create the corresponding down migration.
|
||||
|
||||
### Step 2: Update windmill-common Enums
|
||||
|
||||
#### `backend/windmill-common/src/triggers.rs`
|
||||
|
||||
Add variant to `TriggerKind` enum, and update `to_key()` and `fmt()` implementations.
|
||||
|
||||
#### `backend/windmill-common/src/jobs.rs`
|
||||
|
||||
Add variant to `JobTriggerKind` enum and update the `Display` implementation.
|
||||
|
||||
### Step 3: Backend Service Module
|
||||
|
||||
Create a new directory: `backend/windmill-native-triggers/src/newservice/`
|
||||
|
||||
#### `mod.rs` - Type Definitions
|
||||
|
||||
```rust
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
pub mod external;
|
||||
// pub mod routes; // Only if you need additional service-specific routes
|
||||
|
||||
/// OAuth data deserialized from the three-table pattern.
|
||||
/// The actual structure is built by decrypt_oauth_data() from variable + account + workspace_integrations.
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
pub struct NewServiceOAuthData {
|
||||
pub base_url: String, // from workspace_integrations.oauth_data
|
||||
pub access_token: String, // decrypted from variable table
|
||||
pub refresh_token: Option<String>, // from account table
|
||||
// Note: client_id and client_secret are in OAuthConfig, not here
|
||||
// unless the service needs them at runtime for API calls
|
||||
}
|
||||
|
||||
/// Configuration provided by user when creating/updating a trigger.
|
||||
/// Stored as JSON in native_trigger.service_config.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct NewServiceConfig {
|
||||
// Service-specific configuration fields
|
||||
pub folder_path: String,
|
||||
pub file_filter: Option<String>,
|
||||
}
|
||||
|
||||
/// Data retrieved from the external service about a trigger.
|
||||
/// Returned by the get() method and shown in the UI.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct NewServiceTriggerData {
|
||||
pub folder_path: String,
|
||||
pub file_filter: Option<String>,
|
||||
// Fields that shouldn't affect service_config comparison should use #[serde(skip_serializing)]
|
||||
}
|
||||
|
||||
/// Response from external service when creating a trigger/webhook.
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct CreateTriggerResponse {
|
||||
pub id: String,
|
||||
}
|
||||
|
||||
/// Handler struct (stateless, used for routing)
|
||||
#[derive(Copy, Clone)]
|
||||
pub struct NewService;
|
||||
```
|
||||
|
||||
#### `external.rs` - External Trait Implementation
|
||||
|
||||
```rust
|
||||
use async_trait::async_trait;
|
||||
use reqwest::Method;
|
||||
use sqlx::PgConnection;
|
||||
use std::collections::HashMap;
|
||||
use windmill_common::{
|
||||
error::{Error, Result},
|
||||
BASE_URL, DB,
|
||||
};
|
||||
|
||||
use crate::{
|
||||
generate_webhook_service_url, External, NativeTrigger, NativeTriggerData, ServiceName,
|
||||
sync::{SyncError, TriggerSyncInfo},
|
||||
};
|
||||
use super::{NewService, NewServiceConfig, NewServiceOAuthData, NewServiceTriggerData, CreateTriggerResponse};
|
||||
|
||||
#[async_trait]
|
||||
impl External for NewService {
|
||||
type ServiceConfig = NewServiceConfig;
|
||||
type TriggerData = NewServiceTriggerData;
|
||||
type OAuthData = NewServiceOAuthData;
|
||||
type CreateResponse = CreateTriggerResponse;
|
||||
|
||||
const SERVICE_NAME: ServiceName = ServiceName::NewService;
|
||||
const DISPLAY_NAME: &'static str = "New Service";
|
||||
const SUPPORT_WEBHOOK: bool = true;
|
||||
const TOKEN_ENDPOINT: &'static str = "/oauth/token";
|
||||
const REFRESH_ENDPOINT: &'static str = "/oauth/token";
|
||||
const AUTH_ENDPOINT: &'static str = "/oauth/authorize";
|
||||
|
||||
async fn create(
|
||||
&self,
|
||||
w_id: &str,
|
||||
oauth_data: &Self::OAuthData,
|
||||
webhook_token: &str,
|
||||
data: &NativeTriggerData<Self::ServiceConfig>,
|
||||
db: &DB,
|
||||
tx: &mut PgConnection,
|
||||
) -> Result<Self::CreateResponse> {
|
||||
let base_url = &*BASE_URL.read().await;
|
||||
|
||||
// external_id is None during create (we get it from the response)
|
||||
let webhook_url = generate_webhook_service_url(
|
||||
base_url, w_id, &data.script_path, data.is_flow,
|
||||
None, Self::SERVICE_NAME, webhook_token,
|
||||
);
|
||||
|
||||
let url = format!("{}/api/webhooks/create", oauth_data.base_url);
|
||||
let payload = serde_json::json!({
|
||||
"callback_url": webhook_url,
|
||||
"folder_path": data.service_config.folder_path,
|
||||
});
|
||||
|
||||
let response: CreateTriggerResponse = self
|
||||
.http_client_request(&url, Method::POST, w_id, tx, db, None, Some(&payload))
|
||||
.await?;
|
||||
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
/// Update returns the resolved service_config as JSON.
|
||||
/// For services using the update+get pattern, call self.get() and serialize.
|
||||
async fn update(
|
||||
&self,
|
||||
w_id: &str,
|
||||
oauth_data: &Self::OAuthData,
|
||||
external_id: &str,
|
||||
webhook_token: &str,
|
||||
data: &NativeTriggerData<Self::ServiceConfig>,
|
||||
db: &DB,
|
||||
tx: &mut PgConnection,
|
||||
) -> Result<serde_json::Value> {
|
||||
let base_url = &*BASE_URL.read().await;
|
||||
|
||||
let webhook_url = generate_webhook_service_url(
|
||||
base_url, w_id, &data.script_path, data.is_flow,
|
||||
Some(external_id), Self::SERVICE_NAME, webhook_token,
|
||||
);
|
||||
|
||||
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
|
||||
let payload = serde_json::json!({
|
||||
"callback_url": webhook_url,
|
||||
"folder_path": data.service_config.folder_path,
|
||||
});
|
||||
|
||||
let _: serde_json::Value = self
|
||||
.http_client_request(&url, Method::PUT, w_id, tx, db, None, Some(&payload))
|
||||
.await?;
|
||||
|
||||
// Fetch back the updated state to get the resolved config
|
||||
let trigger_data = self.get(w_id, oauth_data, external_id, db, tx).await?;
|
||||
serde_json::to_value(&trigger_data)
|
||||
.map_err(|e| Error::InternalErr(format!("Failed to serialize trigger data: {}", e)))
|
||||
}
|
||||
|
||||
async fn get(
|
||||
&self,
|
||||
w_id: &str,
|
||||
oauth_data: &Self::OAuthData,
|
||||
external_id: &str,
|
||||
db: &DB,
|
||||
tx: &mut PgConnection,
|
||||
) -> Result<Self::TriggerData> {
|
||||
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
|
||||
self.http_client_request::<_, ()>(&url, Method::GET, w_id, tx, db, None, None).await
|
||||
}
|
||||
|
||||
async fn delete(
|
||||
&self,
|
||||
w_id: &str,
|
||||
oauth_data: &Self::OAuthData,
|
||||
external_id: &str,
|
||||
db: &DB,
|
||||
tx: &mut PgConnection,
|
||||
) -> Result<()> {
|
||||
let url = format!("{}/api/webhooks/{}", oauth_data.base_url, external_id);
|
||||
let _: serde_json::Value = self
|
||||
.http_client_request::<_, ()>(&url, Method::DELETE, w_id, tx, db, None, None)
|
||||
.await
|
||||
.or_else(|e| match &e {
|
||||
Error::InternalErr(msg) if msg.contains("404") => Ok(serde_json::Value::Null),
|
||||
_ => Err(e),
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn exists(
|
||||
&self,
|
||||
w_id: &str,
|
||||
oauth_data: &Self::OAuthData,
|
||||
external_id: &str,
|
||||
db: &DB,
|
||||
tx: &mut PgConnection,
|
||||
) -> Result<bool> {
|
||||
match self.get(w_id, oauth_data, external_id, db, tx).await {
|
||||
Ok(_) => Ok(true),
|
||||
Err(Error::NotFound(_)) => Ok(false),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
/// Background maintenance. Choose the right pattern for your service:
|
||||
/// - For services with queryable external state: use reconcile_with_external_state()
|
||||
/// - For channel-based services with expiration: implement renewal logic
|
||||
async fn maintain_triggers(
|
||||
&self,
|
||||
db: &DB,
|
||||
workspace_id: &str,
|
||||
triggers: &[NativeTrigger],
|
||||
oauth_data: &Self::OAuthData,
|
||||
synced: &mut Vec<TriggerSyncInfo>,
|
||||
errors: &mut Vec<SyncError>,
|
||||
) {
|
||||
// Option A: Reconcile with external state (Nextcloud pattern)
|
||||
// Fetch all triggers from external service and compare with DB
|
||||
let external_triggers = match self.list_all(workspace_id, oauth_data, db).await {
|
||||
Ok(triggers) => triggers,
|
||||
Err(e) => {
|
||||
errors.push(SyncError {
|
||||
resource_path: format!("workspace:{}", workspace_id),
|
||||
error_message: format!("Failed to list triggers: {}", e),
|
||||
error_type: "api_error".to_string(),
|
||||
});
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
// Convert to (external_id, config_json) pairs
|
||||
let external_pairs: Vec<(String, serde_json::Value)> = external_triggers
|
||||
.into_iter()
|
||||
.map(|t| (t.id.clone(), serde_json::to_value(&t).unwrap_or_default()))
|
||||
.collect();
|
||||
|
||||
crate::sync::reconcile_with_external_state(
|
||||
db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
|
||||
).await;
|
||||
}
|
||||
|
||||
fn external_id_and_metadata_from_response(
|
||||
&self,
|
||||
resp: &Self::CreateResponse,
|
||||
) -> (String, Option<serde_json::Value>) {
|
||||
(resp.id.clone(), None)
|
||||
}
|
||||
|
||||
// service_config_from_create_response: NOT overridden (returns None).
|
||||
// This means the handler uses the update+get pattern after create.
|
||||
// Override and return Some(...) to skip the update+get cycle (Google pattern).
|
||||
}
|
||||
|
||||
impl NewService {
|
||||
/// Private helper to list all triggers from the external service.
|
||||
async fn list_all(
|
||||
&self,
|
||||
w_id: &str,
|
||||
oauth_data: &<Self as External>::OAuthData,
|
||||
db: &DB,
|
||||
) -> Result<Vec<<Self as External>::TriggerData>> {
|
||||
// Implementation depends on the external service's API
|
||||
todo!()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 4: Update lib.rs Registry
|
||||
|
||||
In `backend/windmill-native-triggers/src/lib.rs`:
|
||||
|
||||
```rust
|
||||
// Service modules - add new services here:
|
||||
#[cfg(feature = "native_trigger")]
|
||||
pub mod newservice; // <-- Add this
|
||||
|
||||
// ServiceName enum - add variant:
|
||||
pub enum ServiceName {
|
||||
Nextcloud,
|
||||
Google,
|
||||
NewService, // <-- Add this
|
||||
}
|
||||
|
||||
// Then add match arms in ALL ServiceName methods:
|
||||
// as_str(), as_trigger_kind(), as_job_trigger_kind(), token_endpoint(),
|
||||
// auth_endpoint(), oauth_scopes(), resource_type(), extra_auth_params(),
|
||||
// integration_service(), TryFrom<String>, Display
|
||||
```
|
||||
|
||||
### Step 5: Update handler.rs Routes
|
||||
|
||||
In `backend/windmill-native-triggers/src/handler.rs`:
|
||||
|
||||
```rust
|
||||
pub fn generate_native_trigger_routers() -> Router {
|
||||
// ...
|
||||
#[cfg(feature = "native_trigger")]
|
||||
{
|
||||
use crate::newservice::NewService;
|
||||
return router
|
||||
.nest("/nextcloud", service_routes(NextCloud))
|
||||
.nest("/google", service_routes(Google))
|
||||
.nest("/newservice", service_routes(NewService)); // <-- Add this
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Step 6: Update sync.rs
|
||||
|
||||
In `backend/windmill-native-triggers/src/sync.rs`:
|
||||
|
||||
```rust
|
||||
pub async fn sync_all_triggers(db: &DB) -> Result<BackgroundSyncResult> {
|
||||
// ...
|
||||
#[cfg(feature = "native_trigger")]
|
||||
{
|
||||
use crate::newservice::NewService;
|
||||
|
||||
// ... existing service syncs ...
|
||||
|
||||
// New service sync
|
||||
let (service_name, result) = sync_service_triggers(db, NewService).await;
|
||||
total_synced += result.synced_triggers.len();
|
||||
total_errors += result.errors.len();
|
||||
service_results.insert(service_name, result);
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Step 7: Frontend Service Registry
|
||||
|
||||
In `frontend/src/lib/components/triggers/native/utils.ts`:
|
||||
|
||||
Add to `NATIVE_TRIGGER_SERVICES`, `getTriggerIconName()`, and `getServiceIcon()`.
|
||||
|
||||
### Step 8: Frontend Trigger Form Component
|
||||
|
||||
Create: `frontend/src/lib/components/triggers/native/services/newservice/NewServiceTriggerForm.svelte`
|
||||
|
||||
### Step 9: Frontend Icon Component
|
||||
|
||||
Create: `frontend/src/lib/components/icons/NewServiceIcon.svelte`
|
||||
|
||||
### Step 10: Update NativeTriggerEditor
|
||||
|
||||
Check `frontend/src/lib/components/triggers/native/NativeTriggerEditor.svelte` to ensure it dynamically loads form components based on service name.
|
||||
|
||||
### Step 11: Workspace Integration UI
|
||||
|
||||
Add your service to the `supportedServices` map in `frontend/src/lib/components/workspaceSettings/WorkspaceIntegrations.svelte`:
|
||||
|
||||
```typescript
|
||||
const supportedServices: Record<string, ServiceConfig> = {
|
||||
// ... existing services ...
|
||||
newservice: {
|
||||
name: 'newservice',
|
||||
displayName: 'New Service',
|
||||
description: 'Connect to New Service for triggers',
|
||||
icon: NewServiceIcon,
|
||||
docsUrl: 'https://www.windmill.dev/docs/integrations/newservice',
|
||||
requiresBaseUrl: false, // false for cloud services, true for self-hosted
|
||||
setupInstructions: [
|
||||
'Step 1: Create an OAuth app on the service',
|
||||
'Step 2: Configure the redirect URI shown below',
|
||||
'Step 3: Enter the client credentials below'
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 12: Update `frontend/src/lib/components/triggers/utils.ts`
|
||||
|
||||
Update ALL of these maps/functions:
|
||||
1. `triggerIconMap` - import and add icon
|
||||
2. `triggerDisplayNamesMap` - add display name
|
||||
3. `triggerTypeOrder` in `sortTriggers()` - add type
|
||||
4. `getLightConfig()` - add case for your service
|
||||
5. `getTriggerLabel()` - add case for your service
|
||||
6. `jobTriggerKinds` - add to array
|
||||
7. `countPropertyMap` - add count property
|
||||
8. `triggerSaveFunctions` - add save function
|
||||
|
||||
### Step 13: Update TriggersBadge Component
|
||||
|
||||
In `frontend/src/lib/components/graph/renderers/triggers/TriggersBadge.svelte`:
|
||||
|
||||
1. Import the icon
|
||||
2. Add to `baseConfig` with `countKey` (the dynamic `availableNativeServices` loop does NOT set `countKey`)
|
||||
3. Add to the `allTypes` array
|
||||
|
||||
### Step 14: Update TriggersWrapper.svelte
|
||||
|
||||
In `frontend/src/lib/components/triggers/TriggersWrapper.svelte`:
|
||||
|
||||
Add a `{:else if selectedTrigger.type === 'yourservice'}` case that renders `<NativeTriggersPanel service="yourservice" ...>` with the same props pattern as the existing native trigger cases (e.g., `nextcloud`).
|
||||
|
||||
### Step 15: Update AddTriggersButton.svelte
|
||||
|
||||
In `frontend/src/lib/components/triggers/AddTriggersButton.svelte`:
|
||||
|
||||
1. Add `yourserviceAvailable` state variable
|
||||
2. Add `setYourserviceState()` async function using `isServiceAvailable('yourservice', $workspaceStore!)`
|
||||
3. Call it at module level
|
||||
4. Add a dropdown entry to `addTriggerItems` with `hidden: !yourserviceAvailable`
|
||||
|
||||
### Step 16: Update TriggersEditor.svelte Delete Handling
|
||||
|
||||
In `frontend/src/lib/components/triggers/TriggersEditor.svelte`:
|
||||
|
||||
Add your service to the `nativeTriggerServices` map in `deleteDeployedTrigger()`. Native triggers use `NativeTriggerService.deleteNativeTrigger({ workspace, serviceName, externalId })` instead of the standard `path`-based delete.
|
||||
|
||||
### Step 17: Update OpenAPI Spec and Regenerate Types
|
||||
|
||||
Add to `JobTriggerKind` enum in `backend/windmill-api/openapi.yaml`, then:
|
||||
|
||||
```bash
|
||||
cd frontend && npm run generate-backend-client
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Special Patterns
|
||||
|
||||
### Unified Service with `trigger_type` (Google Pattern)
|
||||
|
||||
When a single service handles multiple trigger types (e.g., Google Drive + Calendar share OAuth and API patterns), use a single `ServiceName` variant with a discriminator field:
|
||||
|
||||
```rust
|
||||
pub enum GoogleTriggerType { Drive, Calendar }
|
||||
|
||||
pub struct GoogleServiceConfig {
|
||||
pub trigger_type: GoogleTriggerType,
|
||||
// Drive-specific fields (only used when trigger_type = Drive)
|
||||
pub resource_id: Option<String>,
|
||||
pub resource_name: Option<String>,
|
||||
// Calendar-specific fields (only used when trigger_type = Calendar)
|
||||
pub calendar_id: Option<String>,
|
||||
pub calendar_name: Option<String>,
|
||||
// Metadata set after creation
|
||||
pub google_resource_id: Option<String>,
|
||||
pub expiration: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
Branch in trait methods based on `trigger_type`. Frontend uses a `ToggleButtonGroup` to switch between types. This keeps the codebase simpler (one service, one OAuth flow, one set of routes).
|
||||
|
||||
See `backend/windmill-native-triggers/src/google/` for the reference implementation.
|
||||
|
||||
### Skipping update+get After Create (Google Pattern)
|
||||
|
||||
Override `service_config_from_create_response()` to return `Some(config)` when the external_id is known before the create call:
|
||||
|
||||
```rust
|
||||
fn service_config_from_create_response(
|
||||
&self,
|
||||
data: &NativeTriggerData<Self::ServiceConfig>,
|
||||
resp: &Self::CreateResponse,
|
||||
) -> Option<serde_json::Value> {
|
||||
// Clone input config, add metadata from response
|
||||
let mut config = data.service_config.clone();
|
||||
config.google_resource_id = Some(resp.resource_id.clone());
|
||||
config.expiration = Some(resp.expiration.clone());
|
||||
Some(serde_json::to_value(&config).unwrap())
|
||||
}
|
||||
```
|
||||
|
||||
### Services with Absolute OAuth Endpoints (Google)
|
||||
|
||||
Unlike self-hosted services where OAuth endpoints are relative paths appended to `base_url`, services like Google have absolute URLs:
|
||||
|
||||
```rust
|
||||
// Nextcloud: relative paths
|
||||
ServiceName::Nextcloud => "/apps/oauth2/api/v1/token",
|
||||
// Google: absolute URLs
|
||||
ServiceName::Google => "https://oauth2.googleapis.com/token",
|
||||
```
|
||||
|
||||
The `resolve_endpoint()` function handles both. For services with absolute endpoints:
|
||||
- `base_url` can be empty
|
||||
- `requiresBaseUrl: false` in the frontend workspace integration config
|
||||
- Add `extra_auth_params()` if needed (Google requires `access_type=offline` and `prompt=consent`)
|
||||
|
||||
### Channel-Based Push Notifications with Renewal (Google Pattern)
|
||||
|
||||
For services using expiring watch channels instead of persistent webhooks:
|
||||
|
||||
1. Store expiration in `service_config` (as part of `ServiceConfig`)
|
||||
2. In `maintain_triggers()`, implement renewal logic instead of using `reconcile_with_external_state()`:
|
||||
```rust
|
||||
async fn maintain_triggers(&self, db, workspace_id, triggers, oauth_data, synced, errors) {
|
||||
for trigger in triggers {
|
||||
if should_renew_channel(trigger) {
|
||||
self.renew_channel(db, trigger, oauth_data).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
3. Renewal: best-effort stop old channel, create new one with same external_id, update service_config with new expiration
|
||||
4. Google example: Drive channels expire in 24h (renew when <1h left), Calendar channels expire in 7 days (renew when <1 day left)
|
||||
|
||||
### reconcile_with_external_state (Nextcloud Pattern)
|
||||
|
||||
The reusable function in `sync.rs` compares external triggers with DB state:
|
||||
- Triggers missing externally: sets error "Trigger no longer exists on external service"
|
||||
- Triggers present externally: clears errors, updates service_config if it differs
|
||||
|
||||
Usage in `maintain_triggers()`:
|
||||
```rust
|
||||
let external_pairs: Vec<(String, serde_json::Value)> = /* fetch from external */;
|
||||
crate::sync::reconcile_with_external_state(
|
||||
db, workspace_id, Self::SERVICE_NAME, triggers, &external_pairs, synced, errors,
|
||||
).await;
|
||||
```
|
||||
|
||||
### Webhook Payload Processing
|
||||
|
||||
Override `prepare_webhook()` to parse service-specific payloads into script/flow args:
|
||||
|
||||
```rust
|
||||
async fn prepare_webhook(&self, db, w_id, headers, body, script_path, is_flow) -> Result<PushArgsOwned> {
|
||||
let mut args = HashMap::new();
|
||||
args.insert("event_type".to_string(), Box::new(headers.get("x-event-type").cloned()) as _);
|
||||
args.insert("payload".to_string(), Box::new(serde_json::from_str::<serde_json::Value>(&body)?) as _);
|
||||
Ok(PushArgsOwned { extra: None, args })
|
||||
}
|
||||
```
|
||||
|
||||
Then register in `prepare_native_trigger_args()` in `lib.rs`:
|
||||
```rust
|
||||
pub async fn prepare_native_trigger_args(service_name, db, w_id, headers, body) -> Result<Option<PushArgsOwned>> {
|
||||
match service_name {
|
||||
ServiceName::Google => { /* ... */ Ok(Some(args)) }
|
||||
ServiceName::NewService => { /* ... */ Ok(Some(args)) }
|
||||
ServiceName::Nextcloud => Ok(None), // Uses default body parsing
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Instance-Level OAuth Credentials
|
||||
|
||||
When `workspace_integrations.oauth_data.instance_shared == true`, `decrypt_oauth_data()` reads `client_id` and `client_secret` from instance-level global settings instead of workspace-level. This allows admins to share OAuth app credentials across workspaces.
|
||||
|
||||
The frontend handles this via the `generate_instance_connect_url` endpoint in `workspace_integrations.rs`.
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
- [ ] Database migration runs successfully
|
||||
- [ ] `cargo check -p windmill-native-triggers --features native_trigger` passes
|
||||
- [ ] `npx svelte-check --threshold error` passes (in frontend/)
|
||||
- [ ] Service appears in workspace integrations list
|
||||
- [ ] OAuth flow completes successfully
|
||||
- [ ] Can create a new trigger
|
||||
- [ ] Can view trigger details
|
||||
- [ ] Can update trigger configuration
|
||||
- [ ] Can delete trigger
|
||||
- [ ] Webhook receives and processes payloads
|
||||
- [ ] Background sync works correctly (reconciliation or channel renewal)
|
||||
- [ ] Error handling works (expired tokens, service unavailable)
|
||||
|
||||
---
|
||||
|
||||
## Reference Implementations
|
||||
|
||||
### Nextcloud (Self-Hosted, Update+Get Pattern)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `nextcloud/mod.rs` | Types: NextCloudOAuthData, NextcloudServiceConfig, NextCloudTriggerData |
|
||||
| `nextcloud/external.rs` | External trait: uses update+get pattern, reconcile_with_external_state for sync |
|
||||
| `nextcloud/routes.rs` | Additional route: `GET /events` |
|
||||
|
||||
Key patterns: relative OAuth endpoints, base_url required, list_all + reconcile for sync, update returns JSON from get().
|
||||
|
||||
### Google (Cloud, Unified Service, Short Create)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `google/mod.rs` | Types: GoogleServiceConfig with trigger_type discriminator, GoogleTriggerType enum |
|
||||
| `google/external.rs` | External trait: overrides service_config_from_create_response, channel renewal for sync |
|
||||
| `google/routes.rs` | Additional routes: `GET /calendars`, `GET /drive/files`, `GET /drive/shared_drives` |
|
||||
|
||||
Key patterns: absolute OAuth endpoints, empty base_url, trigger_type for Drive/Calendar, expiring watch channels with renewal, service_config_from_create_response skips update+get, get() reconstructs data from stored service_config (no external "get channel" API).
|
||||
@@ -1,109 +0,0 @@
|
||||
---
|
||||
name: pr
|
||||
description: Open a draft pull request on GitHub. MUST use when you want to create/open a PR.
|
||||
---
|
||||
|
||||
# Pull Request Skill
|
||||
|
||||
Create a draft pull request with a clear title and explicit description of changes.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. **Analyze branch changes**: Understand all commits since diverging from main
|
||||
2. **Push to remote**: Ensure all commits are pushed
|
||||
3. **Create draft PR**: Always open as draft for review before merging
|
||||
|
||||
## PR Title Format
|
||||
|
||||
Follow conventional commit format for the PR title:
|
||||
```
|
||||
<type>: <description>
|
||||
```
|
||||
|
||||
### Types
|
||||
- `feat`: New feature or capability
|
||||
- `fix`: Bug fix
|
||||
- `refactor`: Code restructuring
|
||||
- `docs`: Documentation changes
|
||||
- `chore`: Maintenance tasks
|
||||
- `perf`: Performance improvements
|
||||
|
||||
### Title Rules
|
||||
- Keep under 70 characters
|
||||
- Use lowercase, imperative mood
|
||||
- No period at the end
|
||||
- If `*_ee.rs` files were modified, prefix with `[ee]`: `[ee] <type>: <description>`
|
||||
|
||||
## PR Body Format
|
||||
|
||||
The body MUST be explicit about what changed. Structure:
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
<Clear description of what this PR does and why>
|
||||
|
||||
## Changes
|
||||
- <Specific change 1>
|
||||
- <Specific change 2>
|
||||
- <Specific change 3>
|
||||
|
||||
## Test plan
|
||||
- [ ] <How to verify change 1>
|
||||
- [ ] <How to verify change 2>
|
||||
|
||||
---
|
||||
Generated with [Claude Code](https://claude.com/claude-code)
|
||||
```
|
||||
|
||||
## Execution Steps
|
||||
|
||||
1. Run `git status` to check for uncommitted changes
|
||||
2. Run `git log main..HEAD --oneline` to see all commits in this branch
|
||||
3. Run `git diff main...HEAD` to see the full diff against main
|
||||
4. Check if remote branch exists and is up to date:
|
||||
```bash
|
||||
git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream"
|
||||
```
|
||||
5. Push to remote if needed: `git push -u origin HEAD`
|
||||
6. Create draft PR using gh CLI:
|
||||
```bash
|
||||
gh pr create --draft --title "<type>: <description>" --body "$(cat <<'EOF'
|
||||
## Summary
|
||||
<description>
|
||||
|
||||
## Changes
|
||||
- <change 1>
|
||||
- <change 2>
|
||||
|
||||
## Test plan
|
||||
- [ ] <test 1>
|
||||
- [ ] <test 2>
|
||||
|
||||
---
|
||||
Generated with [Claude Code](https://claude.com/claude-code)
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
7. Return the PR URL to the user
|
||||
|
||||
## EE Companion PR (when `*_ee.rs` files were modified)
|
||||
|
||||
The `*_ee.rs` files in the windmill repo are **symlinks** to `windmill-ee-private` — changes won't appear in `git diff` of the windmill repo. Instead, check the EE repo for uncommitted or unpushed changes.
|
||||
|
||||
Follow the full EE PR workflow in `docs/enterprise.md`. The key PR-specific details:
|
||||
|
||||
1. Find the EE repo/worktree: see "Finding the EE Repo" in `docs/enterprise.md`
|
||||
2. Check for changes: `git -C <ee-path> status --short`
|
||||
- If there are no changes in the EE repo, skip this entire section
|
||||
3. Follow steps 1–5 from the "EE PR Workflow" in `docs/enterprise.md`
|
||||
4. Create the companion PR (title does NOT get the `[ee]` prefix):
|
||||
```bash
|
||||
gh pr create --draft --repo windmill-labs/windmill-ee-private --title "<type>: <description>" --body "$(cat <<'EOF'
|
||||
Companion PR for windmill-labs/windmill#<PR_NUMBER>
|
||||
|
||||
---
|
||||
Generated with [Claude Code](https://claude.com/claude-code)
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
5. Commit `ee-repo-ref.txt` and push the updated windmill branch
|
||||
@@ -1,38 +0,0 @@
|
||||
---
|
||||
name: refine
|
||||
description: End-of-session reflection. Reviews friction encountered during the session and proposes updates to docs/ to capture lessons learned.
|
||||
---
|
||||
|
||||
# Refine Skill
|
||||
|
||||
Reflect on the current session and update documentation with lessons learned.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. **Identify friction**: Review what happened in this session:
|
||||
- Run `git diff main...HEAD --stat` to see what files were touched
|
||||
- Think about: what was slow, what failed, what required multiple attempts, what information was missing or hard to find
|
||||
|
||||
2. **Read current docs**: Read the docs that were relevant to this session:
|
||||
- `docs/validation.md`
|
||||
- `docs/enterprise.md`
|
||||
- `docs/autonomous-mode.md`
|
||||
- Any skills that were invoked
|
||||
|
||||
3. **Propose updates**: For each piece of friction, decide if it warrants a doc update:
|
||||
- **Missing knowledge**: Information you had to discover that should be documented
|
||||
- **Wrong guidance**: Instructions that led you astray
|
||||
- **Missing validation rule**: A check that should be in the validation matrix
|
||||
- **New pattern**: A codebase pattern worth capturing for next time
|
||||
|
||||
4. **Apply updates**: Edit the relevant `docs/` files. Keep changes minimal and specific — add only what would have saved time this session.
|
||||
|
||||
5. **Report**: Summarize what was added/changed and why.
|
||||
|
||||
## Rules
|
||||
|
||||
- Only add knowledge confirmed by this session — no speculative additions
|
||||
- Keep docs concise — add a line or two, not a paragraph
|
||||
- If a whole new doc is needed, create it in `docs/` and add a pointer in `CLAUDE.md`
|
||||
- Don't update skills unless a coding pattern was genuinely wrong
|
||||
- Don't add things Claude already knows — only Windmill-specific knowledge
|
||||
@@ -1,107 +0,0 @@
|
||||
---
|
||||
name: rust-backend
|
||||
description: Rust coding guidelines for the Windmill backend. MUST use when writing or modifying Rust code in the backend directory.
|
||||
---
|
||||
|
||||
# Windmill Rust Patterns
|
||||
|
||||
Apply these Windmill-specific patterns when writing Rust code in `backend/`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
Use `Error` from `windmill_common::error`. Return `Result<T, Error>` or `JsonResult<T>`:
|
||||
|
||||
```rust
|
||||
use windmill_common::error::{Error, Result};
|
||||
|
||||
pub async fn get_job(db: &DB, id: Uuid) -> Result<Job> {
|
||||
sqlx::query_as!(Job, "SELECT id, workspace_id FROM v2_job WHERE id = $1", id)
|
||||
.fetch_optional(db)
|
||||
.await?
|
||||
.ok_or_else(|| Error::NotFound("job not found".to_string()))?;
|
||||
}
|
||||
```
|
||||
|
||||
Never panic in library code. Reserve `.unwrap()` for compile-time guarantees.
|
||||
|
||||
## SQLx Patterns
|
||||
|
||||
**Never use `SELECT *`** — always list columns explicitly. Critical for backwards compatibility when workers lag behind API version:
|
||||
|
||||
```rust
|
||||
// Correct
|
||||
sqlx::query_as!(Job, "SELECT id, workspace_id, path FROM v2_job WHERE id = $1", id)
|
||||
|
||||
// Wrong — breaks when columns are added
|
||||
sqlx::query_as!(Job, "SELECT * FROM v2_job WHERE id = $1", id)
|
||||
```
|
||||
|
||||
Use batch operations to avoid N+1:
|
||||
|
||||
```rust
|
||||
// Preferred — single query with IN clause
|
||||
sqlx::query!("SELECT ... WHERE id = ANY($1)", &ids[..]).fetch_all(db).await?
|
||||
```
|
||||
|
||||
Use transactions for multi-step operations. Parameterize all queries.
|
||||
|
||||
## JSON Handling
|
||||
|
||||
Prefer `Box<serde_json::value::RawValue>` over `serde_json::Value` when storing/passing JSON without inspection:
|
||||
|
||||
```rust
|
||||
pub struct Job {
|
||||
pub args: Option<Box<serde_json::value::RawValue>>,
|
||||
}
|
||||
```
|
||||
|
||||
Only use `serde_json::Value` when you need to inspect or modify the JSON.
|
||||
|
||||
## Serde Optimizations
|
||||
|
||||
```rust
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct Job {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub parent_job: Option<Uuid>,
|
||||
#[serde(skip_serializing_if = "Vec::is_empty")]
|
||||
pub tags: Vec<String>,
|
||||
#[serde(default)]
|
||||
pub priority: i32,
|
||||
}
|
||||
```
|
||||
|
||||
## Async & Concurrency
|
||||
|
||||
Never block the async runtime. Use `spawn_blocking` for CPU-intensive work:
|
||||
|
||||
```rust
|
||||
let result = tokio::task::spawn_blocking(move || expensive_computation(&data)).await?;
|
||||
```
|
||||
|
||||
**Mutex selection**: Prefer `std::sync::Mutex` (or `parking_lot::Mutex`) for data protection. Only use `tokio::sync::Mutex` when holding locks across `.await` points.
|
||||
|
||||
Use `tokio::sync::mpsc` (bounded) for channels. Avoid `std::thread::sleep` in async contexts.
|
||||
|
||||
## Module Structure & Visibility
|
||||
|
||||
- Use `pub(crate)` instead of `pub` when possible
|
||||
- Place new code in the appropriate crate based on functionality
|
||||
- API endpoints go in `windmill-api/src/` organized by domain
|
||||
- Shared functionality goes in `windmill-common/src/`
|
||||
|
||||
## Code Navigation
|
||||
|
||||
Always use rust-analyzer LSP for go-to-definition, find-references, and type info. Do not guess at module paths.
|
||||
|
||||
## Axum Handlers
|
||||
|
||||
Destructure extractors directly in function signatures:
|
||||
|
||||
```rust
|
||||
async fn process_job(
|
||||
Extension(db): Extension<DB>,
|
||||
Path((workspace, job_id)): Path<(String, Uuid)>,
|
||||
Query(pagination): Query<Pagination>,
|
||||
) -> Result<Json<Job>> { ... }
|
||||
```
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
name: svelte-frontend
|
||||
description: Svelte coding guidelines for the Windmill frontend. MUST use when writing or modifying code in the frontend directory.
|
||||
---
|
||||
|
||||
# Windmill Svelte Patterns
|
||||
|
||||
Apply these Windmill-specific patterns when writing Svelte code in `frontend/`. For general Svelte 5 syntax (runes, snippets, event handling), use the Svelte MCP server.
|
||||
|
||||
## Windmill UI Components (MUST use)
|
||||
|
||||
Always use Windmill's design-system components. Never use raw HTML elements.
|
||||
|
||||
### Buttons — `<Button>`
|
||||
|
||||
```svelte
|
||||
<script>
|
||||
import { Button } from '$lib/components/common'
|
||||
import { ChevronLeft } from 'lucide-svelte'
|
||||
</script>
|
||||
|
||||
<Button variant="default" onclick={handleClick}>Label</Button>
|
||||
<Button startIcon={{ icon: ChevronLeft }} iconOnly onclick={prev} />
|
||||
```
|
||||
|
||||
Props: `variant?: 'accent' | 'accent-secondary' | 'default' | 'subtle'`, `unifiedSize?: 'sm' | 'md' | 'lg'`, `startIcon?: { icon: SvelteComponent }`, `iconOnly?: boolean`, `disabled?: boolean`
|
||||
|
||||
### Text inputs — `<TextInput>`
|
||||
|
||||
```svelte
|
||||
<script>
|
||||
import { TextInput } from '$lib/components/common'
|
||||
</script>
|
||||
|
||||
<TextInput bind:value={val} placeholder="Enter value" />
|
||||
```
|
||||
|
||||
Props: `value?: string | number` (bindable), `placeholder?: string`, `disabled?: boolean`, `error?: string | boolean`, `size?: 'sm' | 'md' | 'lg'`
|
||||
|
||||
### Selects — `<Select>`
|
||||
|
||||
```svelte
|
||||
<script>
|
||||
import Select from '$lib/components/select/Select.svelte'
|
||||
</script>
|
||||
|
||||
<Select items={[{ label: 'Jan', value: 1 }]} bind:value={selected} />
|
||||
```
|
||||
|
||||
Props: `items?: Array<{ label?: string; value: any }>`, `value` (bindable), `placeholder?: string`, `clearable?: boolean`, `size?: 'sm' | 'md' | 'lg'`
|
||||
|
||||
### Icons — `lucide-svelte`
|
||||
|
||||
Never write inline SVGs. Import from `lucide-svelte`:
|
||||
|
||||
```svelte
|
||||
<script>
|
||||
import { ChevronLeft, X } from 'lucide-svelte'
|
||||
</script>
|
||||
<ChevronLeft size={16} />
|
||||
```
|
||||
|
||||
## Form Components
|
||||
|
||||
Form components (TextInput, Toggle, Select, etc.) should use the unified size system when placed together.
|
||||
|
||||
## Styling
|
||||
|
||||
- Use Tailwind CSS for all styling — no custom CSS
|
||||
- Use Windmill's theming classes for colors/surfaces (see `frontend/brand-guidelines.md`)
|
||||
- Read component props JSDoc before using them
|
||||
|
||||
## Svelte MCP Server
|
||||
|
||||
Use the Svelte MCP tools when working on Svelte code:
|
||||
|
||||
1. **list-sections**: Call first to discover available docs
|
||||
2. **get-documentation**: Fetch relevant sections based on use_cases
|
||||
3. **svelte-autofixer**: MUST use on all Svelte code before finalizing — keep calling until no issues
|
||||
4. **playground-link**: Only after user confirms and code was NOT written to project files
|
||||
@@ -13,10 +13,10 @@ on:
|
||||
jobs:
|
||||
check-membership:
|
||||
if: |
|
||||
(github.event_name == 'issue_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
|
||||
(github.event_name == 'pull_request_review_comment' && startsWith(github.event.comment.body, '/ai') && !startsWith(github.event.comment.body, '/ai-fast')) ||
|
||||
(github.event_name == 'pull_request_review' && startsWith(github.event.review.body, '/ai') && !startsWith(github.event.review.body, '/ai-fast')) ||
|
||||
(github.event_name == 'issues' && startsWith(github.event.issue.body, '/ai') && !startsWith(github.event.issue.body, '/ai-fast'))
|
||||
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '/ai')) ||
|
||||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '/ai')) ||
|
||||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '/ai')) ||
|
||||
(github.event_name == 'issues' && contains(github.event.issue.body, '/ai'))
|
||||
uses: ./.github/workflows/check-org-membership.yml
|
||||
secrets:
|
||||
access_token: ${{ secrets.ORG_ACCESS_TOKEN }}
|
||||
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "25ac66a1022c41267df199a95f532b0f778c25fbe0f7a7f9734c1f7e536ed6ce"
|
||||
}
|
||||
-16
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO raw_script_temp (workspace_id, hash, content, created_at)\n VALUES ($1, $2, $3, NOW())\n ON CONFLICT (workspace_id, hash) DO UPDATE SET created_at = NOW()",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Bpchar",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "2d523cd0d5b7107b15846b885fa40af492d4c8a8871cef972980150f319fe6ff"
|
||||
}
|
||||
-28
@@ -1,28 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT parent_job, flow_step_id FROM v2_job WHERE id = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "parent_job",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "flow_step_id",
|
||||
"type_info": "Varchar"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9"
|
||||
}
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n path,\n summary,\n description\n FROM\n flow\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
|
||||
"query": "SELECT \n path,\n summary,\n description\n FROM\n flow\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
@@ -31,5 +31,5 @@
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "4615b37cb848f9589622426d291c721e532b230c527deb701e24605c7027e38b"
|
||||
"hash": "33367c42e87e78ae987c0966dc4d445c5eff75b2e2843ffd7a46b03cbaea9ae8"
|
||||
}
|
||||
-29
@@ -1,29 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n f.schema AS \"schema: serde_json::Value\",\n fv.value->>'preprocessor_module' IS NOT NULL AS \"has_preprocessor: bool\"\n FROM flow f\n LEFT JOIN flow_version fv ON fv.id = f.versions[array_length(f.versions, 1)]\n AND fv.workspace_id = f.workspace_id\n WHERE f.path = $1 AND f.workspace_id = $2 AND NOT f.archived",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "schema: serde_json::Value",
|
||||
"type_info": "Json"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "has_preprocessor: bool",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "372ec62fad93831b00d44f6c52a148b6bfa6008e49b2a80bec06c76d9432ec77"
|
||||
}
|
||||
-27
@@ -1,27 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n WITH step_index AS (\n SELECT idx::text AS idx\n FROM v2_job_status,\n jsonb_array_elements(flow_status->'modules') WITH ORDINALITY arr(elem, idx)\n WHERE id = $1\n AND elem->>'id' = $5\n LIMIT 1\n ), completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n $3::text::jsonb,\n CASE WHEN si.idx IS NOT NULL\n THEN jsonb_set(\n s.flow_status,\n ARRAY['modules', (si.idx::int - 1)::text],\n $6::jsonb\n )\n ELSE s.flow_status\n END,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n LEFT JOIN step_index si ON true\n WHERE q.id = $1\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id = $1\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n VALUES ($4, $1, $2)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "x",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Varchar",
|
||||
"Text",
|
||||
"Text",
|
||||
"Text",
|
||||
"Jsonb"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "4461fe84370e7f07b4423ea5e71b913dd6ee9c655f9ec7087cc0fec9b9c3099a"
|
||||
}
|
||||
-16
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO v2_job (id, kind, tag, created_by, permissioned_as, permissioned_as_email, workspace_id, parent_job)\n VALUES ($1, 'noop', 'deno', 'test-user', 'u/test-user', 'test@windmill.dev', $2, $3)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Varchar",
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4538bea4159677d8e653159d7d01649cae08e6ffef668a7cbac49b312bf30766"
|
||||
}
|
||||
-35
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "debounced_times",
|
||||
"type_info": "Int4"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "first_started_at",
|
||||
"type_info": "Timestamptz"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "job_id_to_debounce",
|
||||
"type_info": "Uuid"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618"
|
||||
}
|
||||
-22
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT debounced_times FROM debounce_key WHERE key = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "debounced_times",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "6009cf60c43608bb1c7924dbd0a9cdc01986d787eb24f0fffe4550d939851e22"
|
||||
}
|
||||
-35
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "debounced_times",
|
||||
"type_info": "Int4"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "first_started_at",
|
||||
"type_info": "Timestamptz"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "job_id_to_debounce",
|
||||
"type_info": "Uuid"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073"
|
||||
}
|
||||
-35
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "debounced_times",
|
||||
"type_info": "Int4"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "first_started_at",
|
||||
"type_info": "Timestamptz"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "job_id_to_debounce",
|
||||
"type_info": "Uuid"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b"
|
||||
}
|
||||
-22
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT content FROM raw_script_temp WHERE hash = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "content",
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Bpchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "88ec0ddcc86fb67089b551ccbce7b932800e33cd462a651f7e6716929ee9b6f2"
|
||||
}
|
||||
-29
@@ -1,29 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n schema AS \"schema: serde_json::Value\",\n has_preprocessor\n FROM script\n WHERE path = $1 AND workspace_id = $2\n AND NOT archived AND NOT deleted\n ORDER BY created_at DESC\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "schema: serde_json::Value",
|
||||
"type_info": "Json"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "has_preprocessor",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "909a095d679ee7b8266fc93fe0c598e4b6583daa2b0d7c593fc72ff9b8d1d06b"
|
||||
}
|
||||
+3
-21
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT\n route_path,\n http_method AS \"http_method: _\",\n request_type AS \"request_type: _\",\n workspaced_route,\n summary,\n description,\n authentication_method AS \"authentication_method: _\",\n authentication_resource_path,\n script_path,\n is_flow,\n wrap_body\n FROM\n http_trigger\n WHERE\n path ~ ANY($1) AND\n route_path ~ ANY($2) AND\n workspace_id = $3\n ",
|
||||
"query": "\n SELECT\n route_path,\n http_method AS \"http_method: _\",\n request_type AS \"request_type: _\",\n workspaced_route,\n summary,\n description,\n authentication_method AS \"authentication_method: _\",\n authentication_resource_path\n FROM\n http_trigger\n WHERE\n path ~ ANY($1) AND\n route_path ~ ANY($2) AND\n workspace_id = $3\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
@@ -80,21 +80,6 @@
|
||||
"ordinal": 7,
|
||||
"name": "authentication_resource_path",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 8,
|
||||
"name": "script_path",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 9,
|
||||
"name": "is_flow",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 10,
|
||||
"name": "wrap_body",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
@@ -112,11 +97,8 @@
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
false
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "1cb21a66ffc89ebe53fd8f58690eec4cf11cb4b7816738202b474e8d3ffef427"
|
||||
"hash": "9360d00990822f153ff09c7905ae3180f07d02f38ac12d07a5664d93f160e7ee"
|
||||
}
|
||||
-46
@@ -1,46 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n WITH RECURSIVE chain AS (\n SELECT\n j.id,\n j.parent_job,\n j.flow_step_id,\n 1 AS depth\n FROM v2_job j\n WHERE j.id = $1\n UNION ALL\n SELECT\n pj.id,\n pj.parent_job,\n pj.flow_step_id,\n c.depth + 1\n FROM chain c\n JOIN v2_job pj ON pj.id = c.parent_job\n WHERE c.parent_job IS NOT NULL\n )\n SELECT\n c.id,\n c.parent_job,\n c.flow_step_id,\n EXISTS(SELECT 1 FROM v2_job_queue q WHERE q.id = c.parent_job) AS \"parent_in_queue!\",\n EXISTS(\n SELECT 1 FROM v2_job sib\n WHERE sib.parent_job = c.parent_job\n AND sib.id != c.id\n AND sib.id IN (SELECT sq.id FROM v2_job_queue sq)\n ) AS \"has_other_active_siblings!\"\n FROM chain c\n WHERE c.depth >= 1\n ORDER BY c.depth ASC\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "parent_job",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "flow_step_id",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "parent_in_queue!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 4,
|
||||
"name": "has_other_active_siblings!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb"
|
||||
}
|
||||
-22
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT scheduled_for FROM v2_job_queue WHERE id = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "scheduled_for",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "a9c2019c7bafb172dfb353870238c02673113b806bc3c7a71354c34255ecd31c"
|
||||
}
|
||||
-16
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "UPDATE v2_job_status SET flow_status = (\n SELECT jsonb_set(\n flow_status,\n ARRAY['modules', (idx - 1)::text],\n $2::jsonb\n )\n FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n LIMIT 1\n ) WHERE id = $1 AND (\n SELECT COUNT(*) FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n ) > 0",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Jsonb",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c"
|
||||
}
|
||||
-40
@@ -1,40 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n WITH RECURSIVE chain AS (\n SELECT\n j.id,\n j.parent_job,\n j.flow_step_id,\n 1 AS depth\n FROM v2_job j\n WHERE j.id = $1\n UNION ALL\n SELECT\n pj.id,\n pj.parent_job,\n pj.flow_step_id,\n c.depth + 1\n FROM chain c\n JOIN v2_job pj ON pj.id = c.parent_job\n WHERE c.parent_job IS NOT NULL\n )\n SELECT\n c.id,\n c.parent_job,\n c.flow_step_id,\n EXISTS(SELECT 1 FROM v2_job_queue q WHERE q.id = c.parent_job) AS \"parent_in_queue!\"\n FROM chain c\n WHERE c.depth >= 1\n ORDER BY c.depth ASC\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "parent_job",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "flow_step_id",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "parent_in_queue!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "c79501b30fc28a3ae761579d05f2296e848554e62a00ff7c164109bf3d97f44f"
|
||||
}
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT content FROM script WHERE path = $1 AND workspace_id = $2\n AND archived = false ORDER BY created_at DESC LIMIT 1\n ",
|
||||
"query": "\n SELECT content FROM script WHERE path = $1 AND workspace_id = $2\n AND archived = false ORDER BY created_at DESC LIMIT 1\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
@@ -19,5 +19,5 @@
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "96f6163a164b9ffb4ec52372d7fea158db101f54f0908551b5a4f5e6655e122b"
|
||||
"hash": "c7cae4cf872fce0a989cf89aa35929218a9d459ee1c2b36a28b110e9741ab623"
|
||||
}
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n path,\n summary,\n description\n FROM\n script\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
|
||||
"query": "SELECT \n path,\n summary,\n description\n FROM\n script\n WHERE\n path ~ ANY($1) AND\n workspace_id = $2 AND\n archived is FALSE\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
@@ -31,5 +31,5 @@
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "69b44efb0144fececccafc8a77d040649bb17e239e5afbc27b915efc4c95d54e"
|
||||
"hash": "dc36b46b9eb80cb7c92fa72519d117eda99a6f482a073ccd36a6431ef689a3fd"
|
||||
}
|
||||
-23
@@ -1,23 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT EXISTS(\n SELECT 1 FROM v2_job\n WHERE parent_job = $1 AND id != $2\n AND id IN (SELECT id FROM v2_job_queue)\n ) as has",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "has",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2"
|
||||
}
|
||||
-15
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "UPDATE v2_job_queue SET scheduled_for = $1 WHERE id = $2",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Timestamptz",
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "f9eabfab66ae102c8a61ae5e6349b5b167ff7939828a747ad09bb06c9d6c6d8d"
|
||||
}
|
||||
Generated
+3
-1
@@ -16898,6 +16898,8 @@ dependencies = [
|
||||
"async-recursion",
|
||||
"itertools 0.14.0",
|
||||
"lazy_static",
|
||||
"malachite",
|
||||
"malachite-bigint",
|
||||
"pep440_rs",
|
||||
"phf 0.11.3",
|
||||
"regex",
|
||||
@@ -16954,6 +16956,7 @@ dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
"windmill-parser",
|
||||
"windmill-types",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -17462,7 +17465,6 @@ dependencies = [
|
||||
"strum 0.27.2",
|
||||
"tracing",
|
||||
"uuid",
|
||||
"windmill-parser",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
@@ -1 +1 @@
|
||||
c04f3851c03758662e4936ff4b6e71bc56dbae7e
|
||||
563877bf1c8b4184f638bab51be89b1c0aec6dad
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
DROP INDEX IF EXISTS idx_raw_script_temp_created_at;
|
||||
DROP TABLE IF EXISTS raw_script_temp;
|
||||
@@ -1,11 +0,0 @@
|
||||
-- Temporary storage for raw script content during CLI lock generation
|
||||
-- Content is stored with hash as key, cleaned up after 1 week
|
||||
CREATE TABLE raw_script_temp (
|
||||
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id),
|
||||
hash CHAR(64) NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (workspace_id, hash)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_raw_script_temp_created_at ON raw_script_temp (created_at);
|
||||
@@ -13,19 +13,21 @@ regex-lite.workspace = true
|
||||
|
||||
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
|
||||
regex.workspace = true
|
||||
|
||||
[dependencies]
|
||||
windmill-parser.workspace = true
|
||||
windmill-common.workspace = true
|
||||
rustpython-parser.workspace = true
|
||||
malachite.workspace = true
|
||||
malachite-bigint.workspace = true
|
||||
phf.workspace = true
|
||||
itertools.workspace = true
|
||||
serde_json.workspace = true
|
||||
anyhow.workspace = true
|
||||
lazy_static.workspace = true
|
||||
sqlx.workspace = true
|
||||
async-recursion.workspace = true
|
||||
toml.workspace = true
|
||||
serde.workspace = true
|
||||
pep440_rs.workspace = true
|
||||
tracing.workspace = true
|
||||
|
||||
[dependencies]
|
||||
windmill-parser.workspace = true
|
||||
rustpython-parser.workspace = true
|
||||
phf.workspace = true
|
||||
itertools.workspace = true
|
||||
serde_json.workspace = true
|
||||
anyhow.workspace = true
|
||||
lazy_static.workspace = true
|
||||
|
||||
@@ -8,14 +8,10 @@
|
||||
|
||||
mod mapping;
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
use async_recursion::async_recursion;
|
||||
use itertools::Itertools;
|
||||
use lazy_static::lazy_static;
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
use std::str::FromStr;
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
use std::collections::HashMap;
|
||||
use std::{collections::HashMap, str::FromStr};
|
||||
|
||||
use mapping::{FULL_IMPORTS_MAP, SHORT_IMPORTS_MAP};
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
@@ -28,9 +24,7 @@ use rustpython_parser::{
|
||||
text_size::TextRange,
|
||||
Parse,
|
||||
};
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
use sqlx::{Pool, Postgres};
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
use windmill_common::{
|
||||
error::{self, to_anyhow},
|
||||
worker::{
|
||||
@@ -52,14 +46,10 @@ fn replace_full_import(x: &str) -> Option<String> {
|
||||
FULL_IMPORTS_MAP.get(x).map(|x| (*x).to_owned())
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
lazy_static! {
|
||||
static ref RE: Regex = Regex::new(r"^\#\s?(\S+)\s*$").unwrap();
|
||||
static ref PKG_RE: Regex = Regex::new(r"^([^!=<>]+)(?:[!=<>]|$)").unwrap();
|
||||
}
|
||||
|
||||
lazy_static! {
|
||||
static ref PIN_RE: Regex = Regex::new(r"(?:\s*#\s*(pin|repin):\s*)(\S*)").unwrap();
|
||||
static ref PKG_RE: Regex = Regex::new(r"^([^!=<>]+)(?:[!=<>]|$)").unwrap();
|
||||
// Regex to properly match main function definition at line start,
|
||||
// capturing both sync and async variants
|
||||
static ref DEF_MAIN_RE: Regex = Regex::new(r"(?m)^(async\s+)?def\s+main\s*\(").unwrap();
|
||||
@@ -92,7 +82,7 @@ fn process_import(module: Option<String>, path: &str, level: usize) -> Vec<NImpo
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<String>> {
|
||||
pub fn parse_relative_imports(code: &str, path: &str) -> error::Result<Vec<String>> {
|
||||
let nimports = parse_code_for_imports(code, path)?;
|
||||
return Ok(nimports
|
||||
.into_iter()
|
||||
@@ -104,7 +94,7 @@ pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<Stri
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
||||
pub enum NImport {
|
||||
enum NImport {
|
||||
// Order matters! First we want to resolve all repins
|
||||
|
||||
// manually repinned requirement
|
||||
@@ -144,8 +134,6 @@ pub enum NImport {
|
||||
// Relative imports
|
||||
Relative(String),
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
||||
enum NImportResolved {
|
||||
Repin { pin: ImportPin, key: String },
|
||||
@@ -154,12 +142,12 @@ enum NImportResolved {
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
||||
pub struct ImportPin {
|
||||
pub pkg: String,
|
||||
pub path: String,
|
||||
struct ImportPin {
|
||||
pkg: String,
|
||||
path: String,
|
||||
}
|
||||
|
||||
pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImport>> {
|
||||
fn parse_code_for_imports(code: &str, path: &str) -> error::Result<Vec<NImport>> {
|
||||
// Use regex to safely find the main function definition
|
||||
let mut code = DEF_MAIN_RE
|
||||
.split(code)
|
||||
@@ -187,7 +175,7 @@ pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImp
|
||||
let code_with_fake_main = format!("{}\n\ndef main(): pass", code);
|
||||
|
||||
let ast = Suite::parse(&code_with_fake_main, "main.py").map_err(|e| {
|
||||
anyhow::anyhow!("Error parsing code for imports: {}", e.to_string())
|
||||
error::Error::ExecutionErr(format!("Error parsing code for imports: {}", e.to_string()))
|
||||
})?;
|
||||
// Note: We're still using the original code for finding pins,
|
||||
// as the TextRange values from the parsed AST would be based on code_with_fake_main
|
||||
@@ -268,7 +256,6 @@ pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImp
|
||||
return Ok(nimports);
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
pub async fn parse_python_imports(
|
||||
code: &str,
|
||||
w_id: &str,
|
||||
@@ -277,7 +264,6 @@ pub async fn parse_python_imports(
|
||||
version_specifiers: &mut Vec<pep440_rs::VersionSpecifier>,
|
||||
locked_v: &mut Option<pep440_rs::Version>,
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> error::Result<(Vec<String>, Option<String>)> {
|
||||
let mut compile_error_hint: Option<String> = None;
|
||||
let mut imports = parse_python_imports_inner(
|
||||
@@ -290,7 +276,6 @@ pub async fn parse_python_imports(
|
||||
&mut None,
|
||||
locked_v,
|
||||
raw_workspace_dependencies_o,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?
|
||||
.into_values()
|
||||
@@ -328,7 +313,6 @@ pub async fn parse_python_imports(
|
||||
Ok((imports, compile_error_hint))
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn extract_pkg_name(requirement: &str) -> String {
|
||||
PKG_RE
|
||||
.captures(requirement)
|
||||
@@ -336,7 +320,6 @@ fn extract_pkg_name(requirement: &str) -> String {
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
#[async_recursion]
|
||||
async fn parse_python_imports_inner(
|
||||
code: &str,
|
||||
@@ -348,7 +331,6 @@ async fn parse_python_imports_inner(
|
||||
path_where_annotated_pyv: &mut Option<String>,
|
||||
locked_v: &mut Option<pep440_rs::Version>,
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> error::Result<HashMap<String, NImportResolved>> {
|
||||
tracing::debug!("Parsing python imports for path: {}", path);
|
||||
let PythonAnnotations { py310, py311, py312, py313, .. } = PythonAnnotations::parse(&code);
|
||||
@@ -512,37 +494,17 @@ async fn parse_python_imports_inner(
|
||||
for n in nimports.into_iter() {
|
||||
let mut nested = match n {
|
||||
NImport::Relative(rpath) => {
|
||||
// First try to get content from temp_script_refs cache if available
|
||||
let code_from_cache = if let Some(hash) = temp_script_refs.as_ref().and_then(|dt| dt.get(&rpath)) {
|
||||
tracing::debug!("Found relative import '{}' in temp_script_refs with hash '{}'", rpath, hash);
|
||||
match windmill_common::cache::raw_script_temp::load(hash.clone(), db).await {
|
||||
Ok(content) => Some(content),
|
||||
Err(e) => {
|
||||
tracing::warn!("temp_script_refs hash '{}' not found in cache: {}, falling back to deployed script", hash, e);
|
||||
None
|
||||
}
|
||||
}
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
// Use cached content if available, otherwise fall back to deployed script
|
||||
let code = match code_from_cache {
|
||||
Some(content) => content,
|
||||
None => {
|
||||
sqlx::query_scalar!(
|
||||
r#"
|
||||
SELECT content FROM script WHERE path = $1 AND workspace_id = $2
|
||||
AND archived = false ORDER BY created_at DESC LIMIT 1
|
||||
"#,
|
||||
&rpath,
|
||||
w_id
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await?
|
||||
.unwrap_or_else(|| "".to_string())
|
||||
}
|
||||
};
|
||||
let code = sqlx::query_scalar!(
|
||||
r#"
|
||||
SELECT content FROM script WHERE path = $1 AND workspace_id = $2
|
||||
AND archived = false ORDER BY created_at DESC LIMIT 1
|
||||
"#,
|
||||
&rpath,
|
||||
w_id
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await?
|
||||
.unwrap_or_else(|| "".to_string());
|
||||
|
||||
if already_visited.contains(&rpath) {
|
||||
vec![]
|
||||
@@ -560,7 +522,6 @@ async fn parse_python_imports_inner(
|
||||
path_where_annotated_pyv,
|
||||
locked_v,
|
||||
raw_workspace_dependencies_o,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?
|
||||
.into_values()
|
||||
@@ -685,7 +646,6 @@ async fn parse_python_imports_inner(
|
||||
Ok(final_imports)
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn extract_nimports_from_content(
|
||||
content: &str,
|
||||
hm: &mut HashMap<String, NImportResolved>,
|
||||
|
||||
@@ -26,7 +26,6 @@ def main():
|
||||
&mut vec![],
|
||||
&mut None,
|
||||
&None,
|
||||
&None,
|
||||
)
|
||||
.await?;
|
||||
// println!("{}", serde_json::to_string(&r)?);
|
||||
@@ -68,7 +67,6 @@ def main():
|
||||
&mut vec![],
|
||||
&mut None,
|
||||
&None,
|
||||
&None,
|
||||
)
|
||||
.await?;
|
||||
println!("{}", serde_json::to_string(&r)?);
|
||||
@@ -100,7 +98,6 @@ def main():
|
||||
&mut vec![],
|
||||
&mut None,
|
||||
&None,
|
||||
&None,
|
||||
)
|
||||
.await?;
|
||||
println!("{}", serde_json::to_string(&r)?);
|
||||
|
||||
@@ -16,6 +16,7 @@ regex.workspace = true
|
||||
|
||||
[dependencies]
|
||||
windmill-parser.workspace = true
|
||||
windmill-types.workspace = true
|
||||
anyhow.workspace = true
|
||||
lazy_static.workspace = true
|
||||
serde_json.workspace = true
|
||||
|
||||
@@ -15,7 +15,7 @@ use std::{
|
||||
iter::Peekable,
|
||||
str::CharIndices,
|
||||
};
|
||||
pub use windmill_parser::{s3_mode_extension, Arg, MainArgSignature, ObjectType, S3ModeFormat, Typ};
|
||||
pub use windmill_parser::{Arg, MainArgSignature, ObjectType, Typ};
|
||||
|
||||
pub const SANITIZED_ENUM_STR: &str = "__sanitized_enum__";
|
||||
pub const SANITIZED_RAW_STRING_STR: &str = "__sanitized_raw_string__";
|
||||
@@ -143,6 +143,7 @@ pub fn parse_db_resource(code: &str) -> Option<String> {
|
||||
cap.map(|x| x.get(1).map(|x| x.as_str().to_string()).unwrap())
|
||||
}
|
||||
|
||||
pub use windmill_types::s3::{s3_mode_extension, S3ModeFormat};
|
||||
pub struct S3ModeArgs {
|
||||
pub prefix: Option<String>,
|
||||
pub storage: Option<String>,
|
||||
|
||||
@@ -117,12 +117,6 @@ impl Visit for ImportsFinder {
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse TypeScript/JavaScript code and extract all import paths as raw strings.
|
||||
///
|
||||
/// Returns import paths exactly as written in the code (e.g., `"./module"`, `"../utils"`, `"lodash"`).
|
||||
/// Does not resolve relative paths to absolute Windmill paths.
|
||||
///
|
||||
/// See also: [`parse_relative_imports`] for resolved absolute paths.
|
||||
pub fn parse_expr_for_imports(code: &str, skip_type_only: bool) -> anyhow::Result<Vec<String>> {
|
||||
let cm: Lrc<SourceMap> = Default::default();
|
||||
let fm = cm.new_source_file(FileName::Custom("main.d.ts".into()).into(), code.into());
|
||||
@@ -157,82 +151,6 @@ pub fn parse_expr_for_imports(code: &str, skip_type_only: bool) -> anyhow::Resul
|
||||
Ok(imports)
|
||||
}
|
||||
|
||||
/// Parse TypeScript/JavaScript code and extract relative imports resolved to absolute Windmill paths.
|
||||
///
|
||||
/// Takes the script's Windmill path (e.g., `"f/folder/script"`) and resolves relative imports
|
||||
/// like `"./module"` or `"../utils"` to absolute paths like `"f/folder/module"` or `"f/utils"`.
|
||||
///
|
||||
/// Only returns relative imports (those starting with `./`, `../`, or `/`).
|
||||
/// External package imports (e.g., `"lodash"`) are filtered out.
|
||||
///
|
||||
/// See also: [`parse_expr_for_imports`] for raw import strings without resolution.
|
||||
///
|
||||
/// # Arguments
|
||||
/// * `code` - The TypeScript/JavaScript source code
|
||||
/// * `path` - The Windmill path of the script (e.g., `"f/folder/script"`)
|
||||
///
|
||||
/// # Returns
|
||||
/// A sorted, deduplicated list of resolved absolute Windmill paths.
|
||||
///
|
||||
/// # Examples
|
||||
/// ```ignore
|
||||
/// // Script at "f/folder/script" with: import { x } from "../utils"
|
||||
/// // Returns: ["f/utils"]
|
||||
/// ```
|
||||
pub fn parse_relative_imports(code: &str, path: &str) -> anyhow::Result<Vec<String>> {
|
||||
let imports = parse_expr_for_imports(code, false)?;
|
||||
let script_dir = path.rsplit_once('/').map(|(dir, _)| dir).unwrap_or("");
|
||||
|
||||
let mut resolved: Vec<String> = imports
|
||||
.into_iter()
|
||||
.filter(|imp| is_relative_import(imp))
|
||||
.map(|imp| {
|
||||
// Remove .ts extension if present
|
||||
let imp = imp.strip_suffix(".ts").unwrap_or(&imp);
|
||||
|
||||
if imp.starts_with("/") {
|
||||
// Absolute path (e.g., /f/folder/script) - remove leading slash
|
||||
imp[1..].to_string()
|
||||
} else {
|
||||
// Relative path (e.g., ./script or ../folder/script)
|
||||
let combined = format!("{}/{}", script_dir, imp);
|
||||
normalize_path(&combined)
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
|
||||
resolved.sort();
|
||||
resolved.dedup();
|
||||
Ok(resolved)
|
||||
}
|
||||
|
||||
/// Check if an import path is a relative import (starts with `./`, `../`, or `/`)
|
||||
fn is_relative_import(import_path: &str) -> bool {
|
||||
import_path.starts_with("./")
|
||||
|| import_path.starts_with("../")
|
||||
|| import_path.starts_with("/")
|
||||
}
|
||||
|
||||
/// Normalize a path by resolving `.` and `..` components
|
||||
fn normalize_path(input_path: &str) -> String {
|
||||
let parts: Vec<&str> = input_path.split('/').filter(|p| !p.is_empty()).collect();
|
||||
let mut result: Vec<&str> = Vec::new();
|
||||
|
||||
for part in parts {
|
||||
if part == "." {
|
||||
continue;
|
||||
} else if part == ".." {
|
||||
if !result.is_empty() {
|
||||
result.pop();
|
||||
}
|
||||
} else {
|
||||
result.push(part);
|
||||
}
|
||||
}
|
||||
|
||||
result.join("/")
|
||||
}
|
||||
|
||||
struct OutputFinder {
|
||||
idents: HashSet<(String, String)>,
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
mod tests {
|
||||
use serde_json::json;
|
||||
use windmill_parser::{Arg, MainArgSignature, ObjectProperty, ObjectType, Typ};
|
||||
use windmill_parser_ts::{parse_deno_signature, parse_expr_for_imports, parse_relative_imports};
|
||||
use windmill_parser_ts::{parse_deno_signature, parse_expr_for_imports};
|
||||
|
||||
#[test]
|
||||
fn test_imports_basic() {
|
||||
@@ -798,7 +798,7 @@ mod tests {
|
||||
// Test case where there are exports but no preprocessor
|
||||
let code = r#"
|
||||
export { foo, bar } from "./utils";
|
||||
|
||||
|
||||
export async function main(param: string) {
|
||||
return param;
|
||||
}
|
||||
@@ -806,84 +806,4 @@ mod tests {
|
||||
let sig = parse_deno_signature(code, false, false, None).unwrap();
|
||||
assert_eq!(sig.has_preprocessor, Some(false));
|
||||
}
|
||||
|
||||
// ==========================================================================
|
||||
// Tests for parse_relative_imports
|
||||
// ==========================================================================
|
||||
|
||||
#[test]
|
||||
fn test_relative_imports_dot() {
|
||||
let code = r#"
|
||||
import { helper } from "./helper";
|
||||
export async function main() { return helper(); }
|
||||
"#;
|
||||
let result = parse_relative_imports(code, "f/folder/script").unwrap();
|
||||
assert_eq!(result, vec!["f/folder/helper"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_relative_imports_double_dot() {
|
||||
let code = r#"
|
||||
import { utils } from "../utils/helper";
|
||||
export async function main() { return utils(); }
|
||||
"#;
|
||||
let result = parse_relative_imports(code, "f/folder/subfolder/script").unwrap();
|
||||
assert_eq!(result, vec!["f/folder/utils/helper"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_relative_imports_absolute_path() {
|
||||
let code = r#"
|
||||
import { shared } from "/f/shared/utils";
|
||||
export async function main() { return shared(); }
|
||||
"#;
|
||||
let result = parse_relative_imports(code, "f/folder/script").unwrap();
|
||||
assert_eq!(result, vec!["f/shared/utils"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_relative_imports_mixed() {
|
||||
let code = r#"
|
||||
import { helper } from "./helper";
|
||||
import { utils } from "../utils";
|
||||
import { shared } from "/f/shared/lib";
|
||||
import lodash from "lodash";
|
||||
export async function main() { return helper() + utils() + shared(); }
|
||||
"#;
|
||||
let result = parse_relative_imports(code, "f/folder/script").unwrap();
|
||||
// Should only include relative imports, not external packages like lodash
|
||||
assert_eq!(result, vec!["f/folder/helper", "f/shared/lib", "f/utils"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_relative_imports_with_ts_extension() {
|
||||
let code = r#"
|
||||
import { helper } from "./helper.ts";
|
||||
export async function main() { return helper(); }
|
||||
"#;
|
||||
let result = parse_relative_imports(code, "f/folder/script").unwrap();
|
||||
assert_eq!(result, vec!["f/folder/helper"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_relative_imports_external_only() {
|
||||
let code = r#"
|
||||
import lodash from "lodash";
|
||||
import { something } from "@scope/package";
|
||||
export async function main() { return lodash.map([]); }
|
||||
"#;
|
||||
let result = parse_relative_imports(code, "f/folder/script").unwrap();
|
||||
assert!(result.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_relative_imports_deeply_nested() {
|
||||
let code = r#"
|
||||
import { a } from "../../a";
|
||||
import { b } from "../../../b";
|
||||
export async function main() { return a() + b(); }
|
||||
"#;
|
||||
let result = parse_relative_imports(code, "f/one/two/three/script").unwrap();
|
||||
assert_eq!(result, vec!["f/b", "f/one/a"]);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,7 +40,6 @@ java-parser = [ "dep:windmill-parser-java"]
|
||||
ruby-parser = [ "dep:windmill-parser-ruby"]
|
||||
wac-parser = [ "dep:windmill-parser-wac"]
|
||||
asset-parser = [ "dep:windmill-parser-ts-asset", "dep:windmill-parser-py-asset", "dep:windmill-parser-sql-asset"]
|
||||
py-imports-parser = [ "dep:windmill-parser-py-imports"]
|
||||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
@@ -62,7 +61,6 @@ windmill-parser-wac = { workspace = true, optional = true }
|
||||
windmill-parser-ts-asset = { workspace = true, optional = true }
|
||||
windmill-parser-py-asset = { workspace = true, optional = true }
|
||||
windmill-parser-sql-asset = { workspace = true, optional = true }
|
||||
windmill-parser-py-imports = { workspace = true, optional = true }
|
||||
wasm-bindgen.workspace = true
|
||||
|
||||
serde_json.workspace = true
|
||||
|
||||
@@ -67,12 +67,6 @@ const targets = [
|
||||
features: "asset-parser",
|
||||
env: "default",
|
||||
},
|
||||
{
|
||||
ident: "py-imports",
|
||||
desc: "Python imports"
|
||||
features: "py-imports-parser",
|
||||
env: "default",
|
||||
},
|
||||
# ^^^ Add new entry here ^^^
|
||||
];
|
||||
# NOTE: This is legacy command for building all, but it is not more used
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env nu
|
||||
|
||||
# Build in debug mode specified lang parser to wasm
|
||||
|
||||
# Build in debug mode specified lang parser to wasm
|
||||
# and perform installation to frontend
|
||||
def "main" [
|
||||
lang: string # Example: nu
|
||||
@@ -9,7 +9,4 @@ def "main" [
|
||||
(
|
||||
cd ../../../frontend; npm install ../backend/parsers/windmill-parser-wasm/pkg-($lang)
|
||||
)
|
||||
(
|
||||
cd ../../../cli; bun install ../backend/parsers/windmill-parser-wasm/pkg-($lang)
|
||||
)
|
||||
}
|
||||
|
||||
@@ -36,6 +36,3 @@ popd
|
||||
|
||||
pushd "pkg-asset" && npm publish ${args}
|
||||
popd
|
||||
|
||||
pushd "pkg-py-imports" && npm publish ${args}
|
||||
popd
|
||||
|
||||
@@ -38,8 +38,6 @@ pub fn parse_outputs(code: &str) -> String {
|
||||
return serde_json::to_string(&r).unwrap();
|
||||
}
|
||||
|
||||
/// Parse TypeScript imports and return raw import strings.
|
||||
/// See [`parse_ts_relative_imports`] for resolved absolute paths.
|
||||
#[cfg(feature = "ts-parser")]
|
||||
#[wasm_bindgen]
|
||||
pub fn parse_ts_imports(code: &str) -> String {
|
||||
@@ -52,15 +50,6 @@ pub fn parse_ts_imports(code: &str) -> String {
|
||||
return serde_json::to_string(&r).unwrap();
|
||||
}
|
||||
|
||||
/// Parse TypeScript imports and return relative imports resolved to absolute Windmill paths.
|
||||
/// Throws JS error on parse failure.
|
||||
/// See [`parse_ts_imports`] for raw import strings.
|
||||
#[cfg(feature = "ts-parser")]
|
||||
#[wasm_bindgen]
|
||||
pub fn parse_ts_relative_imports(code: &str, path: &str) -> Result<Vec<String>, String> {
|
||||
windmill_parser_ts::parse_relative_imports(code, path).map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
#[cfg(feature = "bash-parser")]
|
||||
#[wasm_bindgen]
|
||||
pub fn parse_bash(code: &str) -> String {
|
||||
@@ -225,14 +214,6 @@ pub fn parse_assets_py(code: &str) -> String {
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse Python imports and return relative imports resolved to absolute Windmill paths.
|
||||
/// Throws JS error on parse failure.
|
||||
#[cfg(feature = "py-imports-parser")]
|
||||
#[wasm_bindgen]
|
||||
pub fn parse_py_relative_imports(code: &str, path: &str) -> Result<Vec<String>, String> {
|
||||
windmill_parser_py_imports::parse_relative_imports(code, path).map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
#[cfg(feature = "ansible-parser")]
|
||||
#[wasm_bindgen]
|
||||
pub fn parse_assets_ansible(code: &str) -> String {
|
||||
|
||||
@@ -14,23 +14,6 @@ use serde_json::Value;
|
||||
|
||||
pub mod asset_parser;
|
||||
|
||||
/// S3 output format for SQL queries (moved here to avoid pulling sqlx into WASM via windmill-types)
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub enum S3ModeFormat {
|
||||
Json,
|
||||
Csv,
|
||||
Parquet,
|
||||
}
|
||||
|
||||
/// Returns the file extension for the given S3 mode format
|
||||
pub fn s3_mode_extension(format: S3ModeFormat) -> &'static str {
|
||||
match format {
|
||||
S3ModeFormat::Json => "json",
|
||||
S3ModeFormat::Csv => "csv",
|
||||
S3ModeFormat::Parquet => "parquet",
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize, Debug, PartialEq, Default)]
|
||||
pub struct MainArgSignature {
|
||||
pub star_args: bool,
|
||||
|
||||
@@ -312,7 +312,6 @@ async fn cache_hub_scripts(file_path: Option<String>) -> anyhow::Result<()> {
|
||||
"cache_init",
|
||||
"",
|
||||
&mut None,
|
||||
&None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
|
||||
@@ -98,7 +98,7 @@ use windmill_worker::{
|
||||
CARGO_REGISTRIES, INSTANCE_PYTHON_VERSION, JAVA_HOME_DIR, JOB_DEFAULT_TIMEOUT, JOB_ISOLATION,
|
||||
KEEP_JOB_DIR, MAVEN_REPOS, MAVEN_SETTINGS_XML, NO_DEFAULT_MAVEN, NPMRC, NPM_CONFIG_REGISTRY,
|
||||
NSJAIL_AVAILABLE, NUGET_CONFIG, OTEL_TRACING_PROXY_SETTINGS, PIP_EXTRA_INDEX_URL,
|
||||
PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, UNSHARE_PATH, UV_INDEX_STRATEGY,
|
||||
PIP_INDEX_URL, POWERSHELL_REPO_PAT, POWERSHELL_REPO_URL, UV_INDEX_STRATEGY,
|
||||
WORKSPACE_REGISTRIES,
|
||||
};
|
||||
|
||||
@@ -1701,12 +1701,6 @@ pub async fn reload_job_isolation_setting(conn: &Connection) {
|
||||
All jobs will fail until nsjail is installed or the setting is changed."
|
||||
);
|
||||
}
|
||||
if value == JobIsolationLevel::Unshare && UNSHARE_PATH.is_none() {
|
||||
tracing::error!(
|
||||
"job_isolation is set to unshare but the unshare binary is not available on this worker. \
|
||||
Jobs will run without isolation until unshare is installed or the setting is changed."
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn reload_request_size(conn: &Connection) {
|
||||
|
||||
@@ -921,7 +921,6 @@ mod dedicated_worker_protocol {
|
||||
"test-workspace",
|
||||
"f/test/script",
|
||||
LoaderMode::Node,
|
||||
&None,
|
||||
))
|
||||
.expect("build_loader failed");
|
||||
|
||||
@@ -1300,7 +1299,6 @@ mod bun_builder_tests {
|
||||
// Write build.js using the loader and builder constants directly
|
||||
// Parameters are dummy values since tests don't use Windmill relative imports
|
||||
let loader = RELATIVE_BUN_LOADER
|
||||
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", "{}")
|
||||
.replace("W_ID", "test-workspace")
|
||||
.replace("BASE_INTERNAL_URL", "http://localhost:8000")
|
||||
.replace("TOKEN", "test-token")
|
||||
|
||||
@@ -48,7 +48,6 @@ fn flow_module(id: &str, value: FlowModuleValue) -> FlowModule {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -32,7 +32,6 @@ mod prewarmed_isolate_tests {
|
||||
"test-workspace",
|
||||
"f/test/script",
|
||||
LoaderMode::BrowserBundle,
|
||||
&None,
|
||||
)
|
||||
.await
|
||||
.expect("build_loader failed");
|
||||
|
||||
@@ -209,7 +209,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
FlowModule {
|
||||
id: "b".to_string(),
|
||||
@@ -256,7 +255,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
}],
|
||||
modules_node: None,
|
||||
}
|
||||
@@ -277,7 +275,6 @@ async fn test_deno_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
],
|
||||
same_worker: false,
|
||||
@@ -392,7 +389,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
FlowModule {
|
||||
id: "b".to_string(),
|
||||
@@ -448,7 +444,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
FlowModule {
|
||||
id: "e".to_string(),
|
||||
@@ -490,7 +485,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
],
|
||||
modules_node: None,
|
||||
@@ -511,7 +505,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
FlowModule {
|
||||
id: "c".to_string(),
|
||||
@@ -559,7 +552,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
],
|
||||
same_worker: true,
|
||||
|
||||
@@ -33,10 +33,8 @@ use sqlx::{FromRow, Postgres, Transaction};
|
||||
use windmill_audit::audit_oss::{audit_log, AuditAuthorable};
|
||||
use windmill_audit::ActionKind;
|
||||
use windmill_common::assets::{clear_static_asset_usage, AssetUsageKind};
|
||||
use windmill_common::flows::FlowModule;
|
||||
use windmill_common::min_version::{
|
||||
MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2,
|
||||
MIN_VERSION_SUPPORTS_NODE_DEBOUNCING,
|
||||
};
|
||||
use windmill_common::runnable_settings::RunnableSettingsTrait;
|
||||
use windmill_common::utils::query_elems_from_hub;
|
||||
@@ -1578,54 +1576,30 @@ async fn archive_flow_by_path(
|
||||
/// Validates that flow debouncing configuration is supported by all workers
|
||||
/// Returns an error if debouncing is configured but workers are behind required version
|
||||
async fn guard_flow_from_debounce_data(nf: &NewFlow) -> Result<()> {
|
||||
let flow_value = nf.parse_flow_value()?;
|
||||
|
||||
if !MIN_VERSION_SUPPORTS_DEBOUNCING.met().await && !flow_value.debouncing_settings.is_default()
|
||||
if !MIN_VERSION_SUPPORTS_DEBOUNCING.met().await
|
||||
&& !nf.parse_flow_value()?.debouncing_settings.is_default()
|
||||
{
|
||||
tracing::warn!(
|
||||
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
|
||||
);
|
||||
return Err(Error::WorkersAreBehind {
|
||||
feature: "Debouncing".into(),
|
||||
min_version: "1.566.0".into(),
|
||||
});
|
||||
}
|
||||
|
||||
if !MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
|
||||
&& !flow_value.debouncing_settings.is_legacy_compatible()
|
||||
Err(Error::WorkersAreBehind { feature: "Debouncing".into(), min_version: "1.566.0".into() })
|
||||
} else if !MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
|
||||
&& !nf
|
||||
.parse_flow_value()?
|
||||
.debouncing_settings
|
||||
.is_legacy_compatible()
|
||||
&& !*WMDEBUG_FORCE_NO_LEGACY_DEBOUNCING_COMPAT
|
||||
{
|
||||
tracing::warn!(
|
||||
"Flow debouncing configuration rejected: workers are behind minimum required version for debouncing feature"
|
||||
);
|
||||
return Err(Error::WorkersAreBehind {
|
||||
Err(Error::WorkersAreBehind {
|
||||
feature: "V2 Debouncing".into(),
|
||||
min_version: "1.597.0".into(),
|
||||
});
|
||||
}
|
||||
|
||||
// Check node-level debouncing on all modules (including nested branches/loops)
|
||||
let mut has_node_debouncing = false;
|
||||
let check_result = FlowModule::traverse_modules(&flow_value.modules, &mut |m| {
|
||||
if m.debouncing
|
||||
.as_ref()
|
||||
.is_some_and(|d| d.debounce_delay_s.is_some_and(|s| s > 0))
|
||||
{
|
||||
has_node_debouncing = true;
|
||||
}
|
||||
})
|
||||
} else {
|
||||
Ok(())
|
||||
});
|
||||
if let Err(e) = check_result {
|
||||
tracing::warn!("Failed to traverse flow modules for debounce guard: {e}");
|
||||
}
|
||||
if has_node_debouncing && !MIN_VERSION_SUPPORTS_NODE_DEBOUNCING.met().await {
|
||||
return Err(Error::WorkersAreBehind {
|
||||
feature: "Flow node debouncing".into(),
|
||||
min_version: "1.658.0".into(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
@@ -1794,7 +1768,6 @@ mod tests {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
FlowModule {
|
||||
id: "b".to_string(),
|
||||
@@ -1828,7 +1801,6 @@ mod tests {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
FlowModule {
|
||||
id: "c".to_string(),
|
||||
@@ -1862,7 +1834,6 @@ mod tests {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
},
|
||||
],
|
||||
failure_module: Some(Box::new(FlowModule {
|
||||
@@ -1895,7 +1866,6 @@ mod tests {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
})),
|
||||
preprocessor_module: None,
|
||||
same_worker: false,
|
||||
|
||||
@@ -23,5 +23,4 @@ serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
serde_yml.workspace = true
|
||||
sqlx.workspace = true
|
||||
tracing.workspace = true
|
||||
url.workspace = true
|
||||
|
||||
@@ -164,7 +164,6 @@ pub struct FuturePath {
|
||||
summary: Option<String>,
|
||||
description: Option<String>,
|
||||
security_scheme: Option<SecurityScheme>,
|
||||
args_schema: Option<Value>,
|
||||
}
|
||||
|
||||
impl FuturePath {
|
||||
@@ -175,17 +174,8 @@ impl FuturePath {
|
||||
summary: Option<String>,
|
||||
description: Option<String>,
|
||||
security_scheme: Option<SecurityScheme>,
|
||||
args_schema: Option<Value>,
|
||||
) -> FuturePath {
|
||||
FuturePath {
|
||||
route_path,
|
||||
kind,
|
||||
request_type,
|
||||
summary,
|
||||
description,
|
||||
security_scheme,
|
||||
args_schema,
|
||||
}
|
||||
FuturePath { route_path, kind, request_type, summary, description, security_scheme }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -407,21 +397,7 @@ fn generate_paths(
|
||||
);
|
||||
|
||||
if method != Method::GET {
|
||||
if let Some(schema) = &path.args_schema {
|
||||
method_map.insert(
|
||||
"requestBody",
|
||||
serde_json::json!({
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": schema
|
||||
}
|
||||
}
|
||||
}),
|
||||
);
|
||||
} else {
|
||||
method_map.insert("requestBody", generate_default_request());
|
||||
}
|
||||
method_map.insert("requestBody", generate_default_request());
|
||||
} else if is_webhook {
|
||||
method_map.insert(
|
||||
"parameters",
|
||||
@@ -679,76 +655,6 @@ struct GenerateOpenAPI {
|
||||
openapi_spec_format: Format,
|
||||
}
|
||||
|
||||
fn clean_schema_for_openapi(schema: Value) -> Value {
|
||||
if let Value::Object(mut obj) = schema {
|
||||
obj.remove("$schema");
|
||||
obj.remove("order");
|
||||
Value::Object(obj)
|
||||
} else {
|
||||
schema
|
||||
}
|
||||
}
|
||||
|
||||
async fn get_runnable_schema(
|
||||
db: &DB,
|
||||
w_id: &str,
|
||||
script_path: &str,
|
||||
is_flow: bool,
|
||||
) -> Option<Value> {
|
||||
if is_flow {
|
||||
let row = sqlx::query!(
|
||||
r#"SELECT
|
||||
f.schema AS "schema: serde_json::Value",
|
||||
fv.value->>'preprocessor_module' IS NOT NULL AS "has_preprocessor: bool"
|
||||
FROM flow f
|
||||
LEFT JOIN flow_version fv ON fv.id = f.versions[array_length(f.versions, 1)]
|
||||
AND fv.workspace_id = f.workspace_id
|
||||
WHERE f.path = $1 AND f.workspace_id = $2 AND NOT f.archived"#,
|
||||
script_path,
|
||||
w_id,
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
tracing::warn!("Failed to fetch flow schema for {script_path}: {e}");
|
||||
e
|
||||
})
|
||||
.ok()
|
||||
.flatten()?;
|
||||
|
||||
if row.has_preprocessor.unwrap_or(false) {
|
||||
return None;
|
||||
}
|
||||
row.schema.map(clean_schema_for_openapi)
|
||||
} else {
|
||||
let row = sqlx::query!(
|
||||
r#"SELECT
|
||||
schema AS "schema: serde_json::Value",
|
||||
has_preprocessor
|
||||
FROM script
|
||||
WHERE path = $1 AND workspace_id = $2
|
||||
AND NOT archived AND NOT deleted
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 1"#,
|
||||
script_path,
|
||||
w_id,
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
tracing::warn!("Failed to fetch script schema for {script_path}: {e}");
|
||||
e
|
||||
})
|
||||
.ok()
|
||||
.flatten()?;
|
||||
|
||||
if row.has_preprocessor.unwrap_or(false) {
|
||||
return None;
|
||||
}
|
||||
row.schema.map(clean_schema_for_openapi)
|
||||
}
|
||||
}
|
||||
|
||||
async fn http_routes_to_future_paths(
|
||||
db: &DB,
|
||||
user_db: UserDB,
|
||||
@@ -785,9 +691,6 @@ async fn http_routes_to_future_paths(
|
||||
description: Option<String>,
|
||||
authentication_method: AuthenticationMethod,
|
||||
authentication_resource_path: Option<String>,
|
||||
script_path: String,
|
||||
is_flow: bool,
|
||||
wrap_body: bool,
|
||||
}
|
||||
|
||||
http_routes = sqlx::query_as!(
|
||||
@@ -801,10 +704,7 @@ async fn http_routes_to_future_paths(
|
||||
summary,
|
||||
description,
|
||||
authentication_method AS "authentication_method: _",
|
||||
authentication_resource_path,
|
||||
script_path,
|
||||
is_flow,
|
||||
wrap_body
|
||||
authentication_resource_path
|
||||
FROM
|
||||
http_trigger
|
||||
WHERE
|
||||
@@ -864,12 +764,6 @@ async fn http_routes_to_future_paths(
|
||||
HttpMethod::Delete => Method::DELETE,
|
||||
};
|
||||
|
||||
let args_schema = if !http_route.wrap_body {
|
||||
get_runnable_schema(db, w_id, &http_route.script_path, http_route.is_flow).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let future_path = FuturePath::new(
|
||||
route_path,
|
||||
Kind::HttpRoute(HttpRouteConfig::new(method)),
|
||||
@@ -877,7 +771,6 @@ async fn http_routes_to_future_paths(
|
||||
http_route.summary,
|
||||
http_route.description,
|
||||
auth_method,
|
||||
args_schema,
|
||||
);
|
||||
|
||||
openapi_future_paths.push(future_path);
|
||||
@@ -887,7 +780,6 @@ async fn http_routes_to_future_paths(
|
||||
}
|
||||
|
||||
async fn webhook_to_future_paths(
|
||||
db: &DB,
|
||||
pg_pool: &mut PgConnection,
|
||||
webhook_filters: Option<&[WebhookFilter]>,
|
||||
w_id: &str,
|
||||
@@ -913,7 +805,7 @@ async fn webhook_to_future_paths(
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[derive(Debug, Deserialize, Clone, Hash)]
|
||||
struct MinifiedWebhook {
|
||||
path: String,
|
||||
description: Option<String>,
|
||||
@@ -922,7 +814,7 @@ async fn webhook_to_future_paths(
|
||||
|
||||
let webhook_scripts = sqlx::query_as!(
|
||||
MinifiedWebhook,
|
||||
r#"SELECT
|
||||
r#"SELECT
|
||||
path,
|
||||
summary,
|
||||
description
|
||||
@@ -941,7 +833,7 @@ async fn webhook_to_future_paths(
|
||||
|
||||
let webhook_flows = sqlx::query_as!(
|
||||
MinifiedWebhook,
|
||||
r#"SELECT
|
||||
r#"SELECT
|
||||
path,
|
||||
summary,
|
||||
description
|
||||
@@ -961,7 +853,6 @@ async fn webhook_to_future_paths(
|
||||
openapi_future_paths.reserve_exact(webhook_scripts.len() + webhook_flows.len());
|
||||
|
||||
for webhook in webhook_scripts {
|
||||
let args_schema = get_runnable_schema(db, w_id, &webhook.path, false).await;
|
||||
openapi_future_paths.push(FuturePath::new(
|
||||
webhook.path,
|
||||
Kind::Webhook(WebhookConfig::new(RunnableKind::Script)),
|
||||
@@ -969,12 +860,10 @@ async fn webhook_to_future_paths(
|
||||
webhook.summary,
|
||||
webhook.description,
|
||||
Some(SecurityScheme::BearerJwt),
|
||||
args_schema,
|
||||
));
|
||||
}
|
||||
|
||||
for webhook in webhook_flows {
|
||||
let args_schema = get_runnable_schema(db, w_id, &webhook.path, true).await;
|
||||
openapi_future_paths.push(FuturePath::new(
|
||||
webhook.path,
|
||||
Kind::Webhook(WebhookConfig::new(RunnableKind::Flow)),
|
||||
@@ -982,7 +871,6 @@ async fn webhook_to_future_paths(
|
||||
webhook.summary,
|
||||
webhook.description,
|
||||
Some(SecurityScheme::BearerJwt),
|
||||
args_schema,
|
||||
));
|
||||
}
|
||||
}
|
||||
@@ -1010,7 +898,7 @@ async fn generate_openapi_future_path(
|
||||
http_routes_to_future_paths(db, user_db, authed, &mut tx, http_route_filters, w_id).await?;
|
||||
|
||||
openapi_future_paths
|
||||
.append(&mut webhook_to_future_paths(db, &mut tx, webhook_filters, w_id).await?);
|
||||
.append(&mut webhook_to_future_paths(&mut tx, webhook_filters, w_id).await?);
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
|
||||
@@ -239,9 +239,6 @@ pub fn workspaced_service() -> Router {
|
||||
"/history_update/h/:hash/p/*path",
|
||||
post(update_script_history),
|
||||
)
|
||||
// Temporary raw script storage for CLI lock generation
|
||||
.route("/raw_temp/store", post(store_raw_script_temp))
|
||||
.route("/raw_temp/diff", post(diff_raw_scripts_with_deployed))
|
||||
}
|
||||
|
||||
#[derive(Serialize, FromRow)]
|
||||
@@ -1617,9 +1614,6 @@ struct RawScriptByPathQuery {
|
||||
cache_key: Option<String>,
|
||||
// used specifically for python to cache folders on import success to avoid extra db calls on package fetch
|
||||
cache_folders: Option<bool>,
|
||||
// If provided, load content from raw_script_temp table using this hash instead of deployed script.
|
||||
// Used by CLI lock generation to resolve imports from not-yet-deployed scripts.
|
||||
temp_script_hash: Option<String>,
|
||||
}
|
||||
|
||||
struct StringWithLength(String);
|
||||
@@ -1678,16 +1672,6 @@ async fn raw_script_by_path_internal(
|
||||
) -> Result<String> {
|
||||
let path = path.to_path();
|
||||
check_scopes(&authed, || format!("scripts:read:{}", path))?;
|
||||
|
||||
// If temp_script_hash is provided, try loading from temp storage first.
|
||||
// This is used by CLI lock generation to resolve imports from not-yet-deployed scripts.
|
||||
// Falls back to the normal deployed script lookup if not found in temp storage.
|
||||
if let Some(hash) = query.temp_script_hash {
|
||||
if let Ok(content) = windmill_common::cache::raw_script_temp::load(hash, &db).await {
|
||||
return Ok(content);
|
||||
}
|
||||
}
|
||||
|
||||
let cache_path = query
|
||||
.cache_key
|
||||
.map(|x| format!("{w_id}:{path}:{x}{}", if unpin { ":unpinned" } else { "" }));
|
||||
@@ -2479,129 +2463,3 @@ async fn guard_script_from_debounce_data(ns: &NewScript) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Temporary Raw Script Storage for CLI Lock Generation
|
||||
// ============================================================================
|
||||
|
||||
/// Store raw script content temporarily for CLI lock generation.
|
||||
async fn store_raw_script_temp(
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path(w_id): Path<String>,
|
||||
Json(content): Json<String>,
|
||||
) -> Result<Json<String>> {
|
||||
check_scopes(&authed, || "scripts:write".to_string())?;
|
||||
|
||||
let hash = windmill_common::cache::raw_script_temp::compute_hash(&w_id, &content);
|
||||
|
||||
// Store to DB
|
||||
sqlx::query!(
|
||||
"INSERT INTO raw_script_temp (workspace_id, hash, content, created_at)
|
||||
VALUES ($1, $2, $3, NOW())
|
||||
ON CONFLICT (workspace_id, hash) DO UPDATE SET created_at = NOW()",
|
||||
&w_id,
|
||||
&hash,
|
||||
&content
|
||||
)
|
||||
.execute(&db)
|
||||
.await?;
|
||||
|
||||
// Clean up old entries (1 week TTL)
|
||||
sqlx::query!(
|
||||
"DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'"
|
||||
)
|
||||
.execute(&db)
|
||||
.await?;
|
||||
|
||||
Ok(Json(hash))
|
||||
}
|
||||
|
||||
/// Compare local script content hashes with deployed versions.
|
||||
/// Receives a map of path → SHA256(content), returns paths where the hash
|
||||
/// differs from the deployed script (or the script doesn't exist on remote).
|
||||
/// Hash comparison is done entirely in Postgres to avoid transferring content.
|
||||
#[derive(Deserialize)]
|
||||
struct WorkspaceDepDiff {
|
||||
path: String,
|
||||
language: ScriptLang,
|
||||
name: Option<String>,
|
||||
hash: String,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct DiffRequest {
|
||||
scripts: std::collections::HashMap<String, String>,
|
||||
#[serde(default)]
|
||||
workspace_deps: Vec<WorkspaceDepDiff>,
|
||||
}
|
||||
|
||||
async fn diff_raw_scripts_with_deployed(
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path(w_id): Path<String>,
|
||||
Json(req): Json<DiffRequest>,
|
||||
) -> Result<Json<Vec<String>>> {
|
||||
check_scopes(&authed, || "scripts:read".to_string())?;
|
||||
|
||||
let mut matching_set: std::collections::HashSet<String> = std::collections::HashSet::new();
|
||||
let mut all_paths: Vec<String> = Vec::new();
|
||||
|
||||
// --- Scripts ---
|
||||
if !req.scripts.is_empty() {
|
||||
let paths: Vec<String> = req.scripts.keys().cloned().collect();
|
||||
let hashes: Vec<String> = paths.iter().map(|p| req.scripts[p].clone()).collect();
|
||||
|
||||
let matching: Vec<String> = sqlx::query_scalar(
|
||||
"SELECT local.path FROM \
|
||||
unnest($1::text[], $2::text[]) AS local(path, hash) \
|
||||
INNER JOIN LATERAL ( \
|
||||
SELECT encode(sha256(convert_to(s.content, 'UTF8')), 'hex') AS deployed_hash \
|
||||
FROM script s \
|
||||
WHERE s.path = local.path AND s.workspace_id = $3 AND s.archived = false \
|
||||
ORDER BY s.created_at DESC LIMIT 1 \
|
||||
) deployed ON deployed.deployed_hash = local.hash"
|
||||
)
|
||||
.bind(&paths)
|
||||
.bind(&hashes)
|
||||
.bind(&w_id)
|
||||
.fetch_all(&db)
|
||||
.await?;
|
||||
|
||||
matching_set.extend(matching);
|
||||
all_paths.extend(paths);
|
||||
}
|
||||
|
||||
// --- Workspace dependencies ---
|
||||
for dep in &req.workspace_deps {
|
||||
let matching: Option<String> = sqlx::query_scalar(
|
||||
"SELECT $1::text \
|
||||
WHERE EXISTS ( \
|
||||
SELECT 1 FROM workspace_dependencies wd \
|
||||
WHERE wd.workspace_id = $2 AND wd.archived = false \
|
||||
AND wd.language = $3::SCRIPT_LANG \
|
||||
AND wd.name IS NOT DISTINCT FROM $4 \
|
||||
AND encode(sha256(convert_to(wd.content, 'UTF8')), 'hex') = $5 \
|
||||
)"
|
||||
)
|
||||
.bind(&dep.path)
|
||||
.bind(&w_id)
|
||||
.bind(dep.language.as_str())
|
||||
.bind(&dep.name)
|
||||
.bind(&dep.hash)
|
||||
.fetch_optional(&db)
|
||||
.await?;
|
||||
|
||||
if let Some(path) = matching {
|
||||
matching_set.insert(path);
|
||||
}
|
||||
all_paths.push(dep.path.clone());
|
||||
}
|
||||
|
||||
let mismatched: Vec<String> = all_paths
|
||||
.into_iter()
|
||||
.filter(|p| !matching_set.contains(p))
|
||||
.collect();
|
||||
|
||||
Ok(Json(mismatched))
|
||||
}
|
||||
|
||||
@@ -6988,83 +6988,6 @@ paths:
|
||||
type: string
|
||||
format: uuid
|
||||
|
||||
/w/{workspace}/scripts/raw_temp/store:
|
||||
post:
|
||||
summary: store raw script content temporarily for CLI lock generation
|
||||
operationId: storeRawScriptTemp
|
||||
tags:
|
||||
- script
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
requestBody:
|
||||
description: script content to store
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: hash of stored content
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: string
|
||||
|
||||
/w/{workspace}/scripts/raw_temp/diff:
|
||||
post:
|
||||
summary: diff local script hashes against deployed versions
|
||||
operationId: diffRawScriptsWithDeployed
|
||||
tags:
|
||||
- script
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
requestBody:
|
||||
description: scripts and workspace deps to diff against deployed versions
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
required:
|
||||
- scripts
|
||||
properties:
|
||||
scripts:
|
||||
description: map of script path to SHA256 content hash
|
||||
type: object
|
||||
additionalProperties:
|
||||
type: string
|
||||
workspace_deps:
|
||||
description: workspace dependencies to diff
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
required:
|
||||
- path
|
||||
- language
|
||||
- hash
|
||||
properties:
|
||||
path:
|
||||
description: CLI path (e.g. dependencies/package.json)
|
||||
type: string
|
||||
language:
|
||||
$ref: "#/components/schemas/ScriptLang"
|
||||
name:
|
||||
description: named workspace dependency (null for default)
|
||||
type: string
|
||||
hash:
|
||||
description: SHA256 content hash
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: list of paths that differ from deployed versions
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
|
||||
/w/{workspace}/jobs/list_selected_job_groups:
|
||||
# We use post because sending a huge array as a query param can produce
|
||||
# URLs that may be too long
|
||||
|
||||
@@ -2053,7 +2053,6 @@ async fn execute_component(
|
||||
.triggerables_v2
|
||||
.as_ref()
|
||||
.ok_or_else(|| Error::BadRequest(format!("Policy is missing triggerables")))?;
|
||||
|
||||
let policy_triggerables = triggerables_v2
|
||||
.get(path) // start with `path` in case we can avoid the next` format!`.
|
||||
.or_else(|| triggerables_v2.get(&format!("{}:{}", payload.component, &path)))
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
use crate::db::ApiAuthed;
|
||||
use axum::{extract::Path, routing::post, Json, Router};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use windmill_common::{
|
||||
error::{Error, Result},
|
||||
query_builders::try_expand_internal_db_query,
|
||||
scripts::ScriptLang,
|
||||
};
|
||||
|
||||
pub fn workspaced_service() -> Router {
|
||||
Router::new().route("/expand_marker", post(expand_marker))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct ExpandMarkerRequest {
|
||||
language: ScriptLang,
|
||||
content: String,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ExpandMarkerResponse {
|
||||
code: String,
|
||||
}
|
||||
|
||||
async fn expand_marker(
|
||||
_authed: ApiAuthed,
|
||||
Path(_w_id): Path<String>,
|
||||
Json(req): Json<ExpandMarkerRequest>,
|
||||
) -> Result<Json<ExpandMarkerResponse>> {
|
||||
match try_expand_internal_db_query(&req.content, &req.language) {
|
||||
Some(Ok(expanded)) => Ok(Json(ExpandMarkerResponse { code: expanded.code })),
|
||||
Some(Err(msg)) => Err(Error::BadRequest(msg)),
|
||||
None => Err(Error::BadRequest(
|
||||
"Content is not a WM_INTERNAL_DB marker".to_string(),
|
||||
)),
|
||||
}
|
||||
}
|
||||
@@ -5074,10 +5074,6 @@ pub struct RunDependenciesRequest {
|
||||
pub raw_workspace_dependencies: Option<RawWorkspaceDependencies>,
|
||||
#[serde(default)]
|
||||
pub raw_deps: Option<String>,
|
||||
/// Map of script path -> content hash for resolving imports from temp storage.
|
||||
/// Used by CLI to provide local script content during lock generation.
|
||||
#[serde(default)]
|
||||
pub temp_script_refs: Option<HashMap<String, String>>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, Clone, Debug)]
|
||||
@@ -5137,8 +5133,6 @@ async fn run_dependencies_job(
|
||||
let mut hm = HashMap::new();
|
||||
req.raw_workspace_dependencies
|
||||
.map(|v| hm.insert("raw_workspace_dependencies".to_owned(), to_raw_value(&v)));
|
||||
req.temp_script_refs
|
||||
.map(|v| hm.insert("temp_script_refs".to_owned(), to_raw_value(&v)));
|
||||
|
||||
let (uuid, tx) = push(
|
||||
&db,
|
||||
@@ -5189,8 +5183,6 @@ pub struct RunFlowDependenciesRequest {
|
||||
pub raw_workspace_dependencies: Option<RawWorkspaceDependencies>,
|
||||
#[serde(default)]
|
||||
pub raw_deps: Option<HashMap<String, String>>,
|
||||
#[serde(default)]
|
||||
pub temp_script_refs: Option<HashMap<String, String>>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
@@ -5234,10 +5226,6 @@ async fn run_flow_dependencies_job(
|
||||
req.raw_workspace_dependencies
|
||||
.map(|v| args_map.insert("raw_workspace_dependencies".to_string(), to_raw_value(&v)));
|
||||
|
||||
// Add temp_script_refs to args if present (for CLI local import resolution)
|
||||
req.temp_script_refs
|
||||
.map(|v| args_map.insert("temp_script_refs".to_string(), to_raw_value(&v)));
|
||||
|
||||
let (uuid, tx) = push(
|
||||
&db,
|
||||
PushIsolationLevel::IsolatedRoot(db.clone()),
|
||||
|
||||
@@ -98,7 +98,6 @@ mod indexer_oss;
|
||||
mod inkeep_ee;
|
||||
mod inkeep_oss;
|
||||
mod integration;
|
||||
mod internal_db;
|
||||
mod live_migrations;
|
||||
#[cfg(all(feature = "private", feature = "parquet"))]
|
||||
pub mod s3_proxy_ee;
|
||||
@@ -555,7 +554,6 @@ pub async fn run_server(
|
||||
.nest("/groups", groups::workspaced_service())
|
||||
.nest("/groups_history", group_history::workspaced_service())
|
||||
.nest("/inputs", windmill_api_inputs::workspaced_service())
|
||||
.nest("/internal_db", internal_db::workspaced_service())
|
||||
.nest("/job_metrics", job_metrics::workspaced_service())
|
||||
.nest("/job_helpers", job_helpers_service)
|
||||
.nest("/jobs", jobs::workspaced_service())
|
||||
|
||||
@@ -991,38 +991,6 @@ pub mod workspace_dependencies {
|
||||
}
|
||||
}
|
||||
|
||||
/// Temporary raw script content cache for CLI lock generation.
|
||||
pub mod raw_script_temp {
|
||||
use super::*;
|
||||
use crate::DB;
|
||||
|
||||
make_static! {
|
||||
static ref CACHE: { String => String } in "raw_script_temp" <= 10000;
|
||||
}
|
||||
|
||||
/// Compute hash for raw script content (includes workspace_id for isolation).
|
||||
pub fn compute_hash(workspace_id: &str, content: &str) -> String {
|
||||
use sha2::{Digest, Sha256};
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(workspace_id.as_bytes());
|
||||
hasher.update(content.as_bytes());
|
||||
format!("{:x}", hasher.finalize())
|
||||
}
|
||||
|
||||
/// Load content from cache, falling back to DB.
|
||||
pub fn load(hash: String, db: &DB) -> impl Future<Output = error::Result<String>> + '_ {
|
||||
CACHE.get_or_insert_async(hash.clone(), async move {
|
||||
sqlx::query_scalar!(
|
||||
"SELECT content FROM raw_script_temp WHERE hash = $1",
|
||||
&hash
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await?
|
||||
.ok_or_else(|| error::Error::NotFound(format!("raw_script_temp hash: {}", hash)))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const _: () = {
|
||||
impl Import for RawFlow {
|
||||
fn import(src: &impl Storage) -> error::Result<Self> {
|
||||
@@ -1215,8 +1183,7 @@ const _: () = {
|
||||
((u8, ScriptHash), |x| format!("{:02x}-{:016x}", x.0, x.1.0)),
|
||||
(FlowNodeId, |x| format!("{:016x}", x.0)),
|
||||
(AppScriptId, |x| format!("{:016x}", x.0)),
|
||||
((i64, String), |x| format!("{}-{}", x.1, x.0)),
|
||||
(String, |x| x.as_str())
|
||||
((i64, String), |x| format!("{}-{}", x.1, x.0))
|
||||
}
|
||||
|
||||
#[cfg(feature = "scoped_cache")]
|
||||
|
||||
@@ -77,7 +77,6 @@ pub mod oidc_oss;
|
||||
#[cfg(feature = "private")]
|
||||
pub mod otel_ee;
|
||||
pub mod otel_oss;
|
||||
pub mod query_builders;
|
||||
pub mod queue;
|
||||
pub mod result_stream;
|
||||
pub mod runnable_settings;
|
||||
|
||||
@@ -5,7 +5,6 @@ use tokio::sync::RwLock;
|
||||
|
||||
// ============ Feature Definitions ============
|
||||
|
||||
pub const MIN_VERSION_SUPPORTS_NODE_DEBOUNCING: VC = vc(1, 658, 0, "Flow node debouncing");
|
||||
pub const MIN_VERSION_SUPPORTS_TOKEN_HASH: VC = vc(1, 659, 0, "Token hash storage");
|
||||
pub const MIN_VERSION_SUPPORTS_SYNC_JOBS_DEBOUNCING: VC = vc(1, 602, 0, "Sync jobs debouncing");
|
||||
pub const MIN_VERSION_SUPPORTS_DEBOUNCING_V2: VC = vc(1, 597, 0, "Debouncing V2");
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -68,11 +68,7 @@ pub fn initialize_tracing(
|
||||
mode: &Mode,
|
||||
environment: &str,
|
||||
) -> (WorkerGuard, crate::otel_oss::OtelProvider) {
|
||||
let style = if std::env::var("NO_COLOR").is_ok() {
|
||||
"never".into()
|
||||
} else {
|
||||
std::env::var("RUST_LOG_STYLE").unwrap_or_else(|_| "auto".into())
|
||||
};
|
||||
let style = std::env::var("RUST_LOG_STYLE").unwrap_or_else(|_| "auto".into());
|
||||
|
||||
let rust_log_env = std::env::var("RUST_LOG");
|
||||
let rust_log_stdout_env = std::env::var("RUST_LOG_STDOUT");
|
||||
|
||||
@@ -65,8 +65,8 @@ pub mod object_store_reexports {
|
||||
pub use object_store::memory::InMemory;
|
||||
pub use object_store::path::Path;
|
||||
pub use object_store::{
|
||||
Attribute, Attributes, Error as ObjectStoreError, GetResult, ObjectStore, PutMultipartOpts,
|
||||
PutPayload, PutResult, Result as ObjectStoreResult, WriteMultipart,
|
||||
Attribute, Attributes, Error as ObjectStoreError, GetResult, ObjectStore,
|
||||
PutMultipartOpts, PutPayload, PutResult, Result as ObjectStoreResult, WriteMultipart,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -530,7 +530,10 @@ async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result<Arc<d
|
||||
#[cfg(feature = "parquet")]
|
||||
pub fn build_filesystem_client(root_path: &str) -> error::Result<Arc<dyn ObjectStore>> {
|
||||
let store = object_store::local::LocalFileSystem::new_with_prefix(root_path).map_err(|e| {
|
||||
error::Error::internal_err(format!("Error building filesystem object store: {:?}", e))
|
||||
error::Error::internal_err(format!(
|
||||
"Error building filesystem object store: {:?}",
|
||||
e
|
||||
))
|
||||
})?;
|
||||
Ok(Arc::new(store))
|
||||
}
|
||||
@@ -641,45 +644,36 @@ lazy_static::lazy_static! {
|
||||
static ref S3_BUCKET_RESTRICTIONS: Option<HashMap<String, Vec<String>>> = {
|
||||
parse_bucket_restrictions()
|
||||
};
|
||||
static ref AZ_ACCOUNT_NAME_RESTRICTIONS: Option<HashMap<String, Vec<String>>> = {
|
||||
parse_az_account_name_restrictions()
|
||||
};
|
||||
}
|
||||
|
||||
fn parse_bucket_restrictions() -> Option<HashMap<String, Vec<String>>> {
|
||||
let env_var = std::env::var("S3_BUCKETS_WORKSPACE_RESTRICTIONS").ok()?;
|
||||
parse_restrictions_from_str(&env_var, "S3 bucket")
|
||||
parse_bucket_restrictions_from_str(&env_var)
|
||||
}
|
||||
|
||||
fn parse_az_account_name_restrictions() -> Option<HashMap<String, Vec<String>>> {
|
||||
let env_var = std::env::var("AZ_ACCOUNT_NAME_WORKSPACE_RESTRICTIONS").ok()?;
|
||||
parse_restrictions_from_str(&env_var, "Azure account name")
|
||||
}
|
||||
|
||||
fn parse_restrictions_from_str(input: &str, label: &str) -> Option<HashMap<String, Vec<String>>> {
|
||||
fn parse_bucket_restrictions_from_str(input: &str) -> Option<HashMap<String, Vec<String>>> {
|
||||
if input.trim().is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut restrictions = HashMap::new();
|
||||
|
||||
for rule in input.split(';') {
|
||||
let rule = rule.trim();
|
||||
if rule.is_empty() {
|
||||
for bucket_rule in input.split(';') {
|
||||
let bucket_rule = bucket_rule.trim();
|
||||
if bucket_rule.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let parts: Vec<&str> = rule.splitn(2, ':').collect();
|
||||
let parts: Vec<&str> = bucket_rule.splitn(2, ':').collect();
|
||||
if parts.len() != 2 {
|
||||
tracing::warn!(
|
||||
"Invalid {} restriction format: '{}'. Expected 'name:workspace1,workspace2'",
|
||||
label,
|
||||
rule
|
||||
"Invalid bucket restriction format: '{}'. Expected 'bucket:workspace1,workspace2'",
|
||||
bucket_rule
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
let name = parts[0].trim().to_string();
|
||||
let bucket_name = parts[0].trim().to_string();
|
||||
let workspaces: Vec<String> = parts[1]
|
||||
.split(',')
|
||||
.map(|w| w.trim().to_string())
|
||||
@@ -688,33 +682,26 @@ fn parse_restrictions_from_str(input: &str, label: &str) -> Option<HashMap<Strin
|
||||
|
||||
if workspaces.is_empty() {
|
||||
tracing::warn!(
|
||||
"No workspaces specified for {} '{}', skipping restriction",
|
||||
label,
|
||||
name
|
||||
"No workspaces specified for bucket '{}', skipping restriction",
|
||||
bucket_name
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
restrictions.insert(name, workspaces);
|
||||
restrictions.insert(bucket_name, workspaces);
|
||||
}
|
||||
|
||||
if restrictions.is_empty() {
|
||||
None
|
||||
} else {
|
||||
tracing::info!(
|
||||
"{} restrictions loaded for {} entries",
|
||||
label,
|
||||
"S3 bucket restrictions loaded for {} buckets",
|
||||
restrictions.len()
|
||||
);
|
||||
Some(restrictions)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn parse_bucket_restrictions_from_str(input: &str) -> Option<HashMap<String, Vec<String>>> {
|
||||
parse_restrictions_from_str(input, "S3 bucket")
|
||||
}
|
||||
|
||||
pub fn check_bucket_workspace_restriction(
|
||||
bucket_name: &str,
|
||||
workspace_id: &str,
|
||||
@@ -732,23 +719,6 @@ pub fn check_bucket_workspace_restriction(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn check_az_account_name_workspace_restriction(
|
||||
account_name: &str,
|
||||
workspace_id: &str,
|
||||
) -> error::Result<()> {
|
||||
if let Some(ref restrictions) = *AZ_ACCOUNT_NAME_RESTRICTIONS {
|
||||
if let Some(allowed_workspaces) = restrictions.get(account_name) {
|
||||
if !allowed_workspaces.contains(&workspace_id.to_string()) {
|
||||
return Err(error::Error::NotAuthorized(format!(
|
||||
"Workspace '{}' is not authorized to access Azure account '{}'",
|
||||
workspace_id, account_name
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub const DEFAULT_STORAGE: &str = "_default_";
|
||||
|
||||
pub fn bundle(w_id: &str, hash: &str) -> String {
|
||||
@@ -769,8 +739,7 @@ pub async fn upload_artifact_to_store(
|
||||
#[cfg(not(all(feature = "enterprise", feature = "parquet")))]
|
||||
let object_store: Option<()> = None;
|
||||
Ok(
|
||||
if &windmill_common::utils::MODE_AND_ADDONS.mode
|
||||
== &windmill_common::utils::Mode::Standalone
|
||||
if &windmill_common::utils::MODE_AND_ADDONS.mode == &windmill_common::utils::Mode::Standalone
|
||||
&& object_store.is_none()
|
||||
{
|
||||
let path = format!("{}/{}", standalone_dir, path);
|
||||
@@ -852,11 +821,11 @@ pub fn lfs_to_object_store_resource(
|
||||
})?;
|
||||
Ok(ObjectStoreResource::Gcs(gcs_resource))
|
||||
}
|
||||
LargeFileStorage::FilesystemStorage(fs) => {
|
||||
Ok(ObjectStoreResource::Filesystem(FilesystemSettings {
|
||||
LargeFileStorage::FilesystemStorage(fs) => Ok(ObjectStoreResource::Filesystem(
|
||||
FilesystemSettings {
|
||||
root_path: fs.root_path.clone(),
|
||||
}))
|
||||
}
|
||||
},
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1026,9 +995,13 @@ pub async fn convert_json_line_stream<E: Into<anyhow::Error>>(
|
||||
drop(file);
|
||||
|
||||
let ctx = SessionContext::new();
|
||||
ctx.register_json("my_table", path_str, NdJsonReadOptions::default())
|
||||
.await
|
||||
.map_err(to_anyhow)?;
|
||||
ctx.register_json(
|
||||
"my_table",
|
||||
path_str,
|
||||
NdJsonReadOptions::default(),
|
||||
)
|
||||
.await
|
||||
.map_err(to_anyhow)?;
|
||||
|
||||
let df = ctx.sql("SELECT * FROM my_table").await.map_err(to_anyhow)?;
|
||||
let schema = df.schema().clone().into();
|
||||
@@ -1279,7 +1252,8 @@ mod tests {
|
||||
advanced_permissions: None,
|
||||
});
|
||||
// resource_value is ignored for filesystem
|
||||
let result = lfs_to_object_store_resource(&lfs, serde_json::Value::Null).unwrap();
|
||||
let result =
|
||||
lfs_to_object_store_resource(&lfs, serde_json::Value::Null).unwrap();
|
||||
match result {
|
||||
ObjectStoreResource::Filesystem(fs) => {
|
||||
assert_eq!(fs.root_path, "/tmp/mydata");
|
||||
@@ -1305,18 +1279,10 @@ mod tests {
|
||||
port: None,
|
||||
};
|
||||
let result = duckdb_connection_settings_internal(s3).unwrap();
|
||||
assert!(result
|
||||
.connection_settings_str
|
||||
.contains("SET s3_region='eu-west-1'"));
|
||||
assert!(result
|
||||
.connection_settings_str
|
||||
.contains("SET s3_access_key_id='AKIA123'"));
|
||||
assert!(result
|
||||
.connection_settings_str
|
||||
.contains("SET s3_secret_access_key='secret456'"));
|
||||
assert!(result
|
||||
.connection_settings_str
|
||||
.contains("SET s3_url_style='path'"));
|
||||
assert!(result.connection_settings_str.contains("SET s3_region='eu-west-1'"));
|
||||
assert!(result.connection_settings_str.contains("SET s3_access_key_id='AKIA123'"));
|
||||
assert!(result.connection_settings_str.contains("SET s3_secret_access_key='secret456'"));
|
||||
assert!(result.connection_settings_str.contains("SET s3_url_style='path'"));
|
||||
assert!(!result.connection_settings_str.contains("SET s3_use_ssl=0"));
|
||||
assert_eq!(result.s3_bucket, Some("test-bucket".to_string()));
|
||||
assert!(result.azure_container_path.is_none());
|
||||
@@ -1338,9 +1304,7 @@ mod tests {
|
||||
};
|
||||
let result = duckdb_connection_settings_internal(s3).unwrap();
|
||||
assert!(result.connection_settings_str.contains("SET s3_use_ssl=0"));
|
||||
assert!(!result
|
||||
.connection_settings_str
|
||||
.contains("SET s3_url_style='path'"));
|
||||
assert!(!result.connection_settings_str.contains("SET s3_url_style='path'"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1356,12 +1320,8 @@ mod tests {
|
||||
federated_token_file: None,
|
||||
});
|
||||
let result = format_duckdb_connection_settings(resource).unwrap();
|
||||
assert!(result
|
||||
.connection_settings_str
|
||||
.contains("AccountName=myaccount"));
|
||||
assert!(result
|
||||
.connection_settings_str
|
||||
.contains("AccountKey=base64key=="));
|
||||
assert!(result.connection_settings_str.contains("AccountName=myaccount"));
|
||||
assert!(result.connection_settings_str.contains("AccountKey=base64key=="));
|
||||
assert_eq!(
|
||||
result.azure_container_path,
|
||||
Some("az://mycontainer".to_string())
|
||||
@@ -1377,10 +1337,7 @@ mod tests {
|
||||
});
|
||||
let result = format_duckdb_connection_settings(resource);
|
||||
assert!(result.is_err());
|
||||
assert!(result
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("GCS is not supported"));
|
||||
assert!(result.unwrap_err().to_string().contains("GCS is not supported"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1404,8 +1361,10 @@ mod tests {
|
||||
use windmill_common::error::Error;
|
||||
|
||||
// NotFound
|
||||
let err =
|
||||
object_store::Error::NotFound { path: "test/path".into(), source: "missing".into() };
|
||||
let err = object_store::Error::NotFound {
|
||||
path: "test/path".into(),
|
||||
source: "missing".into(),
|
||||
};
|
||||
let mapped = object_store_error_to_error(err);
|
||||
assert!(matches!(mapped, Error::NotFound(_)));
|
||||
|
||||
@@ -1429,8 +1388,10 @@ mod tests {
|
||||
assert!(matches!(mapped, Error::BadRequest(_)));
|
||||
|
||||
// Unauthenticated
|
||||
let err =
|
||||
object_store::Error::Unauthenticated { path: "obj".into(), source: "no creds".into() };
|
||||
let err = object_store::Error::Unauthenticated {
|
||||
path: "obj".into(),
|
||||
source: "no creds".into(),
|
||||
};
|
||||
let mapped = object_store_error_to_error(err);
|
||||
assert!(matches!(mapped, Error::NotAuthorized(_)));
|
||||
}
|
||||
@@ -1572,10 +1533,15 @@ mod tests {
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let resource =
|
||||
ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root.to_string() });
|
||||
let s3_obj =
|
||||
S3Object { s3: "etag.txt".to_string(), storage: None, filename: None, presigned: None };
|
||||
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
|
||||
root_path: root.to_string(),
|
||||
});
|
||||
let s3_obj = S3Object {
|
||||
s3: "etag.txt".to_string(),
|
||||
storage: None,
|
||||
filename: None,
|
||||
presigned: None,
|
||||
};
|
||||
|
||||
let etag = get_etag_or_empty(&resource, s3_obj).await;
|
||||
// LocalFileSystem should return an etag based on file metadata
|
||||
@@ -1588,8 +1554,9 @@ mod tests {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let root = dir.path().to_str().unwrap();
|
||||
|
||||
let resource =
|
||||
ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root.to_string() });
|
||||
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
|
||||
root_path: root.to_string(),
|
||||
});
|
||||
let s3_obj = S3Object {
|
||||
s3: "nonexistent.txt".to_string(),
|
||||
storage: None,
|
||||
@@ -1608,7 +1575,9 @@ mod tests {
|
||||
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let root = dir.path().to_str().unwrap().to_string();
|
||||
let settings = ObjectSettings::Filesystem(FilesystemSettings { root_path: root });
|
||||
let settings = ObjectSettings::Filesystem(FilesystemSettings {
|
||||
root_path: root,
|
||||
});
|
||||
|
||||
let expirable = build_object_store_from_settings(settings, None)
|
||||
.await
|
||||
@@ -1616,7 +1585,10 @@ mod tests {
|
||||
let data = bytes::Bytes::from("end to end via settings");
|
||||
expirable
|
||||
.store
|
||||
.put(&Path::from("e2e.txt"), PutPayload::from(data.clone()))
|
||||
.put(
|
||||
&Path::from("e2e.txt"),
|
||||
PutPayload::from(data.clone()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let result = expirable
|
||||
@@ -1638,7 +1610,9 @@ mod tests {
|
||||
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let root = dir.path().to_str().unwrap().to_string();
|
||||
let resource = ObjectStoreResource::Filesystem(FilesystemSettings { root_path: root });
|
||||
let resource = ObjectStoreResource::Filesystem(FilesystemSettings {
|
||||
root_path: root,
|
||||
});
|
||||
|
||||
let client = build_object_store_client(&resource).await.unwrap();
|
||||
let data = bytes::Bytes::from("end to end via resource");
|
||||
@@ -1660,10 +1634,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_bundle_path_format() {
|
||||
assert_eq!(
|
||||
bundle("my_workspace", "abc123"),
|
||||
"script_bundle/my_workspace/abc123"
|
||||
);
|
||||
assert_eq!(bundle("my_workspace", "abc123"), "script_bundle/my_workspace/abc123");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1675,7 +1646,8 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_parse_bucket_restrictions_single_bucket() {
|
||||
let result = parse_bucket_restrictions_from_str("my-bucket:workspace1,workspace2").unwrap();
|
||||
let result =
|
||||
parse_bucket_restrictions_from_str("my-bucket:workspace1,workspace2").unwrap();
|
||||
assert_eq!(result.len(), 1);
|
||||
assert_eq!(
|
||||
result.get("my-bucket").unwrap(),
|
||||
@@ -1685,13 +1657,19 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_parse_bucket_restrictions_multiple_buckets() {
|
||||
let result = parse_bucket_restrictions_from_str("bucket-a:ws1,ws2;bucket-b:ws3").unwrap();
|
||||
let result = parse_bucket_restrictions_from_str(
|
||||
"bucket-a:ws1,ws2;bucket-b:ws3",
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(result.len(), 2);
|
||||
assert_eq!(
|
||||
result.get("bucket-a").unwrap(),
|
||||
&vec!["ws1".to_string(), "ws2".to_string()]
|
||||
);
|
||||
assert_eq!(result.get("bucket-b").unwrap(), &vec!["ws3".to_string()]);
|
||||
assert_eq!(
|
||||
result.get("bucket-b").unwrap(),
|
||||
&vec!["ws3".to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1703,14 +1681,16 @@ mod tests {
|
||||
#[test]
|
||||
fn test_parse_bucket_restrictions_invalid_format_skipped() {
|
||||
// "no-colon" is invalid, only "valid:ws1" should be parsed
|
||||
let result = parse_bucket_restrictions_from_str("no-colon;valid:ws1").unwrap();
|
||||
let result =
|
||||
parse_bucket_restrictions_from_str("no-colon;valid:ws1").unwrap();
|
||||
assert_eq!(result.len(), 1);
|
||||
assert!(result.contains_key("valid"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_bucket_restrictions_trailing_semicolons() {
|
||||
let result = parse_bucket_restrictions_from_str(";bucket:ws1;;").unwrap();
|
||||
let result =
|
||||
parse_bucket_restrictions_from_str(";bucket:ws1;;").unwrap();
|
||||
assert_eq!(result.len(), 1);
|
||||
assert!(result.contains_key("bucket"));
|
||||
}
|
||||
|
||||
@@ -3100,46 +3100,41 @@ impl PulledJobResult {
|
||||
"Accumulated arguments from debounced jobs in batch"
|
||||
);
|
||||
|
||||
// If the batch query returned no entries (e.g. CE where
|
||||
// v2_job_debounce_batch is never populated), keep the
|
||||
// original value unchanged instead of replacing it with [].
|
||||
if !accumulated_arg.is_empty() {
|
||||
let new_value = to_raw_value(&accumulated_arg);
|
||||
let new_value = to_raw_value(&accumulated_arg);
|
||||
|
||||
let original_value = j
|
||||
.args
|
||||
.as_ref()
|
||||
.and_then(|a| a.get(arg_name_to_accumulate))
|
||||
.map(|v| v.get().to_string())
|
||||
.unwrap_or_else(|| "null".to_string());
|
||||
let original_value = j
|
||||
.args
|
||||
.as_ref()
|
||||
.and_then(|a| a.get(arg_name_to_accumulate))
|
||||
.map(|v| v.get().to_string())
|
||||
.unwrap_or_else(|| "null".to_string());
|
||||
|
||||
append_logs(
|
||||
&j_id,
|
||||
&j.workspace_id,
|
||||
format!(
|
||||
"Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n",
|
||||
&new_value
|
||||
),
|
||||
&(db.into()),
|
||||
append_logs(
|
||||
&j_id,
|
||||
&j.workspace_id,
|
||||
format!(
|
||||
"Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n",
|
||||
&new_value
|
||||
),
|
||||
&(db.into()),
|
||||
)
|
||||
.await;
|
||||
|
||||
j.args
|
||||
.get_or_insert(Json(Default::default()))
|
||||
.as_mut()
|
||||
.insert(arg_name_to_accumulate.to_owned(), new_value);
|
||||
|
||||
// Persist accumulated args to v2_job so that flow steps
|
||||
// re-reading from the DB (via get_mini_pulled_job) see them
|
||||
if let Some(ref args) = j.args {
|
||||
sqlx::query!(
|
||||
"UPDATE v2_job SET args = $2 WHERE id = $1",
|
||||
j_id,
|
||||
args as &Json<HashMap<String, Box<RawValue>>>,
|
||||
)
|
||||
.await;
|
||||
|
||||
j.args
|
||||
.get_or_insert(Json(Default::default()))
|
||||
.as_mut()
|
||||
.insert(arg_name_to_accumulate.to_owned(), new_value);
|
||||
|
||||
// Persist accumulated args to v2_job so that flow steps
|
||||
// re-reading from the DB (via get_mini_pulled_job) see them
|
||||
if let Some(ref args) = j.args {
|
||||
sqlx::query!(
|
||||
"UPDATE v2_job SET args = $2 WHERE id = $1",
|
||||
j_id,
|
||||
args as &Json<HashMap<String, Box<RawValue>>>,
|
||||
)
|
||||
.execute(db)
|
||||
.await?;
|
||||
}
|
||||
.execute(db)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -4429,321 +4429,4 @@ mod debounce {
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Tests for maybe_debounce_flow_node (flow node debouncing)
|
||||
// =========================================================================
|
||||
|
||||
/// Helper: insert a child job with a parent flow.
|
||||
async fn insert_child_job_with_parent(
|
||||
db: &Pool<Postgres>,
|
||||
child_id: Uuid,
|
||||
parent_id: Uuid,
|
||||
workspace_id: &str,
|
||||
) {
|
||||
sqlx::query!(
|
||||
"INSERT INTO v2_job (id, kind, tag, created_by, permissioned_as, permissioned_as_email, workspace_id, parent_job)
|
||||
VALUES ($1, 'noop', 'deno', 'test-user', 'u/test-user', 'test@windmill.dev', $2, $3)",
|
||||
child_id,
|
||||
workspace_id,
|
||||
parent_id,
|
||||
)
|
||||
.execute(db)
|
||||
.await
|
||||
.expect("insert v2_job (child)");
|
||||
|
||||
sqlx::query!(
|
||||
"INSERT INTO v2_job_queue (id, workspace_id, scheduled_for, tag)
|
||||
VALUES ($1, $2, now(), 'deno')",
|
||||
child_id,
|
||||
workspace_id,
|
||||
)
|
||||
.execute(db)
|
||||
.await
|
||||
.expect("insert v2_job_queue (child)");
|
||||
|
||||
sqlx::query!("INSERT INTO v2_job_runtime (id) VALUES ($1)", child_id)
|
||||
.execute(db)
|
||||
.await
|
||||
.expect("insert v2_job_runtime (child)");
|
||||
}
|
||||
|
||||
/// Test: First flow node job in a debounce batch should set scheduled_for
|
||||
/// and create a debounce_key entry. The parent flow should remain in queue.
|
||||
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
|
||||
async fn test_flow_node_debounce_first_job(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
let flow_id = Uuid::new_v4();
|
||||
let child_id = Uuid::new_v4();
|
||||
insert_flow_job(&db, flow_id, "test-workspace", "f/test/my_flow").await;
|
||||
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
|
||||
|
||||
let settings = DebouncingSettings {
|
||||
debounce_delay_s: Some(5),
|
||||
debounce_key: Some("test_flow_node_first".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let args_hm = empty_args();
|
||||
let args = PushArgs::from(&args_hm);
|
||||
let mut tx = db.begin().await?;
|
||||
|
||||
windmill_queue::jobs_ee::maybe_debounce_flow_node(
|
||||
&settings,
|
||||
child_id,
|
||||
flow_id,
|
||||
"f/test/my_flow",
|
||||
"step_a",
|
||||
"test-workspace",
|
||||
&args,
|
||||
&mut tx,
|
||||
&db,
|
||||
)
|
||||
.await?;
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
// Child job should still be in queue with delayed scheduled_for
|
||||
assert!(is_queued(&db, &child_id).await, "child should be queued");
|
||||
let sf = sqlx::query_scalar!(
|
||||
"SELECT scheduled_for FROM v2_job_queue WHERE id = $1",
|
||||
child_id
|
||||
)
|
||||
.fetch_one(&db)
|
||||
.await?;
|
||||
let diff = (sf - Utc::now()).num_seconds();
|
||||
assert!(
|
||||
diff >= 0 && diff <= 6,
|
||||
"scheduled_for should be in the future (up to ~5s), got {diff}s"
|
||||
);
|
||||
|
||||
// Parent flow should still be in queue
|
||||
assert!(
|
||||
is_queued(&db, &flow_id).await,
|
||||
"parent flow should still be queued"
|
||||
);
|
||||
assert!(
|
||||
!is_completed(&db, &flow_id).await,
|
||||
"parent flow should not be completed"
|
||||
);
|
||||
|
||||
// debounce_key should exist
|
||||
let dk = get_debounce_key(&db, "test_flow_node_first").await;
|
||||
assert!(dk.is_some(), "debounce_key entry should exist");
|
||||
let (dk_job, dk_prev, dk_times) = dk.unwrap();
|
||||
assert_eq!(dk_job, child_id);
|
||||
assert!(dk_prev.is_none());
|
||||
assert_eq!(dk_times, 0);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Test: Second flow node job with same key should cancel the first child and
|
||||
/// complete the first parent flow with "Debounced by" result.
|
||||
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
|
||||
async fn test_flow_node_debounce_second_cancels_first(
|
||||
db: Pool<Postgres>,
|
||||
) -> anyhow::Result<()> {
|
||||
// Flow 1 with child 1
|
||||
let flow1 = Uuid::new_v4();
|
||||
let child1 = Uuid::new_v4();
|
||||
insert_flow_job(&db, flow1, "test-workspace", "f/test/my_flow").await;
|
||||
insert_child_job_with_parent(&db, child1, flow1, "test-workspace").await;
|
||||
|
||||
// Flow 2 with child 2
|
||||
let flow2 = Uuid::new_v4();
|
||||
let child2 = Uuid::new_v4();
|
||||
insert_flow_job(&db, flow2, "test-workspace", "f/test/my_flow").await;
|
||||
insert_child_job_with_parent(&db, child2, flow2, "test-workspace").await;
|
||||
|
||||
let settings = DebouncingSettings {
|
||||
debounce_delay_s: Some(5),
|
||||
debounce_key: Some("test_flow_node_cancel".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let args_hm = empty_args();
|
||||
|
||||
// Push child 1
|
||||
{
|
||||
let args = PushArgs::from(&args_hm);
|
||||
let mut tx = db.begin().await?;
|
||||
windmill_queue::jobs_ee::maybe_debounce_flow_node(
|
||||
&settings,
|
||||
child1,
|
||||
flow1,
|
||||
"f/test/my_flow",
|
||||
"step_a",
|
||||
"test-workspace",
|
||||
&args,
|
||||
&mut tx,
|
||||
&db,
|
||||
)
|
||||
.await?;
|
||||
tx.commit().await?;
|
||||
}
|
||||
|
||||
// Push child 2 — should cancel child 1 and complete flow 1
|
||||
{
|
||||
let args = PushArgs::from(&args_hm);
|
||||
let mut tx = db.begin().await?;
|
||||
windmill_queue::jobs_ee::maybe_debounce_flow_node(
|
||||
&settings,
|
||||
child2,
|
||||
flow2,
|
||||
"f/test/my_flow",
|
||||
"step_a",
|
||||
"test-workspace",
|
||||
&args,
|
||||
&mut tx,
|
||||
&db,
|
||||
)
|
||||
.await?;
|
||||
tx.commit().await?;
|
||||
}
|
||||
|
||||
// child1 should be completed (debounced/skipped)
|
||||
assert!(
|
||||
is_completed(&db, &child1).await,
|
||||
"child1 should be completed"
|
||||
);
|
||||
assert!(
|
||||
!is_queued(&db, &child1).await,
|
||||
"child1 should not be in queue"
|
||||
);
|
||||
|
||||
// flow1 (parent of child1) should also be completed
|
||||
assert!(
|
||||
is_completed(&db, &flow1).await,
|
||||
"flow1 should be completed (debounced)"
|
||||
);
|
||||
assert!(
|
||||
!is_queued(&db, &flow1).await,
|
||||
"flow1 should not be in queue"
|
||||
);
|
||||
|
||||
// Check flow1 result contains "Debounced by"
|
||||
let result = sqlx::query_scalar!(
|
||||
"SELECT result::text FROM v2_job_completed WHERE id = $1",
|
||||
flow1
|
||||
)
|
||||
.fetch_one(&db)
|
||||
.await?;
|
||||
assert!(
|
||||
result.as_ref().is_some_and(|r| r.contains("Debounced by")),
|
||||
"flow1 result should contain 'Debounced by', got: {:?}",
|
||||
result
|
||||
);
|
||||
|
||||
// child2 should still be in queue (it's the winner)
|
||||
assert!(
|
||||
is_queued(&db, &child2).await,
|
||||
"child2 should still be queued"
|
||||
);
|
||||
assert!(
|
||||
!is_completed(&db, &child2).await,
|
||||
"child2 should not be completed"
|
||||
);
|
||||
|
||||
// flow2 should still be in queue
|
||||
assert!(is_queued(&db, &flow2).await, "flow2 should still be queued");
|
||||
|
||||
// debounce_key should point to child2
|
||||
let dk = get_debounce_key(&db, "test_flow_node_cancel").await;
|
||||
assert!(dk.is_some());
|
||||
let (dk_job, _, dk_times) = dk.unwrap();
|
||||
assert_eq!(dk_job, child2);
|
||||
assert_eq!(dk_times, 1);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Test: Default debounce key for flow nodes uses $workspace/flow/$path-$step_id.
|
||||
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
|
||||
async fn test_flow_node_debounce_default_key(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
let flow_id = Uuid::new_v4();
|
||||
let child_id = Uuid::new_v4();
|
||||
insert_flow_job(&db, flow_id, "test-workspace", "f/test/my_flow").await;
|
||||
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
|
||||
|
||||
let settings = DebouncingSettings {
|
||||
debounce_delay_s: Some(5),
|
||||
debounce_key: None, // No custom key — use default
|
||||
..Default::default()
|
||||
};
|
||||
let args_hm = empty_args();
|
||||
let args = PushArgs::from(&args_hm);
|
||||
let mut tx = db.begin().await?;
|
||||
|
||||
windmill_queue::jobs_ee::maybe_debounce_flow_node(
|
||||
&settings,
|
||||
child_id,
|
||||
flow_id,
|
||||
"f/test/my_flow",
|
||||
"step_a",
|
||||
"test-workspace",
|
||||
&args,
|
||||
&mut tx,
|
||||
&db,
|
||||
)
|
||||
.await?;
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
// Default key should be: test-workspace/flow/f/test/my_flow-step_a
|
||||
let expected_key = "test-workspace/flow/f/test/my_flow-step_a";
|
||||
let dk = get_debounce_key(&db, expected_key).await;
|
||||
assert!(dk.is_some(), "debounce_key with default key should exist");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Test: Flow node debounce tracks debounced_times counter correctly.
|
||||
/// Each debounce call increments the counter in the debounce_key table.
|
||||
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
|
||||
async fn test_flow_node_debounce_counter_tracking(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
// No limits set so counter never resets
|
||||
let settings = DebouncingSettings {
|
||||
debounce_delay_s: Some(5),
|
||||
debounce_key: Some("test_flow_node_counter".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let args_hm = empty_args();
|
||||
|
||||
// Push 4 jobs. Each subsequent one increments debounced_times.
|
||||
for _ in 0..4 {
|
||||
let flow_id = Uuid::new_v4();
|
||||
let child_id = Uuid::new_v4();
|
||||
insert_flow_job(&db, flow_id, "test-workspace", "f/test/flow").await;
|
||||
insert_child_job_with_parent(&db, child_id, flow_id, "test-workspace").await;
|
||||
|
||||
let args = PushArgs::from(&args_hm);
|
||||
let mut tx = db.begin().await?;
|
||||
windmill_queue::jobs_ee::maybe_debounce_flow_node(
|
||||
&settings,
|
||||
child_id,
|
||||
flow_id,
|
||||
"f/test/flow",
|
||||
"step_a",
|
||||
"test-workspace",
|
||||
&args,
|
||||
&mut tx,
|
||||
&db,
|
||||
)
|
||||
.await?;
|
||||
tx.commit().await?;
|
||||
}
|
||||
|
||||
// After 4 jobs, debounced_times should be 3 (first job creates the entry with 0,
|
||||
// subsequent 3 jobs each increment it)
|
||||
let debounced_times = sqlx::query_scalar!(
|
||||
"SELECT debounced_times FROM debounce_key WHERE key = $1",
|
||||
"test_flow_node_counter"
|
||||
)
|
||||
.fetch_one(&db)
|
||||
.await?;
|
||||
assert_eq!(
|
||||
debounced_times, 3,
|
||||
"debounced_times should be 3 after 4 jobs"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,7 +9,6 @@ name = "windmill_types"
|
||||
path = "src/lib.rs"
|
||||
|
||||
[dependencies]
|
||||
windmill-parser.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
chrono.workspace = true
|
||||
|
||||
@@ -445,8 +445,6 @@ pub struct FlowModule {
|
||||
pub apply_preprocessor: Option<bool>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub pass_flow_input_directly: Option<bool>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub debouncing: Option<DebouncingSettings>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, Serialize, Debug, Clone)]
|
||||
@@ -1119,7 +1117,6 @@ pub fn add_virtual_items_if_necessary(modules: &mut Vec<FlowModule>) {
|
||||
skip_if: None,
|
||||
apply_preprocessor: None,
|
||||
pass_flow_input_directly: None,
|
||||
debouncing: None,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -346,9 +346,20 @@ pub struct DuckdbConnectionSettingsQueryV2 {
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
// Re-export from windmill-parser to keep a single type definition
|
||||
// (windmill-parser is WASM-compatible, windmill-types is not due to sqlx)
|
||||
pub use windmill_parser::{s3_mode_extension, S3ModeFormat};
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub enum S3ModeFormat {
|
||||
Json,
|
||||
Csv,
|
||||
Parquet,
|
||||
}
|
||||
|
||||
pub fn s3_mode_extension(format: S3ModeFormat) -> &'static str {
|
||||
match format {
|
||||
S3ModeFormat::Json => "json",
|
||||
S3ModeFormat::Csv => "csv",
|
||||
S3ModeFormat::Parquet => "parquet",
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
|
||||
@@ -1,11 +1,8 @@
|
||||
// Injected by backend: maps normalized paths to temp storage hashes (or null)
|
||||
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
|
||||
|
||||
const p = {
|
||||
name: "windmill-relative-resolver",
|
||||
async setup(build) {
|
||||
const { writeFileSync, readFileSync, mkdirSync } = await import("fs");
|
||||
const { dirname, resolve, join } = await import("node:path");
|
||||
const { dirname, resolve } = await import("node:path");
|
||||
|
||||
const base_internal_url = "BASE_INTERNAL_URL".replace(
|
||||
"localhost",
|
||||
@@ -98,17 +95,11 @@ const p = {
|
||||
: args.importer.replace(cdir + "/", "");
|
||||
|
||||
const isRelative = !args.path.startsWith("/");
|
||||
const endExt = args.path.endsWith(".ts") ? "" : ".ts";
|
||||
const pathNoExt = args.path.replace(/\.ts$/, "");
|
||||
|
||||
// Lookup temp script hash
|
||||
const normalized = (isRelative ? join(dirname(file_path), pathNoExt) : pathNoExt.slice(1)).replace(/\\/g, "/");
|
||||
const hash = TEMP_SCRIPT_REFS?.[normalized];
|
||||
|
||||
const url = (isRelative
|
||||
let endExt = args.path.endsWith(".ts") ? "" : ".ts";
|
||||
const url = isRelative
|
||||
? `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${file_path}/../${args.path}${endExt}`
|
||||
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`
|
||||
) + (hash ? `?temp_script_hash=${hash}` : "");
|
||||
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`;
|
||||
const file = isRelative
|
||||
? resolve("./" + file_path + "/../" + args.path + ".url")
|
||||
: resolve("./" + args.path + ".url");
|
||||
|
||||
@@ -1,6 +1,3 @@
|
||||
// Injected by backend: maps normalized paths to temp storage hashes (or null)
|
||||
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
|
||||
|
||||
// Windows-specific bun loader that uses a virtual "windmill-url" namespace instead
|
||||
// of writing .url files to disk. This avoids Windows path issues (backslashes in
|
||||
// resolve(), 8.3 short filenames, drive letter prefixes). The virtual namespace
|
||||
@@ -73,13 +70,7 @@ const p = {
|
||||
const rawScriptPath = isAbsolute
|
||||
? `${path}${endExt}`
|
||||
: `${importerPath}/../${path}${endExt}`;
|
||||
const normalized = normalizePath(rawScriptPath);
|
||||
// Look up temp script hash (keys are extensionless paths)
|
||||
const lookupPath = normalized.replace(/\.ts$/, "");
|
||||
const hash = TEMP_SCRIPT_REFS?.[lookupPath];
|
||||
// Encode hash in the path so onLoad can extract it and append to fetch URL
|
||||
const resolvedPath = hash ? `${normalized}?temp_script_hash=${hash}` : normalized;
|
||||
return { path: resolvedPath, namespace: "windmill-url" };
|
||||
return { path: normalizePath(rawScriptPath), namespace: "windmill-url" };
|
||||
}
|
||||
|
||||
build.onLoad({ filter: filterLoad }, async (args) => {
|
||||
@@ -89,13 +80,8 @@ const p = {
|
||||
|
||||
// Load windmill scripts by fetching from the API
|
||||
build.onLoad({ filter: /.*/, namespace: "windmill-url" }, async (args) => {
|
||||
// Extract temp_script_hash if embedded in the path by resolveWindmillImport
|
||||
const [scriptPath, queryString] = args.path.replace(/^windmill-url:/, "").split("?");
|
||||
const hashParam = queryString?.startsWith("temp_script_hash=")
|
||||
? queryString.replace("temp_script_hash=", "")
|
||||
: undefined;
|
||||
const url = `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${scriptPath}`
|
||||
+ (hashParam ? `?temp_script_hash=${hashParam}` : "");
|
||||
const path = args.path.replace(/^windmill-url:/, "");
|
||||
const url = `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${path}`;
|
||||
const req = await fetch(url, {
|
||||
method: "GET",
|
||||
headers: {
|
||||
@@ -138,8 +124,7 @@ const p = {
|
||||
|
||||
// Resolve nested imports from within windmill-url modules
|
||||
build.onResolve({ filter: /\.ts$/, namespace: "windmill-url" }, (args) => {
|
||||
// Strip any query string from the importer path before resolving
|
||||
const importer = args.importer.replace(/^windmill-url:/, "").split("?")[0];
|
||||
const importer = args.importer.replace(/^windmill-url:/, "");
|
||||
return resolveWindmillImport(importer, args.path);
|
||||
});
|
||||
},
|
||||
|
||||
@@ -206,7 +206,6 @@ pub async fn gen_bun_lockfile(
|
||||
workspace_dependencies: &WorkspaceDependenciesPrefetched,
|
||||
npm_mode: bool,
|
||||
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
quiet: bool,
|
||||
) -> Result<Option<String>> {
|
||||
let common_bun_proc_envs: HashMap<String, String> = get_common_bun_proc_envs(None).await;
|
||||
@@ -217,11 +216,6 @@ pub async fn gen_bun_lockfile(
|
||||
gen_bunfig(job_dir, job_id, w_id, db).await?;
|
||||
write_file(job_dir, "package.json", package_json_content.as_str())?;
|
||||
} else {
|
||||
let temp_refs_json = temp_script_refs
|
||||
.as_ref()
|
||||
.and_then(|m| serde_json::to_string(m).ok())
|
||||
.unwrap_or_else(|| "null".to_string());
|
||||
|
||||
let loader = RELATIVE_BUN_LOADER
|
||||
.replace("W_ID", w_id)
|
||||
.replace("BASE_INTERNAL_URL", base_internal_url)
|
||||
@@ -230,8 +224,7 @@ pub async fn gen_bun_lockfile(
|
||||
"CURRENT_PATH",
|
||||
&crate::common::use_flow_root_path(script_path),
|
||||
)
|
||||
.replace("RAW_GET_ENDPOINT", "raw")
|
||||
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", &temp_refs_json);
|
||||
.replace("RAW_GET_ENDPOINT", "raw");
|
||||
|
||||
write_file(
|
||||
&job_dir,
|
||||
@@ -622,15 +615,9 @@ pub async fn build_loader(
|
||||
w_id: &str,
|
||||
current_path: &str,
|
||||
mode: LoaderMode,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> Result<()> {
|
||||
// Use forward slashes in JS strings to avoid backslash escape issues on Windows
|
||||
let job_dir_js = job_dir.replace('\\', "/");
|
||||
let temp_refs_json = temp_script_refs
|
||||
.as_ref()
|
||||
.and_then(|m| serde_json::to_string(m).ok())
|
||||
.unwrap_or_else(|| "null".to_string());
|
||||
|
||||
let loader = RELATIVE_BUN_LOADER
|
||||
.replace("W_ID", w_id)
|
||||
.replace("BASE_INTERNAL_URL", base_internal_url)
|
||||
@@ -639,8 +626,7 @@ pub async fn build_loader(
|
||||
"CURRENT_PATH",
|
||||
&crate::common::use_flow_root_path(current_path),
|
||||
)
|
||||
.replace("RAW_GET_ENDPOINT", "raw_unpinned")
|
||||
.replace("TEMP_SCRIPT_REFS_PLACEHOLDER", &temp_refs_json);
|
||||
.replace("RAW_GET_ENDPOINT", "raw_unpinned");
|
||||
|
||||
if mode == LoaderMode::Node {
|
||||
write_file(
|
||||
@@ -938,7 +924,6 @@ pub async fn prebundle_bun_script(
|
||||
worker_name: &str,
|
||||
token: &str,
|
||||
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> Result<()> {
|
||||
let (local_path, remote_path) =
|
||||
compute_bundle_local_and_remote_path(inner_content, lock, script_path, db, w_id).await;
|
||||
@@ -965,7 +950,6 @@ pub async fn prebundle_bun_script(
|
||||
} else {
|
||||
LoaderMode::BunBundle
|
||||
},
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -1287,7 +1271,6 @@ pub async fn handle_bun_job(
|
||||
workspace_dependencies,
|
||||
annotation.npm,
|
||||
&mut Some(occupancy_metrics),
|
||||
&None,
|
||||
wac_replay_info.is_some(),
|
||||
)
|
||||
.await?;
|
||||
@@ -1656,7 +1639,6 @@ try {{
|
||||
} else {
|
||||
LoaderMode::BunBundle
|
||||
},
|
||||
&None,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -1673,7 +1655,6 @@ try {{
|
||||
} else {
|
||||
LoaderMode::Bun
|
||||
},
|
||||
&None,
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
@@ -3377,7 +3358,6 @@ pub async fn start_worker(
|
||||
w_id,
|
||||
script_path,
|
||||
LoaderMode::BrowserBundle,
|
||||
&None,
|
||||
)
|
||||
.await?;
|
||||
generate_bun_bundle(
|
||||
@@ -3490,7 +3470,6 @@ pub async fn start_worker(
|
||||
.await?,
|
||||
annotation.npm,
|
||||
&mut None,
|
||||
&None,
|
||||
false,
|
||||
)
|
||||
.await?;
|
||||
@@ -3565,7 +3544,6 @@ pub async fn start_worker(
|
||||
} else {
|
||||
LoaderMode::Bun
|
||||
},
|
||||
&None,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
|
||||
@@ -710,13 +710,10 @@ pub fn build_command_with_isolation(program: &str, args: &[&str]) -> Command {
|
||||
cmd.args(args);
|
||||
cmd
|
||||
} else {
|
||||
tracing::error!(
|
||||
"unshare isolation is enabled but UNSHARE_PATH is not available. \
|
||||
Running job without isolation. Check Instance Settings > Job Isolation."
|
||||
panic!(
|
||||
"BUG: unshare isolation is enabled but UNSHARE_PATH is None. \
|
||||
This should have been caught at worker startup."
|
||||
);
|
||||
let mut cmd = Command::new(program);
|
||||
cmd.args(args);
|
||||
cmd
|
||||
}
|
||||
} else {
|
||||
let mut cmd = Command::new(program);
|
||||
@@ -1003,14 +1000,6 @@ pub(crate) async fn get_workspace_s3_resource_path(
|
||||
windmill_object_store::check_bucket_workspace_restriction(bucket, workspace_id)?;
|
||||
}
|
||||
|
||||
// Check Azure account name workspace restrictions
|
||||
if let ObjectStoreResource::Azure(azure_resource) = &object_store_resource {
|
||||
windmill_object_store::check_az_account_name_workspace_restriction(
|
||||
&azure_resource.account_name,
|
||||
workspace_id,
|
||||
)?;
|
||||
}
|
||||
|
||||
Ok(Some(object_store_resource))
|
||||
}
|
||||
|
||||
|
||||
@@ -1348,7 +1348,6 @@ async fn handle_python_deps(
|
||||
&mut version_specifiers,
|
||||
&mut locked_v,
|
||||
&None,
|
||||
&None, // temp_script_refs: only used during CLI lock generation
|
||||
))
|
||||
.await?;
|
||||
|
||||
|
||||
@@ -453,13 +453,8 @@ lazy_static::lazy_static! {
|
||||
);
|
||||
}
|
||||
|
||||
tracing::error!(
|
||||
"unshare test command failed (exit code: {}). stderr: '{}'. flags: '{}'. \
|
||||
Unshare isolation will NOT be available. \
|
||||
If job_isolation is set to 'unshare' in Instance Settings, jobs will run without isolation. \
|
||||
Common causes: user namespaces disabled (sysctl kernel.unprivileged_userns_clone=0), \
|
||||
max_user_namespaces=0, or missing privileges (--mount-proc requires privileged mode).",
|
||||
output.status,
|
||||
tracing::warn!(
|
||||
"unshare test failed: {}. Flags: {}. Set ENABLE_UNSHARE_PID=true to fail on error.",
|
||||
stderr.trim(),
|
||||
flags
|
||||
);
|
||||
@@ -481,15 +476,9 @@ lazy_static::lazy_static! {
|
||||
}
|
||||
|
||||
if e.kind() == std::io::ErrorKind::NotFound {
|
||||
tracing::error!(
|
||||
"unshare binary not found in PATH. Unshare isolation will NOT be available. \
|
||||
Install the util-linux package to enable unshare isolation."
|
||||
);
|
||||
tracing::debug!("unshare binary not found");
|
||||
} else {
|
||||
tracing::error!(
|
||||
"Failed to execute unshare test command: {}. Unshare isolation will NOT be available.",
|
||||
e
|
||||
);
|
||||
tracing::warn!("Failed to test unshare: {}", e);
|
||||
}
|
||||
None
|
||||
}
|
||||
@@ -510,27 +499,23 @@ lazy_static::lazy_static! {
|
||||
},
|
||||
Ok(output) => {
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
tracing::error!(
|
||||
"nsjail test failed (exit code: {}). stderr: '{}'. path: '{}'. \
|
||||
Nsjail sandboxing will NOT be available. \
|
||||
tracing::warn!(
|
||||
"nsjail test failed: {}. \
|
||||
nsjail should be included in all standard windmill images. \
|
||||
If job_isolation is set to 'nsjail_sandboxing' in Instance Settings, jobs will fail.",
|
||||
output.status,
|
||||
stderr.trim(),
|
||||
nsjail_path
|
||||
Check that the nsjail binary is installed and working correctly.",
|
||||
stderr.trim()
|
||||
);
|
||||
None
|
||||
},
|
||||
Err(e) => {
|
||||
if e.kind() == std::io::ErrorKind::NotFound {
|
||||
tracing::error!(
|
||||
"nsjail not found at '{}'. Nsjail sandboxing will NOT be available. \
|
||||
If using a custom image, ensure nsjail is installed.",
|
||||
tracing::info!(
|
||||
"nsjail not found at '{}'. Sandboxing will not be available.",
|
||||
nsjail_path
|
||||
);
|
||||
} else {
|
||||
tracing::error!(
|
||||
"Failed to execute nsjail test at '{}': {}. Nsjail sandboxing will NOT be available.",
|
||||
tracing::warn!(
|
||||
"Failed to test nsjail at '{}': {}.",
|
||||
nsjail_path,
|
||||
e
|
||||
);
|
||||
@@ -1584,27 +1569,10 @@ pub async fn run_worker(
|
||||
);
|
||||
}
|
||||
|
||||
// Force UNSHARE_PATH and NSJAIL_AVAILABLE initialization now for clear startup logging
|
||||
let _ = &*UNSHARE_PATH;
|
||||
let _ = &*NSJAIL_AVAILABLE;
|
||||
|
||||
if (is_unshare_enabled() || *FAVOR_UNSHARE_PID) && UNSHARE_PATH.is_none() {
|
||||
tracing::error!(
|
||||
worker = %worker_name, hostname = %hostname,
|
||||
"Worker is configured to use unshare isolation (FAVOR_UNSHARE_PID={}, job_isolation={:?}) \
|
||||
but unshare is NOT available. Jobs will run without isolation. \
|
||||
See errors above for the specific reason unshare initialization failed.",
|
||||
*FAVOR_UNSHARE_PID,
|
||||
JobIsolationLevel::from_u8(JOB_ISOLATION.load(std::sync::atomic::Ordering::Relaxed))
|
||||
);
|
||||
}
|
||||
if is_sandboxing_enabled() && NSJAIL_AVAILABLE.is_none() {
|
||||
tracing::error!(
|
||||
worker = %worker_name, hostname = %hostname,
|
||||
"Worker is configured to use nsjail sandboxing but nsjail is NOT available. \
|
||||
Jobs requiring sandboxing will fail. \
|
||||
See errors above for the specific reason nsjail initialization failed."
|
||||
);
|
||||
// Force UNSHARE_PATH initialization now to fail-fast if unshare doesn't work
|
||||
// This ensures we panic at startup rather than lazily when first accessed during job execution
|
||||
if is_unshare_enabled() || *ENABLE_UNSHARE_PID || *FAVOR_UNSHARE_PID {
|
||||
let _ = &*UNSHARE_PATH;
|
||||
}
|
||||
|
||||
let start_time = Instant::now();
|
||||
@@ -4209,29 +4177,6 @@ pub async fn run_language_executor(
|
||||
modules: &Option<std::collections::HashMap<String, ScriptModule>>,
|
||||
run_inline: bool,
|
||||
) -> error::Result<Box<RawValue>> {
|
||||
// Expand WM_INTERNAL_DB markers into real SQL before dispatching
|
||||
let expanded_code: String;
|
||||
let mut language = language;
|
||||
let code = if let Some(ref lang) = language {
|
||||
match windmill_common::query_builders::try_expand_internal_db_query(code, lang) {
|
||||
Some(Ok(expanded)) => {
|
||||
if let Some(lang_override) = expanded.language_override {
|
||||
language = Some(lang_override);
|
||||
}
|
||||
expanded_code = expanded.code;
|
||||
&expanded_code
|
||||
}
|
||||
Some(Err(e)) => {
|
||||
return Err(Error::ExecutionErr(format!(
|
||||
"Failed to expand WM_INTERNAL_DB marker: {}",
|
||||
e
|
||||
)));
|
||||
}
|
||||
None => code, // Not a marker, use original code
|
||||
}
|
||||
} else {
|
||||
code
|
||||
};
|
||||
if let Some(modules) = modules {
|
||||
#[cfg(feature = "python")]
|
||||
let base_dir = if language == Some(ScriptLang::Python3) {
|
||||
|
||||
@@ -3687,34 +3687,6 @@ async fn push_next_flow_job(
|
||||
|
||||
tracing::debug!(id = %flow_job.id, root_id = %job_root, "pushed next flow job: {uuid}");
|
||||
|
||||
// Apply flow node debouncing if configured. Skip parallel steps (for-loops, branchall)
|
||||
// where len > 1 — debouncing those would cancel sibling sub-jobs within the same run.
|
||||
#[cfg(feature = "private")]
|
||||
if len == 1 {
|
||||
if let Some(ref debouncing) = module.debouncing {
|
||||
if debouncing.debounce_delay_s.is_some_and(|d| d > 0) {
|
||||
let debounce_args = if let Ok(ref v) = nargs {
|
||||
windmill_queue::PushArgs::from(v.as_ref())
|
||||
} else {
|
||||
windmill_queue::PushArgs::from(&*EHM)
|
||||
};
|
||||
let flow_path = flow_job.runnable_path().to_string();
|
||||
windmill_queue::jobs_ee::maybe_debounce_flow_node(
|
||||
debouncing,
|
||||
uuid,
|
||||
flow_job.id,
|
||||
&flow_path,
|
||||
&module.id,
|
||||
&flow_job.workspace_id,
|
||||
&debounce_args,
|
||||
&mut inner_tx,
|
||||
&db,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if value_with_parallel.type_ == "forloopflow"
|
||||
&& value_with_parallel.parallel.unwrap_or(false)
|
||||
{
|
||||
|
||||
@@ -139,13 +139,6 @@ pub async fn handle_dependency_job(
|
||||
.map(|x| x.get("triggered_by_relative_import").is_some())
|
||||
.unwrap_or_default();
|
||||
|
||||
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
|
||||
let temp_script_refs: Option<HashMap<String, String>> = job
|
||||
.args
|
||||
.as_ref()
|
||||
.and_then(|x| x.get("temp_script_refs"))
|
||||
.and_then(|v| serde_json::from_str(v.get()).ok());
|
||||
|
||||
let content = capture_dependency_job(
|
||||
&job.id,
|
||||
job.script_lang.as_ref().map(|v| Ok(v)).unwrap_or_else(|| {
|
||||
@@ -171,7 +164,6 @@ pub async fn handle_dependency_job(
|
||||
script_path,
|
||||
None,
|
||||
"script",
|
||||
&temp_script_refs,
|
||||
)
|
||||
.await;
|
||||
|
||||
@@ -218,7 +210,6 @@ pub async fn handle_dependency_job(
|
||||
script_path,
|
||||
None,
|
||||
"script",
|
||||
&None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -377,13 +368,6 @@ pub async fn handle_flow_dependency_job(
|
||||
.map(|x| x.get("triggered_by_relative_import").is_some())
|
||||
.unwrap_or_default();
|
||||
|
||||
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
|
||||
let temp_script_refs: Option<HashMap<String, String>> = job
|
||||
.args
|
||||
.as_ref()
|
||||
.and_then(|x| x.get("temp_script_refs"))
|
||||
.and_then(|v| serde_json::from_str(v.get()).ok());
|
||||
|
||||
let version = if skip_flow_update {
|
||||
None
|
||||
} else {
|
||||
@@ -483,7 +467,6 @@ pub async fn handle_flow_dependency_job(
|
||||
&mut dependency_map,
|
||||
&raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
&temp_script_refs,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -698,7 +681,6 @@ async fn lock_flow_value<'c>(
|
||||
dependency_map: &mut ScopedDependencyMap,
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
triggered_by_relative_import: bool,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> Result<(
|
||||
FlowValue,
|
||||
sqlx::Transaction<'c, sqlx::Postgres>,
|
||||
@@ -729,7 +711,6 @@ async fn lock_flow_value<'c>(
|
||||
dependency_map,
|
||||
&raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -761,7 +742,6 @@ async fn lock_flow_value<'c>(
|
||||
dependency_map,
|
||||
&raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -799,7 +779,6 @@ async fn lock_flow_value<'c>(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -838,7 +817,6 @@ async fn lock_modules<'c>(
|
||||
dependency_map: &mut ScopedDependencyMap, // (modules to replace old seq (even unmmodified ones), new transaction, modified ids) )
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
triggered_by_relative_import: bool,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> Result<(
|
||||
Vec<FlowModule>,
|
||||
sqlx::Transaction<'c, sqlx::Postgres>,
|
||||
@@ -894,7 +872,6 @@ async fn lock_modules<'c>(
|
||||
dependency_map,
|
||||
&raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
))
|
||||
.await?;
|
||||
e.value = FlowModuleValue::ForloopFlow {
|
||||
@@ -934,7 +911,6 @@ async fn lock_modules<'c>(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
))
|
||||
.await?;
|
||||
nmodified_ids.extend(inner_modified_ids);
|
||||
@@ -966,7 +942,6 @@ async fn lock_modules<'c>(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
))
|
||||
.await?;
|
||||
e.value = FlowModuleValue::WhileloopFlow {
|
||||
@@ -1003,7 +978,6 @@ async fn lock_modules<'c>(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
))
|
||||
.await?;
|
||||
nmodified_ids.extend(inner_modified_ids);
|
||||
@@ -1034,7 +1008,6 @@ async fn lock_modules<'c>(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
))
|
||||
.await?;
|
||||
errors.extend(ninner_errors);
|
||||
@@ -1105,7 +1078,6 @@ async fn lock_modules<'c>(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
))
|
||||
.await?;
|
||||
|
||||
@@ -1207,7 +1179,6 @@ async fn lock_modules<'c>(
|
||||
job_path,
|
||||
Some(&e.id),
|
||||
"flow",
|
||||
&temp_script_refs,
|
||||
)
|
||||
.await;
|
||||
//
|
||||
@@ -1615,7 +1586,6 @@ async fn lock_modules_app(
|
||||
dependency_map: &mut ScopedDependencyMap,
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
triggered_by_relative_import: bool,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> Result<Value> {
|
||||
match value {
|
||||
Value::Object(mut m) => {
|
||||
@@ -1726,7 +1696,6 @@ async fn lock_modules_app(
|
||||
&job.runnable_path(),
|
||||
container_id.as_deref(),
|
||||
"app",
|
||||
temp_script_refs,
|
||||
)
|
||||
.await;
|
||||
match new_lock {
|
||||
@@ -1804,7 +1773,6 @@ async fn lock_modules_app(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?,
|
||||
);
|
||||
@@ -1833,7 +1801,6 @@ async fn lock_modules_app(
|
||||
dependency_map,
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?,
|
||||
);
|
||||
@@ -1885,13 +1852,6 @@ pub async fn handle_app_dependency_job(
|
||||
.map(|x| x.get("triggered_by_relative_import").is_some())
|
||||
.unwrap_or_default();
|
||||
|
||||
// Extract temp_script_refs from job args (path -> hash mapping for temp storage)
|
||||
let temp_script_refs: Option<HashMap<String, String>> = job
|
||||
.args
|
||||
.as_ref()
|
||||
.and_then(|x| x.get("temp_script_refs"))
|
||||
.and_then(|v| serde_json::from_str(v.get()).ok());
|
||||
|
||||
sqlx::query!(
|
||||
"DELETE FROM workspace_runnable_dependencies WHERE app_path = $1 AND workspace_id = $2",
|
||||
job_path,
|
||||
@@ -1939,7 +1899,6 @@ pub async fn handle_app_dependency_job(
|
||||
&mut dependency_map,
|
||||
&raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
&temp_script_refs,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -2439,8 +2398,6 @@ async fn capture_dependency_job(
|
||||
base_path: &str,
|
||||
step_id: Option<&str>,
|
||||
runnable_type: &str, // "script", "flow", or "app"
|
||||
// Map of script path -> content hash for resolving imports from temp storage (CLI).
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> error::Result<String> {
|
||||
// Check if we can skip relocking:
|
||||
// - Must be triggered by relative import
|
||||
@@ -2499,13 +2456,12 @@ async fn capture_dependency_job(
|
||||
let (mut version_specifiers, mut locked_v) = (vec![], None);
|
||||
let reqs = windmill_parser_py_imports::parse_python_imports(
|
||||
job_raw_code,
|
||||
w_id,
|
||||
&w_id,
|
||||
script_path,
|
||||
&db,
|
||||
&mut version_specifiers,
|
||||
&mut locked_v,
|
||||
raw_workspace_dependencies_o,
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?
|
||||
.0
|
||||
@@ -2629,7 +2585,6 @@ async fn capture_dependency_job(
|
||||
&workspace_dependencies,
|
||||
windmill_common::worker::TypeScriptAnnotations::parse(job_raw_code).npm,
|
||||
&mut Some(occupancy_metrics),
|
||||
temp_script_refs,
|
||||
false,
|
||||
)
|
||||
.await?
|
||||
@@ -2647,7 +2602,6 @@ async fn capture_dependency_job(
|
||||
worker_name,
|
||||
&token,
|
||||
&mut Some(occupancy_metrics),
|
||||
temp_script_refs,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
|
||||
@@ -190,9 +190,9 @@ pub async fn insert_ping(
|
||||
let vcpus = get_vcpus();
|
||||
let memory = get_memory();
|
||||
|
||||
let job_isolation = if crate::is_sandboxing_enabled() && crate::NSJAIL_AVAILABLE.is_some() {
|
||||
let job_isolation = if crate::is_sandboxing_enabled() {
|
||||
Some("nsjail".to_string())
|
||||
} else if crate::is_unshare_enabled() && crate::UNSHARE_PATH.is_some() {
|
||||
} else if crate::is_unshare_enabled() {
|
||||
Some("unshare".to_string())
|
||||
} else {
|
||||
Some("none".to_string())
|
||||
@@ -265,9 +265,9 @@ pub async fn update_worker_ping_from_job(
|
||||
let occupancy_rate_5m = occupancy.as_ref().and_then(|x| x.occupancy_rate_5m);
|
||||
let occupancy_rate_30m = occupancy.as_ref().and_then(|x| x.occupancy_rate_30m);
|
||||
|
||||
let job_isolation = if crate::is_sandboxing_enabled() && crate::NSJAIL_AVAILABLE.is_some() {
|
||||
let job_isolation = if crate::is_sandboxing_enabled() {
|
||||
Some("nsjail".to_string())
|
||||
} else if crate::is_unshare_enabled() && crate::UNSHARE_PATH.is_some() {
|
||||
} else if crate::is_unshare_enabled() {
|
||||
Some("unshare".to_string())
|
||||
} else {
|
||||
Some("none".to_string())
|
||||
|
||||
@@ -12,7 +12,6 @@ const parserPackages = [
|
||||
"windmill-parser-wasm-yaml", "windmill-parser-wasm-csharp",
|
||||
"windmill-parser-wasm-nu", "windmill-parser-wasm-java",
|
||||
"windmill-parser-wasm-ruby",
|
||||
"windmill-parser-wasm-py-imports",
|
||||
];
|
||||
const parserExternals = parserPackages.flatMap(p => ["--external", p]);
|
||||
|
||||
|
||||
+2
-5
@@ -24,11 +24,10 @@
|
||||
"windmill-parser-wasm-nu": "*",
|
||||
"windmill-parser-wasm-php": "*",
|
||||
"windmill-parser-wasm-py": "*",
|
||||
"windmill-parser-wasm-py-imports": "*",
|
||||
"windmill-parser-wasm-regex": "*",
|
||||
"windmill-parser-wasm-ruby": "*",
|
||||
"windmill-parser-wasm-rust": "*",
|
||||
"windmill-parser-wasm-ts": "^1.659.1",
|
||||
"windmill-parser-wasm-ts": "*",
|
||||
"windmill-parser-wasm-yaml": "*",
|
||||
"windmill-yaml-validator": "1.1.1",
|
||||
"ws": "8.18.0",
|
||||
@@ -295,15 +294,13 @@
|
||||
|
||||
"windmill-parser-wasm-py": ["windmill-parser-wasm-py@1.628.3", "", {}, "sha512-TlluqknZpg8cZ+A3m6JFLPseY2PpKtDsxdj26fAnCUzKPtse8TxQR+n0dwC80rfW5TwdWSulvNGRDgcNuf7CTw=="],
|
||||
|
||||
"windmill-parser-wasm-py-imports": ["windmill-parser-wasm-py-imports@1.659.1", "", {}, "sha512-nfnf04WBRf8f/mNIwdvggYOgz3erxrFGjKqULYBH+bKFMlKA6V7eB19m6CXOBkq9rjTp0ZFG+rgsR+Us7JEkyQ=="],
|
||||
|
||||
"windmill-parser-wasm-regex": ["windmill-parser-wasm-regex@1.639.0", "", {}, "sha512-qvYM4sYxB6M0xrqwBljS2fWqOMk6rp++60TRltJnzZDzVaWQrKjTGwNMmfepGAIWy1OGVKp0SCVERhe2P+O6tQ=="],
|
||||
|
||||
"windmill-parser-wasm-ruby": ["windmill-parser-wasm-ruby@1.526.1", "", {}, "sha512-rMBQA8s21wmL2kA5ztRs/ZgVA3ckxe9/NLjxl3iQPL0CX6DlvfaUH0O+AnhpXXDMyBs1Y1SZIhcnbnvsHZ3R8g=="],
|
||||
|
||||
"windmill-parser-wasm-rust": ["windmill-parser-wasm-rust@1.558.1", "", {}, "sha512-21S7lm1KF8zO1187rbq14hzPHII2RdM2+D44MoAh1F6VoaScj+Puq0z5B1O/hwn/95R/a9jBlL2D8jbkXtlD1A=="],
|
||||
|
||||
"windmill-parser-wasm-ts": ["windmill-parser-wasm-ts@1.659.1", "", {}, "sha512-EmXMzOmazC5r29UZh+1TVF9g/N2X51pqK11qDL6xWGeWTIIonhfOZ5nWdGvKQMDUR650fGxehImZzW2v9hNy+w=="],
|
||||
"windmill-parser-wasm-ts": ["windmill-parser-wasm-ts@1.623.1", "", {}, "sha512-FBwi/zXxjhZcCvi04oFdNivazru1ynIqSbafHSArfaaBWesBO3nye9UO/WXUlWZm5a7BExbU+3R/eVJrGaornw=="],
|
||||
|
||||
"windmill-parser-wasm-yaml": ["windmill-parser-wasm-yaml@1.593.0", "", {}, "sha512-Gyx4aR2jsJYuDrD3mCNTmz7LWOQQXPw5yKNCC1xRgUOPfjsD/tINAFfsBLwVOSmlQQcFZO+wHm4KtDtXOcnGVw=="],
|
||||
|
||||
|
||||
+1
-2
@@ -32,11 +32,10 @@
|
||||
"windmill-parser-wasm-nu": "*",
|
||||
"windmill-parser-wasm-php": "*",
|
||||
"windmill-parser-wasm-py": "*",
|
||||
"windmill-parser-wasm-py-imports": "*",
|
||||
"windmill-parser-wasm-regex": "*",
|
||||
"windmill-parser-wasm-ruby": "*",
|
||||
"windmill-parser-wasm-rust": "*",
|
||||
"windmill-parser-wasm-ts": "^1.659.1",
|
||||
"windmill-parser-wasm-ts": "*",
|
||||
"windmill-parser-wasm-yaml": "*",
|
||||
"windmill-yaml-validator": "1.1.1",
|
||||
"ws": "8.18.0",
|
||||
|
||||
@@ -7,7 +7,6 @@ import { yamlParseFile } from "../../utils/yaml.ts";
|
||||
import { stringify as yamlStringify } from "yaml";
|
||||
import { GlobalOptions } from "../../types.ts";
|
||||
import {
|
||||
readLockfile,
|
||||
checkifMetadataUptodate,
|
||||
blueColor,
|
||||
clearGlobalLock,
|
||||
@@ -42,8 +41,6 @@ import { mergeConfigWithConfigFile, SyncOptions } from "../../core/conf.ts";
|
||||
import { resolveWorkspace } from "../../core/context.ts";
|
||||
import { requireLogin } from "../../core/auth.ts";
|
||||
import { getNonDottedPaths } from "../../utils/resource_folders.ts";
|
||||
import { extractRelativeImports } from "../../utils/relative_imports.ts";
|
||||
import { DoubleLinkedDependencyTree } from "../../utils/dependency_tree.ts";
|
||||
|
||||
const TOP_HASH = "__app_hash";
|
||||
export const APP_BACKEND_FOLDER = "backend";
|
||||
@@ -116,9 +113,7 @@ export async function generateAppLocksInternal(
|
||||
defaultTs?: "bun" | "deno";
|
||||
},
|
||||
justUpdateMetadataLock?: boolean,
|
||||
noStaleMessage?: boolean,
|
||||
legacyBehaviour?: boolean,
|
||||
tree?: DoubleLinkedDependencyTree
|
||||
noStaleMessage?: boolean
|
||||
): Promise<string | AppLocksResult | void> {
|
||||
if (appFolder.endsWith(SEP)) {
|
||||
appFolder = appFolder.substring(0, appFolder.length - 1);
|
||||
@@ -130,6 +125,9 @@ export async function generateAppLocksInternal(
|
||||
log.info(`Generating locks for app ${appFolder} at ${remote_path}`);
|
||||
}
|
||||
|
||||
const rawWorkspaceDependencies: Record<string, string> =
|
||||
await getRawWorkspaceDependencies();
|
||||
|
||||
// Read the app file first to filter workspace dependencies
|
||||
const appFilePath = path.join(
|
||||
appFolder,
|
||||
@@ -137,80 +135,35 @@ export async function generateAppLocksInternal(
|
||||
);
|
||||
const appFile = (await yamlParseFile(appFilePath)) as AppFile;
|
||||
|
||||
// Filter workspace dependencies based on inline scripts' languages and annotations
|
||||
const appValue = rawApp ? (appFile as RawAppFile).runnables : (appFile as NormalAppFile).value;
|
||||
const folderNormalized = appFolder.replaceAll(SEP, "/");
|
||||
const filteredDeps = await filterWorkspaceDependenciesForApp(
|
||||
appValue,
|
||||
rawWorkspaceDependencies,
|
||||
appFolder
|
||||
);
|
||||
|
||||
let filteredDeps: Record<string, string> = {};
|
||||
const conf = await readLockfile();
|
||||
let hashes = await generateAppHash(
|
||||
filteredDeps,
|
||||
appFolder,
|
||||
rawApp,
|
||||
opts.defaultTs
|
||||
);
|
||||
|
||||
// New behaviour: tree-based dependency tracking
|
||||
if (!legacyBehaviour && tree) {
|
||||
if (dryRun) {
|
||||
const hashes = await generateAppHash({}, appFolder, rawApp, opts.defaultTs);
|
||||
const isDirectlyStale = !(await checkifMetadataUptodate(appFolder, hashes[TOP_HASH], conf, TOP_HASH));
|
||||
|
||||
// For raw apps in new format, runnables are in separate files under backend/
|
||||
let treeAppValue = structuredClone(appValue);
|
||||
if (rawApp) {
|
||||
const runnablesPath = path.join(appFolder, APP_BACKEND_FOLDER);
|
||||
const runnablesFromFiles = await loadRunnablesFromBackend(runnablesPath);
|
||||
if (Object.keys(runnablesFromFiles).length > 0) {
|
||||
treeAppValue = runnablesFromFiles;
|
||||
}
|
||||
}
|
||||
|
||||
// First pass: add inline scripts as separate nodes, then add app node importing them
|
||||
const inlineScriptPaths: string[] = [];
|
||||
await traverseAndProcessInlineScripts(treeAppValue, async (inlineScript, context) => {
|
||||
if (!inlineScript.content || !inlineScript.language) {
|
||||
return inlineScript;
|
||||
}
|
||||
|
||||
let content = inlineScript.content;
|
||||
// Resolve !inline references
|
||||
if (typeof content === "string" && content.startsWith("!inline ")) {
|
||||
const filePath = appFolder + SEP + content.replace("!inline ", "");
|
||||
try {
|
||||
content = await readFile(filePath, "utf-8");
|
||||
} catch {
|
||||
return inlineScript;
|
||||
}
|
||||
}
|
||||
|
||||
const treePath = folderNormalized + "/" + context.path.join("/");
|
||||
const language = inlineScript.language as ScriptLanguage;
|
||||
const imports = await extractRelativeImports(content, treePath, language);
|
||||
await tree.addNode(treePath, content, language, "", imports, "inline_script", folderNormalized, appFolder, false);
|
||||
inlineScriptPaths.push(treePath);
|
||||
|
||||
return inlineScript;
|
||||
});
|
||||
|
||||
await tree.addNode(folderNormalized, "", "bun", "", inlineScriptPaths, "app", folderNormalized, appFolder, isDirectlyStale, rawApp);
|
||||
return;
|
||||
}
|
||||
// Second pass: get mismatched workspace deps from tree
|
||||
// TODO: pass raw workspace deps more precisely to every inline script lock generation call
|
||||
// (currently we pass the union of all mismatched deps filtered for the whole app)
|
||||
filteredDeps = await filterWorkspaceDependenciesForApp(appValue, tree.getMismatchedWorkspaceDeps(), appFolder);
|
||||
} else {
|
||||
// Legacy behaviour
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
|
||||
filteredDeps = await filterWorkspaceDependenciesForApp(appValue, rawWorkspaceDependencies, appFolder);
|
||||
|
||||
const hashes = await generateAppHash(filteredDeps, appFolder, rawApp, opts.defaultTs);
|
||||
const isDirectlyStale = !(await checkifMetadataUptodate(appFolder, hashes[TOP_HASH], conf, TOP_HASH));
|
||||
|
||||
if (!isDirectlyStale) {
|
||||
if (!noStaleMessage) {
|
||||
log.info(
|
||||
colors.green(`App ${remote_path} metadata is up-to-date, skipping`)
|
||||
);
|
||||
}
|
||||
return;
|
||||
} else if (dryRun) {
|
||||
return remote_path;
|
||||
const conf = await import("../../utils/metadata.ts").then((m) =>
|
||||
m.readLockfile()
|
||||
);
|
||||
if (
|
||||
await checkifMetadataUptodate(appFolder, hashes[TOP_HASH], conf, TOP_HASH)
|
||||
) {
|
||||
if (!noStaleMessage) {
|
||||
log.info(
|
||||
colors.green(`App ${remote_path} metadata is up-to-date, skipping`)
|
||||
);
|
||||
}
|
||||
return;
|
||||
} else if (dryRun) {
|
||||
return remote_path;
|
||||
}
|
||||
|
||||
if (Object.keys(filteredDeps).length > 0 && !noStaleMessage) {
|
||||
@@ -226,8 +179,6 @@ export async function generateAppLocksInternal(
|
||||
let updatedScripts: string[] = [];
|
||||
|
||||
if (!justUpdateMetadataLock) {
|
||||
const hashes = await generateAppHash(filteredDeps, appFolder, rawApp, opts.defaultTs);
|
||||
|
||||
const changedScripts = [];
|
||||
// Find hashes that do not correspond to previous hashes
|
||||
for (const [scriptPath, hash] of Object.entries(hashes)) {
|
||||
@@ -239,13 +190,7 @@ export async function generateAppLocksInternal(
|
||||
}
|
||||
}
|
||||
|
||||
// Get temp_script_refs from tree for relative import resolution
|
||||
const tempScriptRefs = tree?.getTempScriptRefs(folderNormalized);
|
||||
|
||||
// In tree mode, the tree already verified this app is stale (possibly via dependency change).
|
||||
// Per-script hashes only detect content changes, not transitive dependency changes,
|
||||
// so we must regenerate locks for all inline scripts regardless.
|
||||
if (changedScripts.length > 0 || (tree && !legacyBehaviour)) {
|
||||
if (changedScripts.length > 0) {
|
||||
if (!noStaleMessage) {
|
||||
log.info(
|
||||
`Recomputing locks of ${changedScripts.join(", ")} in ${appFolder}`
|
||||
@@ -274,8 +219,7 @@ export async function generateAppLocksInternal(
|
||||
appFolder,
|
||||
filteredDeps,
|
||||
opts.defaultTs,
|
||||
noStaleMessage,
|
||||
tempScriptRefs
|
||||
noStaleMessage
|
||||
);
|
||||
// Note: updateRawAppRunnables now writes each runnable to its own file
|
||||
} else {
|
||||
@@ -292,8 +236,7 @@ export async function generateAppLocksInternal(
|
||||
appFolder,
|
||||
filteredDeps,
|
||||
opts.defaultTs,
|
||||
noStaleMessage,
|
||||
tempScriptRefs
|
||||
noStaleMessage
|
||||
);
|
||||
normalAppFile.value = result.value;
|
||||
updatedScripts = result.updatedScripts;
|
||||
@@ -309,16 +252,15 @@ export async function generateAppLocksInternal(
|
||||
}
|
||||
}
|
||||
|
||||
// Non-legacy mode excludes workspace deps from hash (tracked via tree instead)
|
||||
const depsForHash = (tree && !legacyBehaviour) ? {} : filteredDeps;
|
||||
const finalHashes = await generateAppHash(
|
||||
depsForHash,
|
||||
// Regenerate hashes after updates
|
||||
hashes = await generateAppHash(
|
||||
filteredDeps,
|
||||
appFolder,
|
||||
rawApp,
|
||||
opts.defaultTs
|
||||
);
|
||||
await clearGlobalLock(appFolder);
|
||||
for (const [scriptPath, hash] of Object.entries(finalHashes)) {
|
||||
for (const [scriptPath, hash] of Object.entries(hashes)) {
|
||||
await updateMetadataGlobalLock(appFolder, hash, scriptPath);
|
||||
}
|
||||
if (!noStaleMessage) {
|
||||
@@ -424,8 +366,7 @@ async function updateRawAppRunnables(
|
||||
appFolder: string,
|
||||
rawDeps?: Record<string, string>,
|
||||
defaultTs: "bun" | "deno" = "bun",
|
||||
noStaleMessage?: boolean,
|
||||
tempScriptRefs?: Record<string, string>
|
||||
noStaleMessage?: boolean
|
||||
): Promise<string[]> {
|
||||
const updatedRunnables: string[] = [];
|
||||
const runnablesFolder = path.join(appFolder, APP_BACKEND_FOLDER);
|
||||
@@ -505,8 +446,7 @@ async function updateRawAppRunnables(
|
||||
content,
|
||||
language,
|
||||
`${remotePath}/${runnableId}`,
|
||||
rawDeps,
|
||||
tempScriptRefs
|
||||
rawDeps
|
||||
);
|
||||
|
||||
// Determine file extension for this language
|
||||
@@ -573,8 +513,7 @@ async function updateAppInlineScripts(
|
||||
appFolder: string,
|
||||
rawDeps?: Record<string, string>,
|
||||
defaultTs: "bun" | "deno" = "bun",
|
||||
noStaleMessage?: boolean,
|
||||
tempScriptRefs?: Record<string, string>
|
||||
noStaleMessage?: boolean
|
||||
): Promise<{ value: any; updatedScripts: string[] }> {
|
||||
const pathAssigner = newPathAssigner(defaultTs, { skipInlineScriptSuffix: getNonDottedPaths() });
|
||||
const updatedScripts: string[] = [];
|
||||
@@ -622,8 +561,7 @@ async function updateAppInlineScripts(
|
||||
content,
|
||||
language,
|
||||
scriptPath,
|
||||
rawDeps,
|
||||
tempScriptRefs
|
||||
rawDeps
|
||||
);
|
||||
}
|
||||
// Determine file extension for this language (following extractInlineScriptsForApps pattern)
|
||||
@@ -688,8 +626,7 @@ async function generateInlineScriptLock(
|
||||
content: string,
|
||||
language: string,
|
||||
scriptPath: string,
|
||||
rawWorkspaceDependencies: Record<string, string> | undefined,
|
||||
tempScriptRefs?: Record<string, string>
|
||||
rawWorkspaceDependencies: Record<string, string> | undefined
|
||||
): Promise<string> {
|
||||
// Filter workspace dependencies to only include those matching this script's language and annotations
|
||||
const filteredDeps = rawWorkspaceDependencies
|
||||
@@ -720,9 +657,6 @@ async function generateInlineScriptLock(
|
||||
? filteredDeps
|
||||
: null,
|
||||
entrypoint: scriptPath,
|
||||
...(tempScriptRefs && Object.keys(tempScriptRefs).length > 0
|
||||
? { temp_script_refs: tempScriptRefs }
|
||||
: {}),
|
||||
}),
|
||||
}
|
||||
);
|
||||
|
||||
+190
-31
@@ -6,6 +6,7 @@ import { WebSocket, WebSocketServer } from "ws";
|
||||
|
||||
import * as getPort from "get-port";
|
||||
import * as http from "node:http";
|
||||
import * as https from "node:https";
|
||||
import * as open from "open";
|
||||
import { readFile, realpath } from "node:fs/promises";
|
||||
import { watch } from "node:fs";
|
||||
@@ -32,7 +33,9 @@ import { listSyncCodebases } from "../../utils/codebase.ts";
|
||||
import { createPreviewLocalScriptReader } from "../../utils/local_path_scripts.ts";
|
||||
|
||||
const PORT = 3001;
|
||||
async function dev(opts: GlobalOptions & SyncOptions) {
|
||||
const PROXY_PORT = 3100;
|
||||
|
||||
async function dev(opts: GlobalOptions & SyncOptions & { proxyPort?: number }) {
|
||||
opts = await mergeConfigWithConfigFile(opts);
|
||||
const workspace = await resolveWorkspace(opts);
|
||||
await requireLogin(opts);
|
||||
@@ -106,12 +109,14 @@ async function dev(opts: GlobalOptions & SyncOptions) {
|
||||
codebases,
|
||||
});
|
||||
await replaceAllPathScriptsWithLocal(localFlow.value, localScriptReader, log);
|
||||
const wmFlowPath = localPath.replace(/\.flow\/$/, "").replace(/\/$/, "");
|
||||
currentLastEdit = {
|
||||
type: "flow",
|
||||
flow: localFlow,
|
||||
uriPath: localPath,
|
||||
path: wmFlowPath,
|
||||
};
|
||||
log.info("Updated " + localPath);
|
||||
log.info("Updated " + wmFlowPath);
|
||||
broadcastChanges(currentLastEdit);
|
||||
} else if (typ == "script") {
|
||||
const content = await readFile(cpath, "utf-8");
|
||||
@@ -153,38 +158,49 @@ async function dev(opts: GlobalOptions & SyncOptions) {
|
||||
type: "flow";
|
||||
flow: OpenFlow;
|
||||
uriPath: string;
|
||||
path: string;
|
||||
};
|
||||
|
||||
const connectedClients: Set<WebSocket> = new Set();
|
||||
// Map each connected client to its optional watchPath filter
|
||||
const clientWatchPaths: Map<WebSocket, string | undefined> = new Map();
|
||||
|
||||
// Function to send a message to all connected clients
|
||||
function getEditPath(lastEdit: LastEditScript | LastEditFlow): string {
|
||||
return lastEdit.path;
|
||||
}
|
||||
|
||||
function normalizePath(p: string): string {
|
||||
return p.replace(/\.flow\/?$/, "").replace(/\/$/, "");
|
||||
}
|
||||
|
||||
// Send file changes to clients, filtered by their watchPath
|
||||
function broadcastChanges(lastEdit: LastEditScript | LastEditFlow) {
|
||||
for (const client of connectedClients.values()) {
|
||||
client.send(JSON.stringify(lastEdit));
|
||||
const editPath = normalizePath(getEditPath(lastEdit));
|
||||
const msg = JSON.stringify(lastEdit);
|
||||
for (const [client, watchPath] of clientWatchPaths.entries()) {
|
||||
if (watchPath === undefined || normalizePath(watchPath) === editPath) {
|
||||
client.send(msg);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function startApp() {
|
||||
const server = http.createServer((_req, res) => {
|
||||
res.writeHead(200);
|
||||
res.end();
|
||||
});
|
||||
const wss = new WebSocketServer({ server });
|
||||
|
||||
// WebSocket server event listeners
|
||||
function setupDevWs(wss: WebSocketServer) {
|
||||
wss.on("connection", (ws: WebSocket) => {
|
||||
connectedClients.add(ws);
|
||||
console.log("New client connected");
|
||||
clientWatchPaths.set(ws, undefined);
|
||||
console.log("New dev client connected");
|
||||
|
||||
ws.on("open", () => {
|
||||
if (currentLastEdit) {
|
||||
broadcastChanges(currentLastEdit);
|
||||
// Send the current state to the new client
|
||||
const watchPath = clientWatchPaths.get(ws);
|
||||
if (watchPath === undefined || normalizePath(watchPath) === normalizePath(getEditPath(currentLastEdit))) {
|
||||
ws.send(JSON.stringify(currentLastEdit));
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
ws.on("close", () => {
|
||||
connectedClients.delete(ws);
|
||||
console.log("Client disconnected");
|
||||
clientWatchPaths.delete(ws);
|
||||
console.log("Dev client disconnected");
|
||||
});
|
||||
|
||||
ws.on("message", (message: WebSocket.RawData) => {
|
||||
@@ -198,37 +214,176 @@ async function dev(opts: GlobalOptions & SyncOptions) {
|
||||
|
||||
if (data.type === "load") {
|
||||
loadPaths([data.path]);
|
||||
} else if (data.type === "setWatch") {
|
||||
const path = data.path as string;
|
||||
clientWatchPaths.set(ws, path);
|
||||
console.log(`Client watching: ${path}`);
|
||||
ws.send(JSON.stringify({ type: "watchSet", path }));
|
||||
// Send current state for the watched path if available
|
||||
if (currentLastEdit && normalizePath(getEditPath(currentLastEdit)) === normalizePath(path)) {
|
||||
ws.send(JSON.stringify(currentLastEdit));
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// Start the server
|
||||
const port = await getPort.default({ port: 3001 });
|
||||
const url =
|
||||
async function startLegacyServer(): Promise<number> {
|
||||
const server = http.createServer((_req, res) => {
|
||||
res.writeHead(200);
|
||||
res.end();
|
||||
});
|
||||
const wss = new WebSocketServer({ server });
|
||||
setupDevWs(wss);
|
||||
|
||||
const port = await getPort.default({ port: PORT });
|
||||
return new Promise((resolve) => {
|
||||
server.listen(port, () => {
|
||||
console.log(`Legacy dev server listening on port ${port}`);
|
||||
resolve(port);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function startProxyServer(remoteUrl: string, proxyPort: number, wsPort: number) {
|
||||
const remote = new URL(remoteUrl);
|
||||
const isHttps = remote.protocol === "https:";
|
||||
const remoteHost = remote.hostname;
|
||||
const remotePort = remote.port ? parseInt(remote.port) : (isHttps ? 443 : 80);
|
||||
const httpModule = isHttps ? https : http;
|
||||
|
||||
// Dev WebSocket server (handles /ws_dev path)
|
||||
const devWss = new WebSocketServer({ noServer: true });
|
||||
setupDevWs(devWss);
|
||||
|
||||
// Separate WSS for proxied WebSocket connections (not tracked as dev clients)
|
||||
const proxyWss = new WebSocketServer({ noServer: true });
|
||||
|
||||
const proxyServer = http.createServer((clientReq, clientRes) => {
|
||||
const proxyOpts: http.RequestOptions = {
|
||||
hostname: remoteHost,
|
||||
port: remotePort,
|
||||
path: clientReq.url,
|
||||
method: clientReq.method,
|
||||
headers: {
|
||||
...clientReq.headers,
|
||||
host: remote.host,
|
||||
},
|
||||
};
|
||||
|
||||
const proxyReq = httpModule.request(proxyOpts, (proxyRes) => {
|
||||
// Rewrite Set-Cookie domain to localhost
|
||||
const setCookie = proxyRes.headers["set-cookie"];
|
||||
if (setCookie) {
|
||||
proxyRes.headers["set-cookie"] = setCookie.map((cookie) =>
|
||||
cookie.replace(/domain=[^;]+/gi, "domain=localhost")
|
||||
);
|
||||
}
|
||||
clientRes.writeHead(proxyRes.statusCode ?? 502, proxyRes.headers);
|
||||
proxyRes.pipe(clientRes, { end: true });
|
||||
});
|
||||
|
||||
proxyReq.on("error", (err) => {
|
||||
console.error("Proxy error:", err.message);
|
||||
clientRes.writeHead(502);
|
||||
clientRes.end("Bad Gateway");
|
||||
});
|
||||
|
||||
clientReq.pipe(proxyReq, { end: true });
|
||||
});
|
||||
|
||||
// Handle WebSocket upgrades
|
||||
proxyServer.on("upgrade", (req, socket, head) => {
|
||||
const pathname = req.url?.split("?")[0] ?? "";
|
||||
|
||||
if (pathname === "/ws_dev") {
|
||||
// Handle locally: dev file-change WebSocket
|
||||
devWss.handleUpgrade(req, socket, head, (ws) => {
|
||||
devWss.emit("connection", ws, req);
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Proxy all other WebSocket paths to remote
|
||||
if (pathname.startsWith("/ws/") || pathname.startsWith("/ws_mp/") || pathname.startsWith("/ws_debug/")) {
|
||||
const wsProtocol = isHttps ? "wss" : "ws";
|
||||
const remoteWsUrl = `${wsProtocol}://${remote.host}${req.url}`;
|
||||
const remoteWs = new WebSocket(remoteWsUrl, {
|
||||
headers: {
|
||||
...req.headers,
|
||||
host: remote.host,
|
||||
},
|
||||
});
|
||||
|
||||
remoteWs.on("open", () => {
|
||||
proxyWss.handleUpgrade(req, socket, head, (clientWs) => {
|
||||
clientWs.on("message", (data) => {
|
||||
if (remoteWs.readyState === WebSocket.OPEN) {
|
||||
remoteWs.send(data);
|
||||
}
|
||||
});
|
||||
remoteWs.on("message", (data) => {
|
||||
if (clientWs.readyState === WebSocket.OPEN) {
|
||||
clientWs.send(data);
|
||||
}
|
||||
});
|
||||
clientWs.on("close", () => remoteWs.close());
|
||||
remoteWs.on("close", () => clientWs.close());
|
||||
});
|
||||
});
|
||||
|
||||
remoteWs.on("error", (err) => {
|
||||
console.error("WebSocket proxy error:", err.message);
|
||||
socket.destroy();
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Unknown WS path — destroy
|
||||
socket.destroy();
|
||||
});
|
||||
|
||||
return new Promise<void>((resolve) => {
|
||||
proxyServer.listen(proxyPort, () => {
|
||||
console.log(`Dev proxy listening on http://localhost:${proxyPort}`);
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function startApp() {
|
||||
const wsPort = await startLegacyServer();
|
||||
|
||||
const proxyPort = opts.proxyPort ?? PROXY_PORT;
|
||||
await startProxyServer(workspace.remote, proxyPort, wsPort);
|
||||
|
||||
const legacyUrl =
|
||||
`${workspace.remote}dev?workspace=${workspace.workspaceId}&local=true&wm_token=${workspace.token}` +
|
||||
(port === PORT ? "" : `&port=${port}`);
|
||||
(wsPort === PORT ? "" : `&port=${wsPort}`);
|
||||
|
||||
const proxyUrl =
|
||||
`http://localhost:${proxyPort}/dev?workspace=${workspace.workspaceId}&local=true&wm_token=${workspace.token}`;
|
||||
|
||||
console.log(`\nLegacy dev URL: ${legacyUrl}`);
|
||||
console.log(`Proxy dev URL: ${proxyUrl}`);
|
||||
console.log(`\nTo watch a specific file: ${proxyUrl}&path=<wm_path>`);
|
||||
|
||||
console.log(`Go to ${url}`);
|
||||
try {
|
||||
open.openApp(open.apps.browser, { arguments: [url] }).catch((error) => {
|
||||
open.openApp(open.apps.browser, { arguments: [legacyUrl] }).catch((error) => {
|
||||
console.error(
|
||||
`Failed to open browser, please navigate to ${url}, error: ${error}`
|
||||
`Failed to open browser, please navigate to ${legacyUrl}, error: ${error}`
|
||||
);
|
||||
});
|
||||
console.log("Opened browser for you");
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`Failed to open browser, please navigate to ${url}, ${error}`
|
||||
`Failed to open browser, please navigate to ${legacyUrl}, ${error}`
|
||||
);
|
||||
}
|
||||
|
||||
console.log(
|
||||
"Dev server will automatically point to the last script edited locally"
|
||||
);
|
||||
|
||||
server.listen(port, () => {
|
||||
console.log(`Server listening on port ${port}`);
|
||||
});
|
||||
}
|
||||
|
||||
await Promise.all([startApp(), watchChanges()]);
|
||||
@@ -241,6 +396,10 @@ const command = new Command()
|
||||
"--includes <pattern...:string>",
|
||||
"Filter paths givena glob pattern or path"
|
||||
)
|
||||
.option(
|
||||
"--proxy-port <port:number>",
|
||||
"Port for the localhost reverse proxy (default: 3100)"
|
||||
)
|
||||
.action(dev as any);
|
||||
|
||||
export default command;
|
||||
|
||||
@@ -437,7 +437,6 @@ async function preview(
|
||||
export async function generateLocks(
|
||||
opts: GlobalOptions & {
|
||||
yes?: boolean;
|
||||
dryRun?: boolean;
|
||||
} & SyncOptions,
|
||||
folder: string | undefined
|
||||
) {
|
||||
@@ -488,10 +487,6 @@ export async function generateLocks(
|
||||
}
|
||||
|
||||
if (hasAny) {
|
||||
if (opts.dryRun) {
|
||||
log.info(colors.gray("Dry run complete."));
|
||||
return;
|
||||
}
|
||||
if (
|
||||
!opts.yes &&
|
||||
!(await Confirm.prompt({
|
||||
@@ -597,7 +592,6 @@ const command = new Command()
|
||||
)
|
||||
.arguments("[flow:file]")
|
||||
.option("--yes", "Skip confirmation prompt")
|
||||
.option("--dry-run", "Perform a dry run without making changes")
|
||||
.option(
|
||||
"-i --includes <patterns:file[]>",
|
||||
"Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string)"
|
||||
|
||||
@@ -29,9 +29,10 @@ import { FlowFile } from "./flow.ts";
|
||||
import { FlowValue } from "../../../gen/types.gen.ts";
|
||||
import { replaceInlineScripts } from "../../../windmill-utils-internal/src/inline-scripts/replacer.ts";
|
||||
import { workspaceDependenciesLanguages } from "../../utils/script_common.ts";
|
||||
import { extractNameFromFolder, getFolderSuffix, getNonDottedPaths } from "../../utils/resource_folders.ts";
|
||||
import { extractRelativeImports } from "../../utils/relative_imports.ts";
|
||||
import { DoubleLinkedDependencyTree } from "../../utils/dependency_tree.ts";
|
||||
import {
|
||||
extractNameFromFolder,
|
||||
getNonDottedPaths,
|
||||
} from "../../utils/resource_folders.ts";
|
||||
|
||||
const TOP_HASH = "__flow_hash";
|
||||
async function generateFlowHash(
|
||||
@@ -69,9 +70,7 @@ export async function generateFlowLockInternal(
|
||||
defaultTs?: "bun" | "deno";
|
||||
},
|
||||
justUpdateMetadataLock?: boolean,
|
||||
noStaleMessage?: boolean,
|
||||
legacyBehaviour?: boolean,
|
||||
tree?: DoubleLinkedDependencyTree
|
||||
noStaleMessage?: boolean
|
||||
): Promise<string | FlowLocksResult | void> {
|
||||
if (folder.endsWith(SEP)) {
|
||||
folder = folder.substring(0, folder.length - 1);
|
||||
@@ -81,67 +80,33 @@ export async function generateFlowLockInternal(
|
||||
log.info(`Generating lock for flow ${folder} at ${remote_path}`);
|
||||
}
|
||||
|
||||
// Always get out-of-sync workspace dependencies
|
||||
const rawWorkspaceDependencies: Record<string, string> =
|
||||
await getRawWorkspaceDependencies();
|
||||
|
||||
const flowValue = (await yamlParseFile(
|
||||
folder! + SEP + "flow.yaml"
|
||||
)) as FlowFile;
|
||||
|
||||
const folderNormalized = folder.replaceAll(SEP, "/");
|
||||
const inlineScriptsForTree = extractInlineScriptsForFlows(
|
||||
structuredClone(flowValue.value.modules),
|
||||
{},
|
||||
SEP,
|
||||
// Filter workspace dependencies based on inline scripts' languages and annotations
|
||||
const filteredDeps = await filterWorkspaceDependenciesForFlow(flowValue.value as FlowValue, rawWorkspaceDependencies, folder);
|
||||
|
||||
let hashes = await generateFlowHash(
|
||||
filteredDeps,
|
||||
folder,
|
||||
opts.defaultTs
|
||||
).filter(s => !s.is_lock);
|
||||
);
|
||||
|
||||
let filteredDeps: Record<string, string> = {};
|
||||
const conf = await readLockfile();
|
||||
|
||||
if (!legacyBehaviour && tree) {
|
||||
if (dryRun) {
|
||||
const inlineScriptPaths: string[] = [];
|
||||
for (const script of inlineScriptsForTree) {
|
||||
let content = script.content;
|
||||
if (content.startsWith("!inline ")) {
|
||||
const filePath = folder + SEP + content.replace("!inline ", "");
|
||||
try {
|
||||
content = await readFile(filePath, "utf-8");
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
const treePath = folderNormalized + "/" + path.basename(script.path, path.extname(script.path));
|
||||
const language = script.language as ScriptLanguage;
|
||||
const imports = await extractRelativeImports(content, treePath, language);
|
||||
await tree.addNode(treePath, content, language, "", imports, "inline_script", folderNormalized, folder, false);
|
||||
inlineScriptPaths.push(treePath);
|
||||
}
|
||||
|
||||
const hashes = await generateFlowHash({}, folder, opts.defaultTs);
|
||||
const isDirectlyStale = !(await checkifMetadataUptodate(folder, hashes[TOP_HASH], conf, TOP_HASH));
|
||||
|
||||
await tree.addNode(folderNormalized, "", "bun", "", inlineScriptPaths, "flow", folderNormalized, folder, isDirectlyStale);
|
||||
return;
|
||||
}
|
||||
// Second pass: get mismatched workspace deps from tree
|
||||
filteredDeps = await filterWorkspaceDependenciesForFlow(flowValue.value as FlowValue, tree.getMismatchedWorkspaceDeps(), folder);
|
||||
} else {
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
|
||||
filteredDeps = await filterWorkspaceDependenciesForFlow(flowValue.value as FlowValue, rawWorkspaceDependencies, folder);
|
||||
|
||||
const hashes = await generateFlowHash(filteredDeps, folder, opts.defaultTs);
|
||||
const isDirectlyStale = !(await checkifMetadataUptodate(folder, hashes[TOP_HASH], conf, TOP_HASH));
|
||||
|
||||
if (!isDirectlyStale) {
|
||||
if (!noStaleMessage) {
|
||||
log.info(
|
||||
colors.green(`Flow ${remote_path} metadata is up-to-date, skipping`)
|
||||
);
|
||||
}
|
||||
return;
|
||||
} else if (dryRun) {
|
||||
return remote_path;
|
||||
if (await checkifMetadataUptodate(folder, hashes[TOP_HASH], conf, TOP_HASH)) {
|
||||
if (!noStaleMessage) {
|
||||
log.info(
|
||||
colors.green(`Flow ${remote_path} metadata is up-to-date, skipping`)
|
||||
);
|
||||
}
|
||||
return;
|
||||
} else if (dryRun) {
|
||||
return remote_path;
|
||||
}
|
||||
|
||||
if (Object.keys(filteredDeps).length > 0 && !noStaleMessage) {
|
||||
@@ -157,23 +122,7 @@ export async function generateFlowLockInternal(
|
||||
|
||||
let changedScripts: string[] = [];
|
||||
|
||||
// Build mapping from on-disk file names (hash keys like "a.py") to tree paths
|
||||
// (like "folder/a.inline_script"). The tree uses extractInlineScriptsForFlows without
|
||||
// a path assigner, so paths always have .inline_script suffix, but on-disk files
|
||||
// may not (non-dotted mode).
|
||||
const fileToTreePath = new Map<string, string>();
|
||||
for (const script of inlineScriptsForTree) {
|
||||
const c = script.content;
|
||||
if (c.startsWith("!inline ")) {
|
||||
const fileName = c.replace("!inline ", "");
|
||||
const treePath = folderNormalized + "/" + path.basename(script.path, path.extname(script.path));
|
||||
fileToTreePath.set(fileName, treePath);
|
||||
}
|
||||
}
|
||||
|
||||
if (!justUpdateMetadataLock) {
|
||||
const hashes = await generateFlowHash(filteredDeps, folder, opts.defaultTs);
|
||||
|
||||
//find hashes that do not correspond to previous hashes
|
||||
for (const [path, hash] of Object.entries(hashes)) {
|
||||
if (path == TOP_HASH) {
|
||||
@@ -188,39 +137,27 @@ export async function generateFlowLockInternal(
|
||||
log.info(`Recomputing locks of ${changedScripts.join(", ")} in ${folder}`);
|
||||
}
|
||||
const fileReader = async (path: string) => await readFile(folder + SEP + path, "utf-8");
|
||||
// In tree mode, use the tree's staleness info (which includes transitive dependency changes)
|
||||
// to determine which scripts need relocking, instead of only content-changed ones.
|
||||
const locksToRemove = (tree && !legacyBehaviour)
|
||||
? Object.keys(hashes).filter(k => {
|
||||
if (k === TOP_HASH) return false;
|
||||
const treePath = fileToTreePath.get(k)
|
||||
?? (folderNormalized + "/" + path.basename(k, path.extname(k)));
|
||||
return tree.isStale(treePath);
|
||||
})
|
||||
: changedScripts;
|
||||
await replaceInlineScripts(
|
||||
flowValue.value.modules,
|
||||
fileReader,
|
||||
log,
|
||||
folder + SEP!,
|
||||
SEP,
|
||||
locksToRemove
|
||||
changedScripts
|
||||
);
|
||||
if (flowValue.value.failure_module) {
|
||||
await replaceInlineScripts([flowValue.value.failure_module], fileReader, log, folder + SEP!, SEP, locksToRemove);
|
||||
await replaceInlineScripts([flowValue.value.failure_module], fileReader, log, folder + SEP!, SEP, changedScripts);
|
||||
}
|
||||
if (flowValue.value.preprocessor_module) {
|
||||
await replaceInlineScripts([flowValue.value.preprocessor_module], fileReader, log, folder + SEP!, SEP, locksToRemove);
|
||||
await replaceInlineScripts([flowValue.value.preprocessor_module], fileReader, log, folder + SEP!, SEP, changedScripts);
|
||||
}
|
||||
|
||||
//removeChangedLocks
|
||||
const tempScriptRefs = tree?.getTempScriptRefs(folderNormalized);
|
||||
flowValue.value = await updateFlow(
|
||||
workspace,
|
||||
flowValue.value,
|
||||
remote_path,
|
||||
filteredDeps,
|
||||
tempScriptRefs
|
||||
filteredDeps
|
||||
);
|
||||
|
||||
const lockAssigner = newPathAssigner(opts.defaultTs ?? "bun", {
|
||||
@@ -250,15 +187,13 @@ export async function generateFlowLockInternal(
|
||||
);
|
||||
}
|
||||
|
||||
// Non-legacy mode excludes workspace deps from hash (tracked via tree instead)
|
||||
const depsForHash = (tree && !legacyBehaviour) ? {} : filteredDeps;
|
||||
const finalHashes = await generateFlowHash(
|
||||
depsForHash,
|
||||
hashes = await generateFlowHash(
|
||||
filteredDeps,
|
||||
folder,
|
||||
opts.defaultTs
|
||||
);
|
||||
await clearGlobalLock(folder);
|
||||
for (const [path, hash] of Object.entries(finalHashes)) {
|
||||
for (const [path, hash] of Object.entries(hashes)) {
|
||||
await updateMetadataGlobalLock(folder, hash, path);
|
||||
}
|
||||
if (!noStaleMessage) {
|
||||
@@ -266,16 +201,7 @@ export async function generateFlowLockInternal(
|
||||
}
|
||||
|
||||
// Return the list of updated scripts (extract just the filename from the path)
|
||||
// In tree mode, use the same staleness-aware list we used for lock removal
|
||||
const relocked = (tree && !legacyBehaviour)
|
||||
? Object.keys(finalHashes).filter(k => {
|
||||
if (k === TOP_HASH) return false;
|
||||
const treePath = fileToTreePath.get(k)
|
||||
?? (folderNormalized + "/" + path.basename(k, path.extname(k)));
|
||||
return tree.isStale(treePath);
|
||||
})
|
||||
: changedScripts;
|
||||
const updatedScripts = relocked.map(p => {
|
||||
const updatedScripts = changedScripts.map(p => {
|
||||
const parts = p.split(SEP);
|
||||
return parts[parts.length - 1].replace(/\.[^.]+$/, ""); // Remove extension
|
||||
});
|
||||
@@ -313,8 +239,7 @@ export async function updateFlow(
|
||||
workspace: Workspace,
|
||||
flow_value: FlowValue,
|
||||
remotePath: string,
|
||||
rawWorkspaceDependencies: Record<string, string>,
|
||||
tempScriptRefs?: Record<string, string>
|
||||
rawWorkspaceDependencies: Record<string, string>
|
||||
): Promise<FlowValue | undefined> {
|
||||
let rawResponse;
|
||||
|
||||
@@ -339,9 +264,6 @@ export async function updateFlow(
|
||||
path: remotePath,
|
||||
use_local_lockfiles: true,
|
||||
raw_workspace_dependencies: rawWorkspaceDependencies,
|
||||
...(tempScriptRefs && Object.keys(tempScriptRefs).length > 0
|
||||
? { temp_script_refs: tempScriptRefs }
|
||||
: {}),
|
||||
}),
|
||||
}
|
||||
);
|
||||
@@ -360,9 +282,6 @@ export async function updateFlow(
|
||||
body: JSON.stringify({
|
||||
flow_value,
|
||||
path: remotePath,
|
||||
...(tempScriptRefs && Object.keys(tempScriptRefs).length > 0
|
||||
? { temp_script_refs: tempScriptRefs }
|
||||
: {}),
|
||||
}),
|
||||
}
|
||||
);
|
||||
|
||||
@@ -10,8 +10,6 @@ import * as log from "../../core/log.ts";
|
||||
import {
|
||||
generateScriptMetadataInternal,
|
||||
getRawWorkspaceDependencies,
|
||||
readLockfile,
|
||||
checkifMetadataUptodate,
|
||||
} from "../../utils/metadata.ts";
|
||||
import { generateFlowLockInternal, FlowLocksResult } from "../flow/flow_metadata.ts";
|
||||
import { generateAppLocksInternal, getAppFolders, AppLocksResult } from "../app/app_metadata.ts";
|
||||
@@ -21,20 +19,14 @@ import {
|
||||
ignoreF,
|
||||
} from "../sync/sync.ts";
|
||||
import { exts } from "../script/script.ts";
|
||||
import { isFolderResourcePathAnyFormat, isScriptModulePath, isModuleEntryPoint } from "../../utils/resource_folders.ts";
|
||||
import { isFlowPath, isAppPath, isRawAppPath, isScriptModulePath, isModuleEntryPoint } from "../../utils/resource_folders.ts";
|
||||
import { listSyncCodebases } from "../../utils/codebase.ts";
|
||||
import {
|
||||
DoubleLinkedDependencyTree,
|
||||
uploadScripts,
|
||||
ItemType,
|
||||
} from "../../utils/dependency_tree.ts";
|
||||
|
||||
interface StaleItem {
|
||||
type: ItemType;
|
||||
type: "script" | "flow" | "app";
|
||||
path: string;
|
||||
folder: string;
|
||||
isRawApp?: boolean;
|
||||
staleReason?: string;
|
||||
}
|
||||
|
||||
async function generateMetadata(
|
||||
@@ -46,7 +38,6 @@ async function generateMetadata(
|
||||
skipScripts?: boolean;
|
||||
skipFlows?: boolean;
|
||||
skipApps?: boolean;
|
||||
strictFolderBoundaries?: boolean;
|
||||
} & SyncOptions,
|
||||
folder?: string
|
||||
) {
|
||||
@@ -58,10 +49,12 @@ async function generateMetadata(
|
||||
await requireLogin(opts);
|
||||
opts = await mergeConfigWithConfigFile(opts);
|
||||
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(false);
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies();
|
||||
const codebases = await listSyncCodebases(opts);
|
||||
const ignore = await ignoreF(opts);
|
||||
|
||||
const staleItems: StaleItem[] = [];
|
||||
|
||||
// --schema-only implies skipping flows and apps (they only have locks, no schemas)
|
||||
const skipScripts = opts.skipScripts ?? false;
|
||||
const skipFlows = opts.skipFlows ?? opts.schemaOnly ?? false;
|
||||
@@ -77,11 +70,7 @@ async function generateMetadata(
|
||||
return;
|
||||
}
|
||||
|
||||
log.info(`Checking ${checking.join(", ")}...`);
|
||||
|
||||
// Build dependency tree for relative import tracking
|
||||
const tree = new DoubleLinkedDependencyTree();
|
||||
tree.setWorkspaceDeps(rawWorkspaceDependencies);
|
||||
log.info(colors.gray(`Checking ${checking.join(", ")}...`));
|
||||
|
||||
// === Collect stale scripts ===
|
||||
if (!skipScripts) {
|
||||
@@ -92,7 +81,9 @@ async function generateMetadata(
|
||||
return (
|
||||
(!isD && !exts.some((ext) => p.endsWith(ext))) ||
|
||||
ignore(p, isD) ||
|
||||
isFolderResourcePathAnyFormat(p) ||
|
||||
isFlowPath(p) ||
|
||||
isAppPath(p) ||
|
||||
isRawAppPath(p) ||
|
||||
(isScriptModulePath(p) && !isModuleEntryPoint(p))
|
||||
);
|
||||
},
|
||||
@@ -101,18 +92,19 @@ async function generateMetadata(
|
||||
);
|
||||
|
||||
for (const e of Object.keys(scriptElems)) {
|
||||
await generateScriptMetadataInternal(
|
||||
const candidate = await generateScriptMetadataInternal(
|
||||
e,
|
||||
workspace,
|
||||
opts,
|
||||
true, // dryRun - populate tree
|
||||
true, // dryRun
|
||||
true, // noStaleMessage
|
||||
rawWorkspaceDependencies,
|
||||
codebases,
|
||||
false,
|
||||
false, // legacyBehaviour
|
||||
tree
|
||||
false
|
||||
);
|
||||
if (candidate) {
|
||||
staleItems.push({ type: "script", path: candidate, folder: e });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -134,17 +126,18 @@ async function generateMetadata(
|
||||
)
|
||||
).map((x) => x.substring(0, x.lastIndexOf(SEP)));
|
||||
|
||||
for (const flowFolder of flowElems) {
|
||||
await generateFlowLockInternal(
|
||||
flowFolder,
|
||||
true, // dryRun - populate tree
|
||||
for (const folder of flowElems) {
|
||||
const candidate = await generateFlowLockInternal(
|
||||
folder,
|
||||
true, // dryRun
|
||||
workspace,
|
||||
opts,
|
||||
false,
|
||||
true, // noStaleMessage
|
||||
false, // legacyBehaviour
|
||||
tree
|
||||
true // noStaleMessage
|
||||
);
|
||||
if (candidate) {
|
||||
staleItems.push({ type: "flow", path: candidate, folder });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -168,74 +161,33 @@ async function generateMetadata(
|
||||
const appFolders = getAppFolders(elems, "app.yaml");
|
||||
|
||||
for (const appFolder of rawAppFolders) {
|
||||
await generateAppLocksInternal(
|
||||
const candidate = await generateAppLocksInternal(
|
||||
appFolder,
|
||||
true, // rawApp
|
||||
true, // dryRun - populate tree
|
||||
true, // dryRun
|
||||
workspace,
|
||||
opts,
|
||||
false,
|
||||
true, // noStaleMessage
|
||||
false, // legacyBehaviour
|
||||
tree
|
||||
true // noStaleMessage
|
||||
);
|
||||
if (candidate) {
|
||||
staleItems.push({ type: "app", path: candidate, folder: appFolder, isRawApp: true });
|
||||
}
|
||||
}
|
||||
|
||||
for (const appFolder of appFolders) {
|
||||
await generateAppLocksInternal(
|
||||
const candidate = await generateAppLocksInternal(
|
||||
appFolder,
|
||||
false, // rawApp
|
||||
true, // dryRun - populate tree
|
||||
true, // dryRun
|
||||
workspace,
|
||||
opts,
|
||||
false,
|
||||
true, // noStaleMessage
|
||||
false, // legacyBehaviour
|
||||
tree
|
||||
true // noStaleMessage
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// === Propagate staleness through imports ===
|
||||
tree.propagateStaleness();
|
||||
|
||||
// Upload stale scripts to temp storage so the backend can resolve relative imports.
|
||||
// If this fails (e.g. backend is older and doesn't have /raw_temp endpoints),
|
||||
// degrade gracefully: locks will be generated using deployed script content only.
|
||||
try {
|
||||
await uploadScripts(tree, workspace);
|
||||
} catch (e) {
|
||||
log.warn(colors.yellow(
|
||||
`Failed to upload scripts to temp storage (backend may be too old): ${e}. ` +
|
||||
`Locks will be generated using deployed script versions only — locally modified ` +
|
||||
`relative imports may not be reflected.`
|
||||
));
|
||||
}
|
||||
|
||||
// === Populate staleItems from tree ===
|
||||
const staleItems: StaleItem[] = [];
|
||||
const seenFolders = new Set<string>();
|
||||
|
||||
for (const p of tree.allPaths()) {
|
||||
const staleReason = tree.getStaleReason(p);
|
||||
if (!staleReason) continue;
|
||||
|
||||
const itemType = tree.getItemType(p)!;
|
||||
const itemFolder = tree.getFolder(p)!;
|
||||
|
||||
if (itemType === "dependencies") {
|
||||
staleItems.push({ type: itemType, path: p, folder: itemFolder, staleReason });
|
||||
} else if (itemType === "inline_script") {
|
||||
// Inline scripts are not listed separately — their parent flow/app is stale via propagation
|
||||
continue;
|
||||
} else if (itemType === "script") {
|
||||
const originalPath = tree.getOriginalPath(p)!;
|
||||
staleItems.push({ type: itemType, path: originalPath, folder: itemFolder, staleReason });
|
||||
} else if (!seenFolders.has(itemFolder)) {
|
||||
// Flows/Apps: one entry per folder (dedupe multiple inline scripts)
|
||||
seenFolders.add(itemFolder);
|
||||
const originalPath = tree.getOriginalPath(p)!;
|
||||
staleItems.push({ type: itemType, path: originalPath, folder: itemFolder, isRawApp: tree.getIsRawApp(p), staleReason });
|
||||
if (candidate) {
|
||||
staleItems.push({ type: "app", path: candidate, folder: appFolder, isRawApp: false });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -248,54 +200,11 @@ async function generateMetadata(
|
||||
if (folder.endsWith("/")) {
|
||||
folder = folder.substring(0, folder.length - 1);
|
||||
}
|
||||
// Strip file extension if user passed a specific file path (e.g. f/test/script.ts)
|
||||
const folderNoExt = folder.replace(/\.[^/.]+$/, "");
|
||||
// Check if an item is inside the specified folder
|
||||
const isInsideFolder = (item: StaleItem) => {
|
||||
// Normalize item.folder for comparison (Windows file paths use backslashes)
|
||||
filteredItems = staleItems.filter((item) => {
|
||||
const normalizedFolder = item.folder.replaceAll("\\", "/");
|
||||
const normalizedPath = item.path.replaceAll("\\", "/");
|
||||
return normalizedFolder === folder || normalizedFolder.startsWith(folder + "/")
|
||||
|| normalizedPath === folder || normalizedPath === folderNoExt;
|
||||
};
|
||||
const isPathInFolder = (p: string) => p.startsWith(folder + "/") || p === folder || p === folderNoExt;
|
||||
// Check if a tree path or any of its transitive deps is inside the folder
|
||||
const touchesFolder = (treePath: string) => {
|
||||
if (isPathInFolder(treePath)) return true;
|
||||
let found = false;
|
||||
tree.traverseTransitive(treePath, (importPath) => {
|
||||
if (isPathInFolder(importPath)) {
|
||||
found = true;
|
||||
return true; // stop early
|
||||
}
|
||||
});
|
||||
return found;
|
||||
};
|
||||
|
||||
const isRelevant = (item: StaleItem) => {
|
||||
if (isInsideFolder(item)) return true;
|
||||
if (item.type === "dependencies") return true;
|
||||
const treePath = (item.type === "script"
|
||||
? item.path.replace(/\.[^/.]+$/, "")
|
||||
: item.folder).replaceAll("\\", "/");
|
||||
return touchesFolder(treePath);
|
||||
};
|
||||
|
||||
if (opts.strictFolderBoundaries) {
|
||||
// Strict mode: only items inside the folder
|
||||
filteredItems = staleItems.filter(isInsideFolder);
|
||||
|
||||
// Warn about stale items outside the folder that would be included by default
|
||||
const excludedStale = staleItems.filter((item) => !isInsideFolder(item) && isRelevant(item) && item.type !== "dependencies");
|
||||
for (const item of excludedStale) {
|
||||
const normalizedPath = item.path.replaceAll("\\", "/");
|
||||
log.warn(colors.yellow(
|
||||
`Warning: ${normalizedPath} depends on something inside "${folder}" but is outside it — skipped due to --strict-folder-boundaries. Next generate-metadata will not detect it as stale.`
|
||||
));
|
||||
}
|
||||
} else {
|
||||
// Default: include items inside the folder and any stale importers that transitively depend on it
|
||||
filteredItems = staleItems.filter(isRelevant);
|
||||
}
|
||||
return normalizedFolder === folder || normalizedFolder.startsWith(folder + "/");
|
||||
});
|
||||
}
|
||||
|
||||
// === Show stale items and confirm ===
|
||||
@@ -308,24 +217,28 @@ async function generateMetadata(
|
||||
const scripts = filteredItems.filter((i) => i.type === "script");
|
||||
const flows = filteredItems.filter((i) => i.type === "flow");
|
||||
const apps = filteredItems.filter((i) => i.type === "app");
|
||||
const deps = filteredItems.filter((i) => i.type === "dependencies");
|
||||
|
||||
log.info("");
|
||||
log.info(`Found ${colors.bold(String(filteredItems.length))} item(s) with stale metadata:`);
|
||||
log.info(`Found ${filteredItems.length} item(s) with stale metadata:`);
|
||||
|
||||
const printItems = (label: string, items: StaleItem[]) => {
|
||||
if (items.length === 0) return;
|
||||
log.info(` ${label} (${items.length}):`);
|
||||
for (const item of items) {
|
||||
const reason = item.staleReason ? colors.dim(colors.white(` — ${item.staleReason}`)) : "";
|
||||
log.info(` ~ ${item.path}` + reason);
|
||||
if (scripts.length > 0) {
|
||||
log.info(colors.gray(` Scripts (${scripts.length}):`));
|
||||
for (const item of scripts) {
|
||||
log.info(colors.yellow(` ${item.path}`));
|
||||
}
|
||||
};
|
||||
|
||||
printItems("Workspace dependencies", deps);
|
||||
printItems("Scripts", scripts);
|
||||
printItems("Flows", flows);
|
||||
printItems("Apps", apps);
|
||||
}
|
||||
if (flows.length > 0) {
|
||||
log.info(colors.gray(` Flows (${flows.length}):`));
|
||||
for (const item of flows) {
|
||||
log.info(colors.yellow(` ${item.path}`));
|
||||
}
|
||||
}
|
||||
if (apps.length > 0) {
|
||||
log.info(colors.gray(` Apps (${apps.length}):`));
|
||||
for (const item of apps) {
|
||||
log.info(colors.yellow(` ${item.path}`));
|
||||
}
|
||||
}
|
||||
|
||||
if (opts.dryRun) {
|
||||
return;
|
||||
@@ -346,30 +259,28 @@ async function generateMetadata(
|
||||
log.info("");
|
||||
|
||||
// === Process all stale items with progress counter ===
|
||||
const mismatchedWorkspaceDeps = tree.getMismatchedWorkspaceDeps();
|
||||
const total = filteredItems.length - deps.length;
|
||||
const total = filteredItems.length;
|
||||
const maxWidth = `[${total}/${total}]`.length;
|
||||
let current = 0;
|
||||
|
||||
const formatProgress = (n: number) => {
|
||||
return colors.dim(colors.white(`[${n}/${total}]`.padEnd(maxWidth, " ")));
|
||||
const bracket = `[${n}/${total}]`;
|
||||
return colors.gray(bracket.padEnd(maxWidth, " "));
|
||||
};
|
||||
|
||||
// Process scripts
|
||||
for (const item of scripts) {
|
||||
current++;
|
||||
log.info(`${formatProgress(current)} script ${item.path}`);
|
||||
log.info(`${formatProgress(current)} script ${colors.cyan(item.path)}`);
|
||||
await generateScriptMetadataInternal(
|
||||
item.path, // originalPath with extension
|
||||
item.folder,
|
||||
workspace,
|
||||
opts,
|
||||
false, // dryRun
|
||||
true, // noStaleMessage
|
||||
mismatchedWorkspaceDeps,
|
||||
true, // noStaleMessage - we handle output
|
||||
rawWorkspaceDependencies,
|
||||
codebases,
|
||||
false,
|
||||
false, // legacyBehaviour
|
||||
tree
|
||||
false
|
||||
);
|
||||
}
|
||||
|
||||
@@ -377,49 +288,38 @@ async function generateMetadata(
|
||||
for (const item of flows) {
|
||||
current++;
|
||||
const result = await generateFlowLockInternal(
|
||||
item.folder.replaceAll("/", SEP),
|
||||
item.folder,
|
||||
false, // dryRun
|
||||
workspace,
|
||||
opts,
|
||||
false,
|
||||
true, // noStaleMessage
|
||||
false, // legacyBehaviour
|
||||
tree
|
||||
);
|
||||
const flowResult = result as FlowLocksResult | undefined;
|
||||
const scriptsInfo = flowResult?.updatedScripts?.length
|
||||
? colors.dim(colors.white(`: ${flowResult.updatedScripts.join(", ")}`))
|
||||
true // noStaleMessage - we handle output
|
||||
) as FlowLocksResult | void;
|
||||
const scriptsInfo = result?.updatedScripts?.length
|
||||
? `: ${colors.gray(result.updatedScripts.join(", "))}`
|
||||
: "";
|
||||
log.info(`${formatProgress(current)} flow ${item.path}${scriptsInfo}`);
|
||||
log.info(`${formatProgress(current)} flow ${colors.cyan(item.path)}${scriptsInfo}`);
|
||||
}
|
||||
|
||||
// Process apps
|
||||
for (const item of apps) {
|
||||
current++;
|
||||
const result = await generateAppLocksInternal(
|
||||
item.folder.replaceAll("/", SEP),
|
||||
item.folder,
|
||||
item.isRawApp!, // rawApp
|
||||
false, // dryRun
|
||||
workspace,
|
||||
opts,
|
||||
false,
|
||||
true, // noStaleMessage
|
||||
false, // legacyBehaviour
|
||||
tree
|
||||
);
|
||||
const appResult = result as AppLocksResult | undefined;
|
||||
const scriptsInfo = appResult?.updatedScripts?.length
|
||||
? colors.dim(colors.white(`: ${appResult.updatedScripts.join(", ")}`))
|
||||
true // noStaleMessage - we handle output
|
||||
) as AppLocksResult | void;
|
||||
const scriptsInfo = result?.updatedScripts?.length
|
||||
? `: ${colors.gray(result.updatedScripts.join(", "))}`
|
||||
: "";
|
||||
log.info(`${formatProgress(current)} app ${item.path}${scriptsInfo}`);
|
||||
log.info(`${formatProgress(current)} app ${colors.cyan(item.path)}${scriptsInfo}`);
|
||||
}
|
||||
|
||||
// Persist all stale workspace dep hashes (not just filtered — deps are global, not folder-scoped)
|
||||
const allStaleDeps = staleItems.filter((i) => i.type === "dependencies");
|
||||
await tree.persistDepsHashes(allStaleDeps.map((d) => d.path));
|
||||
|
||||
log.info("");
|
||||
log.info(`Done. Updated ${colors.bold(String(total))} item(s).`);
|
||||
log.info(colors.green(`Done. Updated ${total} item(s).`));
|
||||
}
|
||||
|
||||
const command = new Command()
|
||||
@@ -432,7 +332,6 @@ const command = new Command()
|
||||
.option("--skip-scripts", "Skip processing scripts")
|
||||
.option("--skip-flows", "Skip processing flows")
|
||||
.option("--skip-apps", "Skip processing apps")
|
||||
.option("--strict-folder-boundaries", "Only update items inside the specified folder (requires folder argument)")
|
||||
.option(
|
||||
"-i --includes <patterns:file[]>",
|
||||
"Comma separated patterns to specify which files to include"
|
||||
|
||||
@@ -346,6 +346,30 @@ async function initAction(opts: InitOptions) {
|
||||
log.warn(`Could not create skills: ${skillError}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Create .claude/launch.json for Claude Preview dev server
|
||||
try {
|
||||
const launchJsonPath = ".claude/launch.json";
|
||||
const launchJson = {
|
||||
version: "0.0.1",
|
||||
configurations: [
|
||||
{
|
||||
name: "windmill-dev",
|
||||
runtimeExecutable: "wmill",
|
||||
runtimeArgs: ["dev"],
|
||||
port: 3100,
|
||||
},
|
||||
],
|
||||
};
|
||||
await writeFile(launchJsonPath, JSON.stringify(launchJson, null, 2) + "\n", "utf-8");
|
||||
log.info(colors.green("Created .claude/launch.json"));
|
||||
} catch (launchError) {
|
||||
if (launchError instanceof Error) {
|
||||
log.warn(`Could not create launch.json: ${launchError.message}`);
|
||||
} else {
|
||||
log.warn(`Could not create launch.json: ${launchError}`);
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof Error) {
|
||||
log.warn(`Could not create guidance files: ${error.message}`);
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { GlobalOptions } from "../../types.ts";
|
||||
import { requireLogin } from "../../core/auth.ts";
|
||||
import { resolveWorkspace, validatePath } from "../../core/context.ts";
|
||||
import { readFile, writeFile, stat, mkdir } from "node:fs/promises";
|
||||
import { readFile, writeFile, stat } from "node:fs/promises";
|
||||
import { Buffer } from "node:buffer";
|
||||
import { colors } from "@cliffy/ansi/colors";
|
||||
import { Command } from "@cliffy/command";
|
||||
@@ -130,7 +130,7 @@ async function push(opts: PushOptions, filePath: string) {
|
||||
[],
|
||||
undefined,
|
||||
opts,
|
||||
await getRawWorkspaceDependencies(true),
|
||||
await getRawWorkspaceDependencies(),
|
||||
codebases
|
||||
);
|
||||
log.info(colors.bold.underline.green(`Script ${filePath} pushed`));
|
||||
@@ -1078,11 +1078,6 @@ async function bootstrap(
|
||||
return;
|
||||
}
|
||||
|
||||
// normalize language aliases
|
||||
if (language === ("python" as any)) {
|
||||
language = "python3";
|
||||
}
|
||||
|
||||
const scriptInitialCode = scriptBootstrapCode[language];
|
||||
if (scriptInitialCode === undefined) {
|
||||
throw new Error("Language unknown");
|
||||
@@ -1123,9 +1118,6 @@ async function bootstrap(
|
||||
yamlOptions
|
||||
);
|
||||
|
||||
const parentDir = path.dirname(scriptCodeFileFullPath);
|
||||
await mkdir(parentDir, { recursive: true });
|
||||
|
||||
await writeFile(scriptCodeFileFullPath, scriptInitialCode, {
|
||||
flag: 'wx', encoding: 'utf-8',
|
||||
});
|
||||
@@ -1169,7 +1161,7 @@ export async function generateMetadata(
|
||||
opts = await mergeConfigWithConfigFile(opts);
|
||||
const codebases = await listSyncCodebases(opts);
|
||||
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies();
|
||||
if (scriptPath) {
|
||||
// read script metadata file
|
||||
await generateScriptMetadataInternal(
|
||||
|
||||
@@ -2280,7 +2280,7 @@ export async function pull(
|
||||
|
||||
const tracker: ChangeTracker = await buildTracker(changes);
|
||||
const rawWorkspaceDependencies: Record<string, string> =
|
||||
await getRawWorkspaceDependencies(true);
|
||||
await getRawWorkspaceDependencies();
|
||||
|
||||
for (const change of tracker.scripts) {
|
||||
await generateScriptMetadataInternal(
|
||||
@@ -2611,7 +2611,7 @@ export async function push(
|
||||
false, // els1 (local) is not the remote source
|
||||
);
|
||||
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true);
|
||||
const rawWorkspaceDependencies = await getRawWorkspaceDependencies();
|
||||
|
||||
const tracker: ChangeTracker = await buildTracker(changes);
|
||||
|
||||
@@ -2657,7 +2657,7 @@ export async function push(
|
||||
true,
|
||||
);
|
||||
if (stale) {
|
||||
staleFlows.push(stale as string);
|
||||
staleFlows.push(stale);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2682,7 +2682,7 @@ export async function push(
|
||||
true,
|
||||
);
|
||||
if (stale) {
|
||||
staleApps.push(stale as string);
|
||||
staleApps.push(stale);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2697,7 +2697,7 @@ export async function push(
|
||||
true,
|
||||
);
|
||||
if (stale) {
|
||||
staleApps.push(stale as string);
|
||||
staleApps.push(stale);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+48
-11
File diff suppressed because one or more lines are too long
@@ -1,373 +0,0 @@
|
||||
/**
|
||||
* Double-linked dependency tree for tracking script imports and propagating staleness.
|
||||
*/
|
||||
|
||||
import { Workspace } from "../commands/workspace/workspace.ts";
|
||||
import * as wmill from "../../gen/services.gen.ts";
|
||||
import type { ScriptLang } from "../../gen/types.gen.ts";
|
||||
import { ScriptLanguage } from "./script_common.ts";
|
||||
import {
|
||||
filterWorkspaceDependencies,
|
||||
generateScriptHash,
|
||||
checkifMetadataUptodate,
|
||||
workspaceDependenciesPathToLanguageAndFilename,
|
||||
updateMetadataGlobalLock,
|
||||
} from "./metadata.ts";
|
||||
import { generateHash } from "./utils.ts";
|
||||
|
||||
/**
|
||||
* Diff local scripts against deployed versions, upload only those that differ.
|
||||
* Only uploaded (mismatched) scripts get contentHash set, so flatten() returns
|
||||
* temp_script_refs only for scripts the backend can't resolve from deployed versions.
|
||||
*/
|
||||
export async function uploadScripts(
|
||||
tree: DoubleLinkedDependencyTree,
|
||||
workspace: Workspace
|
||||
): Promise<void> {
|
||||
// Split into scripts vs workspace deps and compute SHA256(content) for each
|
||||
const scriptHashes: Record<string, string> = {};
|
||||
const workspaceDeps: { path: string; language: ScriptLang; name?: string; hash: string }[] = [];
|
||||
|
||||
for (const path of tree.allPaths()) {
|
||||
const content = tree.getContent(path);
|
||||
const itemType = tree.getItemType(path);
|
||||
|
||||
if (itemType === "dependencies") {
|
||||
// Empty string is valid for workspace deps (means "no deps") — only skip undefined
|
||||
if (content === undefined) continue;
|
||||
const info = workspaceDependenciesPathToLanguageAndFilename(path);
|
||||
if (info) {
|
||||
const hash = await generateHash(content);
|
||||
workspaceDeps.push({ path, language: info.language as ScriptLang, name: info.name, hash });
|
||||
}
|
||||
} else if (itemType === "script") {
|
||||
if (!content) continue;
|
||||
const hash = await generateHash(content);
|
||||
scriptHashes[path] = hash;
|
||||
}
|
||||
// Skip inline_script, flow, app — they don't need temp storage uploads
|
||||
}
|
||||
|
||||
if (Object.keys(scriptHashes).length === 0 && workspaceDeps.length === 0) return;
|
||||
|
||||
// Single batch query: find which scripts/deps differ from deployed versions
|
||||
const mismatched = await wmill.diffRawScriptsWithDeployed({
|
||||
workspace: workspace.workspaceId,
|
||||
requestBody: {
|
||||
scripts: scriptHashes,
|
||||
workspace_deps: workspaceDeps,
|
||||
},
|
||||
});
|
||||
|
||||
// Upload only mismatched scripts to temp storage
|
||||
for (const path of mismatched) {
|
||||
const content = tree.getContent(path);
|
||||
const itemType = tree.getItemType(path);
|
||||
|
||||
if (itemType === "dependencies") {
|
||||
// Workspace deps don't need temp storage — just mark as mismatched.
|
||||
// Empty string is valid (means the dep file was emptied locally).
|
||||
if (content !== undefined) {
|
||||
tree.setContentHash(path, "mismatched");
|
||||
}
|
||||
} else if (content) {
|
||||
const hash = await wmill.storeRawScriptTemp({
|
||||
workspace: workspace.workspaceId,
|
||||
requestBody: content,
|
||||
});
|
||||
tree.setContentHash(path, hash);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export type ItemType = "script" | "inline_script" | "flow" | "app" | "dependencies";
|
||||
|
||||
interface DependencyNode {
|
||||
content: string;
|
||||
stalenessHash: string; // Hash for staleness detection (includes deps, content, metadata)
|
||||
contentHash?: string; // Hash for temp storage lookup (content only)
|
||||
language: ScriptLanguage;
|
||||
metadata: string;
|
||||
imports: Set<string>;
|
||||
importedBy: Set<string>;
|
||||
staleReason?: string;
|
||||
// Item metadata for generate-metadata command
|
||||
itemType: ItemType;
|
||||
folder: string; // Folder path (for flows/apps) or remote path (for scripts)
|
||||
originalPath: string; // Original path passed to handler (with extension for scripts)
|
||||
isRawApp?: boolean; // Only set for apps
|
||||
isDirectlyStale: boolean; // True if this item's content changed (vs transitively stale)
|
||||
}
|
||||
|
||||
export class DoubleLinkedDependencyTree {
|
||||
private nodes: Map<string, DependencyNode> = new Map();
|
||||
private workspaceDeps: Record<string, string> = {};
|
||||
|
||||
setWorkspaceDeps(deps: Record<string, string>): void {
|
||||
this.workspaceDeps = deps;
|
||||
}
|
||||
|
||||
async addNode(
|
||||
path: string,
|
||||
content: string,
|
||||
language: ScriptLanguage,
|
||||
metadata: string,
|
||||
imports: string[],
|
||||
itemType: ItemType,
|
||||
folder: string,
|
||||
originalPath: string,
|
||||
isDirectlyStale: boolean,
|
||||
isRawApp?: boolean
|
||||
): Promise<void> {
|
||||
const hasWorkspaceDeps = itemType === "script" || itemType === "inline_script";
|
||||
const filteredDeps = hasWorkspaceDeps
|
||||
? filterWorkspaceDependencies(this.workspaceDeps, content, language)
|
||||
: {};
|
||||
const stalenessHash = await generateScriptHash({}, content, metadata);
|
||||
|
||||
if (!this.nodes.has(path)) {
|
||||
this.nodes.set(path, {
|
||||
content: "", stalenessHash: "", language: "deno", metadata: "",
|
||||
imports: new Set(), importedBy: new Set(),
|
||||
itemType: "script", folder: "", originalPath: "", isDirectlyStale: false,
|
||||
});
|
||||
}
|
||||
const node = this.nodes.get(path)!;
|
||||
node.content = content;
|
||||
node.stalenessHash = stalenessHash;
|
||||
node.language = language;
|
||||
node.metadata = metadata;
|
||||
node.itemType = itemType;
|
||||
node.folder = folder;
|
||||
node.originalPath = originalPath;
|
||||
node.isDirectlyStale = isDirectlyStale;
|
||||
node.isRawApp = isRawApp;
|
||||
|
||||
// Create nodes for referenced workspace deps with content and language.
|
||||
const filteredDepsPaths = Object.keys(filteredDeps);
|
||||
for (const depsPath of filteredDepsPaths) {
|
||||
if (!this.nodes.has(depsPath)) {
|
||||
const depsInfo = workspaceDependenciesPathToLanguageAndFilename(depsPath);
|
||||
const contentHash = await generateHash(filteredDeps[depsPath] + depsPath);
|
||||
const isUpToDate = await checkifMetadataUptodate(depsPath, contentHash, undefined);
|
||||
this.nodes.set(depsPath, {
|
||||
content: filteredDeps[depsPath],
|
||||
stalenessHash: "", language: depsInfo?.language ?? "deno", metadata: "",
|
||||
imports: new Set(), importedBy: new Set(),
|
||||
itemType: "dependencies", folder: "", originalPath: depsPath,
|
||||
isDirectlyStale: !isUpToDate,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const allImports = [...imports, ...filteredDepsPaths];
|
||||
for (const importPath of allImports) {
|
||||
node.imports.add(importPath);
|
||||
|
||||
if (!this.nodes.has(importPath)) {
|
||||
this.nodes.set(importPath, {
|
||||
content: "", stalenessHash: "", language: "deno", metadata: "",
|
||||
imports: new Set(), importedBy: new Set(),
|
||||
itemType: "script", folder: "", originalPath: "", isDirectlyStale: false,
|
||||
});
|
||||
}
|
||||
this.nodes.get(importPath)!.importedBy.add(path);
|
||||
}
|
||||
}
|
||||
|
||||
getContent(path: string): string | undefined {
|
||||
return this.nodes.get(path)?.content;
|
||||
}
|
||||
|
||||
getStalenessHash(path: string): string | undefined {
|
||||
return this.nodes.get(path)?.stalenessHash;
|
||||
}
|
||||
|
||||
getContentHash(path: string): string | undefined {
|
||||
return this.nodes.get(path)?.contentHash;
|
||||
}
|
||||
|
||||
setContentHash(path: string, hash: string): void {
|
||||
const node = this.nodes.get(path);
|
||||
if (node) {
|
||||
node.contentHash = hash;
|
||||
}
|
||||
}
|
||||
|
||||
getLanguage(path: string): ScriptLanguage | undefined {
|
||||
return this.nodes.get(path)?.language;
|
||||
}
|
||||
|
||||
getMetadata(path: string): string | undefined {
|
||||
return this.nodes.get(path)?.metadata;
|
||||
}
|
||||
|
||||
getStaleReason(path: string): string | undefined {
|
||||
return this.nodes.get(path)?.staleReason;
|
||||
}
|
||||
|
||||
getItemType(path: string): ItemType | undefined {
|
||||
return this.nodes.get(path)?.itemType;
|
||||
}
|
||||
|
||||
getFolder(path: string): string | undefined {
|
||||
return this.nodes.get(path)?.folder;
|
||||
}
|
||||
|
||||
getIsRawApp(path: string): boolean | undefined {
|
||||
return this.nodes.get(path)?.isRawApp;
|
||||
}
|
||||
|
||||
getIsDirectlyStale(path: string): boolean {
|
||||
return this.nodes.get(path)?.isDirectlyStale ?? false;
|
||||
}
|
||||
|
||||
getOriginalPath(path: string): string | undefined {
|
||||
return this.nodes.get(path)?.originalPath;
|
||||
}
|
||||
|
||||
getImports(path: string): Set<string> | undefined {
|
||||
return this.nodes.get(path)?.imports;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true if this node has been marked stale (directly or transitively).
|
||||
*/
|
||||
isStale(path: string): boolean {
|
||||
return this.nodes.get(path)?.staleReason !== undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mutates the tree by removing all nodes that are not stale.
|
||||
* Uses BFS on reverse graph (importedBy) to find all stale scripts.
|
||||
* Starts from nodes with isDirectlyStale=true.
|
||||
*/
|
||||
propagateStaleness(): void {
|
||||
// Collect directly stale nodes
|
||||
const directlyStale = new Set<string>();
|
||||
for (const [path, node] of this.nodes.entries()) {
|
||||
if (node.isDirectlyStale) {
|
||||
directlyStale.add(path);
|
||||
node.staleReason = "content changed";
|
||||
}
|
||||
}
|
||||
|
||||
const allStale = new Set(directlyStale);
|
||||
const queue = [...directlyStale];
|
||||
const visited = new Set<string>();
|
||||
|
||||
while (queue.length > 0) {
|
||||
const scriptPath = queue.shift()!;
|
||||
if (visited.has(scriptPath)) continue;
|
||||
visited.add(scriptPath);
|
||||
|
||||
const node = this.nodes.get(scriptPath);
|
||||
if (!node) continue;
|
||||
|
||||
for (const importer of node.importedBy) {
|
||||
if (!allStale.has(importer)) {
|
||||
allStale.add(importer);
|
||||
queue.push(importer);
|
||||
// Set reason for transitively stale scripts
|
||||
const importerNode = this.nodes.get(importer);
|
||||
if (importerNode) importerNode.staleReason = `depends on ${scriptPath}`;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* Walks all transitive imports for a node, calling the callback for each.
|
||||
* Callback may return true to stop traversing that branch.
|
||||
*/
|
||||
traverseTransitive(scriptPath: string, callback: (importPath: string, node: DependencyNode) => boolean | void): void {
|
||||
const queue = [scriptPath];
|
||||
const visited = new Set<string>();
|
||||
|
||||
while (queue.length > 0) {
|
||||
const current = queue.shift()!;
|
||||
if (visited.has(current)) continue;
|
||||
visited.add(current);
|
||||
|
||||
const node = this.nodes.get(current);
|
||||
if (!node) continue;
|
||||
|
||||
for (const importPath of node.imports) {
|
||||
const importNode = this.nodes.get(importPath);
|
||||
if (importNode) {
|
||||
const stop = callback(importPath, importNode);
|
||||
if (!stop) {
|
||||
queue.push(importPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
allPaths(): IterableIterator<string> {
|
||||
return this.nodes.keys();
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns paths of all stale nodes (those with a staleReason).
|
||||
*/
|
||||
*stalePaths(): IterableIterator<string> {
|
||||
for (const [path, node] of this.nodes.entries()) {
|
||||
if (node.staleReason) {
|
||||
yield path;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
has(path: string): boolean {
|
||||
return this.nodes.has(path);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns workspace deps that were uploaded as mismatched with remote.
|
||||
* These need to be passed as raw_workspace_dependencies in job args
|
||||
* so the backend uses local content instead of deployed.
|
||||
*/
|
||||
getMismatchedWorkspaceDeps(): Record<string, string> {
|
||||
const result: Record<string, string> = {};
|
||||
for (const [path, node] of this.nodes.entries()) {
|
||||
if (node.itemType === "dependencies" && node.contentHash && node.content !== undefined) {
|
||||
result[path] = node.content;
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns path → contentHash for all transitive imports that have been uploaded.
|
||||
* Must be called after uploadScripts() has populated contentHash values.
|
||||
*/
|
||||
getTempScriptRefs(scriptPath: string): Record<string, string> {
|
||||
const result: Record<string, string> = {};
|
||||
this.traverseTransitive(scriptPath, (_path, node) => {
|
||||
if (node.contentHash) {
|
||||
result[_path] = node.contentHash;
|
||||
}
|
||||
});
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist workspace dep hashes to wmill-lock.yaml so getRawWorkspaceDependencies
|
||||
* considers them up-to-date on the next run.
|
||||
*/
|
||||
async persistDepsHashes(depsPaths: string[]): Promise<void> {
|
||||
for (const path of depsPaths) {
|
||||
const node = this.nodes.get(path);
|
||||
if (node?.itemType === "dependencies" && node.content !== undefined) {
|
||||
const hash = await generateHash(node.content + path);
|
||||
await updateMetadataGlobalLock(path, hash);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
get size(): number {
|
||||
return this.nodes.size;
|
||||
}
|
||||
}
|
||||
+34
-70
@@ -26,13 +26,11 @@ import { generateHash, readInlinePathSync, getHeaders } from "./utils.ts";
|
||||
import { SyncCodebase } from "./codebase.ts";
|
||||
import { argSigToJsonSchemaType } from "../../windmill-utils-internal/src/parse/parse-schema.ts";
|
||||
import { getIsWin } from "./utils.ts";
|
||||
import { extractRelativeImports } from "./relative_imports.ts";
|
||||
import { DoubleLinkedDependencyTree } from "./dependency_tree.ts";
|
||||
|
||||
const _require = createRequire(import.meta.url);
|
||||
const _parserCache = new Map<string, Promise<any>>();
|
||||
|
||||
export function loadParser(pkgName: string): Promise<any> {
|
||||
function loadParser(pkgName: string): Promise<any> {
|
||||
let p = _parserCache.get(pkgName);
|
||||
if (!p) {
|
||||
p = (async () => {
|
||||
@@ -56,7 +54,7 @@ export class LockfileGenerationError extends Error {
|
||||
}
|
||||
|
||||
|
||||
export async function getRawWorkspaceDependencies(legacyBehaviour: boolean): Promise<Record<string, string>> {
|
||||
export async function getRawWorkspaceDependencies(): Promise<Record<string, string>> {
|
||||
const rawWorkspaceDeps: Record<string, string> = {};
|
||||
|
||||
try {
|
||||
@@ -70,13 +68,11 @@ export async function getRawWorkspaceDependencies(legacyBehaviour: boolean): Pro
|
||||
// Find matching language
|
||||
for (const lang of workspaceDependenciesLanguages) {
|
||||
if (entry.name.endsWith(lang.filename)) {
|
||||
if (legacyBehaviour) {
|
||||
const contentHash = await generateHash(content + filePath);
|
||||
const isUpToDate = await checkifMetadataUptodate(filePath, contentHash, undefined);
|
||||
if (!isUpToDate) {
|
||||
rawWorkspaceDeps[filePath] = content;
|
||||
}
|
||||
} else {
|
||||
// Check if out of sync
|
||||
const contentHash = await generateHash(content + filePath);
|
||||
const isUpToDate = await checkifMetadataUptodate(filePath, contentHash, undefined);
|
||||
|
||||
if (!isUpToDate) {
|
||||
rawWorkspaceDeps[filePath] = content;
|
||||
}
|
||||
break;
|
||||
@@ -190,9 +186,7 @@ export async function generateScriptMetadataInternal(
|
||||
noStaleMessage: boolean,
|
||||
rawWorkspaceDependencies: Record<string, string>,
|
||||
codebases: SyncCodebase[],
|
||||
justUpdateMetadataLock?: boolean,
|
||||
legacyBehaviour?: boolean,
|
||||
tree?: DoubleLinkedDependencyTree
|
||||
justUpdateMetadataLock?: boolean
|
||||
): Promise<string | undefined> {
|
||||
// Detect folder layout: my_script__mod/script.ts
|
||||
const isFolderLayout = isModuleEntryPoint(scriptPath);
|
||||
@@ -228,15 +222,13 @@ export async function generateScriptMetadataInternal(
|
||||
|
||||
const hasModules = existsSync(moduleFolderPath) && statSync(moduleFolderPath).isDirectory();
|
||||
|
||||
// In non-legacy mode, workspace deps are tracked via the tree — exclude from hash
|
||||
const depsForHash = (!legacyBehaviour && tree) ? {} : filteredRawWorkspaceDependencies;
|
||||
let hash = await generateScriptHash(depsForHash, scriptContent, metadataContent);
|
||||
let hash = await generateScriptHash(filteredRawWorkspaceDependencies, scriptContent, metadataContent);
|
||||
|
||||
// Compute per-module hashes for stale detection (like flow inline scripts)
|
||||
let moduleHashes: Record<string, string> = {};
|
||||
if (hasModules) {
|
||||
moduleHashes = await computeModuleHashes(
|
||||
moduleFolderPath, opts.defaultTs, (!legacyBehaviour && tree) ? {} : rawWorkspaceDependencies, isFolderLayout
|
||||
moduleFolderPath, opts.defaultTs, rawWorkspaceDependencies, isFolderLayout
|
||||
);
|
||||
}
|
||||
const hasModuleHashes = Object.keys(moduleHashes).length > 0;
|
||||
@@ -251,43 +243,27 @@ export async function generateScriptMetadataInternal(
|
||||
}
|
||||
|
||||
const conf = await readLockfile();
|
||||
|
||||
// Use checkHash (includes module hashes) so module changes are detected as stale
|
||||
const isDirectlyStale = !(await checkifMetadataUptodate(remotePath, checkHash, conf, checkSubpath));
|
||||
|
||||
// New behaviour: tree-based dependency tracking
|
||||
if (!legacyBehaviour && tree) {
|
||||
if (dryRun) {
|
||||
// First pass: populate tree with script and its imports
|
||||
const imports = await extractRelativeImports(scriptContent, remotePath, language);
|
||||
await tree.addNode(remotePath, scriptContent, language, metadataContent, imports, "script", remotePath, scriptPath, isDirectlyStale);
|
||||
return;
|
||||
if (await checkifMetadataUptodate(remotePath, checkHash, conf, checkSubpath)) {
|
||||
if (!noStaleMessage) {
|
||||
log.info(
|
||||
colors.green(`Script ${remotePath} metadata is up-to-date, skipping`)
|
||||
);
|
||||
}
|
||||
// Second pass: proceed to generate (caller verified this script is stale via tree)
|
||||
} else {
|
||||
// Legacy behaviour: use existing staleness check
|
||||
if (await checkifMetadataUptodate(remotePath, checkHash, conf, checkSubpath)) {
|
||||
if (!noStaleMessage) {
|
||||
log.info(
|
||||
colors.green(`Script ${remotePath} metadata is up-to-date, skipping`)
|
||||
);
|
||||
}
|
||||
return;
|
||||
} else if (dryRun) {
|
||||
let detail = `${remotePath} (${language})`;
|
||||
if (hasModuleHashes) {
|
||||
const changed: string[] = [];
|
||||
for (const [modulePath, moduleHash] of Object.entries(moduleHashes)) {
|
||||
if (!(await checkifMetadataUptodate(remotePath, moduleHash, conf, modulePath))) {
|
||||
changed.push(modulePath);
|
||||
}
|
||||
}
|
||||
if (changed.length > 0) {
|
||||
detail += ` [changed modules: ${changed.join(", ")}]`;
|
||||
return;
|
||||
} else if (dryRun) {
|
||||
let detail = `${remotePath} (${language})`;
|
||||
if (hasModuleHashes) {
|
||||
const changed: string[] = [];
|
||||
for (const [modulePath, moduleHash] of Object.entries(moduleHashes)) {
|
||||
if (!(await checkifMetadataUptodate(remotePath, moduleHash, conf, modulePath))) {
|
||||
changed.push(modulePath);
|
||||
}
|
||||
}
|
||||
return detail;
|
||||
if (changed.length > 0) {
|
||||
detail += ` [changed modules: ${changed.join(", ")}]`;
|
||||
}
|
||||
}
|
||||
return detail;
|
||||
}
|
||||
|
||||
if (!justUpdateMetadataLock && !noStaleMessage) {
|
||||
@@ -312,7 +288,6 @@ export async function generateScriptMetadataInternal(
|
||||
const hasCodebase = findCodebase(scriptPath, codebases) != undefined;
|
||||
|
||||
if (!hasCodebase) {
|
||||
const tempScriptRefs = tree?.getTempScriptRefs(remotePath);
|
||||
const lockPathOverride = isFolderLayout
|
||||
? path.dirname(scriptPath) + "/script.lock"
|
||||
: undefined;
|
||||
@@ -323,7 +298,6 @@ export async function generateScriptMetadataInternal(
|
||||
remotePath,
|
||||
metadataParsedContent,
|
||||
filteredRawWorkspaceDependencies,
|
||||
tempScriptRefs,
|
||||
lockPathOverride,
|
||||
);
|
||||
} else {
|
||||
@@ -384,7 +358,7 @@ export async function generateScriptMetadataInternal(
|
||||
const metadataContentUsedForHash = newMetadataContent;
|
||||
|
||||
hash = await generateScriptHash(
|
||||
depsForHash,
|
||||
filteredRawWorkspaceDependencies,
|
||||
scriptContent,
|
||||
metadataContentUsedForHash
|
||||
);
|
||||
@@ -537,7 +511,6 @@ export async function computeLockCacheKey(
|
||||
scriptContent: string,
|
||||
language: ScriptLanguage,
|
||||
rawWorkspaceDependencies: Record<string, string>,
|
||||
tempScriptRefs?: Record<string, string>
|
||||
): Promise<string> {
|
||||
const annotation = extractWorkspaceDepsAnnotation(scriptContent, language);
|
||||
const annotationStr = annotation
|
||||
@@ -545,10 +518,7 @@ export async function computeLockCacheKey(
|
||||
: "none";
|
||||
const sortedDepsKeys = Object.keys(rawWorkspaceDependencies).sort();
|
||||
const depsStr = sortedDepsKeys.map((k) => `${k}=${rawWorkspaceDependencies[k]}`).join(";");
|
||||
const tempRefsStr = tempScriptRefs
|
||||
? Object.keys(tempScriptRefs).sort().map((k) => `${k}=${tempScriptRefs[k]}`).join(";")
|
||||
: "";
|
||||
return await generateHash(`${language}|${annotationStr}|${depsStr}|${tempRefsStr}`);
|
||||
return await generateHash(`${language}|${annotationStr}|${depsStr}`);
|
||||
}
|
||||
|
||||
const lockCache = new Map<string, string>();
|
||||
@@ -563,15 +533,13 @@ async function fetchScriptLock(
|
||||
language: ScriptLanguage,
|
||||
remotePath: string,
|
||||
rawWorkspaceDependencies: Record<string, string>,
|
||||
tempScriptRefs?: Record<string, string>
|
||||
): Promise<string> {
|
||||
const hasRawDeps = Object.keys(rawWorkspaceDependencies).length > 0;
|
||||
const hasTempRefs = tempScriptRefs && Object.keys(tempScriptRefs).length > 0;
|
||||
const cacheKey = (hasRawDeps || hasTempRefs)
|
||||
? await computeLockCacheKey(scriptContent, language, rawWorkspaceDependencies, tempScriptRefs)
|
||||
const cacheKey = hasRawDeps
|
||||
? await computeLockCacheKey(scriptContent, language, rawWorkspaceDependencies)
|
||||
: undefined;
|
||||
if (cacheKey && lockCache.has(cacheKey)) {
|
||||
log.debug(`Using cached lockfile for ${remotePath}`);
|
||||
log.info(`Using cached lockfile for ${remotePath}`);
|
||||
return lockCache.get(cacheKey)!;
|
||||
}
|
||||
|
||||
@@ -596,8 +564,6 @@ async function fetchScriptLock(
|
||||
raw_workspace_dependencies: Object.keys(rawWorkspaceDependencies).length > 0
|
||||
? rawWorkspaceDependencies : null,
|
||||
entrypoint: remotePath,
|
||||
temp_script_refs: tempScriptRefs && Object.keys(tempScriptRefs).length > 0
|
||||
? tempScriptRefs : null,
|
||||
}),
|
||||
}
|
||||
);
|
||||
@@ -638,7 +604,6 @@ async function updateScriptLock(
|
||||
remotePath: string,
|
||||
metadataContent: Record<string, any>,
|
||||
rawWorkspaceDependencies: Record<string, string>,
|
||||
tempScriptRefs?: Record<string, string>,
|
||||
lockPathOverride?: string,
|
||||
): Promise<void> {
|
||||
if (
|
||||
@@ -656,7 +621,7 @@ async function updateScriptLock(
|
||||
|
||||
if (Object.keys(rawWorkspaceDependencies).length > 0) {
|
||||
const dependencyPaths = Object.keys(rawWorkspaceDependencies).join(', ');
|
||||
log.debug(`Generating script lock for ${remotePath} with raw workspace dependencies: ${dependencyPaths}`);
|
||||
log.info(`Generating script lock for ${remotePath} with raw workspace dependencies: ${dependencyPaths}`);
|
||||
}
|
||||
|
||||
const lock = await fetchScriptLock(
|
||||
@@ -665,7 +630,6 @@ async function updateScriptLock(
|
||||
language,
|
||||
remotePath,
|
||||
rawWorkspaceDependencies,
|
||||
tempScriptRefs
|
||||
);
|
||||
|
||||
const lockPath = lockPathOverride ?? remotePath + ".script.lock";
|
||||
@@ -728,7 +692,7 @@ async function updateModuleLocks(
|
||||
const moduleContent = readFileSync(fullPath, "utf-8");
|
||||
const moduleRemotePath = scriptRemotePath + "/" + relPath;
|
||||
|
||||
log.debug(`Generating lock for module ${relPath}`);
|
||||
log.info(colors.gray(`Generating lock for module ${relPath}`));
|
||||
|
||||
try {
|
||||
const lock = await fetchScriptLock(
|
||||
|
||||
@@ -1,39 +0,0 @@
|
||||
/**
|
||||
* Relative Imports Utilities for CLI
|
||||
*
|
||||
* Provides functions to parse relative imports from TypeScript/Python scripts using WASM.
|
||||
*/
|
||||
|
||||
import { ScriptLanguage } from "./script_common.ts";
|
||||
import { loadParser } from "./metadata.ts";
|
||||
import * as log from "../core/log.ts";
|
||||
|
||||
/**
|
||||
* Extract relative imports from script content based on language.
|
||||
* Returns resolved absolute Windmill paths (e.g., "f/folder/helper").
|
||||
*/
|
||||
export async function extractRelativeImports(
|
||||
code: string,
|
||||
scriptPath: string,
|
||||
language: ScriptLanguage
|
||||
): Promise<string[]> {
|
||||
try {
|
||||
switch (language) {
|
||||
case "bun":
|
||||
case "nativets":
|
||||
case "deno": {
|
||||
const { parse_ts_relative_imports } = await loadParser("windmill-parser-wasm-ts");
|
||||
return parse_ts_relative_imports(code, scriptPath);
|
||||
}
|
||||
case "python3": {
|
||||
const { parse_py_relative_imports } = await loadParser("windmill-parser-wasm-py-imports");
|
||||
return parse_py_relative_imports(code, scriptPath);
|
||||
}
|
||||
default:
|
||||
return [];
|
||||
}
|
||||
} catch (e) {
|
||||
log.warn(`Failed to parse relative imports for ${scriptPath}: ${e}. Dependency tracking for relative imports will be disabled.`);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
@@ -189,26 +189,6 @@ export function isFolderResourcePath(p: string): boolean {
|
||||
return isFlowPath(p) || isAppPath(p) || isRawAppPath(p);
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a path is inside a folder-based resource, checking BOTH dotted (.flow, .app, .raw_app)
|
||||
* and non-dotted (__flow, __app, __raw_app) formats regardless of the global nonDottedPaths setting.
|
||||
* Use this instead of isFolderResourcePath when the config may not yet be loaded or when
|
||||
* you need to handle mixed-format workspaces (e.g. generate-metadata scanning all files).
|
||||
*/
|
||||
export function isFolderResourcePathAnyFormat(p: string): boolean {
|
||||
const n = normalizeSep(p);
|
||||
for (const suffixes of [DOTTED_SUFFIXES, NON_DOTTED_SUFFIXES]) {
|
||||
if (
|
||||
n.includes(suffixes.flow + "/") ||
|
||||
n.includes(suffixes.app + "/") ||
|
||||
n.includes(suffixes.raw_app + "/")
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect the resource type from a path, if any
|
||||
*/
|
||||
|
||||
@@ -63,11 +63,11 @@ export class CargoBackend {
|
||||
|
||||
// Determine default features based on environment
|
||||
// CI mode: minimal features (zip only)
|
||||
// Local mode with license key: full features (zip, private, enterprise, license, python)
|
||||
// Local mode with license key: full features (zip, private, enterprise, license)
|
||||
// Local mode without license key: zip only (EE features reject API calls without valid license)
|
||||
const isCI = process.env["CI_MINIMAL_FEATURES"] === "true";
|
||||
const hasLicenseKey = !!process.env["EE_LICENSE_KEY"];
|
||||
const defaultFeatures = isCI ? ["zip"] : (hasLicenseKey ? ["zip", "private", "enterprise", "license", "python"] : ["zip", "python"]);
|
||||
const defaultFeatures = isCI ? ["zip"] : (hasLicenseKey ? ["zip", "private", "enterprise", "license"] : ["zip"]);
|
||||
|
||||
// Parse additional features from environment variable
|
||||
const envFeatures = process.env["TEST_FEATURES"]?.split(",").filter(f => f.trim()) || [];
|
||||
@@ -328,8 +328,6 @@ export class CargoBackend {
|
||||
SQLX_OFFLINE: "true",
|
||||
// Disable embedding to speed up startup
|
||||
DISABLE_EMBEDDING: "true",
|
||||
// Skip worker version check for workspace deps (workers need time to report version)
|
||||
WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES: "1",
|
||||
// Create default admin user
|
||||
CREATE_SUPERADMIN_IF_NOT_EXISTS: "1",
|
||||
SUPERADMIN_EMAIL: this.config.username,
|
||||
@@ -710,7 +708,6 @@ export class CargoBackend {
|
||||
this.deleteAll("resources"),
|
||||
this.deleteAll("variables"),
|
||||
this.deleteAll("folders"),
|
||||
this.deleteAllWorkspaceDeps(),
|
||||
]);
|
||||
|
||||
console.log("Workspace reset complete");
|
||||
@@ -738,28 +735,6 @@ export class CargoBackend {
|
||||
// Ignore listing failures
|
||||
}
|
||||
}
|
||||
|
||||
private async deleteAllWorkspaceDeps(): Promise<void> {
|
||||
try {
|
||||
const listResponse = await this.apiRequest(`/api/w/${this.config.workspace}/workspace_dependencies/list`);
|
||||
if (!listResponse.ok) return;
|
||||
|
||||
const items = await listResponse.json() as { language: string; name?: string }[];
|
||||
for (const item of items) {
|
||||
try {
|
||||
const nameParam = item.name ? `?name=${encodeURIComponent(item.name)}` : "";
|
||||
await this.apiRequest(
|
||||
`/api/w/${this.config.workspace}/workspace_dependencies/delete/${item.language}${nameParam}`,
|
||||
{ method: "POST" }
|
||||
);
|
||||
} catch {
|
||||
// Ignore individual deletion failures
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Ignore failures
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Global backend instance
|
||||
|
||||
@@ -1,420 +0,0 @@
|
||||
/**
|
||||
* Relative Imports Tests
|
||||
*
|
||||
* E2E tests for the `generate-metadata` command with relative imports:
|
||||
* - Lock files correctly include transitive dependencies
|
||||
* - Staleness propagates through import chains
|
||||
* - Various import patterns handled correctly
|
||||
*/
|
||||
|
||||
import { expect, test } from "bun:test";
|
||||
import { writeFile, readFile, mkdir } from "node:fs/promises";
|
||||
import { withTestBackend } from "./test_backend.ts";
|
||||
|
||||
// TODO: re-enable Python tests on CI if python feature is included by default
|
||||
const isCI = process.env["CI_MINIMAL_FEATURES"] === "true";
|
||||
|
||||
const defaultMetadata = `summary: "Test"
|
||||
schema:
|
||||
type: object
|
||||
properties: {}
|
||||
lock: ""
|
||||
`;
|
||||
|
||||
// =============================================================================
|
||||
// Test 1: TS basic import with npm dependency propagation
|
||||
// =============================================================================
|
||||
|
||||
test("TS: imported script's npm dep appears in importer's lock", { timeout: 60000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
|
||||
includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/test`, { recursive: true });
|
||||
|
||||
const scriptA = `import { helper } from "./script_b.ts";
|
||||
export async function main() { return helper(); }
|
||||
`;
|
||||
const scriptB = `import _ from "lodash";
|
||||
export function helper() { return _.VERSION; }
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
|
||||
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
|
||||
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
|
||||
|
||||
const result = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(result.code, `generate-metadata failed:\nSTDOUT: ${result.stdout}\nSTDERR: ${result.stderr}`).toBe(0);
|
||||
|
||||
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
|
||||
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
|
||||
|
||||
expect(lockB).toContain("lodash");
|
||||
expect(lockA).toContain("lodash");
|
||||
});
|
||||
});
|
||||
|
||||
// =============================================================================
|
||||
// Test 2: TS chained imports - dependency propagates through chain
|
||||
// =============================================================================
|
||||
|
||||
test("TS: chained imports propagate npm deps through entire chain", { timeout: 60000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
|
||||
includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/test`, { recursive: true });
|
||||
|
||||
const scriptA = `import { utilB } from "./script_b.ts";
|
||||
export async function main() { return utilB(); }
|
||||
`;
|
||||
const scriptB = `import { utilC } from "./script_c.ts";
|
||||
export function utilB() { return utilC() + " B"; }
|
||||
`;
|
||||
const scriptC = `import _ from "lodash";
|
||||
export function utilC() { return _.VERSION; }
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
|
||||
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
|
||||
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_c.ts`, scriptC);
|
||||
await writeFile(`${tempDir}/f/test/script_c.script.yaml`, defaultMetadata);
|
||||
|
||||
const result = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(result.code).toBe(0);
|
||||
|
||||
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
|
||||
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
|
||||
const lockC = await readFile(`${tempDir}/f/test/script_c.script.lock`, "utf-8").catch(() => "");
|
||||
|
||||
expect(lockC).toContain("lodash");
|
||||
expect(lockB).toContain("lodash");
|
||||
expect(lockA).toContain("lodash");
|
||||
});
|
||||
});
|
||||
|
||||
// =============================================================================
|
||||
// Test 3: TS circular imports - completes without hanging, locks generated
|
||||
// =============================================================================
|
||||
|
||||
test("TS: circular imports handled gracefully with correct locks", { timeout: 60000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
|
||||
includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/test`, { recursive: true });
|
||||
|
||||
// Circular: A imports B, B imports A, B has npm dep
|
||||
const scriptA = `import { funcB } from "./script_b.ts";
|
||||
export function funcA() { return "A"; }
|
||||
export async function main() { return funcA() + funcB(); }
|
||||
`;
|
||||
const scriptB = `import { funcA } from "./script_a.ts";
|
||||
import _ from "lodash";
|
||||
export function funcB() { return _.VERSION + funcA(); }
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
|
||||
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
|
||||
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
|
||||
|
||||
const result = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(result.code).toBe(0);
|
||||
|
||||
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
|
||||
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
|
||||
|
||||
expect(lockB).toContain("lodash");
|
||||
expect(lockA).toContain("lodash");
|
||||
});
|
||||
});
|
||||
|
||||
// =============================================================================
|
||||
// Test 4: Python basic import with pip dependency propagation
|
||||
// =============================================================================
|
||||
|
||||
test.skipIf(isCI)("Python: imported script's pip dep appears in importer's lock", { timeout: 60000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/test`, { recursive: true });
|
||||
|
||||
const mainPy = `from f.test.helper import helper_func
|
||||
|
||||
def main():
|
||||
return helper_func()
|
||||
`;
|
||||
const helperPy = `import requests
|
||||
|
||||
def helper_func():
|
||||
return requests.__version__
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/test/main.py`, mainPy);
|
||||
await writeFile(`${tempDir}/f/test/main.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/helper.py`, helperPy);
|
||||
await writeFile(`${tempDir}/f/test/helper.script.yaml`, defaultMetadata);
|
||||
|
||||
const result = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
if (result.code !== 0) {
|
||||
console.log("STDOUT:", result.stdout);
|
||||
console.log("STDERR:", result.stderr);
|
||||
}
|
||||
expect(result.code).toBe(0);
|
||||
|
||||
const lockMain = await readFile(`${tempDir}/f/test/main.script.lock`, "utf-8").catch(() => "");
|
||||
const lockHelper = await readFile(`${tempDir}/f/test/helper.script.lock`, "utf-8").catch(() => "");
|
||||
|
||||
expect(lockHelper).toContain("requests");
|
||||
expect(lockMain).toContain("requests");
|
||||
});
|
||||
});
|
||||
|
||||
// =============================================================================
|
||||
// Test 5: Diamond dependency - A imports B and C, both import D
|
||||
// =============================================================================
|
||||
|
||||
test.skipIf(isCI)("Python: diamond dependency pattern propagates correctly", { timeout: 60000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/test`, { recursive: true });
|
||||
|
||||
// Diamond: A -> B, A -> C, B -> D, C -> D
|
||||
const scriptA = `from f.test.script_b import func_b
|
||||
from f.test.script_c import func_c
|
||||
|
||||
def main():
|
||||
return func_b() + func_c()
|
||||
`;
|
||||
const scriptB = `from f.test.script_d import func_d
|
||||
|
||||
def func_b():
|
||||
return "B" + func_d()
|
||||
`;
|
||||
const scriptC = `from f.test.script_d import func_d
|
||||
|
||||
def func_c():
|
||||
return "C" + func_d()
|
||||
`;
|
||||
const scriptD = `import requests
|
||||
|
||||
def func_d():
|
||||
return requests.__version__
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/test/script_a.py`, scriptA);
|
||||
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_b.py`, scriptB);
|
||||
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_c.py`, scriptC);
|
||||
await writeFile(`${tempDir}/f/test/script_c.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_d.py`, scriptD);
|
||||
await writeFile(`${tempDir}/f/test/script_d.script.yaml`, defaultMetadata);
|
||||
|
||||
const result = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(result.code).toBe(0);
|
||||
|
||||
const lockA = await readFile(`${tempDir}/f/test/script_a.script.lock`, "utf-8").catch(() => "");
|
||||
const lockB = await readFile(`${tempDir}/f/test/script_b.script.lock`, "utf-8").catch(() => "");
|
||||
const lockC = await readFile(`${tempDir}/f/test/script_c.script.lock`, "utf-8").catch(() => "");
|
||||
const lockD = await readFile(`${tempDir}/f/test/script_d.script.lock`, "utf-8").catch(() => "");
|
||||
|
||||
expect(lockD).toContain("requests");
|
||||
expect(lockB).toContain("requests");
|
||||
expect(lockC).toContain("requests");
|
||||
expect(lockA).toContain("requests");
|
||||
});
|
||||
});
|
||||
|
||||
// =============================================================================
|
||||
// Test 6: Script isolation - unrelated script not marked stale
|
||||
// =============================================================================
|
||||
|
||||
test("Script isolation: unrelated script not affected by changes", { timeout: 120000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun
|
||||
includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/test`, { recursive: true });
|
||||
|
||||
// A imports B, C is isolated
|
||||
const scriptA = `import { helper } from "./script_b.ts";
|
||||
export async function main() { return helper(); }
|
||||
`;
|
||||
const scriptB = `export function helper() { return "B"; }
|
||||
`;
|
||||
const scriptC = `export async function main() { return "isolated"; }
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/test/script_a.ts`, scriptA);
|
||||
await writeFile(`${tempDir}/f/test/script_a.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_b.ts`, scriptB);
|
||||
await writeFile(`${tempDir}/f/test/script_b.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/script_c.ts`, scriptC);
|
||||
await writeFile(`${tempDir}/f/test/script_c.script.yaml`, defaultMetadata);
|
||||
|
||||
// Generate initial metadata
|
||||
const initial = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(initial.code).toBe(0);
|
||||
|
||||
// Verify all up to date
|
||||
const check1 = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes", "--dry-run"],
|
||||
tempDir
|
||||
);
|
||||
expect(check1.stdout).toContain("All metadata up-to-date");
|
||||
|
||||
// Change script_b
|
||||
await writeFile(`${tempDir}/f/test/script_b.ts`,
|
||||
`export function helper() { return "B changed"; }
|
||||
`);
|
||||
|
||||
// script_a and script_b should be stale, script_c should NOT be mentioned
|
||||
const check2 = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes", "--dry-run"],
|
||||
tempDir
|
||||
);
|
||||
expect(check2.code).toBe(0);
|
||||
expect(check2.stdout).toContain("script_b");
|
||||
expect(check2.stdout).toContain("script_a");
|
||||
expect(check2.stdout).not.toMatch(/script_c/);
|
||||
});
|
||||
});
|
||||
|
||||
// =============================================================================
|
||||
// Test 7: Python relative imports with dot syntax
|
||||
// =============================================================================
|
||||
|
||||
test.skipIf(isCI)("Python: relative imports with dot syntax work correctly", { timeout: 60000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/mymodule`, { recursive: true });
|
||||
|
||||
// Using relative import syntax
|
||||
const mainPy = `from .helper import helper_func
|
||||
|
||||
def main():
|
||||
return helper_func()
|
||||
`;
|
||||
const helperPy = `import requests
|
||||
|
||||
def helper_func():
|
||||
return requests.__version__
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/mymodule/main.py`, mainPy);
|
||||
await writeFile(`${tempDir}/f/mymodule/main.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/mymodule/helper.py`, helperPy);
|
||||
await writeFile(`${tempDir}/f/mymodule/helper.script.yaml`, defaultMetadata);
|
||||
|
||||
const result = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/mymodule/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(result.code).toBe(0);
|
||||
|
||||
const lockMain = await readFile(`${tempDir}/f/mymodule/main.script.lock`, "utf-8").catch(() => "");
|
||||
const lockHelper = await readFile(`${tempDir}/f/mymodule/helper.script.lock`, "utf-8").catch(() => "");
|
||||
|
||||
expect(lockHelper).toContain("requests");
|
||||
expect(lockMain).toContain("requests");
|
||||
});
|
||||
});
|
||||
|
||||
// =============================================================================
|
||||
// Test 8: Adding new import updates importer's lock
|
||||
// =============================================================================
|
||||
|
||||
test.skipIf(isCI)("Python: adding new import updates importer's lock correctly", { timeout: 120000 }, async () => {
|
||||
await withTestBackend(async (backend, tempDir) => {
|
||||
await writeFile(`${tempDir}/wmill.yaml`, `includes: ["**"]
|
||||
excludes: []`);
|
||||
|
||||
await mkdir(`${tempDir}/f/test`, { recursive: true });
|
||||
|
||||
// Initial: main imports helper, helper has no external deps
|
||||
const mainPy = `from f.test.helper import helper_func
|
||||
|
||||
def main():
|
||||
return helper_func()
|
||||
`;
|
||||
const helperPyInitial = `def helper_func():
|
||||
return "no deps"
|
||||
`;
|
||||
|
||||
await writeFile(`${tempDir}/f/test/main.py`, mainPy);
|
||||
await writeFile(`${tempDir}/f/test/main.script.yaml`, defaultMetadata);
|
||||
await writeFile(`${tempDir}/f/test/helper.py`, helperPyInitial);
|
||||
await writeFile(`${tempDir}/f/test/helper.script.yaml`, defaultMetadata);
|
||||
|
||||
// Generate initial locks
|
||||
const initial = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(initial.code).toBe(0);
|
||||
|
||||
// Add new script with pip dep
|
||||
const utilsPy = `import requests
|
||||
|
||||
def get_version():
|
||||
return requests.__version__
|
||||
`;
|
||||
await writeFile(`${tempDir}/f/test/utils.py`, utilsPy);
|
||||
await writeFile(`${tempDir}/f/test/utils.script.yaml`, defaultMetadata);
|
||||
|
||||
// Modify helper to import utils
|
||||
const helperPyWithImport = `from f.test.utils import get_version
|
||||
|
||||
def helper_func():
|
||||
return get_version()
|
||||
`;
|
||||
await writeFile(`${tempDir}/f/test/helper.py`, helperPyWithImport);
|
||||
|
||||
// Regenerate - main should now have requests
|
||||
const afterAdd = await backend.runCLICommand(
|
||||
["generate-metadata", "-i", "f/test/*", "--yes"],
|
||||
tempDir
|
||||
);
|
||||
expect(afterAdd.code).toBe(0);
|
||||
|
||||
const lockUtils = await readFile(`${tempDir}/f/test/utils.script.lock`, "utf-8").catch(() => "");
|
||||
const lockHelper = await readFile(`${tempDir}/f/test/helper.script.lock`, "utf-8").catch(() => "");
|
||||
const lockMain = await readFile(`${tempDir}/f/test/main.script.lock`, "utf-8").catch(() => "");
|
||||
|
||||
expect(lockUtils).toContain("requests");
|
||||
expect(lockHelper).toContain("requests");
|
||||
expect(lockMain).toContain("requests");
|
||||
});
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user