7 Commits

Author SHA1 Message Date
Ruben Fiszel ad53673a28 fix(ai-chat): improve resource-type search tool description and scoring (#10272)
The global-mode search_resource_types tool described its query as a
substring match, but the backend does semantic embedding search. Rewrite
the description so the LLM sends natural-language intent instead of literal
name guesses.

Apply the top-score trim to query_resource_types so a strong match isn't
diluted by weakly-related types that merely clear the 0.75 similarity floor.
The trim was previously inlined only in query_hub_scripts; extract it into a
shared trim_to_top_score helper, use it from both search paths, and add a
unit test pinning the 5% cutoff boundary.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 00:54:18 +02:00
Ruben Fiszel 2318481f4f fix(embeddings): retry on failed init instead of disabling for a day (#10266)
* fix(embeddings): retry on failed init instead of disabling for a day

The embeddings init task treated failure and success identically, so a
transient error at startup left the embeddings DB uninitialized and every
/query_hub_scripts and /query_resource_types call returned "Embeddings db
not initialized":

- If ModelInstance::new() failed (a transient HuggingFace/network error
  surviving its own 5 download retries), the spawned task exited for good
  and embeddings never came up until the next process restart.
- If update_embeddings_db (hub fetch + fill_db) failed, the refresh loop
  still slept the full HUB_EMBEDDINGS_PULLING_INTERVAL_SECS (default 24h)
  before retrying.

Wrap model init in a retry loop, and make update_embeddings_db report
success so the refresh loop backs off by HUB_EMBEDDINGS_RETRY_INTERVAL_SECS
(default 60s, env-configurable) on failure instead of the full pulling
interval.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(embeddings): bound retry backoff to pulling interval

A fixed 60s retry loop was fine for the transient case but spun forever in
an environment where init can never succeed (air-gapped instance with
embeddings left enabled): an error log + HuggingFace fetch storm every ~60s
indefinitely, a behavior regression versus failing once and going quiet.

Decay the retry with exponential backoff capped at
HUB_EMBEDDINGS_PULLING_INTERVAL_SECS, resetting on success. A transient
blip still recovers within ~60s; a permanently-broken env settles into ~1
attempt per pulling interval (default: 1/day).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 23:36:28 +02:00
Ruben Fiszel 6a6295921d fix(embeddings): retry HuggingFace model downloads with backoff (#9597)
Caching the gte-small embedding model fetched config.json / tokenizer.json /
model.safetensors from HuggingFace with no retry, so a single transient
network error ("error sending request for url ...") failed the whole image
build. Wrap each download in a retry loop (up to 5 attempts, exponential
backoff capped at 8s) that logs each retry and surfaces the error only after
the final attempt. No new dependency.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:05:00 +02:00
Ruben Fiszel 1174d7d77f refactor: replace SELECT * with explicit column lists (#9010)
* refactor: replace SELECT * with explicit column lists

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: update sqlx offline query cache

* chore: update sqlx offline query cache

* chore: update sqlx offline query cache with EE support

* chore: update sqlx offline query cache, no deletions

* chore: update sqlx offline query cache after rebase

* fix: correct column names in explicit script query lists

- concurrency_limit → concurrent_limit (matches DB column name)
- runnable_settings → runnable_settings_handle (matches DB column name)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add missing delete_after_secs column to script queries

Also add integration test covering all explicit-column export queries.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: add workspace export integration test covering all explicit-column queries

Covers tarball_workspace (folder, script, resource, resource_type, variable,
schedule, usr, group_) and the mcp_oauth_client SELECT query from windmill-mcp.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: add tarball export integration test covering all explicit-column queries

Single test creates one of each entity type and exercises every runtime-checked
explicit-column query in tarball_workspace. Uses archive_type=tar to avoid
zip feature-gate in CI.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: fix stale verification step and CI contradiction in update-sqlx skill

- Regenerate current_files.txt after EE cache restoration so step 4 reports accurate diff
- Scope "Never use SQLX_OFFLINE=true" to local prepare (CI legitimately uses it)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: remove Co-Authored-By from commit skill template

* refactor: extract SCRIPT_COLUMNS const to single source of truth

Replaces 5 duplicated 44-column lists with a shared const in windmill-types.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-05-04 08:39:45 +00:00
Ruben Fiszel 5b3913052e refactor: convert read-hot globals to AtomicBool/I64 and ArcSwap (#8815)
* refactor: extract load helpers from reload_setting family

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: convert atomic primitive globals to AtomicBool/AtomicI64

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: convert CRITICAL_*/HUB_API_SECRET/INSTANCE_EVENTS_WEBHOOK/JWT_SECRET to ArcSwap

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: pin ee-repo-ref to arcswap-refactor EE branch commit

* refactor: convert BASE_URL/HUB_BASE_URL/MIN_VERSION/LICENSE_KEY*/LICENSE_KEY_ID to ArcSwap

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: convert worker hot-path globals to ArcSwap (WORKER_CONFIG et al)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: pin ee-repo-ref to combined arcswap-urls+worker EE commit

* chore: update ee-repo-ref to d8be8f88cb8898c8f6b27421989d53528223815d

This commit updates the EE repository reference after PR #532 was merged in windmill-ee-private.

Previous ee-repo-ref: c375aaaac9ec0fc0480993627d0defc8054c31a4

New ee-repo-ref: d8be8f88cb8898c8f6b27421989d53528223815d

Automated by sync-ee-ref workflow.

* fix: cleanup unused imports + fix 2 missed WORKER_CONFIG readers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to ce0f8fbbbde09c4a858312d2d8716d224e99042c

This commit updates the EE repository reference after PR #534 was merged in windmill-ee-private.

Previous ee-repo-ref: 450b601b5aba0ca0b2045f4b5071aa8701b4bfb7

New ee-repo-ref: ce0f8fbbbde09c4a858312d2d8716d224e99042c

Automated by sync-ee-ref workflow.

* fix: secret_backend_integration test — BASE_URL.write().await → .store()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: convert APP_WORKSPACED_ROUTE to AtomicBool for symmetry with HTTP_ROUTE_WORKSPACED_ROUTE

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to e587df8 (post-#535 merge)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-04-14 00:04:10 +00:00
Ruben Fiszel e26e437dd1 refactor: extract 12 leaf crates from windmill-api (#7899)
* feat(backend): extract 12 leaf crates from windmill-api to improve incremental compilation

Extract independent modules from windmill-api (90k LOC monolith) into
separate leaf crates to reduce incremental compilation times. Modules
extracted: assets, configs, debug, flow-conversations, inputs,
npm-proxy, openapi, schedule, settings, workers, agent-workers, and
alerting (from windmill-common).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add windmill-api-settings dep to root crate, make ee_oss public

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add agent_workers integration tests

8 tests covering agent worker lifecycle:
- Simple script execution (bun)
- Script with arguments
- Script with logs (verified in job_logs table)
- Script failure handling
- Complex result (nested objects/arrays)
- Agent token creation via API
- Token creation + Initial/MainLoop ping cycle
- Multiple sequential job execution

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 10:31:03 +00:00
Ruben Fiszel 9ff8a85af6 refactor: extract windmill-api into subcrates for parallel compilation (#7845)
* refactor: extract windmill-api into 4 subcrates (api-auth, store, api-sse, api-jobs)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: eliminate refresh_token OnceLock bridge in windmill-store

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: eliminate FromRequestParts OnceLock bridge in windmill-api-auth

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: wire subcrates into workspace and clean up unused re-exports

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve cargo check --all-features errors in subcrate wiring

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* sqlx

* all

* chore: update ee-repo-ref for warning fixes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: extract windmill-trigger crate and expand windmill-api-jobs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-kafka crate from windmill-api

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-postgres crate from windmill-api

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-websocket and windmill-trigger-mqtt crates

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-nats, sqs, gcp, and email crates

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-http crate from windmill-api

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: move token creation and permission helpers to windmill-api-auth

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-native-triggers crate from windmill-api

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* sqlx

* all

* refactor: extract windmill-api-embeddings crate and fix CI warnings

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: resolve type mismatch in oauth2_oss and remaining warnings

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use correct HTTP_CLIENT config in embeddings crate (30s timeout, cert override)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* all

* fix: gate oauth_refresh_ee on oauth2 feature to fix warnings

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* all

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 22:12:55 +00:00