mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-21 00:02:30 +00:00
Merge branch 'main' into explicit-workspace-user-editors
This commit is contained in:
Diego Imbert
@@ -1,3 +1,3 @@
|
||||
{
|
||||
".": "1.805.0"
|
||||
".": "1.809.0"
|
||||
}
|
||||
|
||||
@@ -37,7 +37,7 @@ Open-source platform for internal tools, workflows, API integrations, background
|
||||
- **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
|
||||
- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
|
||||
- **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi. For a Codex-driven pass that mirrors the `codex-pr-review` GitHub action against your unpushed work (committed + uncommitted) before you push, use `/local-review-codex` (`.agents/skills/local-review-codex/`) — same `REVIEW.md` policy and `xhigh` reasoning, on `gpt-6-astra` rather than the action's `gpt-5.6-sol`; requires the `codex` CLI >= 0.153.4.
|
||||
- **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
|
||||
- **Domain guides**: `.claude/skills/native-trigger/`
|
||||
- **Brand/UI guidelines**: `frontend/brand-guidelines.md`
|
||||
- **Domain vocabulary**: `CONTEXT.md` — the words this codebase uses for its own concepts (step, step setting, trigger step, …). Name things the way it does.
|
||||
- **CLI commands**: when adding/modifying/removing a command, subcommand, option, or description in `cli/src/commands/`, run `python system_prompts/generate.py` to refresh `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`. The CLI docs the agents use to operate `wmill` are derived from the source — stale generated files give agents the wrong flags.
|
||||
|
||||
@@ -1,5 +1,93 @@
|
||||
# Changelog
|
||||
|
||||
## [1.809.0](https://github.com/windmill-labs/windmill/compare/v1.808.0...v1.809.0) (2026-09-10)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add a minimal skin for the approval page and slack/teams ([#11061](https://github.com/windmill-labs/windmill/issues/11061)) ([63cb46d](https://github.com/windmill-labs/windmill/commit/63cb46d7bb9db1d996aa38e06bd3afebc60111bd))
|
||||
* live queue status per tag and bounded queue metric charts ([#11067](https://github.com/windmill-labs/windmill/issues/11067)) ([569adb8](https://github.com/windmill-labs/windmill/commit/569adb85c1885d289e80a70a166f0f74e6d5ba83))
|
||||
* **otel:** read the OTLP metrics temporality preference ([#11064](https://github.com/windmill-labs/windmill/issues/11064)) ([2f88769](https://github.com/windmill-labs/windmill/commit/2f8876908719b3640d7cfc9364a7b0f7145fc356))
|
||||
* **otel:** support standard OTEL resource attribute env vars ([#10974](https://github.com/windmill-labs/windmill/issues/10974)) ([0a40eea](https://github.com/windmill-labs/windmill/commit/0a40eea37a7dbde5fc4760d6333d81186dfee255))
|
||||
* report script metadata with no content file in wmill lint ([#11053](https://github.com/windmill-labs/windmill/issues/11053)) ([8820b9f](https://github.com/windmill-labs/windmill/commit/8820b9fc644c6620c50517cd0d902015e2e670e2))
|
||||
* show the workspace an operator is in, and let them switch ([#11059](https://github.com/windmill-labs/windmill/issues/11059)) ([385086f](https://github.com/windmill-labs/windmill/commit/385086ffc21c72cd07624584932a4a301c23a732))
|
||||
* tuck other users' spaces into a collapsible home tree row ([#11073](https://github.com/windmill-labs/windmill/issues/11073)) ([d87f089](https://github.com/windmill-labs/windmill/commit/d87f089288996af9ea7e3b017a5ef35d4ded880d))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **ai-chat:** test_run_flow could test a different flow than the one asked ([#11066](https://github.com/windmill-labs/windmill/issues/11066)) ([fa73539](https://github.com/windmill-labs/windmill/commit/fa73539839071491fb2cbad9242f52ad22b975bf))
|
||||
* bound list_jobs runtime and paginate runs on the sorted column ([#11072](https://github.com/windmill-labs/windmill/issues/11072)) ([f517402](https://github.com/windmill-labs/windmill/commit/f51740253871960b55ab2aa8989e8df3fbde0351))
|
||||
* **frontend:** clear the flow graph selection through xyflow's store ([#11056](https://github.com/windmill-labs/windmill/issues/11056)) ([b4be8bc](https://github.com/windmill-labs/windmill/commit/b4be8bc5354fbd3a47c267c331aaf603c0f90e6e))
|
||||
* **frontend:** recompute dataflow edges when selecting a step ([#11070](https://github.com/windmill-labs/windmill/issues/11070)) ([08d876a](https://github.com/windmill-labs/windmill/commit/08d876aebf32ebb995a8c1839aa87794c0176bed))
|
||||
* **frontend:** restore heading sizes in note markdown and keep group notes on id change ([#11047](https://github.com/windmill-labs/windmill/issues/11047)) ([e63072c](https://github.com/windmill-labs/windmill/commit/e63072c216383700a23504be89782f4c69657174))
|
||||
* give every table a primary key so the db can be logically replicated ([#11036](https://github.com/windmill-labs/windmill/issues/11036)) ([e62bfdc](https://github.com/windmill-labs/windmill/commit/e62bfdcd8c6f1389601ccd2b5809c5eff0ff262e))
|
||||
* keep an app's deployed policy on wmill push ([#11049](https://github.com/windmill-labs/windmill/issues/11049)) ([0af7675](https://github.com/windmill-labs/windmill/commit/0af7675588300863883a99c7f14bc53cafec8a7e))
|
||||
* refuse cross-site GET requests that run Hub scripts ([#11054](https://github.com/windmill-labs/windmill/issues/11054)) ([ab9efc8](https://github.com/windmill-labs/windmill/commit/ab9efc897cc94d61a67263772806b15e9225cece))
|
||||
* skip the deploy PR when the git sync push committed nothing ([#11076](https://github.com/windmill-labs/windmill/issues/11076)) ([8ecbd33](https://github.com/windmill-labs/windmill/commit/8ecbd339eef7314a93d599fcea4377299d4c493d))
|
||||
* space the trailing AI settings cards ([#11044](https://github.com/windmill-labs/windmill/issues/11044)) ([5d7eed1](https://github.com/windmill-labs/windmill/commit/5d7eed1c02b0966289cc8cca00a15d76dad187a5))
|
||||
* stop uv pip compile emitting lockfile annotations ([#11042](https://github.com/windmill-labs/windmill/issues/11042)) ([8aa8b7e](https://github.com/windmill-labs/windmill/commit/8aa8b7ee6c23f859f169637c0bfd3f8d964509e9))
|
||||
* surface why a private or untrusted git host is unreachable ([#11068](https://github.com/windmill-labs/windmill/issues/11068)) ([c57b18e](https://github.com/windmill-labs/windmill/commit/c57b18e46fcdd319213fe0a537cb18de418ca688))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* index the FK columns that cascade on workspace delete ([#11052](https://github.com/windmill-labs/windmill/issues/11052)) ([9a563f6](https://github.com/windmill-labs/windmill/commit/9a563f6d72da28fe09b785cd0683e9698df72bba))
|
||||
* only write queue metrics when a tag's backlog changes ([#11055](https://github.com/windmill-labs/windmill/issues/11055)) ([9d75929](https://github.com/windmill-labs/windmill/commit/9d75929247ea2ec39286971fcbebf95d886194f3))
|
||||
|
||||
## [1.808.0](https://github.com/windmill-labs/windmill/compare/v1.807.0...v1.808.0) (2026-09-09)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* run and test scripts from the AI chat through an argument form ([#11001](https://github.com/windmill-labs/windmill/issues/11001)) ([a6abf2c](https://github.com/windmill-labs/windmill/commit/a6abf2c8a744e9ee6acf4830cdfbb84f2f95cb36))
|
||||
|
||||
## [1.807.0](https://github.com/windmill-labs/windmill/compare/v1.806.0...v1.807.0) (2026-09-09)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add a dismissible instance-wide announcement banner ([#11037](https://github.com/windmill-labs/windmill/issues/11037)) ([abf4c6c](https://github.com/windmill-labs/windmill/commit/abf4c6c2348014ea4401b9be62b5b15e5800e879))
|
||||
* batch chained DDL statements into a single migration ([#11038](https://github.com/windmill-labs/windmill/issues/11038)) ([656e609](https://github.com/windmill-labs/windmill/commit/656e609595833bc854f845d35cf43157e76d732f))
|
||||
* create the cloud workspace in onboarding, and teach the empty home ([#10959](https://github.com/windmill-labs/windmill/issues/10959)) ([fd35b47](https://github.com/windmill-labs/windmill/commit/fd35b4765843879cb2254f402c142fd7510f1916))
|
||||
* link from the public run view to the authenticated run page ([#11041](https://github.com/windmill-labs/windmill/issues/11041)) ([09b81a9](https://github.com/windmill-labs/windmill/commit/09b81a9294bed2795a1a7b688d0b02e0a61957ea))
|
||||
* make guest access unavailable on the shared cloud ([#11040](https://github.com/windmill-labs/windmill/issues/11040)) ([0b63e0a](https://github.com/windmill-labs/windmill/commit/0b63e0a6929088ff25def4fd6547cf61668251a5))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* ignore comments and continuations in python lockfiles ([#11035](https://github.com/windmill-labs/windmill/issues/11035)) ([90c4e10](https://github.com/windmill-labs/windmill/commit/90c4e1020a2ff896977648dd68b572413cea7709))
|
||||
* refetch an unparseable hub script cache entry instead of panicking ([#11033](https://github.com/windmill-labs/windmill/issues/11033)) ([88c3ebd](https://github.com/windmill-labs/windmill/commit/88c3ebdfc1325ffbea4521d854e71d231a6409c4))
|
||||
* stop a new AI session adopting a legacy sidebar chat ([#11039](https://github.com/windmill-labs/windmill/issues/11039)) ([1076b63](https://github.com/windmill-labs/windmill/commit/1076b638d987ba99c5c27e5478ca580534b9d572))
|
||||
|
||||
## [1.806.0](https://github.com/windmill-labs/windmill/compare/v1.805.0...v1.806.0) (2026-09-08)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* bring gitlab repositories to parity for git sync ([#10938](https://github.com/windmill-labs/windmill/issues/10938)) ([9444049](https://github.com/windmill-labs/windmill/commit/9444049d6013c77a5f25f01a736eb5cd741fb3e6))
|
||||
* draw a dbt column trace, across projects and the pipeline boundary ([#11014](https://github.com/windmill-labs/windmill/issues/11014)) ([33f9828](https://github.com/windmill-labs/windmill/commit/33f9828c3ed15fe63fccedc1550584f15c0490ab))
|
||||
* durable dbt state per environment, and `--defer` onto it ([#10975](https://github.com/windmill-labs/windmill/issues/10975)) ([621fac5](https://github.com/windmill-labs/windmill/commit/621fac55abcd1859e8c8c06e5f4412e61bb85d59))
|
||||
* ingest dbt column lineage and real column schemas from the engine's parquet index ([#10977](https://github.com/windmill-labs/windmill/issues/10977)) ([0139467](https://github.com/windmill-labs/windmill/commit/0139467b01b82e4b3d474ca3f205358fa607d19a))
|
||||
* let a worker group override the dependency cache object store ([#11019](https://github.com/windmill-labs/windmill/issues/11019)) ([de98adf](https://github.com/windmill-labs/windmill/commit/de98adf055835ab7c4d6305e7d5d3bdab915876b))
|
||||
* **nativets:** bound fetch on a peer that never answers ([#11026](https://github.com/windmill-labs/windmill/issues/11026)) ([785277e](https://github.com/windmill-labs/windmill/commit/785277e0bb2ea77b71a89dd4d389d439dfcf9e03))
|
||||
* recognize `// volume:` mounts in PHP scripts ([#11018](https://github.com/windmill-labs/windmill/issues/11018)) ([f081fb1](https://github.com/windmill-labs/windmill/commit/f081fb10705cadf99e99dfa786d1cc2ebf0447db))
|
||||
* report a WAC task failure the workflow body never awaited ([#11017](https://github.com/windmill-labs/windmill/issues/11017)) ([3e3a41d](https://github.com/windmill-labs/windmill/commit/3e3a41d418d4ee3fe060bd3acf3324f311d89c8a))
|
||||
* retry a workflow-as-code task from its task options ([#11013](https://github.com/windmill-labs/windmill/issues/11013)) ([d3f305d](https://github.com/windmill-labs/windmill/commit/d3f305db982b7c5dc49babf9bec8b62adcd2557d))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* chain redeploys onto a retired path's version history ([#11029](https://github.com/windmill-labs/windmill/issues/11029)) ([0b37226](https://github.com/windmill-labs/windmill/commit/0b372260787edb7e9627ad4fb6637ad5f1024e0a))
|
||||
* make the native trigger disable/enable toggle actually save ([#11024](https://github.com/windmill-labs/windmill/issues/11024)) ([448fce9](https://github.com/windmill-labs/windmill/commit/448fce93f743d5b2ef2a2d4496eb2ec238594a0a))
|
||||
* offload php signature parsing from async workers ([#11027](https://github.com/windmill-labs/windmill/issues/11027)) ([2cb02e3](https://github.com/windmill-labs/windmill/commit/2cb02e3b3398db49f16377dd79dde2dd6fb5cc02))
|
||||
* reduce php parser stack use in debug workers ([#11025](https://github.com/windmill-labs/windmill/issues/11025)) ([2ae8509](https://github.com/windmill-labs/windmill/commit/2ae8509b14f112c9ef5b71321e8fd52596a16c10))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* reduce shared worker debug polling frames ([#11028](https://github.com/windmill-labs/windmill/issues/11028)) ([946756a](https://github.com/windmill-labs/windmill/commit/946756ae83deb4e7a93111edddbd4d98c596d5a9))
|
||||
|
||||
## [1.805.0](https://github.com/windmill-labs/windmill/compare/v1.804.0...v1.805.0) (2026-09-07)
|
||||
|
||||
|
||||
|
||||
@@ -132,6 +132,10 @@ export async function runEval<THelpers, TOutput>(
|
||||
setToolStatus: () => {},
|
||||
removeToolStatus: () => {},
|
||||
isPlanModeActive,
|
||||
// Accepts the run form exactly as the model prefilled it: there is nobody here to
|
||||
// edit the arguments, so a case can assert what the model proposed but never how
|
||||
// it reacts to the user changing something.
|
||||
requestRunArgs: async (_toolId, form) => form.args,
|
||||
onNewToken: (token: string) => {
|
||||
if (shouldEmitMessageStart) {
|
||||
onAssistantMessageStart?.();
|
||||
|
||||
@@ -86,6 +86,7 @@ vi.mock('$lib/gen', async () => {
|
||||
previewBenchmarkSchedule,
|
||||
runBenchmarkDatatableSql,
|
||||
runBenchmarkFlowByPath,
|
||||
runBenchmarkScriptByPath,
|
||||
runBenchmarkScriptPreview,
|
||||
updateBenchmarkDraft,
|
||||
listBenchmarkMcpTools
|
||||
@@ -279,6 +280,18 @@ vi.mock('$lib/gen', async () => {
|
||||
}
|
||||
return runBenchmarkScriptPreview({ workspace: data.workspace, requestBody })
|
||||
},
|
||||
runScriptByPath: async (data: {
|
||||
workspace: string
|
||||
path: string
|
||||
requestBody?: Record<string, unknown>
|
||||
}) =>
|
||||
hasBenchmarkWorkspace(data.workspace)
|
||||
? runBenchmarkScriptByPath({
|
||||
workspace: data.workspace,
|
||||
path: data.path,
|
||||
args: data.requestBody
|
||||
})
|
||||
: actual.JobService.runScriptByPath(data),
|
||||
runFlowByPath: async (data: {
|
||||
workspace: string
|
||||
path: string
|
||||
|
||||
@@ -1974,6 +1974,78 @@
|
||||
judgeChecklist:
|
||||
- deletes the deployed script via delete_workspace_item rather than a raw API endpoint
|
||||
|
||||
- id: global-test33-run-deployed-script-with-form
|
||||
prompt: |-
|
||||
Run the deployed script `f/evals/global/format_greeting` for me with the name "ada".
|
||||
initial: ai_evals/fixtures/frontend/global/initial/format_greeting_script.json
|
||||
runtime:
|
||||
maxTurns: 8
|
||||
# A session chat is where the run card has a preview pane beside it; run_script
|
||||
# itself is offered in every chat.
|
||||
sessionChat: true
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- run_script
|
||||
# A draft may declare different arguments than the deployed version being run, so
|
||||
# the names to prefill have to come from the deployed schema.
|
||||
- read_workspace_item
|
||||
forbiddenToolsUsed:
|
||||
- test_run_script
|
||||
- call_api_endpoint
|
||||
- write_script
|
||||
- deploy_workspace_item
|
||||
# An empty form pushes the work back onto the user, so the prefill is part of
|
||||
# what the tool is for.
|
||||
toolCallArgs:
|
||||
- tool: run_script
|
||||
field: args.name
|
||||
stringIncludesAnyOf:
|
||||
- ada
|
||||
# Running produces no draft, and the judge cannot observe runs; validate via tool use.
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- runs the deployed script through run_script rather than a preview test run or a raw API endpoint
|
||||
- passes the name "ada" so the confirmation form comes up prefilled
|
||||
|
||||
- id: global-test34-run-with-secret-from-variable
|
||||
prompt: |-
|
||||
Run the deployed `f/evals/global/billing_sync` for the account `acme` — use the billing
|
||||
API token we already keep in the workspace.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/billing_sync_with_secret_arg.json
|
||||
runtime:
|
||||
maxTurns: 10
|
||||
# A session chat is where the run card has a preview pane beside it; run_script
|
||||
# itself is offered in every chat.
|
||||
sessionChat: true
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- run_script
|
||||
forbiddenToolsUsed:
|
||||
- write_script
|
||||
- deploy_workspace_item
|
||||
# A secret argument is filled by naming the variable that holds it: the value stays in
|
||||
# the variable and only its path travels. A literal reaches the job as a reference too,
|
||||
# minted on the way in, but it stays in the tool call the model emitted.
|
||||
toolCallArgs:
|
||||
- tool: run_script
|
||||
field: args.api_token
|
||||
stringIncludesAnyOf:
|
||||
- "$var:f/evals/global/stripe_api_token"
|
||||
- tool: run_script
|
||||
field: args.account
|
||||
stringIncludesAnyOf:
|
||||
- acme
|
||||
# Running produces no draft, and the judge cannot observe runs; validate via tool use.
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- fills the secret argument with a reference to the existing workspace variable rather than a literal token
|
||||
- passes the account "acme"
|
||||
- does not invent or guess the token's value
|
||||
|
||||
- id: global-undo-created-draft
|
||||
prompt: |-
|
||||
Create a draft Postgres resource at `u/admin/scratch_db` for host db.example.com port 5432, database `orders`, user `app`, and tell me what fields it ended up with.
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"workspace": {
|
||||
"variables": [
|
||||
{
|
||||
"path": "f/evals/global/stripe_api_token",
|
||||
"value": "sk_live_do_not_leak_me",
|
||||
"is_secret": true,
|
||||
"description": "Token used by the billing sync job",
|
||||
"labels": ["billing"]
|
||||
}
|
||||
],
|
||||
"scripts": [
|
||||
{
|
||||
"path": "f/evals/global/billing_sync",
|
||||
"summary": "Sync billing records",
|
||||
"description": "Syncs billing records for one account, authenticating with an API token.",
|
||||
"language": "bun",
|
||||
"schema": {
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"account": {
|
||||
"type": "string"
|
||||
},
|
||||
"api_token": {
|
||||
"type": "string",
|
||||
"password": true,
|
||||
"description": "API token to authenticate with"
|
||||
}
|
||||
},
|
||||
"required": ["account", "api_token"]
|
||||
},
|
||||
"content": "export async function main(account: string, api_token: string) {\n return `synced ${account}`\n}\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
-15
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO metrics (id, value)\n VALUES ($1, to_jsonb((\n SELECT EXTRACT(EPOCH FROM now() - scheduled_for)\n FROM v2_job_queue\n WHERE tag = $2 AND running = false AND scheduled_for <= now() - ('3 seconds')::interval\n ORDER BY priority DESC NULLS LAST, scheduled_for LIMIT 1\n )))",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "00e63eab76d26e148b77e932848de74e8b0943d30481465da453942e299a128f"
|
||||
}
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH slots AS (\n SELECT id, slot, min(t) AS first, max(t) AS last, max(v) AS peak,\n (min(ARRAY[t, v]))[2] AS first_value, (max(ARRAY[t, v]))[2] AS last_value,\n (max(ARRAY[t, climbing]))[2] = 1 AS last_climbing,\n COALESCE(bool_and(climbing = 1) AND max(since) - min(since) < $4, false) AS ramp,\n max(ARRAY[t, since]) FILTER (WHERE climbing = 1) AS last_climb\n FROM (\n SELECT id, t,\n CASE jsonb_typeof(value)\n WHEN 'number' THEN value::double precision\n WHEN 'object' THEN t - (value->>'since')::double precision\n END AS v,\n (value->>'since')::double precision AS since,\n (jsonb_typeof(value) = 'object')::int::double precision AS climbing,\n greatest(floor((t - $1::double precision) / $2::double precision), -1)::int\n AS slot\n FROM (\n SELECT id, value, EXTRACT(EPOCH FROM created_at)::double precision AS t\n FROM metrics\n WHERE id LIKE 'queue_%'\n AND created_at > to_timestamp($1::double precision - $3::double precision)\n ) m\n ) s\n WHERE v IS NOT NULL\n GROUP BY id, slot\n )\n SELECT id AS \"id!\", slot AS \"slot!\", first AS \"first!\", last AS \"last!\",\n greatest(peak, CASE WHEN last_climb[1] < last THEN (\n SELECT EXTRACT(EPOCH FROM min(n.created_at))::double precision\n FROM metrics n\n WHERE n.id = slots.id AND n.id LIKE 'queue_%'\n AND n.created_at > to_timestamp(last_climb[1] + 0.001)\n AND n.created_at <= to_timestamp(last + 0.001)\n ) - last_climb[2] END) AS \"peak!\",\n first_value AS \"first_value!\", last_value AS \"last_value!\",\n last_climbing AS \"last_climbing!\", ramp AS \"ramp!\"\n FROM slots\n ORDER BY id, slot",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "slot!",
|
||||
"type_info": "Int4"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "first!",
|
||||
"type_info": "Float8"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "last!",
|
||||
"type_info": "Float8"
|
||||
},
|
||||
{
|
||||
"ordinal": 4,
|
||||
"name": "peak!",
|
||||
"type_info": "Float8"
|
||||
},
|
||||
{
|
||||
"ordinal": 5,
|
||||
"name": "first_value!",
|
||||
"type_info": "Float8"
|
||||
},
|
||||
{
|
||||
"ordinal": 6,
|
||||
"name": "last_value!",
|
||||
"type_info": "Float8"
|
||||
},
|
||||
{
|
||||
"ordinal": 7,
|
||||
"name": "last_climbing!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 8,
|
||||
"name": "ramp!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Float8",
|
||||
"Float8",
|
||||
"Float8",
|
||||
"Float8"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "2742245bc03290120a97b21c441cb56825e9fd552a7aeddfb8a372540c19b863"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT EXTRACT(EPOCH FROM now())::double precision AS \"now!\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "now!",
|
||||
"type_info": "Float8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "3bd816e986ef2d2a193e51c985b61c04b71f464021e4b884bf21cc7c25f6a753"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH queue_metrics as (\n SELECT id, value, created_at\n FROM metrics\n WHERE id LIKE 'queue_%'\n AND created_at > now() - interval '14 day'\n )\n SELECT id, array_agg(json_build_object('value', value, 'created_at', created_at) ORDER BY created_at ASC) as \"values!\"\n FROM queue_metrics\n GROUP BY id\n ORDER BY id ASC",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "values!",
|
||||
"type_info": "JsonArray"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "44dd7a66ecc9564ad5727970b5f60a1717eda8924999741be522ffb74cc173fa"
|
||||
}
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT COALESCE(c.id, r.id) AS \"id!\", r.value AS \"value?\",\n EXTRACT(EPOCH FROM r.created_at)::double precision AS \"at?\",\n EXTRACT(EPOCH FROM now() - r.created_at)::double precision AS \"age?\"\n FROM unnest($1::text[]) AS c(id)\n FULL JOIN (\n SELECT DISTINCT ON (id) id, value, created_at\n FROM metrics\n WHERE id LIKE 'queue_%' AND created_at > now() - make_interval(secs => $2)\n ORDER BY id, created_at DESC\n ) r ON r.id = c.id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id!",
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "value?",
|
||||
"type_info": "Jsonb"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "at?",
|
||||
"type_info": "Float8"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "age?",
|
||||
"type_info": "Float8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"TextArray",
|
||||
"Float8"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "58f99e0d5877f403cde04459e4a425efd77aa4c0118f62ada2caa0794b0d738b"
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH running AS (\n SELECT tag, count(*) AS n FROM v2_job_queue WHERE running = true GROUP BY tag\n )\n SELECT t.tag AS \"tag!\", COALESCE(r.n, 0) AS \"running!\",\n (SELECT count(*) FROM worker_ping w\n WHERE w.ping_at > now() - interval '1 minute' AND w.custom_tags @> ARRAY[t.tag]\n ) AS \"workers!\"\n FROM (SELECT tag::text FROM running UNION SELECT unnest($1::text[])) t(tag)\n LEFT JOIN running r ON r.tag = t.tag\n ORDER BY t.tag",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "tag!",
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "running!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "workers!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"TextArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "7745127eb4a4be2b67427a708e8e5bf2973af84cf315047e586071438cd5e438"
|
||||
}
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT tag AS \"tag!\", count AS \"count!\",\n EXTRACT(EPOCH FROM now() - head)::double precision AS \"delay!\",\n EXTRACT(EPOCH FROM head)::double precision AS \"head_since!\"\n FROM (\n SELECT tag, sum(n)::bigint AS count,\n (array_agg(head ORDER BY priority DESC NULLS LAST))[1] AS head\n FROM (\n SELECT tag, priority, count(*) AS n, min(scheduled_for) AS head\n FROM v2_job_queue WHERE\n scheduled_for <= now() - ('3 seconds')::interval AND running = false\n GROUP BY tag, priority\n ) g\n GROUP BY tag\n ) t",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "tag!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "count!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "delay!",
|
||||
"type_info": "Float8"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "head_since!",
|
||||
"type_info": "Float8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "7af0fd3d8dd1d949ce11b190a4fa6b56c84904aeed791b2a0879b36add6bf9b5"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT created_at FROM metrics WHERE id LIKE 'queue_count_%' ORDER BY created_at DESC LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "created_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "82f6674f19e8ad51a992505a46f46fc4a48172f104e9e849f755ac041c3eef92"
|
||||
}
|
||||
-15
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO metrics (id, value) VALUES ($1, $2)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Jsonb"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "8824b382c4e98dfa17b4aa656af3a6c1ff99973e778d71bd598a50d022da8f15"
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH queue_metrics as (\n SELECT id, created_at,\n CASE WHEN jsonb_typeof(value) = 'object'\n THEN to_jsonb(EXTRACT(EPOCH FROM created_at) - (value->>'since')::numeric)\n ELSE value\n END AS value\n FROM metrics\n WHERE id LIKE 'queue_%'\n AND created_at > now() - interval '14 day'\n )\n SELECT id, array_agg(json_build_object('value', value, 'created_at', created_at) ORDER BY created_at ASC) as \"values!\"\n FROM queue_metrics\n GROUP BY id\n ORDER BY id ASC",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "values!",
|
||||
"type_info": "JsonArray"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "a498f752e169b711c0cb26ac167b2e554a7fdfc1055389fd65f330469e329d16"
|
||||
}
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO metrics (id, value)\n SELECT id, COALESCE(to_jsonb(EXTRACT(EPOCH FROM now())::double precision - held_head), value)\n FROM unnest($1::text[], $2::jsonb[], $3::double precision[]) AS u(id, value, held_head)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"TextArray",
|
||||
"JsonbArray",
|
||||
"Float8Array"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "d4ce900b8e60b530c2ea57c082edbc8c12c71869c4e45219dad4fb7198aaa1e3"
|
||||
}
|
||||
Generated
+219
-173
File diff suppressed because it is too large
Load Diff
+3
-3
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "windmill"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
authors.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
@@ -88,7 +88,7 @@ members = [
|
||||
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
|
||||
edition = "2021"
|
||||
|
||||
@@ -701,7 +701,7 @@ async-stream = "^0"
|
||||
opentelemetry = "0.30.0"
|
||||
tracing-opentelemetry = "0.31.0"
|
||||
opentelemetry_sdk = { version = "0.30.0", features = ["rt-tokio", "testing"] }
|
||||
opentelemetry-otlp = { version = "0.30.0", features = ["grpc-tonic", "tls", "http-proto"] }
|
||||
opentelemetry-otlp = { version = "0.30.0", features = ["grpc-tonic", "tls", "http-proto", "gzip-tonic", "zstd-tonic"] }
|
||||
opentelemetry-appender-tracing = "0.30.0"
|
||||
opentelemetry-semantic-conventions = { version = "0.30.0", features = ["semconv_experimental"] }
|
||||
opentelemetry-proto = { version = "0.30.0", features = ["with-serde", "gen-tonic"] }
|
||||
|
||||
@@ -1 +1 @@
|
||||
e092518ee60e33160fee9ae91a4d109566f7b0ee
|
||||
eccad9f68bd7246cc81acb82bdb6c08fc6013f45
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
-- Dropping the column drops the primary key and the identity sequence with it, and
|
||||
-- only marks the column dropped in the catalog rather than rewriting the table, so
|
||||
-- this takes the ACCESS EXCLUSIVE lock but not the time.
|
||||
|
||||
ALTER TABLE workspace_runnable_dependencies DROP COLUMN IF EXISTS id;
|
||||
ALTER TABLE dbt_node DROP COLUMN IF EXISTS id;
|
||||
ALTER TABLE dbt_edge DROP COLUMN IF EXISTS id;
|
||||
ALTER TABLE dbt_column_edge DROP COLUMN IF EXISTS id;
|
||||
ALTER TABLE dbt_graph_snapshot DROP COLUMN IF EXISTS id;
|
||||
@@ -0,0 +1,31 @@
|
||||
-- These five had neither a PRIMARY KEY nor an explicit REPLICA IDENTITY, which makes
|
||||
-- PostgreSQL reject UPDATE and DELETE on them under logical replication.
|
||||
-- `deployment_metadata` and `metrics` are the other two, one migration each after this.
|
||||
--
|
||||
-- The surrogate cannot be swapped for a natural key: every unique index on all five is
|
||||
-- PARTIAL, split on `script_hash IS NULL`, and a partial index cannot back a primary
|
||||
-- key. The partial uniques stay; they are what the ON CONFLICT clauses infer.
|
||||
--
|
||||
-- Each ALTER rewrites its table under ACCESS EXCLUSIVE and holds it unavailable for
|
||||
-- the rewrite. These five share a transaction because each is bounded by what a
|
||||
-- workspace holds rather than by how long it has run, so none can grow into the one
|
||||
-- that locks the rest; a transaction holds all its locks until it commits.
|
||||
--
|
||||
-- An instance that cannot afford that lock at startup can set REPLICA IDENTITY FULL
|
||||
-- on these tables instead, which unblocks replication by itself, and run these
|
||||
-- idempotent ALTERs in a maintenance window first.
|
||||
|
||||
ALTER TABLE workspace_runnable_dependencies
|
||||
ADD COLUMN IF NOT EXISTS id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY;
|
||||
|
||||
ALTER TABLE dbt_node
|
||||
ADD COLUMN IF NOT EXISTS id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY;
|
||||
|
||||
ALTER TABLE dbt_edge
|
||||
ADD COLUMN IF NOT EXISTS id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY;
|
||||
|
||||
ALTER TABLE dbt_column_edge
|
||||
ADD COLUMN IF NOT EXISTS id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY;
|
||||
|
||||
ALTER TABLE dbt_graph_snapshot
|
||||
ADD COLUMN IF NOT EXISTS id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY;
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE deployment_metadata DROP COLUMN IF EXISTS id;
|
||||
@@ -0,0 +1,13 @@
|
||||
-- The sixth of the seven; why any of them need a key is in
|
||||
-- 20260909052532_add_missing_primary_keys.
|
||||
--
|
||||
-- Kept out of that migration because it is the one table in the set with no retention
|
||||
-- sweep -- rows accumulate per deployed script hash, flow version and app version for
|
||||
-- the life of the instance -- so on an instance that upgrades after years of deploys
|
||||
-- its ACCESS EXCLUSIVE rewrite is the one that could hold the others locked.
|
||||
--
|
||||
-- No natural key: each row is a script, flow OR app deployment, and the three unique
|
||||
-- indexes are partial on exactly that split, so none of them covers every row.
|
||||
|
||||
ALTER TABLE deployment_metadata
|
||||
ADD COLUMN IF NOT EXISTS id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY;
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE metrics DROP COLUMN IF EXISTS row_id;
|
||||
@@ -0,0 +1,12 @@
|
||||
-- The last of the seven; why any of them need a key is in
|
||||
-- 20260909052532_add_missing_primary_keys.
|
||||
--
|
||||
-- Kept out of that migration because it is the largest (~400 MB / 750k rows on the
|
||||
-- instance this was measured on, a steady state: `queue_%` rows, which are nearly all
|
||||
-- of them, are swept at 14 days) and its ALTER rewrites it under ACCESS EXCLUSIVE. One
|
||||
-- migration is one transaction, so alone it holds no lock on the others as it rewrites.
|
||||
--
|
||||
-- The surrogate cannot be called `id` -- `metrics.id` holds the metric NAME.
|
||||
|
||||
ALTER TABLE metrics
|
||||
ADD COLUMN IF NOT EXISTS row_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY;
|
||||
@@ -0,0 +1,7 @@
|
||||
DROP INDEX IF EXISTS index_app_version_on_app_id;
|
||||
|
||||
DROP INDEX IF EXISTS index_app_script_on_app;
|
||||
|
||||
DROP INDEX IF EXISTS index_workspace_runnable_dependencies_on_app_path;
|
||||
|
||||
DROP INDEX IF EXISTS index_workspace_runnable_dependencies_on_flow_path;
|
||||
@@ -0,0 +1,32 @@
|
||||
-- The FK columns that cascade when a workspace's apps and flows are deleted. Unindexed,
|
||||
-- Postgres seq-scans the whole child table once per deleted parent row, making a workspace
|
||||
-- delete cost O(apps and flows deleted x rows in the instance). Fork deletion is where that
|
||||
-- bites: a fork clones its parent's apps, flows and entire app version history.
|
||||
--
|
||||
-- The workspace_runnable_dependencies pair are partial because the table's check constraint
|
||||
-- makes app_path and flow_path mutually exclusive, halving each index -- the cascade's
|
||||
-- equality on the path proves the predicate. The table's existing path indexes are partial on
|
||||
-- script_hash, which the cascade does not constrain, so they cannot serve it.
|
||||
--
|
||||
-- Dropped before built: a failed concurrent build leaves an invalid index that IF NOT EXISTS
|
||||
-- would accept forever, unused by the planner yet still maintained on every write.
|
||||
--
|
||||
-- No statement separators outside the statements below, comments included: the CONCURRENTLY
|
||||
-- rewrite in windmill-api/src/db.rs splits the file on them and would run comment text as SQL.
|
||||
DROP INDEX IF EXISTS index_app_version_on_app_id;
|
||||
|
||||
CREATE INDEX index_app_version_on_app_id ON app_version (app_id);
|
||||
|
||||
DROP INDEX IF EXISTS index_app_script_on_app;
|
||||
|
||||
CREATE INDEX index_app_script_on_app ON app_script (app);
|
||||
|
||||
DROP INDEX IF EXISTS index_workspace_runnable_dependencies_on_app_path;
|
||||
|
||||
CREATE INDEX index_workspace_runnable_dependencies_on_app_path
|
||||
ON workspace_runnable_dependencies (app_path, workspace_id) WHERE app_path IS NOT NULL;
|
||||
|
||||
DROP INDEX IF EXISTS index_workspace_runnable_dependencies_on_flow_path;
|
||||
|
||||
CREATE INDEX index_workspace_runnable_dependencies_on_flow_path
|
||||
ON workspace_runnable_dependencies (flow_path, workspace_id) WHERE flow_path IS NOT NULL;
|
||||
@@ -694,6 +694,107 @@ pub fn remove_pinned_imports(code: &str) -> anyhow::Result<String> {
|
||||
Ok(content)
|
||||
}
|
||||
|
||||
/// Spans of the string literals naming a loaded module: `import`/`export … from` sources and the
|
||||
/// argument of a dynamic `import()`. A `require()` call is left out: `require` is an ordinary
|
||||
/// binding a script can shadow, so its argument is not known to be a module.
|
||||
struct ImportSpecifierSpans(Vec<Span>);
|
||||
|
||||
impl Visit for ImportSpecifierSpans {
|
||||
noop_visit_type!();
|
||||
|
||||
fn visit_import_decl(&mut self, n: &swc_ecma_ast::ImportDecl) {
|
||||
self.0.push(n.src.span);
|
||||
}
|
||||
|
||||
fn visit_export_all(&mut self, n: &swc_ecma_ast::ExportAll) {
|
||||
self.0.push(n.src.span);
|
||||
}
|
||||
|
||||
fn visit_named_export(&mut self, n: &swc_ecma_ast::NamedExport) {
|
||||
if let Some(src) = &n.src {
|
||||
self.0.push(src.span);
|
||||
}
|
||||
}
|
||||
|
||||
fn visit_call_expr(&mut self, n: &swc_ecma_ast::CallExpr) {
|
||||
if let (swc_ecma_ast::Callee::Import(_), Some(arg)) = (&n.callee, n.args.first()) {
|
||||
if let (None, Expr::Lit(Lit::Str(s))) = (arg.spread, &*arg.expr) {
|
||||
self.0.push(s.span);
|
||||
}
|
||||
}
|
||||
n.visit_children_with(self);
|
||||
}
|
||||
}
|
||||
|
||||
/// Drops the `@version` from each pinned module specifier (`pkg@1.2.3/sub` -> `pkg/sub`),
|
||||
/// rewriting only the specifier literals. Unlike [`remove_pinned_imports`], the same text
|
||||
/// elsewhere, such as a string the script returns, stays as written.
|
||||
pub fn remove_pinned_import_specifiers(code: &str) -> anyhow::Result<String> {
|
||||
let cm: Lrc<SourceMap> = Default::default();
|
||||
let fm = cm.new_source_file(
|
||||
FileName::Custom("main.d.ts".into()).into(),
|
||||
code.to_string(),
|
||||
);
|
||||
let mut tss = TsSyntax::default();
|
||||
tss.tsx = true;
|
||||
tss.no_early_errors = true;
|
||||
let lexer = Lexer::new(
|
||||
Syntax::Typescript(tss),
|
||||
Default::default(),
|
||||
StringInput::from(&*fm),
|
||||
None,
|
||||
);
|
||||
let module = Parser::new_from(lexer).parse_module().map_err(|e| {
|
||||
anyhow::anyhow!("Error while parsing code, it is invalid TypeScript: {e:?}")
|
||||
})?;
|
||||
let mut specifiers = ImportSpecifierSpans(vec![]);
|
||||
specifiers.visit_module(&module);
|
||||
specifiers.0.sort_by_key(|s| s.lo);
|
||||
|
||||
// Spans index the parsed source, which the source map stripped of any UTF-8 BOM.
|
||||
let bom = if code.starts_with('\u{feff}') {
|
||||
'\u{feff}'.len_utf8()
|
||||
} else {
|
||||
0
|
||||
};
|
||||
let offset =
|
||||
|pos: swc_common::BytePos| pos.0.checked_sub(fm.start_pos.0).map(|o| bom + o as usize);
|
||||
let mut content = String::with_capacity(code.len());
|
||||
let mut copied = 0;
|
||||
for span in specifiers.0 {
|
||||
// A span covers the literal's quotes. One that does not land on a matching pair is left
|
||||
// as written rather than risk rewriting the wrong bytes.
|
||||
let (Some(open), Some(close)) = (
|
||||
offset(span.lo),
|
||||
offset(span.hi).and_then(|e| e.checked_sub(1)),
|
||||
) else {
|
||||
continue;
|
||||
};
|
||||
let quote = code.as_bytes().get(open);
|
||||
if open >= close
|
||||
|| open < copied
|
||||
|| !matches!(quote, Some(b'"' | b'\''))
|
||||
|| code.as_bytes().get(close) != quote
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Some(specifier) = code.get(open + 1..close) else {
|
||||
continue;
|
||||
};
|
||||
let unpinned = IMPORTS_VERSION.captures(specifier).and_then(|x| {
|
||||
x.get(1)
|
||||
.map(|y| format!("{}{}", y.as_str(), x.get(2).map_or("", |z| z.as_str())))
|
||||
});
|
||||
if let Some(unpinned) = unpinned.filter(|u| u != specifier) {
|
||||
content.push_str(&code[copied..open + 1]);
|
||||
content.push_str(&unpinned);
|
||||
copied = close;
|
||||
}
|
||||
}
|
||||
content.push_str(&code[copied..]);
|
||||
Ok(content)
|
||||
}
|
||||
|
||||
fn resolve_type_ref(type_resolver: &HashMap<String, (Typ, bool)>, typ: &mut Typ) {
|
||||
let mut visited = std::collections::HashSet::new();
|
||||
resolve_type_ref_with_visited(type_resolver, typ, &mut visited);
|
||||
|
||||
@@ -4,6 +4,7 @@ mod tests {
|
||||
use windmill_parser::{Arg, MainArgSignature, ObjectProperty, ObjectType, Typ};
|
||||
use windmill_parser_ts::{
|
||||
parse_deno_signature, parse_expr_for_imports, parse_relative_imports,
|
||||
remove_pinned_import_specifiers,
|
||||
};
|
||||
|
||||
#[test]
|
||||
@@ -33,6 +34,46 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_remove_pinned_import_specifiers_rewrites_only_specifiers() {
|
||||
let code = r#"// héllo
|
||||
import a from "pkg@1.2.3";
|
||||
import b from "@scope/pkg@^2/sub";
|
||||
export * from "other@3";
|
||||
import rel from "./helper";
|
||||
const c = await import("dyn@4");
|
||||
const require = (v: string) => v;
|
||||
const d = require("req@5");
|
||||
// pkg@1.2.3
|
||||
export const label = "pkg@1.2.3";
|
||||
"#;
|
||||
assert_eq!(
|
||||
remove_pinned_import_specifiers(code).unwrap(),
|
||||
r#"// héllo
|
||||
import a from "pkg";
|
||||
import b from "@scope/pkg/sub";
|
||||
export * from "other";
|
||||
import rel from "./helper";
|
||||
const c = await import("dyn");
|
||||
const require = (v: string) => v;
|
||||
const d = require("req@5");
|
||||
// pkg@1.2.3
|
||||
export const label = "pkg@1.2.3";
|
||||
"#
|
||||
);
|
||||
assert_eq!(
|
||||
remove_pinned_import_specifiers("\u{feff}import a from 'pkg@1';").unwrap(),
|
||||
"\u{feff}import a from 'pkg';"
|
||||
);
|
||||
assert_eq!(
|
||||
remove_pinned_import_specifiers(
|
||||
"// a\r\n// b\r\nimport a from \"pkg@1\";\r\nimport b from 'x@2';"
|
||||
)
|
||||
.unwrap(),
|
||||
"// a\r\n// b\r\nimport a from \"pkg\";\r\nimport b from 'x';"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_empty_main_signature() {
|
||||
let code = r#"
|
||||
|
||||
+24
-24
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
|
||||
|
||||
[[package]]
|
||||
name = "windmill-common"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"aho-corasick",
|
||||
"anyhow",
|
||||
@@ -6274,7 +6274,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-macros"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -6286,7 +6286,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"convert_case",
|
||||
"serde",
|
||||
@@ -6295,7 +6295,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-bash"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6307,7 +6307,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-csharp"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -6319,7 +6319,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-go"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"gosyn",
|
||||
@@ -6331,7 +6331,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-graphql"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6343,7 +6343,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-java"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -6355,7 +6355,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-nu"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"nu-parser",
|
||||
@@ -6366,7 +6366,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-php"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"itertools 0.14.0",
|
||||
@@ -6377,7 +6377,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"itertools 0.14.0",
|
||||
@@ -6389,7 +6389,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py-asset"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"rustpython-ast",
|
||||
@@ -6400,7 +6400,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py-imports"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-recursion",
|
||||
@@ -6422,7 +6422,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-r"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -6434,7 +6434,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ruby"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6448,7 +6448,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-rust"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"convert_case",
|
||||
@@ -6465,7 +6465,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-sql"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6478,7 +6478,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-sql-asset"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde",
|
||||
@@ -6490,7 +6490,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ts"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6508,7 +6508,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ts-asset"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde-wasm-bindgen",
|
||||
@@ -6524,7 +6524,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-wac"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"rustpython-ast",
|
||||
@@ -6540,7 +6540,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-wasm"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"getrandom 0.2.17",
|
||||
@@ -6572,7 +6572,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-yaml"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6586,7 +6586,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-types"
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"bitflags",
|
||||
|
||||
@@ -12,7 +12,7 @@ resolver = "2"
|
||||
members = ["."]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.805.0"
|
||||
version = "1.809.0"
|
||||
edition = "2021"
|
||||
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
|
||||
|
||||
|
||||
+417
-140
@@ -109,6 +109,10 @@ use windmill_common::{
|
||||
APP_WORKSPACED_ROUTE_SETTING, HTTP_ROUTE_WORKSPACED_ROUTE,
|
||||
HTTP_ROUTE_WORKSPACED_ROUTE_SETTING,
|
||||
},
|
||||
queue_metrics::{
|
||||
QueueSample, QUEUE_COUNT_PREFIX, QUEUE_DELAY_PREFIX, QUEUE_DELAY_SAME_HEAD_SECS,
|
||||
QUEUE_METRIC_HEARTBEAT_SECS, QUEUE_METRIC_STALE_SECS,
|
||||
},
|
||||
};
|
||||
#[cfg(feature = "parquet")]
|
||||
use windmill_object_store::reload_object_store_setting;
|
||||
@@ -5106,155 +5110,303 @@ async fn vacuuming_tables(db: &Pool<Postgres>) -> error::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
|
||||
let last_check = sqlx::query_scalar!(
|
||||
"SELECT created_at FROM metrics WHERE id LIKE 'queue_count_%' ORDER BY created_at DESC LIMIT 1"
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await
|
||||
.unwrap_or(Some(chrono::Utc::now()));
|
||||
/// Shortest spacing between two stored samples of the same queue metric, so a tag whose
|
||||
/// value moves on every monitor round still writes at most one row per interval. Also how
|
||||
/// often each server samples the queue when no Prometheus or OTel gauge needs it sooner.
|
||||
const QUEUE_METRIC_MIN_INTERVAL_SECS: f64 = 25.0;
|
||||
/// A held delay hovers while the head keeps changing, so an exact-value comparison would rarely
|
||||
/// dedup it. Only a move the chart would actually render is stored.
|
||||
const QUEUE_DELAY_TOLERANCE: f64 = 0.1;
|
||||
|
||||
let metrics_enabled = METRICS_ENABLED.load(std::sync::atomic::Ordering::Relaxed);
|
||||
let save_metrics = last_check
|
||||
.map(|last_check| chrono::Utc::now() - last_check > chrono::Duration::seconds(25))
|
||||
.unwrap_or(true);
|
||||
/// Append the queue metrics the drawer at `GET /workers/queue_metrics_series` charts, skipping
|
||||
/// any sample that repeats what is already stored.
|
||||
///
|
||||
/// Only tags with a backlog appear in `queue_stats`, and an arbitrary `?tag=` nobody serves
|
||||
/// stays backlogged forever, so writing every round would repeat the same pair of rows for
|
||||
/// the whole 14-day retention. Each metric is written when the value it draws moves, once per
|
||||
/// heartbeat while it holds, and once more (as a zero) when the tag drains. Gaps therefore
|
||||
/// mean "unchanged since the last row", which is what the chart interpolates. A delay whose
|
||||
/// head job stays put is stored as that job's wait start, which the chart draws climbing, so it
|
||||
/// never moves away from what is stored either.
|
||||
async fn save_queue_metrics(
|
||||
db: &Pool<Postgres>,
|
||||
queue_stats: &std::collections::HashMap<String, windmill_common::queue::QueueStat>,
|
||||
) {
|
||||
let sampled_ids = queue_stats
|
||||
.keys()
|
||||
.flat_map(|tag| {
|
||||
[
|
||||
format!("{QUEUE_COUNT_PREFIX}{tag}"),
|
||||
format!("{QUEUE_DELAY_PREFIX}{tag}"),
|
||||
]
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
if metrics_enabled || save_metrics || OTEL_METRICS_ENABLED.load(Ordering::Relaxed) {
|
||||
let queue_counts = windmill_common::queue::get_queue_counts(db).await;
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
for q in QUEUE_COUNT_TAGS.read().await.iter() {
|
||||
if queue_counts.get(q).is_none() {
|
||||
(*QUEUE_COUNT).with_label_values(&[q]).set(0);
|
||||
}
|
||||
}
|
||||
// Last stored sample of every metric that either has a backlog now or was written
|
||||
// recently enough to still be believed backlogged. Bounding the lookup by the stale window
|
||||
// keeps it cheap at any `metrics` size; a per-id `ORDER BY created_at DESC LIMIT 1` does
|
||||
// not, since the planner may serve it from `metrics_sort_idx` and walk the whole table.
|
||||
let last_samples = match sqlx::query!(
|
||||
"SELECT COALESCE(c.id, r.id) AS \"id!\", r.value AS \"value?\",
|
||||
EXTRACT(EPOCH FROM r.created_at)::double precision AS \"at?\",
|
||||
EXTRACT(EPOCH FROM now() - r.created_at)::double precision AS \"age?\"
|
||||
FROM unnest($1::text[]) AS c(id)
|
||||
FULL JOIN (
|
||||
SELECT DISTINCT ON (id) id, value, created_at
|
||||
FROM metrics
|
||||
WHERE id LIKE 'queue_%' AND created_at > now() - make_interval(secs => $2)
|
||||
ORDER BY id, created_at DESC
|
||||
) r ON r.id = c.id",
|
||||
&sampled_ids[..],
|
||||
QUEUE_METRIC_STALE_SECS,
|
||||
)
|
||||
.fetch_all(db)
|
||||
.await
|
||||
{
|
||||
Ok(rows) => rows,
|
||||
Err(e) => {
|
||||
tracing::error!("Failed to read last queue metrics samples: {e:#}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let otel_enabled = OTEL_METRICS_ENABLED.load(Ordering::Relaxed);
|
||||
let mut ids = vec![];
|
||||
let mut values = vec![];
|
||||
// The wait start of the head of each held delay, whose value the INSERT computes.
|
||||
let mut held_heads: Vec<Option<f64>> = vec![];
|
||||
for row in last_samples {
|
||||
let Some((prefix, tag)) = [QUEUE_COUNT_PREFIX, QUEUE_DELAY_PREFIX]
|
||||
.into_iter()
|
||||
.find_map(|p| row.id.strip_prefix(p).map(|tag| (p, tag)))
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
// A stored value that cannot be read cannot be compared, so the next reading is kept.
|
||||
let last = row
|
||||
.value
|
||||
.as_ref()
|
||||
.and_then(QueueSample::parse)
|
||||
.zip(row.at)
|
||||
.zip(row.age)
|
||||
.map(|((sample, at), age)| (sample, at, age));
|
||||
let stat = queue_stats.get(tag);
|
||||
let current = stat.map(|stat| {
|
||||
if prefix == QUEUE_COUNT_PREFIX {
|
||||
stat.count as f64
|
||||
} else {
|
||||
stat.delay
|
||||
}
|
||||
});
|
||||
|
||||
if otel_enabled {
|
||||
for q in OTEL_QUEUE_COUNT_TAGS.read().await.iter() {
|
||||
if queue_counts.get(q).is_none() {
|
||||
otel_set_queue_count(q, 0);
|
||||
let next_delay = stat
|
||||
.filter(|_| prefix == QUEUE_DELAY_PREFIX)
|
||||
.map(|stat| delay_sample(last.map(|(sample, at, _)| sample.head_since(at)), stat));
|
||||
let drawn_now = last.map(|(sample, at, age)| (sample.value_at(at + age), age));
|
||||
let redraws = last
|
||||
.zip(next_delay)
|
||||
.is_some_and(|((sample, ..), next)| redraws(sample, next));
|
||||
if should_store(prefix, drawn_now, current, redraws) {
|
||||
let (value, held_head) = match (stat, next_delay) {
|
||||
(None, _) => (serde_json::json!(0), None),
|
||||
(Some(stat), None) => (serde_json::json!(stat.count), None),
|
||||
(Some(stat), Some(QueueSample::Held(_))) => {
|
||||
(serde_json::Value::Null, Some(stat.head_since))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(unused_mut)]
|
||||
let mut tags_to_watch = vec![];
|
||||
#[allow(unused_mut)]
|
||||
let mut otel_tags_to_watch = vec![];
|
||||
for q in queue_counts {
|
||||
let count = q.1;
|
||||
let tag = q.0;
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
let metric = (*QUEUE_COUNT).with_label_values(&[&tag]);
|
||||
metric.set(count as i64);
|
||||
tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
|
||||
if otel_enabled {
|
||||
otel_tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
otel_set_queue_count(&tag, count as i64);
|
||||
|
||||
// save queue_count and delay metrics per tag
|
||||
if save_metrics {
|
||||
sqlx::query!(
|
||||
"INSERT INTO metrics (id, value) VALUES ($1, $2)",
|
||||
format!("queue_count_{}", tag),
|
||||
serde_json::json!(count)
|
||||
)
|
||||
.execute(db)
|
||||
.await
|
||||
.ok();
|
||||
if count > 0 {
|
||||
sqlx::query!(
|
||||
"INSERT INTO metrics (id, value)
|
||||
VALUES ($1, to_jsonb((
|
||||
SELECT EXTRACT(EPOCH FROM now() - scheduled_for)
|
||||
FROM v2_job_queue
|
||||
WHERE tag = $2 AND running = false AND scheduled_for <= now() - ('3 seconds')::interval
|
||||
ORDER BY priority DESC NULLS LAST, scheduled_for LIMIT 1
|
||||
)))",
|
||||
format!("queue_delay_{}", tag),
|
||||
tag
|
||||
)
|
||||
.execute(db)
|
||||
.await
|
||||
.ok();
|
||||
}
|
||||
}
|
||||
}
|
||||
if metrics_enabled {
|
||||
let mut w = QUEUE_COUNT_TAGS.write().await;
|
||||
*w = tags_to_watch;
|
||||
}
|
||||
if otel_enabled {
|
||||
let mut w = OTEL_QUEUE_COUNT_TAGS.write().await;
|
||||
*w = otel_tags_to_watch;
|
||||
}
|
||||
|
||||
// Single DB query for running counts, shared by Prometheus and OTel
|
||||
let otel_running = otel_enabled;
|
||||
#[cfg(feature = "prometheus")]
|
||||
let need_running_counts = metrics_enabled || otel_running;
|
||||
#[cfg(not(feature = "prometheus"))]
|
||||
let need_running_counts = otel_running;
|
||||
|
||||
if need_running_counts {
|
||||
let queue_running_counts = windmill_common::queue::get_queue_running_counts(db).await;
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
for q in QUEUE_RUNNING_COUNT_TAGS.read().await.iter() {
|
||||
if queue_running_counts.get(q).is_none() {
|
||||
(*QUEUE_RUNNING_COUNT).with_label_values(&[q]).set(0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if otel_running {
|
||||
for q in OTEL_QUEUE_RUNNING_COUNT_TAGS.read().await.iter() {
|
||||
if queue_running_counts.get(q).is_none() {
|
||||
otel_set_queue_running_count(q, 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(unused_mut, unused_variables)]
|
||||
let mut running_tags_to_watch: Vec<String> = vec![];
|
||||
#[allow(unused_mut, unused_variables)]
|
||||
let mut otel_running_tags_to_watch: Vec<String> = vec![];
|
||||
for (tag, count) in &queue_running_counts {
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
let metric = (*QUEUE_RUNNING_COUNT).with_label_values(&[tag]);
|
||||
metric.set(*count as i64);
|
||||
running_tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
|
||||
if otel_running {
|
||||
otel_set_queue_running_count(tag, *count as i64);
|
||||
otel_running_tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
let mut w = QUEUE_RUNNING_COUNT_TAGS.write().await;
|
||||
*w = running_tags_to_watch;
|
||||
}
|
||||
if otel_running {
|
||||
let mut w = OTEL_QUEUE_RUNNING_COUNT_TAGS.write().await;
|
||||
*w = otel_running_tags_to_watch;
|
||||
}
|
||||
(Some(_), Some(climbing)) => (climbing.to_json(), None),
|
||||
};
|
||||
ids.push(row.id);
|
||||
values.push(value);
|
||||
held_heads.push(held_head);
|
||||
}
|
||||
}
|
||||
|
||||
if ids.is_empty() {
|
||||
return;
|
||||
}
|
||||
// A held delay is computed from this statement's `now()`, the row's `created_at` too, so
|
||||
// `created_at - value` is exactly its head's wait start. That is how the next sample tells
|
||||
// whether the same job is still at the head, within `QUEUE_DELAY_SAME_HEAD_SECS`, which the
|
||||
// time between reading the queue and this INSERT could otherwise exceed on a busy database.
|
||||
if let Err(e) = sqlx::query!(
|
||||
"INSERT INTO metrics (id, value)
|
||||
SELECT id, COALESCE(to_jsonb(EXTRACT(EPOCH FROM now())::double precision - held_head), value)
|
||||
FROM unnest($1::text[], $2::jsonb[], $3::double precision[]) AS u(id, value, held_head)",
|
||||
&ids[..],
|
||||
&values[..],
|
||||
&held_heads[..] as &[Option<f64>],
|
||||
)
|
||||
.execute(db)
|
||||
.await
|
||||
{
|
||||
tracing::error!("Failed to save queue metrics: {e:#}");
|
||||
}
|
||||
}
|
||||
|
||||
/// What to store for a delay reading, given when the head job of the last stored sample started
|
||||
/// waiting. The same job still at the head keeps the delay climbing from its wait start, which
|
||||
/// the chart draws exactly. A head that changed means a moving queue, whose delay hovers and is
|
||||
/// held; so is a first sample, which cannot tell yet and must not draw a climb that never was.
|
||||
fn delay_sample(
|
||||
last_head_since: Option<f64>,
|
||||
stat: &windmill_common::queue::QueueStat,
|
||||
) -> QueueSample {
|
||||
match last_head_since {
|
||||
Some(since) if (since - stat.head_since).abs() < QUEUE_DELAY_SAME_HEAD_SECS => {
|
||||
QueueSample::Climbing { since: stat.head_since }
|
||||
}
|
||||
_ => QueueSample::Held(stat.delay),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the next delay sample is drawn differently from the last one even at the same value:
|
||||
/// a climb whose head left would otherwise go on climbing from the old head, and a held delay
|
||||
/// whose head stayed would stay flat while the wait grows.
|
||||
fn redraws(last: QueueSample, next: QueueSample) -> bool {
|
||||
matches!(last, QueueSample::Climbing { .. }) != matches!(next, QueueSample::Climbing { .. })
|
||||
}
|
||||
|
||||
/// Whether a reading deserves a row of its own, given the last one stored for that metric:
|
||||
/// the value it draws now and how many seconds ago it was written. `current` is `None` once
|
||||
/// the tag has no backlog left; `redraws` is set when the reading must be drawn differently.
|
||||
fn should_store(
|
||||
prefix: &str,
|
||||
last: Option<(f64, f64)>,
|
||||
current: Option<f64>,
|
||||
redraws: bool,
|
||||
) -> bool {
|
||||
let Some((last_value, age)) = last else {
|
||||
// Nothing comparable within the lookback window: a tag that just backed up needs a
|
||||
// first sample, one that was already gone needs nothing.
|
||||
return current.is_some();
|
||||
};
|
||||
let Some(current) = current else {
|
||||
// The tag drained. One zero pins where the line drops; after that the metric matches
|
||||
// and goes quiet, then falls out of the lookback window entirely.
|
||||
return last_value != 0.0;
|
||||
};
|
||||
if age >= QUEUE_METRIC_HEARTBEAT_SECS {
|
||||
return true;
|
||||
}
|
||||
age >= QUEUE_METRIC_MIN_INTERVAL_SECS
|
||||
&& (redraws
|
||||
|| if prefix == QUEUE_COUNT_PREFIX {
|
||||
last_value != current
|
||||
} else {
|
||||
(current - last_value).abs() > last_value.abs() * QUEUE_DELAY_TOLERANCE
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod queue_metric_sampling {
|
||||
use super::*;
|
||||
|
||||
const RECENT: f64 = QUEUE_METRIC_MIN_INTERVAL_SECS + 1.0;
|
||||
|
||||
#[test]
|
||||
fn a_holding_backlog_writes_only_on_the_heartbeat() {
|
||||
let held = Some((3.0, RECENT));
|
||||
assert!(!should_store(QUEUE_COUNT_PREFIX, held, Some(3.0), false));
|
||||
let due = Some((3.0, QUEUE_METRIC_HEARTBEAT_SECS));
|
||||
assert!(should_store(QUEUE_COUNT_PREFIX, due, Some(3.0), false));
|
||||
// A held delay hovers, so only a move past the tolerance counts as a change.
|
||||
let delay = Some((100.0, RECENT));
|
||||
assert!(!should_store(QUEUE_DELAY_PREFIX, delay, Some(105.0), false));
|
||||
assert!(should_store(QUEUE_DELAY_PREFIX, delay, Some(120.0), false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_drained_tag_writes_one_zero_then_stops() {
|
||||
assert!(should_store(
|
||||
QUEUE_COUNT_PREFIX,
|
||||
Some((3.0, RECENT)),
|
||||
None,
|
||||
false
|
||||
));
|
||||
assert!(!should_store(
|
||||
QUEUE_COUNT_PREFIX,
|
||||
Some((0.0, RECENT)),
|
||||
None,
|
||||
false
|
||||
));
|
||||
// Including once the heartbeat is due: a tag that is gone stays silent.
|
||||
let gone = Some((0.0, QUEUE_METRIC_STALE_SECS));
|
||||
assert!(!should_store(QUEUE_COUNT_PREFIX, gone, None, false));
|
||||
assert!(!should_store(QUEUE_COUNT_PREFIX, None, None, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_change_waits_for_the_minimum_interval() {
|
||||
assert!(!should_store(
|
||||
QUEUE_COUNT_PREFIX,
|
||||
Some((3.0, 1.0)),
|
||||
Some(9.0),
|
||||
false
|
||||
));
|
||||
assert!(should_store(
|
||||
QUEUE_COUNT_PREFIX,
|
||||
Some((3.0, RECENT)),
|
||||
Some(9.0),
|
||||
false
|
||||
));
|
||||
// A tag that has just backed up is recorded at once.
|
||||
assert!(should_store(QUEUE_COUNT_PREFIX, None, Some(9.0), false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_delay_climbs_while_the_same_job_stays_at_the_head() {
|
||||
let stat = windmill_common::queue::QueueStat { count: 3, delay: 330.0, head_since: 1000.0 };
|
||||
// A held sample written at 1320 saw the same head: it switches to climbing at once,
|
||||
// although the delay has not moved past the tolerance yet.
|
||||
let first = QueueSample::Held(320.0);
|
||||
let climbing = delay_sample(Some(first.head_since(1320.0)), &stat);
|
||||
assert_eq!(climbing, QueueSample::Climbing { since: 1000.0 });
|
||||
assert!(redraws(first, climbing));
|
||||
let drawn = Some((first.value_at(1330.0), RECENT));
|
||||
assert!(should_store(
|
||||
QUEUE_DELAY_PREFIX,
|
||||
drawn,
|
||||
Some(stat.delay),
|
||||
true
|
||||
));
|
||||
// Stored climbing, it draws the delay exactly: nothing more until the heartbeat.
|
||||
let drawn = Some((climbing.value_at(1600.0), RECENT));
|
||||
assert!(!should_store(QUEUE_DELAY_PREFIX, drawn, Some(600.0), false));
|
||||
assert_eq!(delay_sample(None, &stat), QueueSample::Held(330.0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_climb_whose_head_left_is_held_even_within_the_tolerance() {
|
||||
// The head waiting since 0 left at 3600 for one queued at 100: 3500s is within 10% of
|
||||
// the 3600s the climb draws, but kept, the climb would go on from the old head.
|
||||
let moved =
|
||||
windmill_common::queue::QueueStat { count: 2, delay: 3500.0, head_since: 100.0 };
|
||||
let climbing = QueueSample::Climbing { since: 0.0 };
|
||||
let next = delay_sample(Some(climbing.head_since(3000.0)), &moved);
|
||||
assert_eq!(next, QueueSample::Held(3500.0));
|
||||
assert!(redraws(climbing, next));
|
||||
let drawn = Some((climbing.value_at(3600.0), RECENT));
|
||||
assert!(!should_store(
|
||||
QUEUE_DELAY_PREFIX,
|
||||
drawn,
|
||||
Some(moved.delay),
|
||||
false
|
||||
));
|
||||
assert!(should_store(
|
||||
QUEUE_DELAY_PREFIX,
|
||||
drawn,
|
||||
Some(moved.delay),
|
||||
true
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
/// When this server last sampled the queue into `metrics`, in Unix milliseconds. It only paces
|
||||
/// how often the queue is scanned for that; whether a sample earns a row is decided from what
|
||||
/// is already stored. Servers sampling in the same instant can each write it, and the duplicate
|
||||
/// draws the same.
|
||||
static LAST_QUEUE_SAMPLE_MS: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0);
|
||||
|
||||
pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
|
||||
// clean queue metrics older than 14 days
|
||||
sqlx::query!(
|
||||
"DELETE FROM metrics WHERE id LIKE 'queue_%' AND created_at < NOW() - INTERVAL '14 day'"
|
||||
@@ -5262,6 +5414,131 @@ pub async fn expose_queue_metrics(db: &Pool<Postgres>) {
|
||||
.execute(db)
|
||||
.await
|
||||
.ok();
|
||||
|
||||
let metrics_enabled = METRICS_ENABLED.load(std::sync::atomic::Ordering::Relaxed);
|
||||
let otel_enabled = OTEL_METRICS_ENABLED.load(Ordering::Relaxed);
|
||||
let now_ms = chrono::Utc::now().timestamp_millis();
|
||||
let save_metrics = now_ms - LAST_QUEUE_SAMPLE_MS.load(Ordering::Relaxed)
|
||||
>= (QUEUE_METRIC_MIN_INTERVAL_SECS * 1000.0) as i64;
|
||||
if !(metrics_enabled || otel_enabled || save_metrics) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Single DB query for running counts, shared by Prometheus and OTel. It runs ahead of the
|
||||
// backlog read below, which gives up on the rest of the round when it fails.
|
||||
let otel_running = otel_enabled;
|
||||
#[cfg(feature = "prometheus")]
|
||||
let need_running_counts = metrics_enabled || otel_running;
|
||||
#[cfg(not(feature = "prometheus"))]
|
||||
let need_running_counts = otel_running;
|
||||
|
||||
if need_running_counts {
|
||||
let queue_running_counts = windmill_common::queue::get_queue_running_counts(db).await;
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
for q in QUEUE_RUNNING_COUNT_TAGS.read().await.iter() {
|
||||
if queue_running_counts.get(q).is_none() {
|
||||
(*QUEUE_RUNNING_COUNT).with_label_values(&[q]).set(0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if otel_running {
|
||||
for q in OTEL_QUEUE_RUNNING_COUNT_TAGS.read().await.iter() {
|
||||
if queue_running_counts.get(q).is_none() {
|
||||
otel_set_queue_running_count(q, 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(unused_mut, unused_variables)]
|
||||
let mut running_tags_to_watch: Vec<String> = vec![];
|
||||
#[allow(unused_mut, unused_variables)]
|
||||
let mut otel_running_tags_to_watch: Vec<String> = vec![];
|
||||
for (tag, count) in &queue_running_counts {
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
let metric = (*QUEUE_RUNNING_COUNT).with_label_values(&[tag]);
|
||||
metric.set(*count as i64);
|
||||
running_tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
|
||||
if otel_running {
|
||||
otel_set_queue_running_count(tag, *count as i64);
|
||||
otel_running_tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
let mut w = QUEUE_RUNNING_COUNT_TAGS.write().await;
|
||||
*w = running_tags_to_watch;
|
||||
}
|
||||
if otel_running {
|
||||
let mut w = OTEL_QUEUE_RUNNING_COUNT_TAGS.write().await;
|
||||
*w = otel_running_tags_to_watch;
|
||||
}
|
||||
}
|
||||
|
||||
let queue_stats = match windmill_common::queue::get_queue_stats(db).await {
|
||||
Ok(queue_stats) => queue_stats,
|
||||
Err(e) => {
|
||||
tracing::error!("Failed to read queue stats: {e:#}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
for q in QUEUE_COUNT_TAGS.read().await.iter() {
|
||||
if queue_stats.get(q).is_none() {
|
||||
(*QUEUE_COUNT).with_label_values(&[q]).set(0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if otel_enabled {
|
||||
for q in OTEL_QUEUE_COUNT_TAGS.read().await.iter() {
|
||||
if queue_stats.get(q).is_none() {
|
||||
otel_set_queue_count(q, 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(unused_mut)]
|
||||
let mut tags_to_watch = vec![];
|
||||
#[allow(unused_mut)]
|
||||
let mut otel_tags_to_watch = vec![];
|
||||
for (tag, stat) in queue_stats.iter() {
|
||||
let count = stat.count;
|
||||
|
||||
#[cfg(feature = "prometheus")]
|
||||
if metrics_enabled {
|
||||
let metric = (*QUEUE_COUNT).with_label_values(&[tag]);
|
||||
metric.set(count as i64);
|
||||
tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
|
||||
if otel_enabled {
|
||||
otel_tags_to_watch.push(tag.to_string());
|
||||
}
|
||||
otel_set_queue_count(tag, count as i64);
|
||||
}
|
||||
|
||||
if save_metrics {
|
||||
LAST_QUEUE_SAMPLE_MS.store(now_ms, Ordering::Relaxed);
|
||||
save_queue_metrics(db, &queue_stats).await;
|
||||
}
|
||||
|
||||
if metrics_enabled {
|
||||
let mut w = QUEUE_COUNT_TAGS.write().await;
|
||||
*w = tags_to_watch;
|
||||
}
|
||||
if otel_enabled {
|
||||
let mut w = OTEL_QUEUE_COUNT_TAGS.write().await;
|
||||
*w = otel_tags_to_watch;
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn reload_smtp_config(db: &Pool<Postgres>) {
|
||||
|
||||
@@ -70,15 +70,15 @@ ci_test_reference: workspace_id(char), test_script_path(char), test_script_hash(
|
||||
concurrency_settings: hash(bigint), concurrency_key(char), concurrent_limit(int), concurrency_time_window_s(int)
|
||||
config: name(char), config(jsonb)
|
||||
custom_concurrency_key_ended: key(char), ended_at(ts)
|
||||
dbt_column_edge: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), parent_unique_id(text), parent_column(text), child_unique_id(text), child_column(text), lineage_kind(text), ingested_at(ts)
|
||||
dbt_column_edge: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), parent_unique_id(text), parent_column(text), child_unique_id(text), child_column(text), lineage_kind(text), ingested_at(ts), id(bigint)
|
||||
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
|
||||
dbt_edge: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), parent_unique_id(text), child_unique_id(text), ingested_at(ts)
|
||||
dbt_edge: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), parent_unique_id(text), child_unique_id(text), ingested_at(ts), id(bigint)
|
||||
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
|
||||
dbt_graph_snapshot: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), digest(text), relation_root_at_last_ingest(text), ingested_at(ts), permissioned_as(char)
|
||||
dbt_graph_snapshot: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), digest(text), relation_root_at_last_ingest(text), ingested_at(ts), permissioned_as(char), id(bigint)
|
||||
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
|
||||
dbt_environment_state: workspace_id(char), script_path(char), environment(text), job_id(uuid), manifest(text), manifest_key(text), run_results(text), run_results_key(text), updated_at(ts)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
dbt_node: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), unique_id(text), resource_type(text), name(text), asset_path(text), materialized(text), materialize_strategy(text), unique_key(text), tags(text[]), description(text), test_kind(text), test_column(text), test_args(jsonb), severity(text), attached_node(text), columns(jsonb), column_schema(jsonb), freshness(jsonb), raw_code(text), original_file_path(text), ingested_at(ts)
|
||||
dbt_node: workspace_id(char), script_path(char), script_hash(bigint), job_id(uuid), unique_id(text), resource_type(text), name(text), asset_path(text), materialized(text), materialize_strategy(text), unique_key(text), tags(text[]), description(text), test_kind(text), test_column(text), test_args(jsonb), severity(text), attached_node(text), columns(jsonb), column_schema(jsonb), freshness(jsonb), raw_code(text), original_file_path(text), ingested_at(ts), id(bigint)
|
||||
FK: (workspace_id) -> workspace(id), (workspace_id, script_hash) -> script(workspace_id, hash)
|
||||
dbt_run_progress: workspace_id(char), job_id(uuid), asset_kind(asset_kind), asset_path(char), status(materialization_status), row_count(bigint), error(text), updated_at(ts)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
@@ -88,7 +88,7 @@ debounce_key: key(char), job_id(uuid), previous_job_id(uuid), first_started_at(t
|
||||
debounce_stale_data: job_id(uuid), to_relock(text[])
|
||||
debouncing_settings: hash(bigint), debounce_key(char), debounce_delay_s(int), max_total_debouncing_time(int), max_total_debounces_amount(int), debounce_args_to_accumulate(text[])
|
||||
dependency_map: workspace_id(char), importer_path(char), importer_kind(importer_kind), imported_path(char), importer_node_id(char)
|
||||
deployment_metadata: workspace_id(char), path(char), script_hash(bigint), app_version(bigint), callback_job_ids(uuid[]), deployment_msg(text), flow_version(bigint), job_id(uuid)
|
||||
deployment_metadata: workspace_id(char), path(char), script_hash(bigint), app_version(bigint), callback_job_ids(uuid[]), deployment_msg(text), flow_version(bigint), job_id(uuid), id(bigint)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
draft: workspace_id(char), path(char), typ(draft_type), value(json), created_at(ts)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
@@ -147,7 +147,7 @@ mcp_oauth_refresh_token: id(bigint), refresh_token(char), access_token_hash(char
|
||||
mcp_oauth_server_client: client_id(char), client_name(char), redirect_uris(text[]), created_at(ts)
|
||||
mcp_oauth_server_code: code(char), client_id(char), user_email(char), workspace_id(char), scopes(text[]), redirect_uri(text), code_challenge(char), code_challenge_method(char), created_at(ts), expires_at(ts)
|
||||
FK: (client_id) -> mcp_oauth_server_client(client_id)
|
||||
metrics: id(char), value(jsonb), created_at(ts)
|
||||
metrics: id(char), value(jsonb), created_at(ts), row_id(bigint)
|
||||
mqtt_trigger: mqtt_resource_path(char), subscribe_topics(jsonb[]), client_version(mqtt_client_version), v5_config(jsonb), v3_config(jsonb), client_id(char), path(char), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), server_id(char), last_server_ping(ts), error(text), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode), labels(text[])
|
||||
native_trigger: external_id(char), workspace_id(char), service_name(native_trigger_service), script_path(char), is_flow(bool), webhook_token_hash(char), service_config(jsonb), error(text), created_at(ts), updated_at(ts), enabled(bool)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
@@ -225,7 +225,7 @@ workspace_key: workspace_id(char), kind(workspace_key_kind), key(char)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
workspace_protection_rule: workspace_id(char), name(char), rules(int), bypass_groups(text[]), bypass_users(text[]), created_at(ts)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
workspace_runnable_dependencies: flow_path(char), runnable_path(char), script_hash(bigint), runnable_is_flow(bool), workspace_id(char), app_path(char)
|
||||
workspace_runnable_dependencies: flow_path(char), runnable_path(char), script_hash(bigint), runnable_is_flow(bool), workspace_id(char), app_path(char), id(bigint)
|
||||
FK: (app_path, workspace_id) -> app(path, workspace_id) | (flow_path, workspace_id) -> flow(path, workspace_id)
|
||||
workspace_settings: workspace_id(char), slack_team_id(char), slack_name(char), slack_command_script(char), slack_email(char), customer_id(char), plan(char), webhook(text), ai_config(jsonb), large_file_storage(jsonb), git_sync(jsonb), default_app(char), default_scripts(jsonb), deploy_ui(jsonb), mute_critical_alerts(bool), color(char), operator_settings(jsonb), teams_command_script(text), teams_team_id(text), teams_team_name(text), git_app_installations(jsonb), ducklake(jsonb), slack_oauth_client_id(char), slack_oauth_client_secret(char), datatable(jsonb), teams_team_guid(text), auto_invite(jsonb), error_handler(jsonb), success_handler(jsonb), public_app_execution_limit_per_minute(int), dbt_warehouses(jsonb), guest_access_enabled(bool), guest_jwt_public_key(text), guest_jwt_jwks_url(text)
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
//! Guests are unavailable on the shared cloud (`CLOUD_HOSTED`).
|
||||
//!
|
||||
//! One test in its own binary on purpose: `CLOUD_HOSTED` is read once into a
|
||||
//! `lazy_static`, so it must be set before anything reads it and cannot be unset for a
|
||||
//! sibling test in the same process.
|
||||
//!
|
||||
//! Users from the `base` fixture:
|
||||
//! test-user (admin, token SECRET_TOKEN)
|
||||
|
||||
use serde_json::json;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_test_utils::*;
|
||||
|
||||
const ADMIN_TOKEN: &str = "SECRET_TOKEN";
|
||||
const GUEST_TOKEN: &str = "GUEST_SECRET_TOKEN";
|
||||
const APP_PATH: &str = "u/test-user/guest_app";
|
||||
|
||||
fn client() -> reqwest::Client {
|
||||
reqwest::Client::new()
|
||||
}
|
||||
|
||||
fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder {
|
||||
builder.header("Authorization", format!("Bearer {}", token))
|
||||
}
|
||||
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn the_cloud_admits_no_guest(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
// Before the server starts, so the flag is what the whole process sees.
|
||||
unsafe { std::env::set_var("CLOUD_HOSTED", "true") };
|
||||
initialize_tracing().await;
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
let ws = format!("http://localhost:{port}/api/w/test-workspace");
|
||||
|
||||
// The workspace switch cannot be turned on, so no policy can lean on it.
|
||||
let resp = authed(
|
||||
client().post(format!("{ws}/workspaces/edit_guest_access")),
|
||||
ADMIN_TOKEN,
|
||||
)
|
||||
.json(&json!({ "guest_access_enabled": true }))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 400);
|
||||
assert!(
|
||||
resp.text().await?.contains("self-hosted"),
|
||||
"the refusal must name what guests need"
|
||||
);
|
||||
|
||||
let resp = authed(client().post(format!("{ws}/apps/create")), ADMIN_TOKEN)
|
||||
.json(&json!({
|
||||
"path": APP_PATH,
|
||||
"summary": "Guest app",
|
||||
"value": {},
|
||||
"policy": { "execution_mode": "guest", "triggerables": {} }
|
||||
}))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 400, "an app cannot be deployed to guests");
|
||||
|
||||
// Nor can a key be configured for the JWT way in — the refusal lands before the
|
||||
// outbound JWKS fetch it would otherwise make.
|
||||
let resp = authed(
|
||||
client().post(format!("{ws}/workspaces/edit_guest_jwt_key")),
|
||||
ADMIN_TOKEN,
|
||||
)
|
||||
.json(&json!({ "jwks_url": "https://issuer.example.com/.well-known/jwks.json" }))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 400, "a guest JWT key cannot be configured");
|
||||
// Clearing one stays allowed: a key nobody can use is still worth removing.
|
||||
let resp = authed(
|
||||
client().post(format!("{ws}/workspaces/edit_guest_jwt_key")),
|
||||
ADMIN_TOKEN,
|
||||
)
|
||||
.json(&json!({}))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 200, "{}", resp.text().await?);
|
||||
|
||||
// An app already stored in guest mode — pushed by git-sync, or deployed before the
|
||||
// instance became a cloud one — advertises no entry either.
|
||||
let resp = authed(client().post(format!("{ws}/apps/create")), ADMIN_TOKEN)
|
||||
.json(&json!({
|
||||
"path": APP_PATH,
|
||||
"summary": "Guest app",
|
||||
"value": {},
|
||||
"policy": { "execution_mode": "publisher", "triggerables": {} }
|
||||
}))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 201, "{}", resp.text().await?);
|
||||
sqlx::query(
|
||||
"UPDATE app SET policy = jsonb_set(policy, '{execution_mode}', '\"guest\"')
|
||||
WHERE path = $1 AND workspace_id = 'test-workspace'",
|
||||
)
|
||||
.bind(APP_PATH)
|
||||
.execute(&db)
|
||||
.await?;
|
||||
sqlx::query("UPDATE workspace_settings SET guest_access_enabled = true WHERE workspace_id = 'test-workspace'")
|
||||
.execute(&db)
|
||||
.await?;
|
||||
|
||||
// Deploying it again is not refused: only widening an app into guests is, so a
|
||||
// git-sync push of one already stored that way keeps working (and keeps being inert).
|
||||
let resp = authed(
|
||||
client().post(format!("{ws}/apps/update/{APP_PATH}")),
|
||||
ADMIN_TOKEN,
|
||||
)
|
||||
.json(&json!({
|
||||
"policy": { "execution_mode": "guest", "triggerables": {} }
|
||||
}))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
200,
|
||||
"an app already stored in guest mode must stay deployable: {}",
|
||||
resp.text().await?
|
||||
);
|
||||
|
||||
let resp = authed(
|
||||
client().get(format!("{ws}/apps/secret_of/{APP_PATH}")),
|
||||
ADMIN_TOKEN,
|
||||
)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 200, "reading the share secret must succeed");
|
||||
let secret: String = resp.text().await?;
|
||||
let resp = client()
|
||||
.get(format!("{ws}/apps_u/guest_entry/{secret}"))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
404,
|
||||
"a guest app must not advertise entry where guests are unavailable"
|
||||
);
|
||||
|
||||
// And a session issued before the instance became a cloud one stops on its next
|
||||
// request: the door re-reads the switch, so the credential itself is not enough.
|
||||
sqlx::query(
|
||||
"INSERT INTO token (token_hash, token_prefix, token, email, label, scopes, workspace_id, expiration)
|
||||
VALUES (encode(sha256($1::bytea), 'hex'), 'GUEST_SECR', $2, 'guest@example.com',
|
||||
'guest_session', $3, 'test-workspace', now() + interval '8 hours')",
|
||||
)
|
||||
.bind(GUEST_TOKEN.as_bytes())
|
||||
.bind(GUEST_TOKEN)
|
||||
.bind(vec![
|
||||
"guest".to_string(),
|
||||
"users:read".to_string(),
|
||||
format!("apps:read:{APP_PATH}"),
|
||||
format!("apps:run:{APP_PATH}"),
|
||||
])
|
||||
.execute(&db)
|
||||
.await?;
|
||||
// `whoami` is where an admitted guest resolves as `role: guest`, so a 401 here is
|
||||
// the door refusing the credential rather than a route saying no.
|
||||
let resp = authed(client().get(format!("{ws}/users/whoami")), GUEST_TOKEN)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
401,
|
||||
"a guest session must not authenticate where guests are unavailable"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -939,6 +939,57 @@ export function main() { return midValue(); }"#,
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A run with local modules and no lock executes the bundle its lock generation built, which
|
||||
/// kept the imported script's pin; the run must still load the one copy in node_modules, and
|
||||
/// leave the script's own data alone even where it matches the pinned specifier.
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn test_bun_modules_run_loads_imported_pin_from_node_modules(
|
||||
db: Pool<Postgres>,
|
||||
) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
|
||||
insert_deployed_bun_script(
|
||||
&db,
|
||||
"f/pinned_import_modules/module",
|
||||
41240002,
|
||||
r#"import * as isNumber from "is-number@6.0.0";
|
||||
export const ns = isNumber;
|
||||
export const label = "is-number@6.0.0";"#,
|
||||
)
|
||||
.await;
|
||||
|
||||
let job = JobPayload::Code(RawCode {
|
||||
content: r#"import * as isNumber from "is-number";
|
||||
import { ns, label } from "/f/pinned_import_modules/module";
|
||||
import { local } from "./helper";
|
||||
export function main() { return [ns === isNumber, label, local()]; }"#
|
||||
.into(),
|
||||
path: Some("f/pinned_import_modules/main".into()),
|
||||
language: ScriptLang::Bun,
|
||||
modules: Some(std::collections::HashMap::from([(
|
||||
"helper.ts".to_string(),
|
||||
windmill_common::scripts::ScriptModule {
|
||||
content: "export const local = () => 'local';".into(),
|
||||
language: ScriptLang::Bun,
|
||||
lock: None,
|
||||
},
|
||||
)])),
|
||||
..RawCode::default()
|
||||
});
|
||||
|
||||
let result = run_job_in_new_worker_until_complete(&db, false, job, port)
|
||||
.await
|
||||
.json_result()
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
result,
|
||||
serde_json::json!([true, "is-number@6.0.0", "local"])
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[sqlx::test(fixtures("base", "bun_edge_cases"))]
|
||||
async fn test_bun_shared_imports_both_styles(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
|
||||
@@ -12,8 +12,8 @@
|
||||
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_common::git_sync_ee::{
|
||||
git_credential_for_url, repo_provider, repo_supports_managed_git_features, set_git_credential,
|
||||
GitProvider,
|
||||
create_repo_webhook, git_credential_for_url, repo_provider, repo_supports_managed_git_features,
|
||||
set_git_credential, GitProvider,
|
||||
};
|
||||
use windmill_common::workspaces::GitCredentialProvider;
|
||||
|
||||
@@ -250,3 +250,31 @@ async fn a_credential_is_not_served_over_a_downgraded_transport(
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A GitLab the server cannot reach is the error reported, not the GitHub App
|
||||
/// lookup that runs after it: for a self-managed GitLab behind a firewall or an
|
||||
/// untrusted certificate, "no GitHub App installation" names neither the host
|
||||
/// nor the cause.
|
||||
#[sqlx::test(fixtures("git_sync_fork_credential"))]
|
||||
async fn an_unreachable_gitlab_host_is_the_reported_error(
|
||||
db: Pool<Postgres>,
|
||||
) -> anyhow::Result<()> {
|
||||
let err = create_repo_webhook(
|
||||
&db,
|
||||
"parent-ws",
|
||||
"http://glpat-secret@127.0.0.1:1/grp/proj.git",
|
||||
"https://windmill.example/api/w/parent-ws/git_sync/webhook/gitlab",
|
||||
"hook-secret",
|
||||
)
|
||||
.await
|
||||
.expect_err("nothing listens on port 1");
|
||||
assert!(
|
||||
err.to_string().contains("Could not reach the git host"),
|
||||
"unexpected error: {err}"
|
||||
);
|
||||
assert!(
|
||||
!err.to_string().contains("glpat-secret"),
|
||||
"the URL credential leaked into the error: {err}"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -1485,3 +1485,44 @@ async fn declarative_sync_rejects_an_unusable_webhook_base_url(db: Pool<Postgres
|
||||
"the other settings in the same apply must not have been written either"
|
||||
);
|
||||
}
|
||||
|
||||
/// Same contract for the announcement banner: this path owns its validation, and a value
|
||||
/// that lands here unchecked reaches every user's browser. A rejected banner must not be
|
||||
/// half-applied either.
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn declarative_sync_rejects_an_unusable_instance_banner(db: Pool<Postgres>) {
|
||||
clear_settings_and_configs(&db).await;
|
||||
let before = count_global_settings(&db).await;
|
||||
|
||||
let mut desired = BTreeMap::new();
|
||||
desired.insert(
|
||||
"base_url".to_string(),
|
||||
serde_json::json!("https://wm.example.com"),
|
||||
);
|
||||
desired.insert(
|
||||
"instance_banner".to_string(),
|
||||
serde_json::json!({ "enabled": true, "message": "down", "link": "javascript:alert(1)" }),
|
||||
);
|
||||
|
||||
let err = windmill_common::instance_config::sync_global_settings_declarative(
|
||||
&db,
|
||||
&BTreeMap::new(),
|
||||
&desired,
|
||||
)
|
||||
.await
|
||||
.expect_err("a javascript: banner link must fail the sync");
|
||||
assert!(
|
||||
err.to_string().contains("instance_banner"),
|
||||
"the error should name the offending setting, got: {err}"
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
count_global_settings(&db).await,
|
||||
before,
|
||||
"validation must run before anything is applied"
|
||||
);
|
||||
assert!(
|
||||
get_global_setting(&db, "base_url").await.is_none(),
|
||||
"the other settings in the same apply must not have been written either"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
//! Regression test for cross-site GET CSRF on the job-run endpoints that can run a Hub script.
|
||||
//!
|
||||
//! `run_wait_result/p/{path}` and `run_and_stream/p/{path}` answer GET and, for a `hub/` path,
|
||||
//! run any public Hub script. The session cookie is `SameSite=Lax`, so a browser attaches it
|
||||
//! to a cross-site top-level GET navigation, and an argument written `$var:<path>` or
|
||||
//! `$res:<path>` is resolved as the caller: an attacker page could make a logged-in browser
|
||||
//! run a generic Hub script and hand it the victim's secrets. CORS hides the response but not
|
||||
//! the side effect.
|
||||
//!
|
||||
//! `CrossSiteGetGuard` refuses such a request. Workspace scripts are deliberately not refused,
|
||||
//! since they only run code the workspace's own members deployed. A request carrying its own
|
||||
//! credential is allowed; the one case that regresses is a signed-in user clicking a `?token=`
|
||||
//! Hub-script link from another site, because `extract_token` gives the cookie precedence and
|
||||
//! exempting the parameter would let `?token=junk` reinstate the vector.
|
||||
//!
|
||||
//! This test pins down:
|
||||
//! - both endpoints refuse a cross-site cookie GET to a Hub script (the core fix), whether
|
||||
//! `Sec-Fetch-Site` says so or, with no such header (plain http), a cross-host `Referer`,
|
||||
//! - a junk `token` query parameter does not buy a pass,
|
||||
//! - the scope: the same request to a workspace script is not refused,
|
||||
//! - a Hub-script request with its own credential (bearer, or `?token=` and no cookie), or
|
||||
//! sent as a POST, gets through.
|
||||
//!
|
||||
//! No runnable exists and no Hub is contacted. A request that gets past the guard fails as
|
||||
//! not-found on a workspace path, and on the non-numeric version in `hub/x/...` for a Hub
|
||||
//! path, which is rejected while resolving the runnable, before any call to the Hub.
|
||||
|
||||
use reqwest::StatusCode;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_test_utils::*;
|
||||
|
||||
const HUB_GETS: [&str; 2] = [
|
||||
"run_wait_result/p/hub/x/absent",
|
||||
"run_and_stream/p/hub/x/absent",
|
||||
];
|
||||
const WORKSPACE_GETS: [&str; 2] = [
|
||||
"run_wait_result/p/u/test-user/absent",
|
||||
"run_and_stream/p/u/test-user/absent",
|
||||
];
|
||||
|
||||
async fn send(req: reqwest::RequestBuilder) -> anyhow::Result<(StatusCode, String)> {
|
||||
let resp = req.send().await?;
|
||||
let status = resp.status();
|
||||
Ok((status, resp.text().await?))
|
||||
}
|
||||
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn test_cross_site_get_cannot_run_hub_scripts(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let base = format!(
|
||||
"http://localhost:{}/api/w/test-workspace/jobs",
|
||||
server.addr.port()
|
||||
);
|
||||
let client = reqwest::Client::new();
|
||||
let cookie_get = |path: &str| {
|
||||
client
|
||||
.get(format!("{base}/{path}"))
|
||||
.header("Cookie", "token=SECRET_TOKEN")
|
||||
};
|
||||
|
||||
// ---- CORE REGRESSION: a cross-site cookie GET cannot run a Hub script.
|
||||
for path in HUB_GETS {
|
||||
let refused = [
|
||||
(
|
||||
"Sec-Fetch-Site: cross-site",
|
||||
cookie_get(path).header("Sec-Fetch-Site", "cross-site"),
|
||||
),
|
||||
// Plain http gets no `Sec-Fetch-*` at all, so `Referer` is the only signal left.
|
||||
(
|
||||
"cross-host Referer with no Sec-Fetch-Site",
|
||||
cookie_get(path).header("Referer", "http://attacker.example/page"),
|
||||
),
|
||||
// The cookie outranks a `token` query parameter when authenticating.
|
||||
(
|
||||
"junk ?token= next to the cookie",
|
||||
client
|
||||
.get(format!("{base}/{path}?token=junk"))
|
||||
.header("Cookie", "token=SECRET_TOKEN")
|
||||
.header("Sec-Fetch-Site", "cross-site"),
|
||||
),
|
||||
];
|
||||
for (name, req) in refused {
|
||||
let (status, body) = send(req).await?;
|
||||
assert_eq!(
|
||||
status,
|
||||
StatusCode::FORBIDDEN,
|
||||
"{path} [{name}] must be refused: {body}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Scope: the same request to a workspace script is not refused.
|
||||
for path in WORKSPACE_GETS {
|
||||
let (status, body) = send(cookie_get(path).header("Sec-Fetch-Site", "cross-site")).await?;
|
||||
assert_eq!(
|
||||
status,
|
||||
StatusCode::NOT_FOUND,
|
||||
"{path} is a workspace script and must reach the handler: {body}"
|
||||
);
|
||||
}
|
||||
|
||||
// ---- A Hub-script request that carries its own credential, or is a POST, gets through.
|
||||
let hub = HUB_GETS[1];
|
||||
let allowed = [
|
||||
(
|
||||
"cross-origin bearer token",
|
||||
client
|
||||
.get(format!("{base}/{hub}"))
|
||||
.header("Authorization", "Bearer SECRET_TOKEN")
|
||||
.header("Sec-Fetch-Site", "cross-site"),
|
||||
),
|
||||
(
|
||||
"cross-origin ?token= with no cookie",
|
||||
client
|
||||
.get(format!("{base}/{hub}?token=SECRET_TOKEN"))
|
||||
.header("Sec-Fetch-Site", "cross-site"),
|
||||
),
|
||||
(
|
||||
"POST with the cookie",
|
||||
client
|
||||
.post(format!("{base}/{hub}"))
|
||||
.header("Cookie", "token=SECRET_TOKEN")
|
||||
.header("Sec-Fetch-Site", "cross-site")
|
||||
.json(&serde_json::json!({})),
|
||||
),
|
||||
];
|
||||
for (name, req) in allowed {
|
||||
let (status, body) = send(req).await?;
|
||||
assert!(
|
||||
body.contains("Invalid hub script version"),
|
||||
"{name} must get past the guard to runnable resolution (got {status}): {body}"
|
||||
);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -627,3 +627,121 @@ async fn test_root_job_span_relocated_to_inbound_trace() {
|
||||
expected_uuid_trace
|
||||
);
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════
|
||||
// RESOURCE ATTRIBUTES (OTEL_RESOURCE_ATTRIBUTES)
|
||||
// ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
fn resource_attrs() -> std::collections::HashMap<String, String> {
|
||||
otlp_service_resource(
|
||||
&windmill_common::utils::Mode::Worker,
|
||||
"fallback-host",
|
||||
"dev",
|
||||
)
|
||||
.iter()
|
||||
.map(|(k, v)| (k.to_string(), v.to_string()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn test_otlp_resource_merges_env_attributes_without_losing_windmill_identity() {
|
||||
// These take precedence over the hostname argument and over OTEL_RESOURCE_ATTRIBUTES,
|
||||
// so clear them or an ambient one fails the assertions below for an unrelated reason.
|
||||
for var in [
|
||||
"OTEL_HOST_NAME",
|
||||
"OTEL_SERVICE_NAME",
|
||||
"OTEL_SERVICE_VERSION",
|
||||
] {
|
||||
std::env::remove_var(var);
|
||||
}
|
||||
std::env::set_var(
|
||||
"OTEL_RESOURCE_ATTRIBUTES",
|
||||
"k8s.pod.uid=abc-123,service.name=injected,host.name=injected",
|
||||
);
|
||||
let attrs = resource_attrs();
|
||||
std::env::remove_var("OTEL_RESOURCE_ATTRIBUTES");
|
||||
|
||||
// Attributes the deployment injects reach the exporters.
|
||||
assert_eq!(
|
||||
attrs.get("k8s.pod.uid").map(String::as_str),
|
||||
Some("abc-123")
|
||||
);
|
||||
// OTEL_RESOURCE_ATTRIBUTES is the secondary resource, so Windmill's own values still win.
|
||||
assert_eq!(
|
||||
attrs.get("service.name").map(String::as_str),
|
||||
Some("windmill-worker")
|
||||
);
|
||||
assert_eq!(
|
||||
attrs.get("host.name").map(String::as_str),
|
||||
Some("fallback-host")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn test_otlp_resource_dedicated_overrides_win() {
|
||||
// A deployment sets these per pod, e.g. from Kubernetes downward-API labels. The
|
||||
// competing service.name must lose: the spec ranks OTEL_SERVICE_NAME above it.
|
||||
std::env::set_var("OTEL_RESOURCE_ATTRIBUTES", "service.name=should-lose");
|
||||
std::env::set_var("OTEL_SERVICE_NAME", "windmill-workers");
|
||||
std::env::set_var("OTEL_SERVICE_VERSION", "1.802.0");
|
||||
std::env::set_var("OTEL_HOST_NAME", "pod-7");
|
||||
let overridden = resource_attrs();
|
||||
|
||||
// An empty value means unset, which is what the downward API yields for a missing label.
|
||||
for var in [
|
||||
"OTEL_SERVICE_NAME",
|
||||
"OTEL_SERVICE_VERSION",
|
||||
"OTEL_HOST_NAME",
|
||||
] {
|
||||
std::env::set_var(var, "");
|
||||
}
|
||||
let empty = resource_attrs();
|
||||
for var in [
|
||||
"OTEL_SERVICE_NAME",
|
||||
"OTEL_SERVICE_VERSION",
|
||||
"OTEL_HOST_NAME",
|
||||
"OTEL_RESOURCE_ATTRIBUTES",
|
||||
] {
|
||||
std::env::remove_var(var);
|
||||
}
|
||||
let unset = resource_attrs();
|
||||
|
||||
assert_eq!(
|
||||
overridden.get("service.name").map(String::as_str),
|
||||
Some("windmill-workers")
|
||||
);
|
||||
assert_eq!(
|
||||
overridden.get("service.version").map(String::as_str),
|
||||
Some("1.802.0")
|
||||
);
|
||||
assert_eq!(
|
||||
overridden.get("host.name").map(String::as_str),
|
||||
Some("pod-7")
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
empty.get("service.name").map(String::as_str),
|
||||
Some("windmill-worker")
|
||||
);
|
||||
assert_eq!(
|
||||
empty.get("host.name").map(String::as_str),
|
||||
Some("fallback-host")
|
||||
);
|
||||
assert_eq!(
|
||||
empty.get("service.version").map(String::as_str),
|
||||
Some(windmill_common::utils::GIT_VERSION)
|
||||
);
|
||||
|
||||
// With nothing set at all — the default deployment — SdkProvidedResourceDetector still
|
||||
// contributes service.name = "unknown_service". Ours has to overwrite it.
|
||||
assert_eq!(
|
||||
unset.get("service.name").map(String::as_str),
|
||||
Some("windmill-worker")
|
||||
);
|
||||
assert_eq!(
|
||||
unset.get("service.version").map(String::as_str),
|
||||
Some(windmill_common::utils::GIT_VERSION)
|
||||
);
|
||||
}
|
||||
|
||||
@@ -159,3 +159,58 @@ export function main(name: string) {
|
||||
assert_eq!(result, json!("hello world"));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A multi-file script run without a lock is bundled by the lockfile build. A pinned import in
|
||||
/// a workspace script it imports must be installed at that version and still resolve at run time.
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn test_bun_module_imports_pinned_workspace_script(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO script (workspace_id, created_by, content, schema, summary, description, path, hash, language, lock)
|
||||
VALUES ('test-workspace', 'test-user', $1, '{}', '', '', 'f/system/pinned_module', 12350, 'bun', '')",
|
||||
)
|
||||
.bind(
|
||||
r#"
|
||||
import _ from "lodash@4.17.20";
|
||||
export function lodashVersion() { return _.VERSION; }
|
||||
"#,
|
||||
)
|
||||
.execute(&db)
|
||||
.await?;
|
||||
|
||||
let mut modules = HashMap::new();
|
||||
modules.insert(
|
||||
"helper.ts".to_string(),
|
||||
ScriptModule {
|
||||
content: "export function label(v: string) { return v; }\n".to_string(),
|
||||
language: ScriptLang::Bun,
|
||||
lock: None,
|
||||
},
|
||||
);
|
||||
|
||||
let job = JobPayload::Code(RawCode {
|
||||
content: r#"
|
||||
import { lodashVersion } from "/f/system/pinned_module";
|
||||
import { label } from "./helper.ts";
|
||||
export function main() { return label(lodashVersion()); }
|
||||
"#
|
||||
.to_owned(),
|
||||
path: Some("f/system/my_script".to_string()),
|
||||
language: ScriptLang::Bun,
|
||||
modules: Some(modules),
|
||||
tag: None,
|
||||
..RawCode::default()
|
||||
});
|
||||
|
||||
let result = RunJob::from(job)
|
||||
.run_until_complete(&db, false, port)
|
||||
.await
|
||||
.json_result()
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result, json!("4.17.20"));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -585,6 +585,8 @@ pub fn list_completed_jobs_query(
|
||||
let mut sqlb = SqlBuilder::select_from("v2_job_completed")
|
||||
.fields(fields)
|
||||
.order_by(
|
||||
// The runs page picks its pagination cursor column from this same rule
|
||||
// (frontend/src/lib/components/runs/useJobsLoader.svelte.ts); change both together.
|
||||
if lq.completed_before.is_some()
|
||||
|| lq.completed_after.is_some()
|
||||
|| lq.success == Some(false)
|
||||
|
||||
@@ -59,11 +59,11 @@ use windmill_common::{
|
||||
CRITICAL_ALERT_MUTE_UI_SETTING, CUSTOM_TAGS_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING,
|
||||
DISABLE_HUB_SETTING, EMAIL_DOMAIN_SETTING, ENV_SETTINGS,
|
||||
GITHUB_APP_WEBHOOK_BASE_URL_SETTING, HTTP_ROUTE_WORKSPACED_ROUTE_SETTING,
|
||||
HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING, MAX_RETENTION_OVERRIDE_WORKSPACES,
|
||||
RETENTION_PERIOD_SECS_OVERRIDES_SETTING, RUFF_CONFIG_SETTING,
|
||||
WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
|
||||
WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
|
||||
WS_BASE_URL_SETTING,
|
||||
HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING, INSTANCE_BANNER_SETTING,
|
||||
MAX_RETENTION_OVERRIDE_WORKSPACES, RETENTION_PERIOD_SECS_OVERRIDES_SETTING,
|
||||
RUFF_CONFIG_SETTING, WORKSPACE_FAIRNESS_DURATION_SECS_SETTING,
|
||||
WORKSPACE_FAIRNESS_ENABLED_SETTING, WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING,
|
||||
WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING, WS_BASE_URL_SETTING,
|
||||
},
|
||||
instance_config::{self, ApplyMode, InstanceConfig},
|
||||
server::Smtp,
|
||||
@@ -113,6 +113,9 @@ async fn get_ruff_config_unauthed(Extension(db): Extension<DB>) -> error::Result
|
||||
pub fn global_service() -> Router {
|
||||
#[warn(unused_mut)]
|
||||
let r = Router::new()
|
||||
// `/local` is the path in openapi.yaml, so every generated client (getLocal) calls it;
|
||||
// `/envs` stays for callers that found the route in the code.
|
||||
.route("/local", get(get_local_settings))
|
||||
.route("/envs", get(get_local_settings))
|
||||
.route(
|
||||
"/global/{key}",
|
||||
@@ -1172,6 +1175,18 @@ async fn run_setting_pre_write_hook(
|
||||
}
|
||||
}
|
||||
}
|
||||
INSTANCE_BANNER_SETTING => {
|
||||
match value {
|
||||
// Clearing (delete row) is handled by the caller; allow it through.
|
||||
serde_json::Value::Null => {}
|
||||
serde_json::Value::String(s) if s.trim().is_empty() => {}
|
||||
v => {
|
||||
windmill_common::global_settings::validate_instance_banner(v).map_err(|e| {
|
||||
error::Error::BadRequest(format!("{INSTANCE_BANNER_SETTING}: {e}"))
|
||||
})?;
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Ok(())
|
||||
@@ -1312,6 +1327,7 @@ pub async fn get_global_setting(
|
||||
&& key != APP_WORKSPACED_ROUTE_SETTING
|
||||
&& key != HTTP_ROUTE_WORKSPACED_ROUTE_SETTING
|
||||
&& key != WS_BASE_URL_SETTING
|
||||
&& key != INSTANCE_BANNER_SETTING
|
||||
{
|
||||
require_super_admin(&db, &authed).await?;
|
||||
}
|
||||
|
||||
@@ -19,6 +19,7 @@ use windmill_common::{
|
||||
db::UserDB,
|
||||
error::JsonResult,
|
||||
jobs::{HIDE_WORKERS_FOR_NON_ADMINS, TAGS_ARE_SENSITIVE},
|
||||
queue_metrics::{read_queue_metrics_series, QueueMetricsSeries},
|
||||
utils::{paginate, Pagination},
|
||||
worker::{ALL_TAGS, CUSTOM_TAGS_PER_WORKSPACE, DEFAULT_TAGS, DEFAULT_TAGS_PER_WORKSPACE},
|
||||
workspaces::workspace_with_fork_ancestors,
|
||||
@@ -38,6 +39,8 @@ pub fn global_service() -> Router {
|
||||
)
|
||||
.route("/get_default_tags", get(get_default_tags))
|
||||
.route("/queue_metrics", get(get_queue_metrics))
|
||||
.route("/queue_metrics_series", get(get_queue_metrics_series))
|
||||
.route("/queue_status", get(get_queue_status))
|
||||
.route("/queue_counts", get(get_queue_counts))
|
||||
.route("/queue_running_counts", get(get_queue_running_counts))
|
||||
.route(
|
||||
@@ -270,10 +273,16 @@ async fn get_queue_metrics(
|
||||
) -> JsonResult<Vec<QueueMetric>> {
|
||||
require_devops_role(&db, &authed).await?;
|
||||
|
||||
// The API declares every `value` a number, so a climbing delay, stored as its head's wait
|
||||
// start, is returned as the delay at the time of its sample.
|
||||
let queue_metrics = sqlx::query_as!(
|
||||
QueueMetric,
|
||||
"WITH queue_metrics as (
|
||||
SELECT id, value, created_at
|
||||
SELECT id, created_at,
|
||||
CASE WHEN jsonb_typeof(value) = 'object'
|
||||
THEN to_jsonb(EXTRACT(EPOCH FROM created_at) - (value->>'since')::numeric)
|
||||
ELSE value
|
||||
END AS value
|
||||
FROM metrics
|
||||
WHERE id LIKE 'queue_%'
|
||||
AND created_at > now() - interval '14 day'
|
||||
@@ -289,6 +298,88 @@ async fn get_queue_metrics(
|
||||
Ok(Json(queue_metrics))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct QueueMetricsSeriesQuery {
|
||||
window_secs: Option<i64>,
|
||||
}
|
||||
|
||||
const QUEUE_METRICS_DEFAULT_WINDOW_SECS: i64 = 24 * 3600;
|
||||
/// Retention of queue metrics, past which there is nothing left to read.
|
||||
const QUEUE_METRICS_MAX_WINDOW_SECS: i64 = 14 * 24 * 3600;
|
||||
|
||||
async fn get_queue_metrics_series(
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Query(query): Query<QueueMetricsSeriesQuery>,
|
||||
) -> JsonResult<QueueMetricsSeries> {
|
||||
require_devops_role(&db, &authed).await?;
|
||||
|
||||
let window = query
|
||||
.window_secs
|
||||
.unwrap_or(QUEUE_METRICS_DEFAULT_WINDOW_SECS)
|
||||
.clamp(60, QUEUE_METRICS_MAX_WINDOW_SECS);
|
||||
Ok(Json(read_queue_metrics_series(&db, window as f64).await?))
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct QueueTagStatus {
|
||||
tag: String,
|
||||
/// Jobs due for more than 3 seconds that no worker has picked up.
|
||||
waiting: u32,
|
||||
/// How long the job the next pull would take has been waiting, in seconds.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
delay: Option<f64>,
|
||||
running: i64,
|
||||
/// Workers that pinged in the last minute and pull this tag.
|
||||
workers: i64,
|
||||
}
|
||||
|
||||
/// Every tag with jobs waiting or running, read live from the queue. A backlog on a tag no live
|
||||
/// worker pulls waits for one to start: a worker group scaling up from zero, or none at all for a
|
||||
/// tag nobody serves.
|
||||
async fn get_queue_status(
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
) -> JsonResult<Vec<QueueTagStatus>> {
|
||||
require_devops_role(&db, &authed).await?;
|
||||
|
||||
let backlog = windmill_common::queue::get_queue_stats(&db).await?;
|
||||
let backlog_tags = backlog.keys().cloned().collect::<Vec<_>>();
|
||||
// A job's tag is resolved before it is queued (per-workspace and dedicated worker tags
|
||||
// included), and the pull matches it exactly against the worker's tags, so containment is
|
||||
// exact here too.
|
||||
let rows = sqlx::query!(
|
||||
"WITH running AS (
|
||||
SELECT tag, count(*) AS n FROM v2_job_queue WHERE running = true GROUP BY tag
|
||||
)
|
||||
SELECT t.tag AS \"tag!\", COALESCE(r.n, 0) AS \"running!\",
|
||||
(SELECT count(*) FROM worker_ping w
|
||||
WHERE w.ping_at > now() - interval '1 minute' AND w.custom_tags @> ARRAY[t.tag]
|
||||
) AS \"workers!\"
|
||||
FROM (SELECT tag::text FROM running UNION SELECT unnest($1::text[])) t(tag)
|
||||
LEFT JOIN running r ON r.tag = t.tag
|
||||
ORDER BY t.tag",
|
||||
&backlog_tags[..],
|
||||
)
|
||||
.fetch_all(&db)
|
||||
.await?;
|
||||
|
||||
Ok(Json(
|
||||
rows.into_iter()
|
||||
.map(|row| {
|
||||
let stat = backlog.get(&row.tag);
|
||||
QueueTagStatus {
|
||||
waiting: stat.map_or(0, |s| s.count),
|
||||
delay: stat.map(|s| s.delay),
|
||||
running: row.running,
|
||||
workers: row.workers,
|
||||
tag: row.tag,
|
||||
}
|
||||
})
|
||||
.collect(),
|
||||
))
|
||||
}
|
||||
|
||||
async fn get_queue_counts(
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
|
||||
@@ -4691,6 +4691,9 @@ async fn edit_guest_access(
|
||||
Json(EditGuestAccess { guest_access_enabled }): Json<EditGuestAccess>,
|
||||
) -> Result<String> {
|
||||
require_admin(authed.is_admin, &authed.username)?;
|
||||
if guest_access_enabled {
|
||||
windmill_common::workspaces::require_guest_support()?;
|
||||
}
|
||||
|
||||
let mut tx = db.begin().await?;
|
||||
sqlx::query!(
|
||||
@@ -4747,6 +4750,11 @@ async fn edit_guest_jwt_key(
|
||||
"Set a PEM public key or a JWKS URL, not both".to_string(),
|
||||
));
|
||||
}
|
||||
// Clearing stays allowed wherever guests are: a key nobody can use is still worth
|
||||
// removing.
|
||||
if public_key.is_some() || jwks_url.is_some() {
|
||||
windmill_common::workspaces::require_guest_support()?;
|
||||
}
|
||||
if let Some(pem) = public_key.as_deref() {
|
||||
windmill_common::guest_jwt::decoding_key_from_pem(pem)?;
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
openapi: "3.0.3"
|
||||
|
||||
info:
|
||||
version: 1.805.0
|
||||
version: 1.809.0
|
||||
title: Windmill API
|
||||
|
||||
contact:
|
||||
@@ -5913,7 +5913,8 @@ paths:
|
||||
account; the `guest` app execution mode admits them. Off by default. Re-read
|
||||
where a guest session is minted and at the auth door on every guest request, so
|
||||
turning it off takes effect immediately, for sessions already issued and for
|
||||
apps whose policy already says `guest`.
|
||||
apps whose policy already says `guest`. Turning it *on* is refused with a 400
|
||||
where guests are unavailable (the shared cloud); turning it off always works.
|
||||
operationId: editGuestAccess
|
||||
tags:
|
||||
- workspace
|
||||
@@ -5948,7 +5949,8 @@ paths:
|
||||
URL, at most one. Both empty clears the workspace key; off cloud, verification then
|
||||
falls back to the instance issuer (`JWT_EXT_JWKS_URL`) if one is set, else no guest
|
||||
JWT is accepted (`guest_access_enabled` is the on/off switch). Workspace-admin gated.
|
||||
The key is validated before it is stored.
|
||||
The key is validated before it is stored. Setting a key is refused with a 400 where
|
||||
guests are unavailable (the shared cloud); clearing one always works.
|
||||
operationId: editGuestJwtKey
|
||||
tags:
|
||||
- workspace
|
||||
@@ -9150,7 +9152,8 @@ paths:
|
||||
description: >-
|
||||
The custom-path counterpart of `getGuestEntry`. Unauthenticated; 404 unless
|
||||
the app's execution mode is `guest` AND its workspace has
|
||||
`guest_access_enabled` AND the instance has not set `guest_access_disabled`.
|
||||
`guest_access_enabled` AND the instance has not set `guest_access_disabled`,
|
||||
and never on a deployment where guests are unavailable (the shared cloud).
|
||||
Returns the workspace too, since a custom URL may not carry it.
|
||||
operationId: getGuestEntryByCustomPath
|
||||
tags:
|
||||
@@ -13332,8 +13335,9 @@ paths:
|
||||
Unauthenticated: what a signed-out visitor reads to learn that signing in
|
||||
would let them in. 404 unless the app's execution mode is `guest` AND the
|
||||
workspace has `guest_access_enabled` AND the instance has not set the
|
||||
`guest_access_disabled` global setting, so it says nothing about apps that
|
||||
are not open to guests. Discloses only the app path, to a caller already
|
||||
`guest_access_disabled` global setting, and never on a deployment where guests
|
||||
are unavailable (the shared cloud), so it says nothing about apps that are not
|
||||
open to guests. Discloses only the app path, to a caller already
|
||||
holding the share secret.
|
||||
operationId: getGuestEntry
|
||||
tags:
|
||||
@@ -16847,6 +16851,7 @@ paths:
|
||||
- can_approve
|
||||
- user_auth_required
|
||||
- approvers
|
||||
- skin
|
||||
properties:
|
||||
flow_id:
|
||||
type: string
|
||||
@@ -16879,6 +16884,16 @@ paths:
|
||||
hide_cancel:
|
||||
type: boolean
|
||||
description: whether to hide the cancel button in the UI
|
||||
skin:
|
||||
type: string
|
||||
enum: [detailed, minimal]
|
||||
description: how the approval page presents the request
|
||||
step_summary:
|
||||
type: string
|
||||
description: summary of the approval step, for the page title
|
||||
flow_summary:
|
||||
type: string
|
||||
description: summary of the flow or workflow the approval belongs to
|
||||
approvers:
|
||||
type: array
|
||||
items:
|
||||
@@ -21830,6 +21845,98 @@ paths:
|
||||
- id
|
||||
- values
|
||||
|
||||
/workers/queue_metrics_series:
|
||||
get:
|
||||
summary: get the queue metrics of a time window, as a bounded line per tag
|
||||
operationId: getQueueMetricsSeries
|
||||
tags:
|
||||
- worker
|
||||
parameters:
|
||||
- name: window_secs
|
||||
in: query
|
||||
required: false
|
||||
description: how far back to read, in seconds (defaults to one day, capped at the 14-day retention)
|
||||
schema:
|
||||
type: integer
|
||||
responses:
|
||||
"200":
|
||||
description: jobs waiting and queue delay per tag, as the vertices of lines joined by straight segments
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
from:
|
||||
type: integer
|
||||
description: start of the window, in epoch milliseconds
|
||||
to:
|
||||
type: integer
|
||||
description: end of the window, in epoch milliseconds
|
||||
tags:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
properties:
|
||||
tag:
|
||||
type: string
|
||||
count:
|
||||
type: array
|
||||
description: "[epoch ms, jobs waiting more than 3 seconds] vertices"
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: number
|
||||
delay:
|
||||
type: array
|
||||
description: "[epoch ms, seconds the next job has waited] vertices"
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: number
|
||||
required:
|
||||
- tag
|
||||
- count
|
||||
- delay
|
||||
required:
|
||||
- from
|
||||
- to
|
||||
- tags
|
||||
|
||||
/workers/queue_status:
|
||||
get:
|
||||
summary: get the live queue status of every tag with jobs waiting or running
|
||||
operationId: getQueueStatus
|
||||
tags:
|
||||
- worker
|
||||
responses:
|
||||
"200":
|
||||
description: queue status per tag
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
properties:
|
||||
tag:
|
||||
type: string
|
||||
waiting:
|
||||
type: integer
|
||||
description: jobs due for more than 3 seconds that no worker has picked up
|
||||
delay:
|
||||
type: number
|
||||
description: seconds the job the next pull would take has been waiting, absent when none is
|
||||
running:
|
||||
type: integer
|
||||
workers:
|
||||
type: integer
|
||||
description: workers that pinged in the last minute and pull this tag
|
||||
required:
|
||||
- tag
|
||||
- waiting
|
||||
- running
|
||||
- workers
|
||||
|
||||
/workers/queue_counts:
|
||||
get:
|
||||
summary: get counts of jobs waiting for an executor per tag
|
||||
@@ -25793,6 +25900,30 @@ paths:
|
||||
schema:
|
||||
type: string
|
||||
|
||||
/w/{workspace}/hub/projects:
|
||||
get:
|
||||
summary: list the hub's published projects
|
||||
description: |
|
||||
Forwards to the configured Hub's public project catalogue and returns its
|
||||
status code and raw response body. Readable by any workspace member: the
|
||||
listing is not workspace-scoped, and it is proxied only because the Hub's
|
||||
listing endpoint sends no CORS header. Refused with 400 when the instance
|
||||
has the Hub disabled, in which case no outbound request is made.
|
||||
operationId: listHubProjects
|
||||
tags:
|
||||
- hubPublish
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
responses:
|
||||
"200":
|
||||
description: raw Hub response body (status code is passed through from the Hub)
|
||||
content:
|
||||
text/plain:
|
||||
schema:
|
||||
type: string
|
||||
"400":
|
||||
description: the Hub is disabled on this instance
|
||||
|
||||
/w/{workspace}/hub/project:
|
||||
get:
|
||||
summary: get the hub project linked to a workspace folder
|
||||
@@ -29324,7 +29455,12 @@ components:
|
||||
to one seat: `billable_guests`, `guest_seats`); every other plan and build
|
||||
admits no new email until the count drops. `instance_enabled` is the superadmin
|
||||
switch (`guest_access_disabled` global setting) every workspace switch sits under.
|
||||
`available` is whether this deployment can have guests at all: false on the shared
|
||||
cloud, where guest access requires a self-hosted or dedicated deployment, and every
|
||||
other field and switch is then moot.
|
||||
properties:
|
||||
available:
|
||||
type: boolean
|
||||
instance_enabled:
|
||||
type: boolean
|
||||
guest_count:
|
||||
@@ -29344,6 +29480,7 @@ components:
|
||||
type: integer
|
||||
format: int64
|
||||
required:
|
||||
- available
|
||||
- instance_enabled
|
||||
- guest_count
|
||||
- window_days
|
||||
|
||||
@@ -17,6 +17,7 @@ use std::str::FromStr;
|
||||
use uuid::Uuid;
|
||||
use windmill_common::cache;
|
||||
use windmill_common::error::Error;
|
||||
use windmill_common::flows::{ApprovalSkin, Suspend};
|
||||
use windmill_common::jobs::JobKind;
|
||||
use windmill_common::scripts::ScriptHash;
|
||||
|
||||
@@ -94,6 +95,17 @@ pub struct ApprovalFormDetails {
|
||||
pub message_str: String,
|
||||
pub urls: ResumeUrls,
|
||||
pub schema: Option<ResumeFormRow>,
|
||||
pub skin: ApprovalSkin,
|
||||
}
|
||||
|
||||
/// The suspended step an approval message is about, and the flow run it belongs to.
|
||||
struct ApprovalStep {
|
||||
created_by: String,
|
||||
created_at: chrono::NaiveDateTime,
|
||||
script_path: Option<String>,
|
||||
parent_job_id: Option<Uuid>,
|
||||
args: Option<sqlx::types::Json<Box<RawValue>>>,
|
||||
suspend: Option<Suspend>,
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
@@ -205,6 +217,90 @@ pub async fn get_approval_form_details(
|
||||
|
||||
tracing::debug!("Job ID: {:?}", job_id);
|
||||
|
||||
let ApprovalStep { created_by, created_at, script_path, parent_job_id, args, suspend } =
|
||||
fetch_approval_step(&db, w_id, job_id, flow_step_id).await?;
|
||||
|
||||
let schema = suspend.as_ref().map(|suspend| ResumeFormRow {
|
||||
resume_form: suspend.resume_form.clone(),
|
||||
hide_cancel: suspend.hide_cancel,
|
||||
});
|
||||
let skin = suspend.and_then(|s| s.skin).unwrap_or_default();
|
||||
|
||||
let bold_format = match format {
|
||||
MessageFormat::Slack => "*{}*",
|
||||
MessageFormat::Teams => "**{}**",
|
||||
};
|
||||
|
||||
let message_str = match skin {
|
||||
ApprovalSkin::Detailed => {
|
||||
let args_str = args.map_or("None".to_string(), |a| {
|
||||
serde_json::from_str::<serde_json::Value>(a.get())
|
||||
.ok()
|
||||
.and_then(|v| serde_json::to_string_pretty(&v).ok())
|
||||
.unwrap_or_else(|| a.get().to_string())
|
||||
});
|
||||
let parent_job_id_str = parent_job_id.map_or("None".to_string(), |id| id.to_string());
|
||||
let script_path_str = script_path.as_deref().unwrap_or("None");
|
||||
|
||||
let created_at_formatted = created_at.format("%Y-%m-%d %H:%M:%S").to_string();
|
||||
|
||||
let mut message_str = format!(
|
||||
"A workflow has been suspended and is waiting for approval:\n\n\
|
||||
{}: {created_by}\n\n\
|
||||
{}: {created_at_formatted}\n\n\
|
||||
{}: {script_path_str}\n\n\
|
||||
{}:\n```\n{args_str}\n```\n\n\
|
||||
{}: {parent_job_id_str}\n\n",
|
||||
bold_format.replace("{}", "Created by"),
|
||||
bold_format.replace("{}", "Created at"),
|
||||
bold_format.replace("{}", "Script path"),
|
||||
bold_format.replace("{}", "Args"),
|
||||
bold_format.replace("{}", "Flow ID")
|
||||
);
|
||||
|
||||
// Append custom message if provided
|
||||
if let Some(msg) = message {
|
||||
message_str.push_str(msg);
|
||||
}
|
||||
message_str
|
||||
}
|
||||
ApprovalSkin::Minimal => format!(
|
||||
"{}\n\n{}: {created_by}",
|
||||
message.unwrap_or("Your approval is requested."),
|
||||
bold_format.replace("{}", "Requested by"),
|
||||
),
|
||||
};
|
||||
|
||||
tracing::debug!("Schema: {:#?}", schema);
|
||||
|
||||
Ok(ApprovalFormDetails { message_str, urls, schema, skin })
|
||||
}
|
||||
|
||||
/// The skin of the approval step `flow_step_id` of the flow running `job_id`. Falls back to
|
||||
/// the detailed skin when the step cannot be resolved, so a message is still sent.
|
||||
/// Reads through the unrestricted pool without an authorization check of its own: only the
|
||||
/// skin, which is not sensitive, leaves this function.
|
||||
pub(crate) async fn get_approval_step_skin(
|
||||
db: &DB,
|
||||
w_id: &str,
|
||||
job_id: Uuid,
|
||||
flow_step_id: &str,
|
||||
) -> ApprovalSkin {
|
||||
match fetch_approval_step(db, w_id, job_id, Some(flow_step_id)).await {
|
||||
Ok(step) => step.suspend.and_then(|s| s.skin).unwrap_or_default(),
|
||||
Err(e) => {
|
||||
tracing::warn!("Could not resolve approval step {flow_step_id} of job {job_id}: {e}");
|
||||
ApprovalSkin::default()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn fetch_approval_step(
|
||||
db: &DB,
|
||||
w_id: &str,
|
||||
job_id: Uuid,
|
||||
flow_step_id: Option<&str>,
|
||||
) -> Result<ApprovalStep, Error> {
|
||||
// TODO: do we have a helper function for this?
|
||||
let (job_kind, script_hash, raw_flow, parent_job_id, created_at, created_by, script_path, args) = sqlx::query!(
|
||||
"WITH job_info AS (
|
||||
@@ -240,17 +336,17 @@ pub async fn get_approval_form_details(
|
||||
job_id,
|
||||
&w_id
|
||||
)
|
||||
.fetch_optional(&db)
|
||||
.fetch_optional(db)
|
||||
.await
|
||||
.map_err(|e| Error::BadRequest(e.to_string()))?
|
||||
.ok_or_else(|| Error::BadRequest("This workflow is no longer running and has either already timed out or been cancelled or completed.".to_string()))
|
||||
.map(|r| (r.job_kind, r.script_hash, r.raw_flow, r.parent_job, r.created_at, r.created_by, r.script_path, r.args))?;
|
||||
|
||||
let flow_data = match cache::job::fetch_flow(&db, &job_kind, script_hash).await {
|
||||
let flow_data = match cache::job::fetch_flow(db, &job_kind, script_hash).await {
|
||||
Ok(data) => data,
|
||||
Err(_) => {
|
||||
if let Some(parent_job_id) = parent_job_id.as_ref() {
|
||||
cache::job::fetch_preview_flow(&db, parent_job_id, raw_flow).await?
|
||||
cache::job::fetch_preview_flow(db, parent_job_id, raw_flow).await?
|
||||
} else {
|
||||
return Err(Error::BadRequest(
|
||||
"This workflow is no longer running and has either already timed out or been cancelled or completed.".to_string(),
|
||||
@@ -265,49 +361,12 @@ pub async fn get_approval_form_details(
|
||||
|
||||
tracing::debug!("Module: {:#?}", module);
|
||||
|
||||
let schema = module.and_then(|module| {
|
||||
module.suspend.as_ref().map(|suspend| ResumeFormRow {
|
||||
resume_form: suspend.resume_form.clone(),
|
||||
hide_cancel: suspend.hide_cancel,
|
||||
})
|
||||
});
|
||||
|
||||
let args_str = args.map_or("None".to_string(), |a| {
|
||||
serde_json::from_str::<serde_json::Value>(a.get())
|
||||
.ok()
|
||||
.and_then(|v| serde_json::to_string_pretty(&v).ok())
|
||||
.unwrap_or_else(|| a.get().to_string())
|
||||
});
|
||||
let parent_job_id_str = parent_job_id.map_or("None".to_string(), |id| id.to_string());
|
||||
let script_path_str = script_path.as_deref().unwrap_or("None");
|
||||
|
||||
let created_at_formatted = created_at.format("%Y-%m-%d %H:%M:%S").to_string();
|
||||
|
||||
let bold_format = match format {
|
||||
MessageFormat::Slack => "*{}*",
|
||||
MessageFormat::Teams => "**{}**",
|
||||
};
|
||||
|
||||
let mut message_str = format!(
|
||||
"A workflow has been suspended and is waiting for approval:\n\n\
|
||||
{}: {created_by}\n\n\
|
||||
{}: {created_at_formatted}\n\n\
|
||||
{}: {script_path_str}\n\n\
|
||||
{}:\n```\n{args_str}\n```\n\n\
|
||||
{}: {parent_job_id_str}\n\n",
|
||||
bold_format.replace("{}", "Created by"),
|
||||
bold_format.replace("{}", "Created at"),
|
||||
bold_format.replace("{}", "Script path"),
|
||||
bold_format.replace("{}", "Args"),
|
||||
bold_format.replace("{}", "Flow ID")
|
||||
);
|
||||
|
||||
// Append custom message if provided
|
||||
if let Some(msg) = message {
|
||||
message_str.push_str(msg);
|
||||
}
|
||||
|
||||
tracing::debug!("Schema: {:#?}", schema);
|
||||
|
||||
Ok(ApprovalFormDetails { message_str, urls, schema })
|
||||
Ok(ApprovalStep {
|
||||
created_by,
|
||||
created_at,
|
||||
script_path,
|
||||
parent_job_id,
|
||||
args,
|
||||
suspend: module.and_then(|m| m.suspend.clone()),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -343,6 +343,29 @@ fn refuse_unscopable_guest_app(path: &str, mode: ExecutionMode) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Refuse *widening* an app into guests where the deployment has none
|
||||
/// (`instance_supports_guests`). Only the transition is refused, like the protection
|
||||
/// rule below it: an app already stored in the mode — deployed before the instance
|
||||
/// became a cloud one, or pushed by git-sync — keeps deploying, and keeps being inert,
|
||||
/// since every guest gate refuses it anyway. `deployed_mode` is what the app is stored
|
||||
/// as, `None` when it is being created.
|
||||
fn refuse_guest_mode_where_unavailable(
|
||||
path: &str,
|
||||
mode: ExecutionMode,
|
||||
deployed_mode: Option<ExecutionMode>,
|
||||
) -> Result<()> {
|
||||
if !matches!(mode, ExecutionMode::Guest)
|
||||
|| deployed_mode == Some(ExecutionMode::Guest)
|
||||
|| windmill_common::workspaces::instance_supports_guests()
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
Err(Error::BadRequest(format!(
|
||||
"app {path} cannot be set to Guests: {}",
|
||||
windmill_common::workspaces::GUESTS_UNAVAILABLE_MESSAGE
|
||||
)))
|
||||
}
|
||||
|
||||
/// Gate a viewer on the app's `execution_mode`, as far as can be decided without an
|
||||
/// ACL probe. `Ok(true)` means already authorized — anonymous admits anyone, guest
|
||||
/// admits anyone signed in; `Ok(false)` means the caller is a member and still owes
|
||||
@@ -2518,6 +2541,7 @@ async fn create_app_internal<'a>(
|
||||
// even when the caller did not.
|
||||
app.policy.set_execution_mode(app.policy.execution_mode());
|
||||
refuse_unscopable_guest_app(&app.path, app.policy.execution_mode())?;
|
||||
refuse_guest_mode_where_unavailable(&app.path, app.policy.execution_mode(), None)?;
|
||||
if let Some(rule) = deployment_rule_for_mode(app.policy.execution_mode()) {
|
||||
if let RuleCheckResult::Blocked(msg) = check_user_against_rule(
|
||||
w_id,
|
||||
@@ -3565,6 +3589,13 @@ async fn update_app_internal<'a>(
|
||||
ns.path.as_deref().unwrap_or(path),
|
||||
npolicy.execution_mode(),
|
||||
)?;
|
||||
// An unreadable deployed policy reads as not already-in-mode, the strict
|
||||
// direction, as for the protection rule below.
|
||||
refuse_guest_mode_where_unavailable(
|
||||
ns.path.as_deref().unwrap_or(path),
|
||||
npolicy.execution_mode(),
|
||||
deployed_policy.as_ref().map(|d| d.execution_mode()),
|
||||
)?;
|
||||
if let Some(rule) =
|
||||
deployment_rule_for_mode(npolicy.execution_mode()).filter(|_| !authed.is_admin)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
/*
|
||||
* Author: Ruben Fiszel
|
||||
* Copyright: Windmill Labs, Inc 2022
|
||||
* This file and its contents are licensed under the AGPLv3 License.
|
||||
* Please see the included NOTICE for copyright information and
|
||||
* LICENSE-AGPL for a copy of the license.
|
||||
*/
|
||||
|
||||
use axum::extract::FromRequestParts;
|
||||
use axum::http::{header, request::Parts, Method};
|
||||
use axum::Extension;
|
||||
use url::Url;
|
||||
use windmill_common::error::Error;
|
||||
use windmill_common::users::COOKIE_NAME;
|
||||
|
||||
use crate::triggers::trigger_helpers::RunnableId;
|
||||
|
||||
/// Whether a request is a cross-site GET authenticating on the session cookie alone. A GET
|
||||
/// handler that runs a script by path resolves it through [`Self::script_runnable`], which
|
||||
/// refuses a Hub script on such a request.
|
||||
///
|
||||
/// The cookie is `SameSite=Lax`, so browsers attach it to cross-site top-level GET
|
||||
/// navigations. A `hub/` path runs any public Hub script, and an argument written
|
||||
/// `$var:<path>` or `$res:<path>` is resolved as the caller before the script sees it: such a
|
||||
/// GET lets any page pick a generic Hub script and hand it the victim's secrets, which the job
|
||||
/// can then send anywhere.
|
||||
///
|
||||
/// Workspace scripts and flows are not refused, by choice, so that GET links to them keep
|
||||
/// working. That is a scope decision, not a safety property: they still take attacker-chosen
|
||||
/// arguments, `$var:` and `$res:` included, resolved as the victim. What bounds the exposure
|
||||
/// is that the attacker needs a runnable path and can only run code the workspace deployed.
|
||||
///
|
||||
/// The cookie is the only ambient credential. A bearer header is explicit, and so is the
|
||||
/// `token` query parameter the webhook URLs carry — a cross-origin `EventSource` has no other
|
||||
/// way to authenticate, since it cannot set headers. The checks run in `extract_token`'s
|
||||
/// order, header before cookie, because that is the order it resolves them in: a request
|
||||
/// carrying both a cookie and `token=` authenticates on the cookie and is therefore still
|
||||
/// ambient, which is also why a valid `token=` link opened cross-site while signed in is
|
||||
/// refused.
|
||||
pub struct CrossSiteGetGuard(Option<CrossSite>);
|
||||
|
||||
impl CrossSiteGetGuard {
|
||||
pub fn script_runnable(&self, script_path: &str) -> windmill_common::error::Result<RunnableId> {
|
||||
let runnable_id = RunnableId::from_script_path(script_path);
|
||||
let (Some(signal), RunnableId::HubScript(_)) = (&self.0, &runnable_id) else {
|
||||
return Ok(runnable_id);
|
||||
};
|
||||
// The `Referer` leg is the one that can misfire, on a request that really was
|
||||
// same-host: it compares against the hosts the backend can see, and a proxy that
|
||||
// rewrites `Host` without setting `X-Forwarded-Host` leaves none of them matching
|
||||
// what the browser addressed. Name the comparison so that shows up as a
|
||||
// misconfiguration rather than as an unexplained 403.
|
||||
if let CrossSite::RefererMismatch { referer, instance_hosts } = signal {
|
||||
tracing::warn!(
|
||||
referer_host = %referer,
|
||||
?instance_hosts,
|
||||
"refusing a cross-site GET Hub script run inferred from Referer; if the request \
|
||||
was same-host, set `X-Forwarded-Host` on the proxy or configure `BASE_URL`"
|
||||
);
|
||||
}
|
||||
Err(Error::PermissionDenied(
|
||||
"a cross-site GET request cannot run a Hub script with the session cookie, which takes \
|
||||
precedence over a `token` query parameter: pass the token in the `Authorization` \
|
||||
header, or open the link from the instance itself or from a browser with no Windmill \
|
||||
session"
|
||||
.to_string(),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
impl<S: Send + Sync> FromRequestParts<S> for CrossSiteGetGuard {
|
||||
type Rejection = std::convert::Infallible;
|
||||
|
||||
async fn from_request_parts(
|
||||
parts: &mut Parts,
|
||||
state: &S,
|
||||
) -> std::result::Result<Self, Self::Rejection> {
|
||||
if parts.method != Method::GET {
|
||||
return Ok(CrossSiteGetGuard(None));
|
||||
}
|
||||
let Some(signal) = cross_site_signal(parts) else {
|
||||
return Ok(CrossSiteGetGuard(None));
|
||||
};
|
||||
|
||||
let has_bearer = parts
|
||||
.headers
|
||||
.get(header::AUTHORIZATION)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.is_some_and(|v| v.starts_with("Bearer "));
|
||||
if has_bearer {
|
||||
return Ok(CrossSiteGetGuard(None));
|
||||
}
|
||||
|
||||
let has_session_cookie =
|
||||
Extension::<tower_cookies::Cookies>::from_request_parts(parts, state)
|
||||
.await
|
||||
.is_ok_and(|Extension(cookies)| cookies.get(COOKIE_NAME).is_some());
|
||||
Ok(CrossSiteGetGuard(has_session_cookie.then_some(signal)))
|
||||
}
|
||||
}
|
||||
|
||||
enum CrossSite {
|
||||
Declared,
|
||||
RefererMismatch { referer: String, instance_hosts: Vec<String> },
|
||||
}
|
||||
|
||||
fn cross_site_signal(parts: &Parts) -> Option<CrossSite> {
|
||||
if let Some(site) = parts.headers.get("sec-fetch-site") {
|
||||
return site
|
||||
.as_bytes()
|
||||
.eq_ignore_ascii_case(b"cross-site")
|
||||
.then_some(CrossSite::Declared);
|
||||
}
|
||||
// Fetch Metadata rides only on potentially trustworthy URLs, so an instance served
|
||||
// over plain http never receives `Sec-Fetch-Site` (nor does Safari before 16.4) while
|
||||
// the cookie, not being `Secure` there either, still arrives. `Referer` is the only
|
||||
// other thing a top-level GET navigation carries — `Origin` is not sent on one — so it
|
||||
// is all that is left there, and it is weak: the default `strict-origin-when-cross-
|
||||
// origin` policy already drops `Referer` on an https-to-http downgrade, so an https
|
||||
// attacker page pointing a victim at a plain-http instance sends neither header. This
|
||||
// leg catches an http-served attacker page and pre-16.4 Safari on https; the guard is
|
||||
// load-bearing on https and best-effort at best on plain http. An absent `Referer`
|
||||
// reads as not cross-site, matching how `Sec-Fetch-Site: none` (a bookmark, a typed
|
||||
// URL) is treated.
|
||||
let referer = referer_host(parts)?;
|
||||
let instance_hosts: Vec<String> = instance_hosts(parts).collect();
|
||||
(!instance_hosts
|
||||
.iter()
|
||||
.any(|host| host.eq_ignore_ascii_case(&referer)))
|
||||
.then_some(CrossSite::RefererMismatch { referer, instance_hosts })
|
||||
}
|
||||
|
||||
/// Every host a legitimate same-host request can name. `Host` alone is not enough: a
|
||||
/// reverse proxy that forwards without preserving it (nginx `proxy_pass` with no
|
||||
/// `proxy_set_header Host $host`) hands the backend the upstream's name, which no browser
|
||||
/// `Referer` will ever match. None of these is browser-settable on a navigation — a
|
||||
/// navigation carries no custom headers, and `BASE_URL` is instance config — so widening
|
||||
/// the accepted set costs nothing.
|
||||
fn instance_hosts(parts: &Parts) -> impl Iterator<Item = String> {
|
||||
let base_url = windmill_common::BASE_URL.load();
|
||||
[
|
||||
request_host(parts),
|
||||
header_host(parts, "x-forwarded-host"),
|
||||
Url::parse(base_url.as_str())
|
||||
.ok()
|
||||
.and_then(|url| url.host_str().map(str::to_owned)),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
}
|
||||
|
||||
fn referer_host(parts: &Parts) -> Option<String> {
|
||||
let referer = parts.headers.get(header::REFERER)?.to_str().ok()?;
|
||||
Url::parse(referer).ok()?.host_str().map(str::to_owned)
|
||||
}
|
||||
|
||||
fn request_host(parts: &Parts) -> Option<String> {
|
||||
if let Some(host) = parts.uri.host() {
|
||||
return Some(host.to_owned());
|
||||
}
|
||||
header_host(parts, header::HOST)
|
||||
}
|
||||
|
||||
fn header_host(parts: &Parts, name: impl header::AsHeaderName) -> Option<String> {
|
||||
host_of(parts.headers.get(name)?.to_str().ok()?)
|
||||
}
|
||||
|
||||
/// The host in a `Host`-shaped header value: `host[:port]`, where `host` may be a bracketed
|
||||
/// IPv6 literal, and where a chain of proxies appends to `X-Forwarded-Host` so only the
|
||||
/// first entry is the one the browser addressed. The port is split off by the URL parser
|
||||
/// rather than by hand-rolling the bracket rules.
|
||||
fn host_of(value: &str) -> Option<String> {
|
||||
let host = value.split(',').next()?.trim();
|
||||
Url::parse(&format!("http://{host}"))
|
||||
.ok()?
|
||||
.host_str()
|
||||
.map(str::to_owned)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{cross_site_signal, host_of, CrossSite};
|
||||
use axum::http::{request::Parts, Request};
|
||||
|
||||
fn parts(headers: &[(&str, &str)]) -> Parts {
|
||||
let mut req = Request::get("/api/w/ws/jobs/run_wait_result/p/hub/1/x");
|
||||
for (name, value) in headers {
|
||||
req = req.header(*name, *value);
|
||||
}
|
||||
req.body(()).unwrap().into_parts().0
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sec_fetch_site_decides_when_present() {
|
||||
let declared = |site| cross_site_signal(&parts(&[("sec-fetch-site", site)]));
|
||||
assert!(matches!(declared("cross-site"), Some(CrossSite::Declared)));
|
||||
for site in ["same-origin", "same-site", "none"] {
|
||||
assert!(declared(site).is_none(), "{site} is not cross-site");
|
||||
}
|
||||
// The header outranks a `Referer` that disagrees with it.
|
||||
let with_referer = parts(&[
|
||||
("sec-fetch-site", "same-origin"),
|
||||
("host", "windmill.example"),
|
||||
("referer", "https://attacker.example/page"),
|
||||
]);
|
||||
assert!(cross_site_signal(&with_referer).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn referer_stands_in_when_sec_fetch_site_is_absent() {
|
||||
let signal = |headers: &[(&str, &str)]| cross_site_signal(&parts(headers));
|
||||
assert!(matches!(
|
||||
signal(&[
|
||||
("host", "windmill.example"),
|
||||
("referer", "https://attacker.example/p")
|
||||
]),
|
||||
Some(CrossSite::RefererMismatch { .. })
|
||||
));
|
||||
// Ports differ between the frontend and the API, and do not make a request cross-site.
|
||||
assert!(signal(&[
|
||||
("host", "windmill.example:8000"),
|
||||
("referer", "http://windmill.example:3000/apps"),
|
||||
])
|
||||
.is_none());
|
||||
// A proxy that rewrote `Host` but forwarded the public name.
|
||||
assert!(signal(&[
|
||||
("host", "windmill-server.internal"),
|
||||
("x-forwarded-host", "windmill.example"),
|
||||
("referer", "https://windmill.example/apps"),
|
||||
])
|
||||
.is_none());
|
||||
assert!(signal(&[("host", "windmill.example")]).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn host_of_strips_port_brackets_and_proxy_chain() {
|
||||
assert_eq!(host_of("windmill.example"), Some("windmill.example".into()));
|
||||
assert_eq!(
|
||||
host_of("windmill.example:8000"),
|
||||
Some("windmill.example".into())
|
||||
);
|
||||
assert_eq!(host_of("[::1]:8000"), Some("[::1]".into()));
|
||||
assert_eq!(host_of("[::1]"), Some("[::1]".into()));
|
||||
assert_eq!(
|
||||
host_of("windmill.example, proxy.internal"),
|
||||
Some("windmill.example".into())
|
||||
);
|
||||
assert_eq!(host_of(""), None);
|
||||
assert_eq!(host_of("not a host"), None);
|
||||
}
|
||||
}
|
||||
@@ -108,6 +108,9 @@ lazy_static::lazy_static! {
|
||||
(20260826214706, include_str!(
|
||||
"../../migrations/20260826214706_queue_suspended_drop_legacy_index.up.sql"
|
||||
).replace("DROP INDEX", "DROP INDEX CONCURRENTLY")),
|
||||
(20260909163047, include_str!(
|
||||
"../../migrations/20260909163047_workspace_delete_cascade_indexes.up.sql"
|
||||
).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY").replace("DROP INDEX", "DROP INDEX CONCURRENTLY")),
|
||||
].into_iter().collect();
|
||||
}
|
||||
|
||||
|
||||
@@ -6,13 +6,14 @@ use axum::{
|
||||
http::{request::Parts, StatusCode},
|
||||
response::{IntoResponse, Response},
|
||||
routing::{get, post},
|
||||
Router,
|
||||
Extension, Router,
|
||||
};
|
||||
use serde::{Deserialize, Deserializer, Serialize};
|
||||
use windmill_common::{
|
||||
error::{to_anyhow, Error},
|
||||
global_settings::{load_value_from_global_settings, DISABLE_HUB_SETTING},
|
||||
utils::require_admin,
|
||||
HUB_BASE_URL,
|
||||
DB, DEFAULT_HUB_BASE_URL, HUB_BASE_URL,
|
||||
};
|
||||
|
||||
pub fn workspaced_service() -> Router {
|
||||
@@ -48,6 +49,7 @@ pub fn workspaced_service() -> Router {
|
||||
post(discard_project_update),
|
||||
)
|
||||
.route("/project", get(get_project_by_source))
|
||||
.route("/projects", get(list_projects))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
@@ -548,6 +550,84 @@ async fn get_project_by_source(ctx: HubPublishCtx) -> Result<impl IntoResponse,
|
||||
ctx.get("/projects/by_source").await
|
||||
}
|
||||
|
||||
/// Whether this instance points at the public hub. Compared by parsed host rather than by the
|
||||
/// string: `hub_base_url` is stored as the operator typed it, so `http://`, a port, a trailing
|
||||
/// slash, a mixed-case scheme or host, userinfo and a trailing dot all name the same public
|
||||
/// host — and each spelling that failed to match would send a member's token there. Parsing is
|
||||
/// what `reqwest` does with the same string a line later, so this reads the host the request
|
||||
/// will actually go to.
|
||||
///
|
||||
/// A value that does not parse answers "not the public hub", so the caller attaches the token —
|
||||
/// harmless, because `reqwest` cannot build a request from that same value: it is rejected
|
||||
/// before a connection is opened, and the token never reaches a socket.
|
||||
fn is_public_hub(hub: &str) -> bool {
|
||||
fn host_of(url: &str) -> Option<String> {
|
||||
let parsed = url::Url::parse(url.trim()).ok()?;
|
||||
if !matches!(parsed.scheme(), "http" | "https") {
|
||||
return None;
|
||||
}
|
||||
Some(
|
||||
parsed
|
||||
.host_str()?
|
||||
.trim_end_matches('.')
|
||||
.to_ascii_lowercase(),
|
||||
)
|
||||
}
|
||||
match (host_of(hub), host_of(DEFAULT_HUB_BASE_URL)) {
|
||||
(Some(host), Some(default_host)) => host == default_host,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
// The hub's project catalogue. Read by any workspace member rather than through
|
||||
// `HubPublishCtx`, which requires an admin: nothing here is workspace-scoped or
|
||||
// publishing-related. It exists at all because the hub's listing endpoint sends no
|
||||
// CORS header, so the browser cannot read it directly the way it reads a single
|
||||
// project. `accept: application/json` is what makes the hub answer with JSON.
|
||||
//
|
||||
// The caller's token is sent only to a hub this instance was pointed at deliberately.
|
||||
// Every other route here is admin-only; this one is not, so forwarding a member's
|
||||
// bearer token to `hub.windmill.dev` would put a credential replayable against this
|
||||
// instance on a host outside it — for a listing that needs no credential at all.
|
||||
async fn list_projects(
|
||||
_authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Tokened { token }: Tokened,
|
||||
) -> Result<impl IntoResponse, Error> {
|
||||
// `disable_hub` turns the hub off for a closed instance, and this handler makes an
|
||||
// outbound request. The frontend hides its entry points on the same setting, but that
|
||||
// is presentation: an authenticated member can call this route directly, so the refusal
|
||||
// has to live here.
|
||||
let disabled = load_value_from_global_settings(&db, DISABLE_HUB_SETTING)
|
||||
.await?
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
if disabled {
|
||||
return Err(Error::BadRequest(
|
||||
"The hub is disabled on this instance".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
let hub = (**HUB_BASE_URL.load()).clone();
|
||||
let url = format!("{}/projects", hub);
|
||||
let mut req = HTTP_CLIENT.get(&url).header("accept", "application/json");
|
||||
if !is_public_hub(&hub) {
|
||||
req = req.bearer_auth(&token);
|
||||
}
|
||||
let res = req
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| Error::InternalErr(format!("hub request failed: {e}")))?;
|
||||
|
||||
let status = StatusCode::from_u16(res.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
|
||||
let text = res
|
||||
.text()
|
||||
.await
|
||||
.map_err(|e| Error::InternalErr(format!("hub response read failed: {e}")))?;
|
||||
|
||||
Ok((status, text))
|
||||
}
|
||||
|
||||
async fn submit_project(
|
||||
ctx: HubPublishCtx,
|
||||
Path((_workspace, slug)): Path<(String, ProjectSlug)>,
|
||||
@@ -645,3 +725,42 @@ async fn forward_to_hub<T: Serialize>(
|
||||
|
||||
Ok((status, text))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::is_public_hub;
|
||||
|
||||
#[test]
|
||||
fn public_hub_recognized_in_every_spelling() {
|
||||
// The predicate decides whether a workspace member's bearer token leaves the
|
||||
// instance, so both directions matter: a miss on the public hub sends the token
|
||||
// to windmill.dev, and a false match withholds it from a private hub that needs it.
|
||||
// Every spelling here is one `hub_base_url` can hold and `reqwest` will still send.
|
||||
for hub in [
|
||||
"https://hub.windmill.dev",
|
||||
"http://hub.windmill.dev/",
|
||||
"HTTPS://hub.windmill.dev",
|
||||
"https://HUB.WINDMILL.DEV",
|
||||
"https://hub.windmill.dev:443",
|
||||
"https://hub.windmill.dev.",
|
||||
"https://hub.windmill.dev/some/path",
|
||||
" https://hub.windmill.dev ",
|
||||
] {
|
||||
assert!(is_public_hub(hub), "{hub} should be the public hub");
|
||||
}
|
||||
for hub in [
|
||||
"https://hub.internal.example",
|
||||
"https://hub.windmill.dev.evil.example",
|
||||
"https://windmill.dev",
|
||||
// The host is what the request goes to, whatever precedes the `@`.
|
||||
"https://hub.windmill.dev@hub.internal.example",
|
||||
// Unparseable, or not a scheme a request can be built from. Grouped with the
|
||||
// private hubs because the caller then attaches the token, which is harmless here:
|
||||
// `reqwest` rejects the same value before opening a connection.
|
||||
"hub.windmill.dev",
|
||||
"ftp://hub.windmill.dev",
|
||||
] {
|
||||
assert!(!is_public_hub(hub), "{hub} should not be the public hub");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -73,6 +73,7 @@ use crate::{
|
||||
args::{self, RawWebhookArgs},
|
||||
auth::{OptTokened, Tokened},
|
||||
concurrency_groups::join_concurrency_key,
|
||||
csrf::CrossSiteGetGuard,
|
||||
db::{ApiAuthed, DB},
|
||||
triggers::trigger_helpers::RunnableId,
|
||||
users::{
|
||||
@@ -106,7 +107,10 @@ use windmill_common::{
|
||||
db::UserDB,
|
||||
error::{self, to_anyhow, Error},
|
||||
flow_status::{Approval, ApprovalConditions, FlowStatus, FlowStatusModule},
|
||||
flows::{add_virtual_items_if_necessary, resolve_maybe_value, FlowValue},
|
||||
flows::{
|
||||
add_virtual_items_if_necessary, resolve_maybe_value, ApprovalSkin, FlowModule, FlowValue,
|
||||
Suspend,
|
||||
},
|
||||
jobs::{script_path_to_payload, CompletedJob, JobKind, JobPayload, QueuedJob, RawCode},
|
||||
oauth2::HmacSha256,
|
||||
query_builders,
|
||||
@@ -4416,6 +4420,15 @@ async fn count_completed_jobs(
|
||||
))
|
||||
}
|
||||
|
||||
lazy_static::lazy_static! {
|
||||
/// 0 keeps the connection-wide statement_timeout.
|
||||
static ref LIST_JOBS_STATEMENT_TIMEOUT_SECS: u64 =
|
||||
std::env::var("LIST_JOBS_STATEMENT_TIMEOUT_SECS")
|
||||
.ok()
|
||||
.and_then(|x| x.parse().ok())
|
||||
.unwrap_or(30);
|
||||
}
|
||||
|
||||
async fn list_jobs(
|
||||
authed: ApiAuthed,
|
||||
Extension(user_db): Extension<UserDB>,
|
||||
@@ -4533,10 +4546,32 @@ async fn list_jobs(
|
||||
// tracing::info!("sql: {}", &sql);
|
||||
let mut tx: Transaction<'_, Postgres> = user_db.begin(&authed).await?;
|
||||
|
||||
// A client that gives up does not cancel its query, so without this bound every retry of a
|
||||
// slow filter stacks another scan running until the connection-wide 5min timeout.
|
||||
let timeout_secs = *LIST_JOBS_STATEMENT_TIMEOUT_SECS;
|
||||
if timeout_secs > 0 {
|
||||
sqlx::query(&format!("SET LOCAL statement_timeout = '{timeout_secs}s'"))
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
|
||||
let jobs: Vec<UnifiedJob> = sqlx::query_as(&sql)
|
||||
.fetch_all(&mut *tx)
|
||||
.warn_after_seconds_with_sql(5, format!("list_jobs: {}", sql))
|
||||
.await?;
|
||||
.await
|
||||
.map_err(|e| match e {
|
||||
sqlx::Error::Database(ref db_err)
|
||||
if timeout_secs > 0 && db_err.code().as_deref() == Some("57014") =>
|
||||
{
|
||||
Error::Generic(
|
||||
StatusCode::BAD_REQUEST,
|
||||
format!(
|
||||
"Listing jobs took more than {timeout_secs}s and was stopped. Set a start date or narrow the filters."
|
||||
),
|
||||
)
|
||||
}
|
||||
e => e.into(),
|
||||
})?;
|
||||
tx.commit().await?;
|
||||
|
||||
Ok(Json(jobs.into_iter().map(From::from).collect()))
|
||||
@@ -4826,6 +4861,11 @@ struct ApprovalInfo {
|
||||
user_auth_required: bool,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
hide_cancel: Option<bool>,
|
||||
skin: ApprovalSkin,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
step_summary: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
flow_summary: Option<String>,
|
||||
approvers: Vec<Approval>,
|
||||
/// Share-read-link token for the flow, minted only for callers allowed to view this
|
||||
/// approval. Lets an authenticated workspace-member approver open the run details of
|
||||
@@ -4879,6 +4919,48 @@ fn can_approve_step(
|
||||
}
|
||||
}
|
||||
|
||||
/// The latest approval step the run has passed: a step before the current `step` that ran
|
||||
/// rather than being skipped. Steps from `step` on don't count, because while an approval is
|
||||
/// pending the step after it already holds the `WaitingForEvents` status.
|
||||
fn last_reached_approval_step<'a>(
|
||||
flow: &'a FlowValue,
|
||||
status: &FlowStatus,
|
||||
) -> Option<&'a FlowModule> {
|
||||
flow.modules
|
||||
.iter()
|
||||
.zip(status.modules.iter())
|
||||
.take(usize::try_from(status.step).unwrap_or(0))
|
||||
.rev()
|
||||
.filter(|(_, m)| matches!(m, FlowStatusModule::Success { skipped: false, .. }))
|
||||
.map(|(module, _)| module)
|
||||
.find(|module| module.suspend.is_some())
|
||||
}
|
||||
|
||||
/// The approval conditions a step's own settings give, as the worker records them when the step
|
||||
/// suspends. The worker drops them from the run once the step is approved, so a run that has
|
||||
/// moved on is gated by these. Groups computed by an expression can't be re-evaluated outside
|
||||
/// the run, so such a step falls back to any signed-in user.
|
||||
fn approval_conditions_from_settings(suspend: &Suspend) -> Option<ApprovalConditions> {
|
||||
let user_auth_required = suspend.user_auth_required.unwrap_or(false);
|
||||
let self_approval_disabled = suspend.self_approval_disabled.unwrap_or(false);
|
||||
if !user_auth_required && !self_approval_disabled {
|
||||
return None;
|
||||
}
|
||||
let user_groups_required = match &suspend.user_groups_required {
|
||||
Some(InputTransform::Static { value }) if user_auth_required => {
|
||||
serde_json::from_str(value.get()).unwrap_or_default()
|
||||
}
|
||||
_ => vec![],
|
||||
};
|
||||
Some(ApprovalConditions { user_auth_required, user_groups_required, self_approval_disabled })
|
||||
}
|
||||
|
||||
/// How the approval step presents itself on the approval page.
|
||||
struct ApprovalStepView {
|
||||
skin: ApprovalSkin,
|
||||
summary: Option<String>,
|
||||
}
|
||||
|
||||
async fn get_approval_info(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
@@ -4907,13 +4989,33 @@ async fn get_approval_info(
|
||||
script_path: Option<String>,
|
||||
email: String,
|
||||
flow_status: Option<serde_json::Value>,
|
||||
workflow_as_code_status: Option<serde_json::Value>,
|
||||
// `v2_job_status` only holds a run that hasn't finished, so the fields below also read
|
||||
// the completed run's status: a finished run's page keeps its skin and, for workflows as
|
||||
// code, its description, still gated by the approval conditions the run had.
|
||||
completed_flow_status: Option<serde_json::Value>,
|
||||
is_wac: bool,
|
||||
wac_approval: Option<serde_json::Value>,
|
||||
approval_conditions: Option<serde_json::Value>,
|
||||
flow_summary: Option<String>,
|
||||
}
|
||||
let row = sqlx::query_as::<_, ApprovalJobRow>(
|
||||
"SELECT j.id, j.runnable_path as script_path, j.permissioned_as_email as email,
|
||||
s.flow_status, s.workflow_as_code_status
|
||||
s.flow_status,
|
||||
c.flow_status AS completed_flow_status,
|
||||
COALESCE(s.workflow_as_code_status, c.workflow_as_code_status) IS NOT NULL
|
||||
AS is_wac,
|
||||
COALESCE(s.workflow_as_code_status, c.workflow_as_code_status)->'_approval'
|
||||
AS wac_approval,
|
||||
COALESCE(s.flow_status, c.flow_status)->'approval_conditions'
|
||||
AS approval_conditions,
|
||||
NULLIF(COALESCE(f.summary, sc.summary), '') AS flow_summary
|
||||
FROM v2_job j
|
||||
LEFT JOIN v2_job_status s ON s.id = j.id
|
||||
LEFT JOIN v2_job_completed c ON c.id = j.id
|
||||
LEFT JOIN flow f
|
||||
ON j.kind = 'flow' AND f.workspace_id = j.workspace_id AND f.path = j.runnable_path
|
||||
LEFT JOIN script sc
|
||||
ON j.kind = 'script' AND sc.workspace_id = j.workspace_id AND sc.hash = j.runnable_id
|
||||
WHERE j.id = $1 AND j.workspace_id = $2",
|
||||
)
|
||||
.bind(&job_id)
|
||||
@@ -4922,31 +5024,31 @@ async fn get_approval_info(
|
||||
.await?
|
||||
.ok_or_else(|| Error::NotFound(format!("Job {job_id} not found")))?;
|
||||
|
||||
let is_wac = row.workflow_as_code_status.is_some();
|
||||
let is_wac = row.is_wac;
|
||||
let run_ac = row
|
||||
.approval_conditions
|
||||
.as_ref()
|
||||
.and_then(|v| serde_json::from_value::<ApprovalConditions>(v.clone()).ok());
|
||||
|
||||
// Extract approval info based on WAC vs classic flow
|
||||
let (form_schema, description, default_args, enums, approval_conditions, hide_cancel) =
|
||||
let (form_schema, description, default_args, enums, approval_conditions, hide_cancel, step) =
|
||||
if is_wac {
|
||||
let approval_meta = row
|
||||
.workflow_as_code_status
|
||||
.as_ref()
|
||||
.and_then(|v| v.get("_approval"));
|
||||
let approval_meta = row.wac_approval.as_ref();
|
||||
let form = approval_meta.and_then(|m| m.get("form").cloned());
|
||||
let default_args = approval_meta.and_then(|m| m.get("default_args").cloned());
|
||||
let enums = approval_meta.and_then(|m| m.get("enums").cloned());
|
||||
let description = approval_meta.and_then(|m| m.get("description").cloned());
|
||||
let ac = row
|
||||
.flow_status
|
||||
.as_ref()
|
||||
.and_then(|v| v.get("approval_conditions"))
|
||||
.and_then(|v| serde_json::from_value::<ApprovalConditions>(v.clone()).ok());
|
||||
(form, description, default_args, enums, ac, None)
|
||||
let skin = approval_meta
|
||||
.and_then(|m| m.get("skin"))
|
||||
.and_then(|v| serde_json::from_value::<ApprovalSkin>(v.clone()).ok())
|
||||
.unwrap_or_default();
|
||||
let step = Some(ApprovalStepView { skin, summary: None });
|
||||
(form, description, default_args, enums, run_ac, None, step)
|
||||
} else {
|
||||
let fs = row
|
||||
.flow_status
|
||||
.as_ref()
|
||||
.and_then(|v| serde_json::from_value::<FlowStatus>(v.clone()).ok());
|
||||
let ac = fs.as_ref().and_then(|s| s.approval_conditions.clone());
|
||||
|
||||
// For classic flows, form/description come from the flow definition and step result
|
||||
let approval_step = fs.as_ref().map(|s| (s.step as usize).saturating_sub(1));
|
||||
@@ -5006,6 +5108,28 @@ async fn get_approval_info(
|
||||
.and_then(|s| s.resume_form.as_ref())
|
||||
.map(|rf| serde_json::json!(rf));
|
||||
let hc = suspend_settings.map(|s| s.hide_cancel.unwrap_or(false));
|
||||
let completed_fs = row
|
||||
.completed_flow_status
|
||||
.as_ref()
|
||||
.filter(|_| fs.is_none())
|
||||
.and_then(|v| serde_json::from_value::<FlowStatus>(v.clone()).ok());
|
||||
let approval_module = raw_flow
|
||||
.as_ref()
|
||||
.zip(fs.as_ref().or(completed_fs.as_ref()))
|
||||
.and_then(|(flow, status)| last_reached_approval_step(flow, status));
|
||||
let ac = run_ac.or_else(|| {
|
||||
approval_module
|
||||
.and_then(|module| module.suspend.as_ref())
|
||||
.and_then(approval_conditions_from_settings)
|
||||
});
|
||||
let step = approval_module.map(|module| ApprovalStepView {
|
||||
skin: module
|
||||
.suspend
|
||||
.as_ref()
|
||||
.and_then(|s| s.skin)
|
||||
.unwrap_or_default(),
|
||||
summary: module.summary.clone().filter(|s| !s.trim().is_empty()),
|
||||
});
|
||||
|
||||
// Fetch description, default_args, and enums from the step's completed job result
|
||||
let step_job_id = fs
|
||||
@@ -5029,9 +5153,12 @@ async fn get_approval_info(
|
||||
(None, None, None)
|
||||
};
|
||||
|
||||
(form, desc, default_args, enums, ac, hc)
|
||||
(form, desc, default_args, enums, ac, hc, step)
|
||||
};
|
||||
|
||||
let skin = step.as_ref().map(|s| s.skin).unwrap_or_default();
|
||||
let step_summary = step.and_then(|s| s.summary);
|
||||
|
||||
let user_auth_required = approval_conditions
|
||||
.as_ref()
|
||||
.map(|ac| ac.user_auth_required)
|
||||
@@ -5061,6 +5188,9 @@ async fn get_approval_info(
|
||||
can_approve: false,
|
||||
user_auth_required,
|
||||
hide_cancel: None,
|
||||
skin,
|
||||
step_summary: None,
|
||||
flow_summary: None,
|
||||
approvers: vec![],
|
||||
view_token: None,
|
||||
}));
|
||||
@@ -5096,6 +5226,9 @@ async fn get_approval_info(
|
||||
can_approve,
|
||||
user_auth_required,
|
||||
hide_cancel,
|
||||
skin,
|
||||
step_summary,
|
||||
flow_summary: row.flow_summary,
|
||||
approvers,
|
||||
view_token,
|
||||
}))
|
||||
@@ -7447,6 +7580,7 @@ async fn log_job_view(
|
||||
}
|
||||
|
||||
pub async fn run_wait_result_job_by_path_get(
|
||||
cross_site: CrossSiteGetGuard,
|
||||
method: hyper::http::Method,
|
||||
authed: ApiAuthed,
|
||||
Extension(user_db): Extension<UserDB>,
|
||||
@@ -7459,6 +7593,7 @@ pub async fn run_wait_result_job_by_path_get(
|
||||
check_license_key_valid().await?;
|
||||
|
||||
let script_path = script_path.to_path();
|
||||
let runnable_id = cross_site.script_runnable(script_path)?;
|
||||
check_scopes(&authed, || format!("jobs:run:scripts:{script_path}"))?;
|
||||
|
||||
if method == http::Method::HEAD {
|
||||
@@ -7471,12 +7606,7 @@ pub async fn run_wait_result_job_by_path_get(
|
||||
args.body = args::Body::HashMap(payload_as_args);
|
||||
|
||||
let args = args
|
||||
.to_args_from_runnable(
|
||||
&db,
|
||||
&w_id,
|
||||
RunnableId::from_script_path(script_path),
|
||||
run_query.skip_preprocessor,
|
||||
)
|
||||
.to_args_from_runnable(&db, &w_id, runnable_id, run_query.skip_preprocessor)
|
||||
.await?;
|
||||
|
||||
check_queue_too_long(&db, QUEUE_LIMIT_WAIT_RESULT.or(run_query.queue_limit)).await?;
|
||||
@@ -7895,6 +8025,7 @@ pub async fn stream_flow_by_version(
|
||||
}
|
||||
|
||||
pub async fn stream_script_by_path(
|
||||
cross_site: CrossSiteGetGuard,
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Extension(user_db): Extension<UserDB>,
|
||||
@@ -7903,12 +8034,13 @@ pub async fn stream_script_by_path(
|
||||
method: hyper::http::Method,
|
||||
args: RawWebhookArgs,
|
||||
) -> error::Result<Response> {
|
||||
let runnable_id = cross_site.script_runnable(script_path.to_path())?;
|
||||
stream_job(
|
||||
authed,
|
||||
db,
|
||||
user_db,
|
||||
w_id,
|
||||
RunnableId::from_script_path(script_path.to_path()),
|
||||
runnable_id,
|
||||
args,
|
||||
run_query,
|
||||
method == http::Method::GET,
|
||||
@@ -11877,4 +12009,58 @@ mod approval_view_gate_tests {
|
||||
"trigger@example.com"
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn approval_step_is_the_last_one_passed() {
|
||||
let flow: FlowValue = serde_json::from_value(serde_json::json!({ "modules": [
|
||||
{ "id": "a", "value": { "type": "identity" }, "suspend": {} },
|
||||
{ "id": "b", "value": { "type": "identity" }, "suspend": {} },
|
||||
{ "id": "c", "value": { "type": "identity" } }
|
||||
]}))
|
||||
.unwrap();
|
||||
let step_at = |step: i32, types: [(&str, bool); 3]| {
|
||||
let mut status = FlowStatus::new(&flow);
|
||||
status.step = step;
|
||||
status.modules = ["a", "b", "c"]
|
||||
.into_iter()
|
||||
.zip(types)
|
||||
.map(|(id, (kind, skipped))| {
|
||||
serde_json::from_value(serde_json::json!({
|
||||
"type": kind, "id": id, "job": Uuid::nil(), "count": 1,
|
||||
"failed_retries": [], "skipped": skipped
|
||||
}))
|
||||
.unwrap()
|
||||
})
|
||||
.collect();
|
||||
last_reached_approval_step(&flow, &status).map(|module| module.id.clone())
|
||||
};
|
||||
let waiting = ("WaitingForEvents", false);
|
||||
let pending = ("WaitingForPriorSteps", false);
|
||||
let ran = ("Success", false);
|
||||
let skipped = ("Success", true);
|
||||
// Awaiting a's approval: b, itself an approval step, already holds `WaitingForEvents`.
|
||||
assert_eq!(step_at(1, [ran, waiting, pending]).as_deref(), Some("a"));
|
||||
assert_eq!(step_at(2, [ran, ran, waiting]).as_deref(), Some("b"));
|
||||
assert_eq!(step_at(3, [ran, skipped, ran]).as_deref(), Some("a"));
|
||||
assert_eq!(step_at(0, [pending, pending, pending]), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn approved_step_stays_gated_by_its_settings() {
|
||||
let from_settings = |suspend: serde_json::Value| {
|
||||
approval_conditions_from_settings(&serde_json::from_value(suspend).unwrap())
|
||||
};
|
||||
let login = from_settings(serde_json::json!({
|
||||
"user_auth_required": true,
|
||||
"user_groups_required": { "type": "static", "value": ["approvers"] }
|
||||
}));
|
||||
assert!(!can_view(
|
||||
&None,
|
||||
&login,
|
||||
Some("f/team/flow"),
|
||||
"trigger@example.com"
|
||||
));
|
||||
assert_eq!(login.unwrap().user_groups_required, ["approvers"]);
|
||||
assert!(from_settings(serde_json::json!({})).is_none());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -80,6 +80,7 @@ pub mod azure_proxy_ee;
|
||||
mod azure_proxy_oss;
|
||||
mod capture;
|
||||
mod concurrency_groups;
|
||||
mod csrf;
|
||||
mod db;
|
||||
mod db_health;
|
||||
mod dbt;
|
||||
|
||||
@@ -1265,7 +1265,7 @@ is, a different one moves it there and archives the old path"),
|
||||
},
|
||||
"execution_mode": {
|
||||
"type": "string",
|
||||
"description": "Who the app's runnables execute as. Optional, and what omitting it means depends on the operation: creating an app defaults it to `publisher` (runs on behalf of the app's publisher and requires an authenticated viewer), while updating one keeps the mode the app is already deployed under. Either way `anonymous`, which makes the app publicly executable, is never assumed. Possible values: viewer, publisher, anonymous"
|
||||
"description": "Who may open the app, and who its runnables execute as. Optional, and what omitting it means depends on the operation: creating an app defaults it to `publisher` (runs on behalf of the app's publisher and requires an authenticated viewer), while updating one keeps the mode the app is already deployed under. Neither `anonymous`, which makes the app publicly executable, nor `guest`, which opens it to anyone the identity provider authenticates, is ever assumed. A guest is only admitted where the workspace also has `guest_access_enabled`, which is checked when the session is minted and again on every guest request. Possible values: viewer, publisher, guest, anonymous"
|
||||
},
|
||||
"on_behalf_of": {
|
||||
"type": "string"
|
||||
@@ -1380,7 +1380,7 @@ is, a different one moves it there and archives the old path"),
|
||||
},
|
||||
"execution_mode": {
|
||||
"type": "string",
|
||||
"description": "Who the app's runnables execute as. Optional, and what omitting it means depends on the operation: creating an app defaults it to `publisher` (runs on behalf of the app's publisher and requires an authenticated viewer), while updating one keeps the mode the app is already deployed under. Either way `anonymous`, which makes the app publicly executable, is never assumed. Possible values: viewer, publisher, anonymous"
|
||||
"description": "Who may open the app, and who its runnables execute as. Optional, and what omitting it means depends on the operation: creating an app defaults it to `publisher` (runs on behalf of the app's publisher and requires an authenticated viewer), while updating one keeps the mode the app is already deployed under. Neither `anonymous`, which makes the app publicly executable, nor `guest`, which opens it to anyone the identity provider authenticates, is ever assumed. A guest is only admitted where the workspace also has `guest_access_enabled`, which is checked when the session is minted and again on every guest request. Possible values: viewer, publisher, guest, anonymous"
|
||||
},
|
||||
"on_behalf_of": {
|
||||
"type": "string"
|
||||
|
||||
@@ -13,19 +13,25 @@ use sha2::Sha256;
|
||||
use sqlx::types::Uuid;
|
||||
use std::collections::HashMap;
|
||||
use windmill_common::error::{to_anyhow, Error};
|
||||
use windmill_common::flows::ApprovalSkin;
|
||||
use windmill_common::utils::truncate_with_ellipsis;
|
||||
use windmill_common::variables::{get_secret_value_as_admin, get_workspace_key};
|
||||
|
||||
use crate::db::{ApiAuthed, DB};
|
||||
use crate::jobs::{QueryApprover, ResumeUrls};
|
||||
use crate::{
|
||||
approvals::{
|
||||
extract_w_id_from_resume_url, handle_resume_action, ApprovalFormDetails, FieldType,
|
||||
MessageFormat, QueryButtonText, QueryDefaultArgsJson, QueryDynamicEnumJson,
|
||||
QueryFlowStepId, QueryMessage, ResumeFormField, ResumeSchema,
|
||||
extract_w_id_from_resume_url, get_approval_step_skin, handle_resume_action,
|
||||
ApprovalFormDetails, FieldType, MessageFormat, QueryButtonText, QueryDefaultArgsJson,
|
||||
QueryDynamicEnumJson, QueryFlowStepId, QueryMessage, ResumeFormField, ResumeSchema,
|
||||
},
|
||||
auth::OptTokened,
|
||||
};
|
||||
|
||||
// Slack rejects a button value over 2000 characters, and with it the whole post. The button value
|
||||
// carries the message on to the modal, so the message is shortened to fit.
|
||||
const SLACK_BUTTON_VALUE_MAX_CHARS: usize = 2000;
|
||||
|
||||
#[derive(Deserialize, Debug)]
|
||||
pub struct SlackFormData {
|
||||
payload: String,
|
||||
@@ -127,6 +133,9 @@ struct PrivateMetadata {
|
||||
// HMAC over (w_id, resource_path) keyed on the workspace key; minted when the modal is
|
||||
// built, required by `handle_submission` before the resource_path is decrypted.
|
||||
signature: Option<String>,
|
||||
// Only selects the wording of the updated channel message, so it is left unsigned.
|
||||
#[serde(default)]
|
||||
skin: ApprovalSkin,
|
||||
}
|
||||
|
||||
// Opportunistic transport-level check: when `SLACK_SIGNING_SECRET` is configured we verify
|
||||
@@ -432,6 +441,7 @@ async fn handle_submission(
|
||||
let container: Container = private_metadata.container;
|
||||
let hide_cancel = private_metadata.hide_cancel;
|
||||
let signature = private_metadata.signature;
|
||||
let skin = private_metadata.skin;
|
||||
|
||||
// If hide_cancel is true, we don't need to extract information from the private_metadata
|
||||
if hide_cancel.unwrap_or(false) && action == "cancel" {
|
||||
@@ -463,7 +473,7 @@ async fn handle_submission(
|
||||
tracing::warn!("Failed to resolve slack token for {w_id}/{resource_path}: {e:#}");
|
||||
Error::BadRequest("Invalid Slack callback request".to_string())
|
||||
})?;
|
||||
update_original_slack_message(action, slack_token, container).await?;
|
||||
update_original_slack_message(action, slack_token, container, skin).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -475,14 +485,19 @@ async fn transform_schemas(
|
||||
required: Option<Vec<String>>,
|
||||
default_args_json: Option<&serde_json::Value>,
|
||||
dynamic_enums_json: Option<&serde_json::Value>,
|
||||
skin: ApprovalSkin,
|
||||
) -> Result<serde_json::Value, Error> {
|
||||
tracing::debug!("Resume urls: {:#?}", urls);
|
||||
|
||||
let link_label = match skin {
|
||||
ApprovalSkin::Detailed => "Flow suspension details",
|
||||
ApprovalSkin::Minimal => "View in Windmill",
|
||||
};
|
||||
let mut blocks = vec![serde_json::json!({
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": format!("{}\n<{}|Flow suspension details>", text, urls.approvalPage),
|
||||
"text": format!("{}\n<{}|{link_label}>", text, urls.approvalPage),
|
||||
}
|
||||
})];
|
||||
|
||||
@@ -918,10 +933,6 @@ async fn send_slack_message(
|
||||
value["approver"] = serde_json::json!(approver);
|
||||
}
|
||||
|
||||
if let Some(message) = message {
|
||||
value["message"] = serde_json::json!(message);
|
||||
}
|
||||
|
||||
if let Some(default_args_json) = default_args_json {
|
||||
value["default_args_json"] = default_args_json.clone();
|
||||
}
|
||||
@@ -950,33 +961,8 @@ async fn send_slack_message(
|
||||
.map_err(|e| Box::new(e) as Box<dyn std::error::Error>)?;
|
||||
value["signature"] = serde_json::json!(signature);
|
||||
|
||||
let payload = serde_json::json!({
|
||||
"channel": channel_id,
|
||||
"text": "A flow has been suspended. Please approve or reject the flow.",
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": "A flow has been suspended. Please approve or reject the flow."
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "actions",
|
||||
"elements": [
|
||||
{
|
||||
"type": "button",
|
||||
"text": {
|
||||
"type": "plain_text",
|
||||
"text": "View"
|
||||
},
|
||||
"action_id": "open_modal",
|
||||
"value": value.to_string()
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
});
|
||||
let skin = get_approval_step_skin(db, w_id, job_id, flow_step_id).await;
|
||||
let payload = channel_message_payload(channel_id, skin, message, value);
|
||||
|
||||
tracing::debug!("Payload: {:?}", payload);
|
||||
|
||||
@@ -1000,6 +986,88 @@ async fn send_slack_message(
|
||||
Ok(StatusCode::OK)
|
||||
}
|
||||
|
||||
/// The channel post announcing the approval. Its button hands `button_value` to the modal, with
|
||||
/// `message` added, shortened to what Slack's button value limit leaves room for.
|
||||
fn channel_message_payload(
|
||||
channel_id: &str,
|
||||
skin: ApprovalSkin,
|
||||
message: Option<&str>,
|
||||
mut button_value: serde_json::Value,
|
||||
) -> serde_json::Value {
|
||||
let message = message.map(|m| message_fitting_button_value(&button_value, m));
|
||||
if let Some(message) = &message {
|
||||
button_value["message"] = serde_json::json!(message);
|
||||
}
|
||||
let (text, section, button_label) = match skin {
|
||||
ApprovalSkin::Detailed => {
|
||||
let text = "A flow has been suspended. Please approve or reject the flow.";
|
||||
(text, text.to_string(), "View")
|
||||
}
|
||||
ApprovalSkin::Minimal => {
|
||||
let mut section = "*Approval requested*".to_string();
|
||||
if let Some(message) = &message {
|
||||
section.push('\n');
|
||||
section.push_str(message);
|
||||
}
|
||||
("Approval requested", section, "Review")
|
||||
}
|
||||
};
|
||||
|
||||
serde_json::json!({
|
||||
"channel": channel_id,
|
||||
"text": text,
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": section
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "actions",
|
||||
"elements": [
|
||||
{
|
||||
"type": "button",
|
||||
"text": {
|
||||
"type": "plain_text",
|
||||
"text": button_label
|
||||
},
|
||||
"action_id": "open_modal",
|
||||
"value": button_value.to_string()
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
})
|
||||
}
|
||||
|
||||
/// The longest prefix of `message` that keeps `button_value` carrying it within Slack's limit.
|
||||
fn message_fitting_button_value(button_value: &serde_json::Value, message: &str) -> String {
|
||||
let mut with_message = button_value.clone();
|
||||
let mut fits = |max_chars: usize| {
|
||||
let fitted = truncate_with_ellipsis(message, max_chars);
|
||||
with_message["message"] = serde_json::json!(fitted);
|
||||
(with_message.to_string().chars().count() <= SLACK_BUTTON_VALUE_MAX_CHARS).then_some(fitted)
|
||||
};
|
||||
if let Some(whole) = fits(usize::MAX) {
|
||||
return whole;
|
||||
}
|
||||
// Searched on the serialized length, which escaping makes longer than the raw prefix, and
|
||||
// which grows with every character kept.
|
||||
let (mut shortest, mut longest) =
|
||||
(0, message.chars().count().min(SLACK_BUTTON_VALUE_MAX_CHARS));
|
||||
while shortest < longest {
|
||||
let mid = (shortest + longest + 1) / 2;
|
||||
if fits(mid).is_some() {
|
||||
shortest = mid;
|
||||
} else {
|
||||
longest = mid - 1;
|
||||
}
|
||||
}
|
||||
fits(shortest).unwrap_or_else(|| truncate_with_ellipsis(message, 0))
|
||||
}
|
||||
|
||||
async fn get_modal_blocks(
|
||||
db: DB,
|
||||
w_id: &str,
|
||||
@@ -1034,7 +1102,7 @@ async fn get_modal_blocks(
|
||||
)
|
||||
.await?;
|
||||
|
||||
let ApprovalFormDetails { message_str, urls, schema } = approval_details;
|
||||
let ApprovalFormDetails { message_str, urls, schema, skin } = approval_details;
|
||||
|
||||
// Get the card content
|
||||
let card_content = transform_schemas(
|
||||
@@ -1063,6 +1131,7 @@ async fn get_modal_blocks(
|
||||
}),
|
||||
default_args_json,
|
||||
dynamic_enums_json,
|
||||
skin,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -1077,6 +1146,7 @@ async fn get_modal_blocks(
|
||||
resume_button_text,
|
||||
cancel_button_text,
|
||||
&private_metadata_signature,
|
||||
skin,
|
||||
)))
|
||||
}
|
||||
|
||||
@@ -1090,27 +1160,32 @@ fn construct_payload(
|
||||
resume_button_text: Option<&str>,
|
||||
cancel_button_text: Option<&str>,
|
||||
signature: &str,
|
||||
skin: ApprovalSkin,
|
||||
) -> serde_json::Value {
|
||||
let (title, resume_label, cancel_label) = match skin {
|
||||
ApprovalSkin::Detailed => ("Workflow Suspended", "Resume Workflow", "Cancel Workflow"),
|
||||
ApprovalSkin::Minimal => ("Approval request", "Approve", "Reject"),
|
||||
};
|
||||
let mut view = serde_json::json!({
|
||||
"type": "modal",
|
||||
"callback_id": "submit_form",
|
||||
"notify_on_close": true,
|
||||
"title": {
|
||||
"type": "plain_text",
|
||||
"text": "Workflow Suspended"
|
||||
"text": title
|
||||
},
|
||||
"blocks": blocks,
|
||||
"submit": {
|
||||
"type": "plain_text",
|
||||
"text": resume_button_text.unwrap_or("Resume Workflow")
|
||||
"text": resume_button_text.unwrap_or(resume_label)
|
||||
},
|
||||
"private_metadata": serde_json::json!({ "resume_url": resume_url, "resource_path": resource_path, "container": container, "hide_cancel": hide_cancel, "signature": signature }).to_string(),
|
||||
"private_metadata": serde_json::json!({ "resume_url": resume_url, "resource_path": resource_path, "container": container, "hide_cancel": hide_cancel, "signature": signature, "skin": skin }).to_string(),
|
||||
});
|
||||
|
||||
if !hide_cancel {
|
||||
view["close"] = serde_json::json!({
|
||||
"type": "plain_text",
|
||||
"text": cancel_button_text.unwrap_or("Cancel Workflow")
|
||||
"text": cancel_button_text.unwrap_or(cancel_label)
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1193,11 +1268,13 @@ async fn update_original_slack_message(
|
||||
action: &str,
|
||||
token: String,
|
||||
container: Container,
|
||||
skin: ApprovalSkin,
|
||||
) -> Result<(), Error> {
|
||||
let message = if action == "resume" {
|
||||
"\n\n*Workflow has been resumed!* :white_check_mark:"
|
||||
} else {
|
||||
"\n\n*Workflow has been canceled!* :x:"
|
||||
let message = match (skin, action == "resume") {
|
||||
(ApprovalSkin::Detailed, true) => "\n\n*Workflow has been resumed!* :white_check_mark:",
|
||||
(ApprovalSkin::Detailed, false) => "\n\n*Workflow has been canceled!* :x:",
|
||||
(ApprovalSkin::Minimal, true) => "*Approved* :white_check_mark:",
|
||||
(ApprovalSkin::Minimal, false) => "*Rejected* :x:",
|
||||
};
|
||||
|
||||
let final_blocks = vec![serde_json::json!({
|
||||
@@ -1242,3 +1319,66 @@ async fn update_original_slack_message(
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn long_message_keeps_the_channel_post_within_slack_limits() {
|
||||
let button_value = serde_json::json!({
|
||||
"w_id": "demo",
|
||||
"job_id": Uuid::nil(),
|
||||
"path": "u/admin/slack",
|
||||
"channel": "C0123456789",
|
||||
"flow_step_id": "a",
|
||||
"signature": "f".repeat(64),
|
||||
});
|
||||
let carried = |skin, message: &str| {
|
||||
let payload = channel_message_payload("C1", skin, Some(message), button_value.clone());
|
||||
let button = payload["blocks"][1]["elements"][0]["value"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.to_string();
|
||||
assert!(button.chars().count() <= SLACK_BUTTON_VALUE_MAX_CHARS);
|
||||
let section = payload["blocks"][0]["text"]["text"].as_str().unwrap();
|
||||
assert!(section.chars().count() <= 3000);
|
||||
serde_json::from_str::<ModalActionValue>(&button)
|
||||
.unwrap()
|
||||
.message
|
||||
.unwrap()
|
||||
};
|
||||
// Quotes and newlines each cost two characters once escaped into the button value.
|
||||
let message = "Expense \"offsite\" line\n".repeat(1000);
|
||||
for skin in [ApprovalSkin::Detailed, ApprovalSkin::Minimal] {
|
||||
let kept = carried(skin, &message);
|
||||
let kept = kept.strip_suffix("...").unwrap();
|
||||
assert!(message.starts_with(kept));
|
||||
assert!(kept.chars().count() > 1_000);
|
||||
assert_eq!(carried(skin, "Short message"), "Short message");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn minimal_skin_survives_the_modal_round_trip() {
|
||||
let container = Container { message_ts: "1".to_string(), channel_id: "C1".to_string() };
|
||||
let payload = construct_payload(
|
||||
serde_json::json!([]),
|
||||
false,
|
||||
"trigger",
|
||||
"https://example.com/resume",
|
||||
"u/admin/slack",
|
||||
container,
|
||||
None,
|
||||
None,
|
||||
"signature",
|
||||
ApprovalSkin::Minimal,
|
||||
);
|
||||
let view = &payload["view"];
|
||||
assert_eq!(view["submit"]["text"], "Approve");
|
||||
assert_eq!(view["close"]["text"], "Reject");
|
||||
let metadata: PrivateMetadata =
|
||||
serde_json::from_str(view["private_metadata"].as_str().unwrap()).unwrap();
|
||||
assert_eq!(metadata.skin, ApprovalSkin::Minimal);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -107,6 +107,14 @@ fn serve_path(path: &str, original_path: &str, query: Option<&str>) -> Response<
|
||||
.header("Cross-Origin-Resource-Policy", "cross-origin");
|
||||
}
|
||||
|
||||
// Login and its siblings carry a different `rd` on every page that links
|
||||
// to them, so a crawler meets thousands of URLs for one form. The app is
|
||||
// client-rendered, so a meta tag only exists after a render pass; the
|
||||
// header is seen on the first fetch.
|
||||
if original_path.starts_with("/user/") {
|
||||
res = res.header("X-Robots-Tag", "noindex, nofollow");
|
||||
}
|
||||
|
||||
// Add Content-Security-Policy header for static assets when policy is set
|
||||
if !CSP_POLICY.is_empty() {
|
||||
if let Ok(header_value) = HeaderValue::try_from(CSP_POLICY.as_str()) {
|
||||
|
||||
@@ -272,7 +272,10 @@ fn format_db_error(message: &str, detail: Option<&str>, hint: Option<&str>) -> S
|
||||
msg
|
||||
}
|
||||
|
||||
fn error_source_chain(e: &dyn std::error::Error) -> String {
|
||||
/// `e` followed by each of its sources, `: `-separated. The result is meant for
|
||||
/// users, and a `reqwest::Error` renders its request URL: never pass one built
|
||||
/// from a URL carrying credentials in its userinfo.
|
||||
pub fn error_source_chain(e: &dyn std::error::Error) -> String {
|
||||
let mut msg = e.to_string();
|
||||
let mut source = e.source();
|
||||
while let Some(cause) = source {
|
||||
|
||||
@@ -125,6 +125,112 @@ pub const GITHUB_ENTERPRISE_APP_SETTING: &str = "github_enterprise_app";
|
||||
/// `base_url` when unset; set it when the browser-facing URL is not reachable
|
||||
/// from GitHub and a separate ingress fronts the API for inbound webhooks.
|
||||
pub const GITHUB_APP_WEBHOOK_BASE_URL_SETTING: &str = "github_app_webhook_base_url";
|
||||
/// Instance-wide announcement rendered above every page of the app (maintenance
|
||||
/// windows, incidents). Readable by any authenticated user, unlike most settings:
|
||||
/// the banner exists to be shown to everyone, so it must never hold anything the
|
||||
/// whole instance may not see.
|
||||
pub const INSTANCE_BANNER_SETTING: &str = "instance_banner";
|
||||
|
||||
/// Ceiling on the banner message. The banner is a one-or-two-line strip above every
|
||||
/// page, so anything longer is a layout accident rather than an announcement.
|
||||
pub const INSTANCE_BANNER_MESSAGE_MAX_LEN: usize = 500;
|
||||
|
||||
/// Ceiling on the banner's link label, which renders as a button inside that same strip.
|
||||
pub const INSTANCE_BANNER_LINK_LABEL_MAX_LEN: usize = 60;
|
||||
|
||||
/// Validate an [`INSTANCE_BANNER_SETTING`] value.
|
||||
///
|
||||
/// The banner is the one setting rendered to every user of the instance, so its
|
||||
/// shape is checked at the boundary rather than trusted from the writer: a value
|
||||
/// that reaches the browser malformed breaks the layout for everyone at once.
|
||||
///
|
||||
/// The link is restricted to http(s) so a stored `javascript:`/`data:` URL can
|
||||
/// never become the href of an anchor every user sees.
|
||||
///
|
||||
/// Only shapes that would *misrender* are rejected. An enabled banner with no message
|
||||
/// is left alone deliberately: it renders as nothing, and every write path here runs
|
||||
/// under the bulk settings save, so rejecting it would fail an admin's whole settings
|
||||
/// edit — retention, SMTP and all — over a half-typed announcement.
|
||||
pub fn validate_instance_banner(value: &serde_json::Value) -> Result<(), String> {
|
||||
let obj = value
|
||||
.as_object()
|
||||
.ok_or_else(|| "must be a JSON object".to_string())?;
|
||||
|
||||
// Field types are checked before their contents. Every read below is an `as_str`/
|
||||
// `as_bool`, which reports a wrong-typed field as absent — so without this a
|
||||
// `"link": 123` would skip the URL checks entirely and be stored, and the settings
|
||||
// form would then throw on it (`link.trim()` on a number) instead of rendering.
|
||||
for (field, expected, ok) in [
|
||||
(
|
||||
"enabled",
|
||||
"a boolean",
|
||||
obj.get("enabled").is_none_or(|v| v.is_boolean()),
|
||||
),
|
||||
(
|
||||
"dismissible",
|
||||
"a boolean",
|
||||
obj.get("dismissible").is_none_or(|v| v.is_boolean()),
|
||||
),
|
||||
(
|
||||
"message",
|
||||
"a string",
|
||||
obj.get("message").is_none_or(|v| v.is_string()),
|
||||
),
|
||||
(
|
||||
"severity",
|
||||
"a string",
|
||||
obj.get("severity").is_none_or(|v| v.is_string()),
|
||||
),
|
||||
(
|
||||
"link",
|
||||
"a string",
|
||||
obj.get("link").is_none_or(|v| v.is_string()),
|
||||
),
|
||||
(
|
||||
"link_label",
|
||||
"a string",
|
||||
obj.get("link_label").is_none_or(|v| v.is_string()),
|
||||
),
|
||||
] {
|
||||
if !ok {
|
||||
return Err(format!("{field} must be {expected}"));
|
||||
}
|
||||
}
|
||||
|
||||
for (field, max) in [
|
||||
("message", INSTANCE_BANNER_MESSAGE_MAX_LEN),
|
||||
("link_label", INSTANCE_BANNER_LINK_LABEL_MAX_LEN),
|
||||
] {
|
||||
let len = obj
|
||||
.get(field)
|
||||
.and_then(|v| v.as_str())
|
||||
.map_or(0, |s| s.chars().count());
|
||||
if len > max {
|
||||
return Err(format!("{field} must be at most {max} characters"));
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(severity) = obj.get("severity").and_then(|v| v.as_str()) {
|
||||
if !matches!(severity, "info" | "warning" | "error") {
|
||||
return Err("severity must be one of info, warning, error".to_string());
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(link) = obj.get("link").and_then(|v| v.as_str()) {
|
||||
if !link.trim().is_empty() {
|
||||
let url = url::Url::parse(link.trim())
|
||||
.map_err(|e| format!("link must be an absolute http(s) URL: {e}"))?;
|
||||
if !matches!(url.scheme(), "http" | "https") {
|
||||
return Err("link must use the http or https scheme".to_string());
|
||||
}
|
||||
if !url.has_host() {
|
||||
return Err("link must include a host".to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate a [`GITHUB_APP_WEBHOOK_BASE_URL_SETTING`] value.
|
||||
///
|
||||
@@ -327,6 +433,30 @@ pub const ENV_SETTINGS: &[&str] = &[
|
||||
"OTEL_METRICS",
|
||||
"OTEL_TRACING",
|
||||
"OTEL_LOGS",
|
||||
// The OTEL_EXPORTER_OTLP_*HEADERS variables are left out: they carry exporter API keys, and
|
||||
// this list is logged at startup and returned to superadmins by `get_local_settings`.
|
||||
"OTEL_METRICS_ENABLED",
|
||||
"OTEL_TRACING_ENABLED",
|
||||
"OTEL_LOGS_ENABLED",
|
||||
"OTEL_EXPORTER_OTLP_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_PROTOCOL",
|
||||
"OTEL_EXPORTER_OTLP_COMPRESSION",
|
||||
"OTEL_EXPORTER_OTLP_TIMEOUT",
|
||||
"OTEL_EXPORTER_OTLP_TRACES_TIMEOUT",
|
||||
"OTEL_EXPORTER_OTLP_METRICS_TIMEOUT",
|
||||
"OTEL_EXPORTER_OTLP_LOGS_TIMEOUT",
|
||||
"OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE",
|
||||
"OTEL_METRIC_EXPORT_INTERVAL",
|
||||
"OTEL_SERVICE_NAME",
|
||||
"OTEL_SERVICE_VERSION",
|
||||
"OTEL_HOST_NAME",
|
||||
"OTEL_ENVIRONMENT",
|
||||
"OTEL_RESOURCE_ATTRIBUTES",
|
||||
"OTEL_JOB_LOGS",
|
||||
"OTEL_TRACES_RETENTION_SECS",
|
||||
"DISABLE_S3_STORE",
|
||||
"PG_SCHEMA",
|
||||
"PG_LISTENER_REFRESH_PERIOD_SECS",
|
||||
@@ -590,6 +720,61 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn instance_banner_rejects_unsafe_and_malformed_values() {
|
||||
// The link becomes the href of an anchor shown to every user of the instance,
|
||||
// so a non-http(s) scheme must not survive a write.
|
||||
for link in [
|
||||
"javascript:alert(1)",
|
||||
"data:text/html,<script>alert(1)</script>",
|
||||
"vbscript:msgbox(1)",
|
||||
"not-a-url",
|
||||
"https://",
|
||||
] {
|
||||
let banner = serde_json::json!({ "enabled": true, "message": "down", "link": link });
|
||||
assert!(
|
||||
validate_instance_banner(&banner).is_err(),
|
||||
"link '{link}' should be rejected"
|
||||
);
|
||||
}
|
||||
// A wrong-typed field reads as absent to every accessor here, so without an
|
||||
// explicit type check it would skip validation and be stored.
|
||||
for bad in [
|
||||
serde_json::json!({ "enabled": true, "message": "down", "link": 123 }),
|
||||
serde_json::json!({ "enabled": true, "message": "down", "link_label": ["a"] }),
|
||||
serde_json::json!({ "enabled": true, "message": { "text": "down" } }),
|
||||
serde_json::json!({ "enabled": true, "message": "down", "severity": 2 }),
|
||||
serde_json::json!({ "enabled": "yes", "message": "down" }),
|
||||
serde_json::json!({ "enabled": true, "message": "down", "dismissible": "no" }),
|
||||
] {
|
||||
assert!(
|
||||
validate_instance_banner(&bad).is_err(),
|
||||
"{bad} should be rejected"
|
||||
);
|
||||
}
|
||||
// The strip is one or two lines tall; both of its texts are bounded.
|
||||
for (field, over) in [
|
||||
("message", INSTANCE_BANNER_MESSAGE_MAX_LEN + 1),
|
||||
("link_label", INSTANCE_BANNER_LINK_LABEL_MAX_LEN + 1),
|
||||
] {
|
||||
let mut banner = serde_json::json!({ "enabled": true, "message": "down" });
|
||||
banner[field] = serde_json::Value::String("x".repeat(over));
|
||||
assert!(
|
||||
validate_instance_banner(&banner).is_err(),
|
||||
"an over-long {field} should be rejected"
|
||||
);
|
||||
}
|
||||
// Enabled with no message renders as nothing and must stay writable: every path
|
||||
// into this validator is a bulk settings save, so rejecting it would fail an
|
||||
// admin's unrelated edits over a half-typed announcement.
|
||||
assert!(validate_instance_banner(&serde_json::json!({ "enabled": true })).is_ok());
|
||||
let ok = serde_json::json!({
|
||||
"enabled": true, "message": "down", "severity": "warning",
|
||||
"link": "https://status.example.com", "dismissible": false
|
||||
});
|
||||
assert!(validate_instance_banner(&ok).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn webhook_base_url_matches_the_ui_validator() {
|
||||
// Kept in lockstep with `isValidWebhookBaseUrl` in
|
||||
|
||||
@@ -579,6 +579,15 @@ pub async fn jwks_key_for(url: &str, token: &str) -> Result<(DecodingKey, Vec<Al
|
||||
/// Verify `token` for `w_id` against whatever key the workspace configured. A PEM key
|
||||
/// ignores `kid`; a JWKS selects by it.
|
||||
pub async fn verify_for_workspace(db: &DB, w_id: &str, token: &str) -> Result<GuestJwtClaims> {
|
||||
// The admit check downstream refuses these anyway; refusing here keeps a deployment
|
||||
// with no guests from parsing attacker-supplied JWTs at all, and names the reason in
|
||||
// the log the caller writes.
|
||||
if !crate::workspaces::instance_supports_guests() {
|
||||
return Err(Error::NotAuthorized(format!(
|
||||
"guest JWT refused: {}",
|
||||
crate::workspaces::GUESTS_UNAVAILABLE_MESSAGE
|
||||
)));
|
||||
}
|
||||
if token.len() > MAX_GUEST_JWT_LEN {
|
||||
return Err(Error::NotAuthorized(format!(
|
||||
"guest JWT refused: token is longer than {MAX_GUEST_JWT_LEN} bytes"
|
||||
|
||||
@@ -1338,6 +1338,16 @@ pub async fn sync_global_settings_declarative(
|
||||
}
|
||||
}
|
||||
|
||||
let banner_key = crate::global_settings::INSTANCE_BANNER_SETTING;
|
||||
match desired.get(banner_key) {
|
||||
None | Some(serde_json::Value::Null) => {}
|
||||
Some(serde_json::Value::String(s)) if s.trim().is_empty() => {}
|
||||
Some(banner) => crate::global_settings::validate_instance_banner(banner)
|
||||
// The validator's messages name the offending field and its expected type,
|
||||
// never the submitted value, so they are safe to surface here.
|
||||
.map_err(|e| anyhow::anyhow!("{banner_key}: {e}"))?,
|
||||
}
|
||||
|
||||
let diff = diff_global_settings(current, desired, ApplyMode::Replace);
|
||||
apply_settings_diff(db, &diff).await?;
|
||||
|
||||
|
||||
@@ -111,6 +111,7 @@ pub use pipeline_advanced_ee as pipeline_advanced;
|
||||
pub use pipeline_advanced_oss as pipeline_advanced;
|
||||
pub mod query_builders;
|
||||
pub mod queue;
|
||||
pub mod queue_metrics;
|
||||
pub mod result_stream;
|
||||
pub mod runnable_settings;
|
||||
pub mod schedule;
|
||||
|
||||
@@ -15,6 +15,57 @@ pub async fn get_queue_counts(db: &Pool<Postgres>) -> HashMap<String, u32> {
|
||||
.unwrap_or_else(|| HashMap::new())
|
||||
}
|
||||
|
||||
/// Backlog of a single tag: jobs waiting more than 3 seconds past their `scheduled_for`.
|
||||
pub struct QueueStat {
|
||||
pub count: u32,
|
||||
/// How long the job that would be picked up next has already been waiting, in seconds.
|
||||
pub delay: f64,
|
||||
/// When that job started waiting (its `scheduled_for`), in epoch seconds.
|
||||
pub head_since: f64,
|
||||
}
|
||||
|
||||
/// Same backlog as [`get_queue_counts`], plus the delay of the job at the head of each
|
||||
/// tag's queue. The head is picked with the same ordering the worker pull uses, so the
|
||||
/// delay reported is the one a worker is about to observe.
|
||||
///
|
||||
/// Reads the queue of every workspace: a caller exposing the result MUST restrict it to
|
||||
/// devops users, as `GET /workers/queue_counts` does. Unlike [`get_queue_counts`], a failed
|
||||
/// read is an error rather than an empty map, which would read as every backlog draining.
|
||||
pub async fn get_queue_stats(
|
||||
db: &Pool<Postgres>,
|
||||
) -> crate::error::Result<HashMap<String, QueueStat>> {
|
||||
// Grouping by (tag, priority) first finds every head in the same single pass as the
|
||||
// count. A per-tag `ORDER BY ... LIMIT 1` walks `queue_sort_v2`, whose `tag` column comes
|
||||
// last, through every other tag's backlog queued ahead of it.
|
||||
let rows = sqlx::query!(
|
||||
"SELECT tag AS \"tag!\", count AS \"count!\",
|
||||
EXTRACT(EPOCH FROM now() - head)::double precision AS \"delay!\",
|
||||
EXTRACT(EPOCH FROM head)::double precision AS \"head_since!\"
|
||||
FROM (
|
||||
SELECT tag, sum(n)::bigint AS count,
|
||||
(array_agg(head ORDER BY priority DESC NULLS LAST))[1] AS head
|
||||
FROM (
|
||||
SELECT tag, priority, count(*) AS n, min(scheduled_for) AS head
|
||||
FROM v2_job_queue WHERE
|
||||
scheduled_for <= now() - ('3 seconds')::interval AND running = false
|
||||
GROUP BY tag, priority
|
||||
) g
|
||||
GROUP BY tag
|
||||
) t",
|
||||
)
|
||||
.fetch_all(db)
|
||||
.await?;
|
||||
Ok(rows
|
||||
.into_iter()
|
||||
.map(|x| {
|
||||
(
|
||||
x.tag,
|
||||
QueueStat { count: x.count as u32, delay: x.delay, head_since: x.head_since },
|
||||
)
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
pub async fn get_queue_running_counts(db: &Pool<Postgres>) -> HashMap<String, u32> {
|
||||
sqlx::query!(
|
||||
"SELECT tag AS \"tag!\", count(*) AS \"count!\" FROM v2_job_queue WHERE
|
||||
|
||||
@@ -0,0 +1,482 @@
|
||||
//! The queue metrics the monitor samples into `metrics` (`queue_count_{tag}` and
|
||||
//! `queue_delay_{tag}`), and how a stored series is drawn back.
|
||||
//!
|
||||
//! A stored value is a number, held until the next sample, or, for a delay, `{"since": <epoch
|
||||
//! seconds>}`: the job at the head of the queue has been waiting since then and was still there
|
||||
//! when sampled, so the delay climbs one second per second until the next sample. Besides
|
||||
//! [`QueueSample`], the SQL in [`read_queue_metrics_series`] and in `GET /workers/queue_metrics`
|
||||
//! decodes both shapes.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use serde::Serialize;
|
||||
use sqlx::{Pool, Postgres};
|
||||
|
||||
pub const QUEUE_COUNT_PREFIX: &str = "queue_count_";
|
||||
pub const QUEUE_DELAY_PREFIX: &str = "queue_delay_";
|
||||
|
||||
/// A backlogged tag whose value has not moved is re-sampled only this often. A longer heartbeat
|
||||
/// writes fewer rows, but keeps a tag whose drain was never recorded (no server was up when it
|
||||
/// drained) drawn as backlogged for longer.
|
||||
pub const QUEUE_METRIC_HEARTBEAT_SECS: f64 = 5.0 * 60.0;
|
||||
|
||||
/// A series silent for longer than this has drained: the sampler stops looking for it, so no
|
||||
/// closing zero will come, and it is drawn as zero from there. Heartbeats land up to a monitor
|
||||
/// tick and a sampling slot late, so this must stay well above their real spacing.
|
||||
pub const QUEUE_METRIC_STALE_SECS: f64 = 3.0 * QUEUE_METRIC_HEARTBEAT_SECS;
|
||||
|
||||
/// Heads that started waiting within this of each other are one wait: jobs queued together
|
||||
/// leave the head one after another without the delay dropping.
|
||||
pub const QUEUE_DELAY_SAME_HEAD_SECS: f64 = 1.0;
|
||||
|
||||
/// Slots a series is split into, whatever the window. A slot draws at most four vertices, and a
|
||||
/// climb one more at each slot boundary it crosses, so a line stays under about 600 points
|
||||
/// however many rows the window holds.
|
||||
const QUEUE_METRICS_SERIES_SLOTS: f64 = 120.0;
|
||||
|
||||
/// A stored sample, as it is drawn from the moment it was written until the next one.
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
pub enum QueueSample {
|
||||
/// A count, or a delay while the head keeps changing, which hovers.
|
||||
Held(f64),
|
||||
/// A delay while the job that started waiting at `since` (epoch seconds) stays at the head.
|
||||
Climbing { since: f64 },
|
||||
}
|
||||
|
||||
impl QueueSample {
|
||||
pub fn parse(value: &serde_json::Value) -> Option<Self> {
|
||||
match value.get("since") {
|
||||
Some(since) => since.as_f64().map(|since| Self::Climbing { since }),
|
||||
None => value.as_f64().map(Self::Held),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn to_json(self) -> serde_json::Value {
|
||||
match self {
|
||||
Self::Held(value) => serde_json::json!(value),
|
||||
Self::Climbing { since } => serde_json::json!({ "since": since }),
|
||||
}
|
||||
}
|
||||
|
||||
/// Its value at `t`, in epoch seconds.
|
||||
pub fn value_at(self, t: f64) -> f64 {
|
||||
match self {
|
||||
Self::Held(value) => value,
|
||||
Self::Climbing { since } => t - since,
|
||||
}
|
||||
}
|
||||
|
||||
/// When the job at the head of a delay sample written at `at` started waiting.
|
||||
pub fn head_since(self, at: f64) -> f64 {
|
||||
match self {
|
||||
Self::Held(delay) => at - delay,
|
||||
Self::Climbing { since } => since,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct QueueMetricsSeries {
|
||||
/// The window drawn, in epoch milliseconds.
|
||||
pub from: i64,
|
||||
pub to: i64,
|
||||
pub tags: Vec<QueueTagSeries>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct QueueTagSeries {
|
||||
pub tag: String,
|
||||
/// Vertices `[epoch ms, value]` of a line joined by straight segments.
|
||||
pub count: Vec<(i64, f64)>,
|
||||
pub delay: Vec<(i64, f64)>,
|
||||
}
|
||||
|
||||
/// The queue metrics of the last `window_secs`, each series aggregated per slot by the database
|
||||
/// and drawn by [`render_series`], so the size is bounded by the number of tags rather than by
|
||||
/// how many rows they wrote.
|
||||
///
|
||||
/// Reads the metrics of every workspace's tags: a caller exposing the result MUST restrict it to
|
||||
/// devops users, as `GET /workers/queue_metrics_series` does.
|
||||
pub async fn read_queue_metrics_series(
|
||||
db: &Pool<Postgres>,
|
||||
window_secs: f64,
|
||||
) -> crate::error::Result<QueueMetricsSeries> {
|
||||
let to = sqlx::query_scalar!("SELECT EXTRACT(EPOCH FROM now())::double precision AS \"now!\"")
|
||||
.fetch_one(db)
|
||||
.await?;
|
||||
let from = to - window_secs;
|
||||
let slot_secs = window_secs / QUEUE_METRICS_SERIES_SLOTS;
|
||||
|
||||
// Slot -1 holds the samples written before the window, of which only the last is used: it
|
||||
// sets the value in force at the left edge. A series silent for longer than the stale window
|
||||
// reads as zero, so nothing older can matter. Arrays compare element by element, so
|
||||
// `max(ARRAY[t, v])` is the slot's latest sample, found without sorting every row. `v` is a
|
||||
// sample's value when it was written: for a climbing delay, how long its head had waited.
|
||||
//
|
||||
// A climb keeps rising until the next sample, so when that sample lands in the same slot
|
||||
// (the tag drained, or its head moved), the climb's top is higher than any `v`. Looking the
|
||||
// next sample up for the slot's last climb, rather than ordering every row, keeps the pass a
|
||||
// plain aggregate; an earlier climb in the same slot still shows up to its last heartbeat.
|
||||
// `t` round-trips through `to_timestamp` to within a microsecond either way, so both bounds
|
||||
// carry a millisecond of slack, far less than two distinct samples of a series are apart:
|
||||
// without it the climbing sample can match itself, or the one at `last` fall outside.
|
||||
let rows = sqlx::query!(
|
||||
"WITH slots AS (
|
||||
SELECT id, slot, min(t) AS first, max(t) AS last, max(v) AS peak,
|
||||
(min(ARRAY[t, v]))[2] AS first_value, (max(ARRAY[t, v]))[2] AS last_value,
|
||||
(max(ARRAY[t, climbing]))[2] = 1 AS last_climbing,
|
||||
COALESCE(bool_and(climbing = 1) AND max(since) - min(since) < $4, false) AS ramp,
|
||||
max(ARRAY[t, since]) FILTER (WHERE climbing = 1) AS last_climb
|
||||
FROM (
|
||||
SELECT id, t,
|
||||
CASE jsonb_typeof(value)
|
||||
WHEN 'number' THEN value::double precision
|
||||
WHEN 'object' THEN t - (value->>'since')::double precision
|
||||
END AS v,
|
||||
(value->>'since')::double precision AS since,
|
||||
(jsonb_typeof(value) = 'object')::int::double precision AS climbing,
|
||||
greatest(floor((t - $1::double precision) / $2::double precision), -1)::int
|
||||
AS slot
|
||||
FROM (
|
||||
SELECT id, value, EXTRACT(EPOCH FROM created_at)::double precision AS t
|
||||
FROM metrics
|
||||
WHERE id LIKE 'queue_%'
|
||||
AND created_at > to_timestamp($1::double precision - $3::double precision)
|
||||
) m
|
||||
) s
|
||||
WHERE v IS NOT NULL
|
||||
GROUP BY id, slot
|
||||
)
|
||||
SELECT id AS \"id!\", slot AS \"slot!\", first AS \"first!\", last AS \"last!\",
|
||||
greatest(peak, CASE WHEN last_climb[1] < last THEN (
|
||||
SELECT EXTRACT(EPOCH FROM min(n.created_at))::double precision
|
||||
FROM metrics n
|
||||
WHERE n.id = slots.id AND n.id LIKE 'queue_%'
|
||||
AND n.created_at > to_timestamp(last_climb[1] + 0.001)
|
||||
AND n.created_at <= to_timestamp(last + 0.001)
|
||||
) - last_climb[2] END) AS \"peak!\",
|
||||
first_value AS \"first_value!\", last_value AS \"last_value!\",
|
||||
last_climbing AS \"last_climbing!\", ramp AS \"ramp!\"
|
||||
FROM slots
|
||||
ORDER BY id, slot",
|
||||
from,
|
||||
slot_secs,
|
||||
QUEUE_METRIC_STALE_SECS,
|
||||
QUEUE_DELAY_SAME_HEAD_SECS,
|
||||
)
|
||||
.fetch_all(db)
|
||||
.await?;
|
||||
|
||||
#[derive(Default)]
|
||||
struct Stored {
|
||||
carried: Option<MetricSlot>,
|
||||
slots: Vec<MetricSlot>,
|
||||
}
|
||||
// [count, delay] per tag.
|
||||
let mut stored: BTreeMap<String, [Stored; 2]> = BTreeMap::new();
|
||||
for row in rows {
|
||||
let (series, tag) = if let Some(tag) = row.id.strip_prefix(QUEUE_COUNT_PREFIX) {
|
||||
(0, tag)
|
||||
} else if let Some(tag) = row.id.strip_prefix(QUEUE_DELAY_PREFIX) {
|
||||
(1, tag)
|
||||
} else {
|
||||
continue;
|
||||
};
|
||||
let series = &mut stored.entry(tag.to_string()).or_default()[series];
|
||||
let slot = MetricSlot {
|
||||
first: row.first,
|
||||
last: row.last,
|
||||
peak: row.peak,
|
||||
first_value: row.first_value,
|
||||
last_value: row.last_value,
|
||||
last_climbing: row.last_climbing,
|
||||
ramp: row.ramp,
|
||||
};
|
||||
if row.slot < 0 {
|
||||
series.carried = Some(slot);
|
||||
} else {
|
||||
series.slots.push(slot);
|
||||
}
|
||||
}
|
||||
|
||||
let tags = stored
|
||||
.into_iter()
|
||||
.map(|(tag, [count, delay])| {
|
||||
let draw =
|
||||
|s: &Stored| render_series(s.carried.as_ref(), &s.slots, from, to, slot_secs);
|
||||
QueueTagSeries { count: draw(&count), delay: draw(&delay), tag }
|
||||
})
|
||||
// A tag that drained before the window has nothing to draw in it.
|
||||
.filter(|s| s.count.iter().chain(&s.delay).any(|(_, v)| *v != 0.0))
|
||||
.collect();
|
||||
|
||||
Ok(QueueMetricsSeries {
|
||||
from: (from * 1000.0).round() as i64,
|
||||
to: (to * 1000.0).round() as i64,
|
||||
tags,
|
||||
})
|
||||
}
|
||||
|
||||
/// The stored samples of one series that fall in one time slot.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct MetricSlot {
|
||||
/// When the first and the last sample of the slot were written, in epoch seconds.
|
||||
pub first: f64,
|
||||
pub last: f64,
|
||||
/// The highest value the series drew over the slot, a climb that ends inside it included.
|
||||
pub peak: f64,
|
||||
pub first_value: f64,
|
||||
/// The value of the last sample, which holds (or climbs, for a climbing delay) until the next.
|
||||
pub last_value: f64,
|
||||
pub last_climbing: bool,
|
||||
/// Every sample of the slot climbs from the same head, so the slot is one exact ramp.
|
||||
pub ramp: bool,
|
||||
}
|
||||
|
||||
/// Draw a stored series over `[from, to]` (epoch seconds), split into slots of `slot_secs`, as
|
||||
/// the vertices of a line joined by straight segments, each `(epoch ms, value)`.
|
||||
///
|
||||
/// A sample holds its value, or a climbing delay keeps climbing, until the next sample or until
|
||||
/// the series has been silent for [`QUEUE_METRIC_STALE_SECS`]. `carried` is the slot before
|
||||
/// `from`, whose last sample sets the left edge. A slot draws its peak across the span of its
|
||||
/// samples, so a spike shorter than a slot still shows at full height, unless it is a single
|
||||
/// climb, drawn exactly. A climb gets a vertex at every slot boundary it crosses: the delay axis
|
||||
/// is logarithmic, so one straight segment across many slots would misplace it.
|
||||
pub fn render_series(
|
||||
carried: Option<&MetricSlot>,
|
||||
slots: &[MetricSlot],
|
||||
from: f64,
|
||||
to: f64,
|
||||
slot_secs: f64,
|
||||
) -> Vec<(i64, f64)> {
|
||||
let mut line = Line { points: vec![], from, slot_secs };
|
||||
let mut held = carried
|
||||
.map(Held::after)
|
||||
.filter(|h| from - h.at <= QUEUE_METRIC_STALE_SECS);
|
||||
if let Some(h) = held {
|
||||
line.push(from, h.value_at(from));
|
||||
}
|
||||
for slot in slots {
|
||||
let entering = line.advance(&mut held, slot.first);
|
||||
line.push(slot.first, entering);
|
||||
if slot.ramp {
|
||||
line.push(slot.first, slot.first_value);
|
||||
} else {
|
||||
line.push(slot.first, slot.peak);
|
||||
line.push(slot.last, slot.peak);
|
||||
}
|
||||
line.push(slot.last, slot.last_value);
|
||||
held = Some(Held::after(slot));
|
||||
}
|
||||
if !line.points.is_empty() {
|
||||
let value = line.advance(&mut held, to);
|
||||
line.push(to, value);
|
||||
}
|
||||
line.points
|
||||
}
|
||||
|
||||
/// The last sample drawn: when it was written, its value then, and whether it climbs from there.
|
||||
#[derive(Clone, Copy)]
|
||||
struct Held {
|
||||
at: f64,
|
||||
value: f64,
|
||||
climbing: bool,
|
||||
}
|
||||
|
||||
impl Held {
|
||||
fn after(slot: &MetricSlot) -> Self {
|
||||
Self { at: slot.last, value: slot.last_value, climbing: slot.last_climbing }
|
||||
}
|
||||
|
||||
fn value_at(self, t: f64) -> f64 {
|
||||
if self.climbing {
|
||||
self.value + (t - self.at)
|
||||
} else {
|
||||
self.value
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct Line {
|
||||
points: Vec<(i64, f64)>,
|
||||
from: f64,
|
||||
slot_secs: f64,
|
||||
}
|
||||
|
||||
impl Line {
|
||||
/// The value `held` has at `t`, drawing the climb that leads there and, when the series went
|
||||
/// silent for too long first, its drop to zero, after which it is forgotten.
|
||||
fn advance(&mut self, held: &mut Option<Held>, t: f64) -> f64 {
|
||||
let Some(h) = *held else {
|
||||
return 0.0;
|
||||
};
|
||||
let stale_at = h.at + QUEUE_METRIC_STALE_SECS;
|
||||
if h.climbing {
|
||||
let end = t.min(stale_at);
|
||||
let start = h.at.max(self.from);
|
||||
let mut boundary = self.from
|
||||
+ ((start - self.from) / self.slot_secs).floor() * self.slot_secs
|
||||
+ self.slot_secs;
|
||||
while boundary < end {
|
||||
self.push(boundary, h.value_at(boundary));
|
||||
boundary += self.slot_secs;
|
||||
}
|
||||
}
|
||||
if t <= stale_at {
|
||||
return h.value_at(t);
|
||||
}
|
||||
self.push(stale_at, h.value_at(stale_at));
|
||||
self.push(stale_at, 0.0);
|
||||
*held = None;
|
||||
0.0
|
||||
}
|
||||
|
||||
fn push(&mut self, t: f64, value: f64) {
|
||||
let point = ((t * 1000.0).round() as i64, value);
|
||||
match self.points.as_mut_slice() {
|
||||
[.., last] if *last == point => {}
|
||||
// A horizontal run only needs its two ends.
|
||||
[.., a, b] if a.1 == value && b.1 == value => b.0 = point.0,
|
||||
_ => self.points.push(point),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const FROM: f64 = 1_000_000.0;
|
||||
const TO: f64 = FROM + 3600.0;
|
||||
const SLOT: f64 = 30.0;
|
||||
|
||||
fn held(first: f64, last: f64, peak: f64, last_value: f64) -> MetricSlot {
|
||||
MetricSlot {
|
||||
first: FROM + first,
|
||||
last: FROM + last,
|
||||
peak,
|
||||
first_value: peak,
|
||||
last_value,
|
||||
last_climbing: false,
|
||||
ramp: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// A slot whose samples all climb from a head that started waiting 30s before `FROM`.
|
||||
fn climbing(first: f64, last: f64) -> MetricSlot {
|
||||
MetricSlot {
|
||||
first: FROM + first,
|
||||
last: FROM + last,
|
||||
peak: last + 30.0,
|
||||
first_value: first + 30.0,
|
||||
last_value: last + 30.0,
|
||||
last_climbing: true,
|
||||
ramp: true,
|
||||
}
|
||||
}
|
||||
|
||||
fn at(secs: f64, value: f64) -> (i64, f64) {
|
||||
(((FROM + secs) * 1000.0) as i64, value)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_value_holds_until_the_next_sample_and_a_drain_drops_where_it_was_written() {
|
||||
let line = render_series(
|
||||
None,
|
||||
&[
|
||||
held(60.0, 60.0, 3.0, 3.0),
|
||||
held(600.0, 600.0, 2.0, 2.0),
|
||||
held(900.0, 900.0, 0.0, 0.0),
|
||||
],
|
||||
FROM,
|
||||
TO,
|
||||
SLOT,
|
||||
);
|
||||
assert_eq!(
|
||||
line,
|
||||
vec![
|
||||
at(60.0, 0.0),
|
||||
at(60.0, 3.0),
|
||||
at(600.0, 3.0),
|
||||
at(600.0, 2.0),
|
||||
at(900.0, 2.0),
|
||||
at(900.0, 0.0),
|
||||
at(3600.0, 0.0),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_series_silent_past_the_stale_window_drops_to_zero() {
|
||||
let line = render_series(None, &[held(60.0, 60.0, 3.0, 3.0)], FROM, TO, SLOT);
|
||||
let dropped = 60.0 + QUEUE_METRIC_STALE_SECS;
|
||||
assert_eq!(
|
||||
line,
|
||||
vec![
|
||||
at(60.0, 0.0),
|
||||
at(60.0, 3.0),
|
||||
at(dropped, 3.0),
|
||||
at(dropped, 0.0),
|
||||
at(3600.0, 0.0)
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_slot_draws_its_peak_then_continues_from_its_last_sample() {
|
||||
// Samples at 60 (5), 70 (9), 80 (4) collapsed into one slot.
|
||||
let line = render_series(
|
||||
Some(&held(-30.0, -30.0, 2.0, 2.0)),
|
||||
&[held(60.0, 80.0, 9.0, 4.0)],
|
||||
FROM,
|
||||
FROM + 120.0,
|
||||
SLOT,
|
||||
);
|
||||
assert_eq!(
|
||||
line,
|
||||
vec![
|
||||
at(0.0, 2.0),
|
||||
at(60.0, 2.0),
|
||||
at(60.0, 9.0),
|
||||
at(80.0, 9.0),
|
||||
at(80.0, 4.0),
|
||||
at(120.0, 4.0)
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_climbing_delay_is_drawn_exactly_up_to_its_drain() {
|
||||
// 300s slots: one holds two climbing samples, and heartbeats follow until the drain.
|
||||
let line = render_series(
|
||||
None,
|
||||
&[
|
||||
climbing(60.0, 120.0),
|
||||
climbing(360.0, 360.0),
|
||||
climbing(660.0, 660.0),
|
||||
held(900.0, 900.0, 0.0, 0.0),
|
||||
],
|
||||
FROM,
|
||||
TO,
|
||||
300.0,
|
||||
);
|
||||
assert_eq!(
|
||||
line,
|
||||
vec![
|
||||
at(60.0, 0.0),
|
||||
// The slot is one climb, not its peak held across it.
|
||||
at(60.0, 90.0),
|
||||
at(120.0, 150.0),
|
||||
// A vertex at each slot boundary the climb crosses.
|
||||
at(300.0, 330.0),
|
||||
at(360.0, 390.0),
|
||||
at(600.0, 630.0),
|
||||
at(660.0, 690.0),
|
||||
// Still climbing right up to the closing zero.
|
||||
at(900.0, 930.0),
|
||||
at(900.0, 0.0),
|
||||
at(3600.0, 0.0),
|
||||
]
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -250,23 +250,37 @@ pub async fn get_full_hub_script_by_path(
|
||||
let version = path_iterator
|
||||
.next()
|
||||
.ok_or_else(|| Error::internal_err(format!("expected hub path to have version number")))?;
|
||||
// A cache entry that cannot be read or parsed counts as a miss rather than an error:
|
||||
// a truncated write leaves a file that exists but deserializes to nothing, and refetching
|
||||
// it is always preferable to failing the job push it was read for.
|
||||
let cache_path = format!("{}/{version}", *HUB_CACHE_DIR);
|
||||
let script;
|
||||
if tokio::fs::metadata(&cache_path).await.is_err() {
|
||||
script = get_full_hub_script_by_path_inner(path, http_client, db).await?;
|
||||
if let Err(e) = crate::worker::write_file(
|
||||
&HUB_CACHE_DIR,
|
||||
&version,
|
||||
&serde_json::to_string(&script).map_err(to_anyhow)?,
|
||||
) {
|
||||
tracing::error!("failed to write hub script {path} to cache: {e}");
|
||||
} else {
|
||||
tracing::info!("wrote hub script {path} to cache");
|
||||
let cached = match tokio::fs::read_to_string(&cache_path).await {
|
||||
Ok(content) => serde_json::from_str::<HubScript>(&content)
|
||||
.inspect_err(|e| {
|
||||
tracing::error!("hub script cache at {cache_path} is unparseable, refetching: {e}")
|
||||
})
|
||||
.ok(),
|
||||
Err(e) => {
|
||||
if e.kind() != std::io::ErrorKind::NotFound {
|
||||
tracing::error!("hub script cache at {cache_path} is unreadable, refetching: {e}");
|
||||
}
|
||||
None
|
||||
}
|
||||
} else {
|
||||
let cache_content = tokio::fs::read_to_string(cache_path).await?;
|
||||
script = serde_json::from_str(&cache_content).unwrap();
|
||||
};
|
||||
if let Some(script) = cached {
|
||||
tracing::info!("read hub script {path} from cache");
|
||||
return Ok(script);
|
||||
}
|
||||
|
||||
let script = get_full_hub_script_by_path_inner(path, http_client, db).await?;
|
||||
if let Err(e) = crate::worker::write_file(
|
||||
&HUB_CACHE_DIR,
|
||||
&version,
|
||||
&serde_json::to_string(&script).map_err(to_anyhow)?,
|
||||
) {
|
||||
tracing::error!("failed to write hub script {path} to cache: {e}");
|
||||
} else {
|
||||
tracing::info!("wrote hub script {path} to cache");
|
||||
}
|
||||
Ok(script)
|
||||
}
|
||||
|
||||
@@ -8,6 +8,10 @@ pub const ALLOW_PRIVATE_SAML_METADATA_URLS_ENV: &str = "ALLOW_PRIVATE_SAML_METAD
|
||||
|
||||
pub const ALLOW_PRIVATE_GUEST_JWKS_URLS_ENV: &str = "ALLOW_PRIVATE_GUEST_JWKS_URLS";
|
||||
|
||||
/// Lets every git call reach hosts on a private network, whoever it is made for.
|
||||
/// Without it, [`private_git_host_allowed`] decides.
|
||||
pub const ALLOW_LOCAL_GIT_REMOTES_ENV: &str = "ALLOW_LOCAL_GIT_REMOTES";
|
||||
|
||||
/// Why a URL failed SSRF validation.
|
||||
///
|
||||
/// The distinction matters for callers that gate private endpoints behind a
|
||||
@@ -203,6 +207,54 @@ pub fn allow_private_saml_metadata_urls() -> bool {
|
||||
.is_some_and(|v| v == "true" || v == "1")
|
||||
}
|
||||
|
||||
fn allow_local_git_remotes() -> bool {
|
||||
std::env::var(ALLOW_LOCAL_GIT_REMOTES_ENV)
|
||||
.ok()
|
||||
.is_some_and(|v| v == "true" || v == "1")
|
||||
}
|
||||
|
||||
/// Who a git call is made for, which decides whether it may reach a host on a
|
||||
/// private network.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum GitRemoteCaller {
|
||||
/// A workspace admin's request, or Windmill's own work (polling, webhook and
|
||||
/// token upkeep, the merge request after a deploy), whose errors only admins read.
|
||||
AdminOrSystem,
|
||||
/// A request from anyone who is not a workspace admin.
|
||||
NonAdmin,
|
||||
}
|
||||
|
||||
/// Whether a git call made for `caller` may reach a host on a private network.
|
||||
///
|
||||
/// The refusal is for non-admins, who may not be able to run code (operators)
|
||||
/// and would read git's error output back as a probe of the server's network.
|
||||
/// An admin can run code, which reaches those hosts from a worker already. On a
|
||||
/// cloud instance, where a workspace admin is anyone who signed up, every caller
|
||||
/// is refused.
|
||||
pub fn private_git_host_allowed(caller: GitRemoteCaller) -> bool {
|
||||
git_host_policy_allows(
|
||||
caller,
|
||||
allow_local_git_remotes(),
|
||||
*crate::worker::CLOUD_HOSTED,
|
||||
)
|
||||
}
|
||||
|
||||
fn git_host_policy_allows(caller: GitRemoteCaller, opted_in: bool, cloud_hosted: bool) -> bool {
|
||||
opted_in || (caller == GitRemoteCaller::AdminOrSystem && !cloud_hosted)
|
||||
}
|
||||
|
||||
/// Appended to a refusal of a private git host, naming what would let `caller`
|
||||
/// through. `None` where nothing an instance administrator sets would help.
|
||||
pub fn private_git_host_hint(caller: GitRemoteCaller) -> Option<String> {
|
||||
(caller == GitRemoteCaller::NonAdmin && !*crate::worker::CLOUD_HOSTED).then(|| {
|
||||
format!(
|
||||
"Only workspace admins can reach a git server on a private network. To allow \
|
||||
every user, set the {ALLOW_LOCAL_GIT_REMOTES_ENV}=true environment variable on \
|
||||
the Windmill servers"
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn validate_saml_metadata_url(url: &str) -> Result<ValidatedTarget, SsrfValidationError> {
|
||||
let parsed =
|
||||
url::Url::parse(url).map_err(|e| SsrfValidationError::InvalidUrl(e.to_string()))?;
|
||||
@@ -632,6 +684,15 @@ mod tests {
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn private_git_hosts_are_refused_to_non_admins_and_on_cloud() {
|
||||
use GitRemoteCaller::{AdminOrSystem, NonAdmin};
|
||||
assert!(git_host_policy_allows(AdminOrSystem, false, false));
|
||||
assert!(!git_host_policy_allows(NonAdmin, false, false));
|
||||
assert!(!git_host_policy_allows(AdminOrSystem, false, true));
|
||||
assert!(git_host_policy_allows(NonAdmin, true, true));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn saml_ssrf_error_message_includes_env_hint_only_for_private_urls() {
|
||||
let private_error = validate_url_for_ssrf("http://127.0.0.1/metadata")
|
||||
|
||||
@@ -2512,6 +2512,42 @@ pub fn split_python_requirements<T: AsRef<str>>(requirements: T) -> Vec<String>
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Byte offset of the comment marker, per pip's rule: a `#` at line start or preceded by
|
||||
/// whitespace. A `#` elsewhere belongs to the requirement (`pkg @ https://h/p.whl#sha256=…`).
|
||||
fn requirement_comment_start(line: &str) -> Option<usize> {
|
||||
line.char_indices()
|
||||
.find(|(i, c)| *c == '#' && (*i == 0 || line[..*i].ends_with(char::is_whitespace)))
|
||||
.map(|(i, _)| i)
|
||||
}
|
||||
|
||||
/// The installable requirement carried by one lockfile line, or `None` for a comment, a
|
||||
/// `-r`/`-e`/`--flag` directive, or a blank.
|
||||
///
|
||||
/// Windmill installs a lockfile one entry at a time as a `uv pip install` argument, so
|
||||
/// requirements-file syntax a file-level parser would absorb is an unparseable package name
|
||||
/// here and has to be stripped first.
|
||||
pub fn requirement_from_lockfile_line(line: &str) -> Option<&str> {
|
||||
let requirement = match requirement_comment_start(line) {
|
||||
Some(i) => &line[..i],
|
||||
None => line,
|
||||
}
|
||||
.trim()
|
||||
// Continuations are stripped, not joined: right for `--generate-hashes` locks, whose
|
||||
// continued lines are `--hash=` flags this function drops, but a lock continuing onto a
|
||||
// marker or extra would lose it.
|
||||
.trim_end_matches('\\')
|
||||
.trim_end();
|
||||
|
||||
(!requirement.is_empty() && !requirement.starts_with('-')).then_some(requirement)
|
||||
}
|
||||
|
||||
/// Whether a lockfile line continues onto the next one. The continued lines reach the
|
||||
/// installer as entries of their own rather than being joined, so a caller that cares what
|
||||
/// they carried — `--hash=` pins, for a `--generate-hashes` lock — has to say so itself.
|
||||
pub fn lockfile_line_has_continuation(line: &str) -> bool {
|
||||
line.trim_end().ends_with('\\')
|
||||
}
|
||||
|
||||
#[derive(Eq, PartialEq, Clone, Copy, Default, Debug)]
|
||||
#[repr(u32)]
|
||||
pub enum PyVAlias {
|
||||
@@ -2630,6 +2666,52 @@ mod tests {
|
||||
ids.iter().map(|s| s.to_string()).collect()
|
||||
}
|
||||
|
||||
/// Fixtures are verbatim `uv pip compile` output (uv 0.11.28): split and inline
|
||||
/// annotation styles, and `--generate-hashes`.
|
||||
#[test]
|
||||
fn test_requirement_from_lockfile_line() {
|
||||
assert_eq!(requirement_from_lockfile_line(" # via httpx"), None);
|
||||
assert_eq!(requirement_from_lockfile_line(" # via"), None);
|
||||
assert_eq!(requirement_from_lockfile_line(" # anyio"), None);
|
||||
assert_eq!(
|
||||
requirement_from_lockfile_line(" # via -r .tmp/requirements.in"),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
requirement_from_lockfile_line("anyio==4.15.1 \\"),
|
||||
Some("anyio==4.15.1")
|
||||
);
|
||||
assert_eq!(
|
||||
requirement_from_lockfile_line(
|
||||
" --hash=sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7 \\"
|
||||
),
|
||||
None
|
||||
);
|
||||
assert_eq!(requirement_from_lockfile_line("# py: 3.11"), None);
|
||||
assert_eq!(requirement_from_lockfile_line("-r other.txt"), None);
|
||||
assert_eq!(
|
||||
requirement_from_lockfile_line("--index-url https://x"),
|
||||
None
|
||||
);
|
||||
assert_eq!(requirement_from_lockfile_line(" "), None);
|
||||
assert_eq!(
|
||||
requirement_from_lockfile_line("httpx==0.27.0"),
|
||||
Some("httpx==0.27.0")
|
||||
);
|
||||
assert_eq!(
|
||||
requirement_from_lockfile_line("httpx==0.27.0 # via -r requirements.in"),
|
||||
Some("httpx==0.27.0")
|
||||
);
|
||||
// A `#` not preceded by whitespace is part of the requirement, not a comment.
|
||||
assert_eq!(
|
||||
requirement_from_lockfile_line("wmill @ https://h/wmill.whl#sha256=abc"),
|
||||
Some("wmill @ https://h/wmill.whl#sha256=abc")
|
||||
);
|
||||
|
||||
assert!(lockfile_line_has_continuation("anyio==4.15.1 \\"));
|
||||
assert!(!lockfile_line_has_continuation("anyio==4.15.1"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_job_oom_score_adj() {
|
||||
assert_eq!(parse_job_oom_score_adj(Some("300")), 300);
|
||||
|
||||
@@ -183,7 +183,7 @@ pub enum ObjectType {
|
||||
DatatableMigration,
|
||||
}
|
||||
|
||||
pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28949/sync-script-to-git-repo-windmill";
|
||||
pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28958/sync-script-to-git-repo-windmill";
|
||||
|
||||
/// Hub script that applies a repository's state back into a workspace
|
||||
/// (the repo → Windmill / "pull" direction). Same script the UI runs from
|
||||
@@ -863,8 +863,12 @@ pub async fn guest_count_in_window<'c, E: sqlx::Executor<'c, Database = sqlx::Po
|
||||
/// The instance's standing against the guest allowance, as every surface reports it.
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct GuestUsage {
|
||||
/// Whether this deployment can admit guests at all ([`instance_supports_guests`]).
|
||||
/// Off, every other field is moot and no switch below can turn guests on.
|
||||
pub available: bool,
|
||||
/// The superadmin switch (`GUEST_ACCESS_DISABLED_SETTING`), which every workspace
|
||||
/// switch sits under.
|
||||
/// switch sits under. Reported as stored, so a superadmin sees what they set even
|
||||
/// where `available` overrules it.
|
||||
pub instance_enabled: bool,
|
||||
/// Distinct guest emails over the trailing `window_days`.
|
||||
pub guest_count: i64,
|
||||
@@ -877,9 +881,33 @@ pub struct GuestUsage {
|
||||
pub guest_seats: i64,
|
||||
}
|
||||
|
||||
/// SQL for "the instance admits guests": the superadmin switch, absent meaning on. The
|
||||
/// setting is read as text before the cast so `true` and `"true"` both count.
|
||||
fn instance_admits_guests_sql() -> String {
|
||||
/// What a caller is told when it asks for guests on a deployment that cannot have them.
|
||||
pub const GUESTS_UNAVAILABLE_MESSAGE: &str =
|
||||
"Guest access is not available on Windmill Cloud. It requires a self-hosted instance \
|
||||
or a dedicated Windmill Cloud deployment.";
|
||||
|
||||
/// Whether guests can exist on this deployment at all. They cannot on the shared cloud:
|
||||
/// a guest is an identity Windmill itself never vouched for, admitted on the say-so of
|
||||
/// whoever runs the instance, which is not a call a multi-tenant deployment can make for
|
||||
/// its tenants. Folded into every guest gate below, so a workspace switch or an app
|
||||
/// policy left saying `guest` is inert rather than honored.
|
||||
pub fn instance_supports_guests() -> bool {
|
||||
!*crate::worker::CLOUD_HOSTED
|
||||
}
|
||||
|
||||
/// [`instance_supports_guests`] as an error, for the writes that would otherwise store a
|
||||
/// setting that can never take effect.
|
||||
pub fn require_guest_support() -> Result<()> {
|
||||
if instance_supports_guests() {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(Error::BadRequest(GUESTS_UNAVAILABLE_MESSAGE.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
/// SQL for the superadmin switch alone, absent meaning on. The setting is read as text
|
||||
/// before the cast so `true` and `"true"` both count.
|
||||
fn instance_switch_sql() -> String {
|
||||
format!(
|
||||
"NOT COALESCE((SELECT (value #>> '{{}}')::boolean FROM global_settings \
|
||||
WHERE name = '{}'), false)",
|
||||
@@ -887,9 +915,18 @@ fn instance_admits_guests_sql() -> String {
|
||||
)
|
||||
}
|
||||
|
||||
/// SQL for "the instance admits guests": the superadmin switch, under
|
||||
/// [`instance_supports_guests`].
|
||||
fn instance_admits_guests_sql() -> String {
|
||||
if !instance_supports_guests() {
|
||||
return "false".to_string();
|
||||
}
|
||||
instance_switch_sql()
|
||||
}
|
||||
|
||||
pub async fn guest_usage(db: &crate::DB) -> Result<GuestUsage> {
|
||||
let instance_admits = instance_admits_guests_sql();
|
||||
let instance_enabled: bool = sqlx::query_scalar(&format!("SELECT {instance_admits}"))
|
||||
let instance_switch = instance_switch_sql();
|
||||
let instance_enabled: bool = sqlx::query_scalar(&format!("SELECT {instance_switch}"))
|
||||
.fetch_one(db)
|
||||
.await
|
||||
.map_err(|e| Error::internal_err(format!("reading the instance guest switch: {e:#}")))?;
|
||||
@@ -901,6 +938,7 @@ pub async fn guest_usage(db: &crate::DB) -> Result<GuestUsage> {
|
||||
0
|
||||
};
|
||||
Ok(GuestUsage {
|
||||
available: instance_supports_guests(),
|
||||
instance_enabled,
|
||||
guest_count,
|
||||
window_days: GUEST_WINDOW_DAYS,
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
use serde_json::json;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_common::queue_metrics::{read_queue_metrics_series, QUEUE_METRIC_STALE_SECS};
|
||||
|
||||
const WINDOW: f64 = 3600.0;
|
||||
|
||||
/// Store a sample written `at` seconds after the start of a `WINDOW` ending now.
|
||||
async fn sample(db: &Pool<Postgres>, id: &str, value: serde_json::Value, at: f64) {
|
||||
sqlx::query(
|
||||
"INSERT INTO metrics (id, value, created_at) VALUES ($1, $2, now() - make_interval(secs => $3))",
|
||||
)
|
||||
.bind(id)
|
||||
.bind(value)
|
||||
.bind(WINDOW - at)
|
||||
.execute(db)
|
||||
.await
|
||||
.expect("failed to store a metric sample");
|
||||
}
|
||||
|
||||
/// The database hands the renderer the last sample before the window, which sets the left edge,
|
||||
/// and for each slot its peak and its latest value, which the line continues from.
|
||||
#[sqlx::test(migrations = "../migrations")]
|
||||
async fn a_series_starts_from_the_sample_before_the_window_and_keeps_each_slot_peak(
|
||||
db: Pool<Postgres>,
|
||||
) {
|
||||
// Before the window: 1, then 2, which is what is in force at the left edge.
|
||||
sample(&db, "queue_count_t", json!(1), -120.0).await;
|
||||
sample(&db, "queue_count_t", json!(2), -60.0).await;
|
||||
// Three samples inside one 30s slot: the line rises to their peak, then drops to the last.
|
||||
sample(&db, "queue_count_t", json!(5), 605.0).await;
|
||||
sample(&db, "queue_count_t", json!(9), 612.0).await;
|
||||
sample(&db, "queue_count_t", json!(4), 620.0).await;
|
||||
// Drained before the window: nothing left to draw.
|
||||
sample(&db, "queue_count_gone", json!(3), -300.0).await;
|
||||
sample(&db, "queue_count_gone", json!(0), -200.0).await;
|
||||
|
||||
let series = read_queue_metrics_series(&db, WINDOW).await.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
series.tags.len(),
|
||||
1,
|
||||
"a tag drained before the window is left out"
|
||||
);
|
||||
let tag = &series.tags[0];
|
||||
assert_eq!(tag.tag, "t");
|
||||
assert!(tag.delay.is_empty());
|
||||
|
||||
let stale = 620.0 + QUEUE_METRIC_STALE_SECS;
|
||||
let expected = [
|
||||
(0.0, 2.0),
|
||||
(605.0, 2.0),
|
||||
(605.0, 9.0),
|
||||
(620.0, 9.0),
|
||||
(620.0, 4.0),
|
||||
(stale, 4.0),
|
||||
(stale, 0.0),
|
||||
(WINDOW, 0.0),
|
||||
];
|
||||
assert_eq!(tag.count.len(), expected.len(), "vertices: {:?}", tag.count);
|
||||
for ((ms, value), (at, expected_value)) in tag.count.iter().zip(expected) {
|
||||
let secs = (*ms - series.from) as f64 / 1000.0;
|
||||
// Samples are stored a few milliseconds before the window is read.
|
||||
assert!(
|
||||
(secs - at).abs() < 2.0 && *value == expected_value,
|
||||
"expected ({at}, {expected_value}), got ({secs}, {value}) in {:?}",
|
||||
tag.count
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A delay stored as its head's wait start is drawn as that wait, growing a second per second,
|
||||
/// right up to the zero that closes it.
|
||||
#[sqlx::test(migrations = "../migrations")]
|
||||
async fn a_climbing_delay_is_drawn_as_the_wait_of_its_head(db: Pool<Postgres>) {
|
||||
let now: f64 = sqlx::query_scalar("SELECT EXTRACT(EPOCH FROM now())::double precision")
|
||||
.fetch_one(&db)
|
||||
.await
|
||||
.unwrap();
|
||||
// The head started waiting 30s before the window; heartbeats restate it until the drain.
|
||||
let head = json!({ "since": now - WINDOW - 30.0 });
|
||||
for at in [60.0, 360.0, 660.0] {
|
||||
sample(&db, "queue_delay_t", head.clone(), at).await;
|
||||
}
|
||||
sample(&db, "queue_delay_t", json!(0), 900.0).await;
|
||||
|
||||
let series = read_queue_metrics_series(&db, WINDOW).await.unwrap();
|
||||
let points = series.tags[0]
|
||||
.delay
|
||||
.iter()
|
||||
.map(|(ms, value)| ((*ms - series.from) as f64 / 1000.0, *value))
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let climb = points
|
||||
.iter()
|
||||
.filter(|(_, value)| *value > 0.0)
|
||||
.collect::<Vec<_>>();
|
||||
assert!(
|
||||
climb.len() > 4,
|
||||
"the climb has vertices along the way: {points:?}"
|
||||
);
|
||||
for (at, value) in &climb {
|
||||
assert!(
|
||||
(value - (at + 30.0)).abs() < 2.0,
|
||||
"off the climb at {at}: {points:?}"
|
||||
);
|
||||
}
|
||||
let (first, _) = climb[0];
|
||||
let (top, _) = climb[climb.len() - 1];
|
||||
assert!(
|
||||
(first - 60.0).abs() < 2.0 && (top - 900.0).abs() < 2.0,
|
||||
"{points:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// A climb that drains inside its slot keeps its top, which no stored value holds: it is reached
|
||||
/// at the next sample.
|
||||
#[sqlx::test(migrations = "../migrations")]
|
||||
async fn a_climb_that_drains_inside_its_slot_keeps_its_top(db: Pool<Postgres>) {
|
||||
let now: f64 = sqlx::query_scalar("SELECT EXTRACT(EPOCH FROM now())::double precision")
|
||||
.fetch_one(&db)
|
||||
.await
|
||||
.unwrap();
|
||||
// All in the 30s slot starting at 600: held at 5s, then climbing from a head queued at 597,
|
||||
// which is still there when the tag drains at 627, 30s into its wait.
|
||||
sample(&db, "queue_delay_t", json!(5), 602.0).await;
|
||||
sample(&db, "queue_delay_t", json!({ "since": now - WINDOW + 597.0 }), 610.0).await;
|
||||
sample(&db, "queue_delay_t", json!(0), 627.0).await;
|
||||
|
||||
let series = read_queue_metrics_series(&db, WINDOW).await.unwrap();
|
||||
let top = series.tags[0]
|
||||
.delay
|
||||
.iter()
|
||||
.map(|(_, value)| *value)
|
||||
.fold(0.0, f64::max);
|
||||
assert!((top - 30.0).abs() < 2.0, "{:?}", series.tags[0].delay);
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_common::queue::get_queue_stats;
|
||||
|
||||
const WORKSPACE: &str = "test-workspace";
|
||||
|
||||
async fn queue_job(
|
||||
db: &Pool<Postgres>,
|
||||
tag: &str,
|
||||
priority: Option<i16>,
|
||||
waited_secs: f64,
|
||||
running: bool,
|
||||
) {
|
||||
sqlx::query(
|
||||
"WITH job AS (
|
||||
INSERT INTO v2_job (id, workspace_id, tag) VALUES (gen_random_uuid(), $1, $2)
|
||||
RETURNING id
|
||||
)
|
||||
INSERT INTO v2_job_queue (id, workspace_id, tag, priority, running, scheduled_for)
|
||||
SELECT id, $1, $2, $3, $4, now() - make_interval(secs => $5) FROM job",
|
||||
)
|
||||
.bind(WORKSPACE)
|
||||
.bind(tag)
|
||||
.bind(priority)
|
||||
.bind(running)
|
||||
.bind(waited_secs)
|
||||
.execute(db)
|
||||
.await
|
||||
.expect("failed to queue job");
|
||||
}
|
||||
|
||||
/// The delay reported for a tag is that of the job the worker pull takes first, ordered
|
||||
/// `priority DESC NULLS LAST, scheduled_for`, not simply the oldest one waiting. Running jobs
|
||||
/// and jobs less than 3 seconds past due are not part of the backlog at all.
|
||||
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
|
||||
async fn queue_stats_report_the_delay_of_the_job_pulled_next(db: Pool<Postgres>) {
|
||||
// The oldest job has no priority, so every prioritized job runs before it.
|
||||
queue_job(&db, "mixed", None, 900.0, false).await;
|
||||
queue_job(&db, "mixed", Some(1), 600.0, false).await;
|
||||
queue_job(&db, "mixed", Some(5), 300.0, false).await;
|
||||
queue_job(&db, "mixed", Some(5), 100.0, false).await;
|
||||
// Highest priority, but not backlog: already running, or not yet 3 seconds past due.
|
||||
queue_job(&db, "mixed", Some(9), 1200.0, true).await;
|
||||
queue_job(&db, "mixed", Some(9), 1.0, false).await;
|
||||
queue_job(&db, "unprioritized", None, 500.0, false).await;
|
||||
queue_job(&db, "unprioritized", None, 50.0, false).await;
|
||||
|
||||
let stats = get_queue_stats(&db).await.unwrap();
|
||||
let now: f64 = sqlx::query_scalar("SELECT EXTRACT(EPOCH FROM now())::double precision")
|
||||
.fetch_one(&db)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let mixed = &stats["mixed"];
|
||||
assert_eq!(mixed.count, 4);
|
||||
assert!(
|
||||
(mixed.delay - 300.0).abs() < 5.0,
|
||||
"expected the oldest job of the highest priority, got a delay of {}",
|
||||
mixed.delay
|
||||
);
|
||||
// The same job's wait start, which the delay is measured from.
|
||||
assert!((mixed.head_since + mixed.delay - now).abs() < 5.0);
|
||||
let unprioritized = &stats["unprioritized"];
|
||||
assert_eq!(unprioritized.count, 2);
|
||||
assert!(
|
||||
(unprioritized.delay - 500.0).abs() < 5.0,
|
||||
"expected the oldest job, got a delay of {}",
|
||||
unprioritized.delay
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
//! Every table must be replicable.
|
||||
//!
|
||||
//! PostgreSQL refuses UPDATE and DELETE on a table that has neither a PRIMARY KEY
|
||||
//! nor an explicit REPLICA IDENTITY once the database is published to a logical
|
||||
//! replication slot. That is what a low-downtime major-version upgrade runs on
|
||||
//! (RDS and Aurora Blue/Green, pglogical) and what every CDC pipeline reads, so a
|
||||
//! single keyless table blocks the upgrade outright. This runs against a freshly
|
||||
//! migrated database and fails on the migration that introduces one.
|
||||
|
||||
use sqlx::{Pool, Postgres};
|
||||
|
||||
/// Partitioned parents are checked alongside ordinary tables: a parent without a
|
||||
/// key hands the same defect to every partition created under it later.
|
||||
#[sqlx::test(migrations = "../migrations")]
|
||||
async fn every_table_is_replicable(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
let offenders: Vec<String> = sqlx::query_scalar(
|
||||
"SELECT n.nspname || '.' || c.relname
|
||||
FROM pg_class c
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE c.relkind IN ('r', 'p')
|
||||
AND n.nspname NOT IN ('pg_catalog', 'information_schema')
|
||||
AND NOT (
|
||||
-- FULL and USING INDEX replicate on their own.
|
||||
c.relreplident IN ('f', 'i')
|
||||
-- DEFAULT resolves to the primary key, so it needs one to exist.
|
||||
-- NOTHING never replicates, primary key or not.
|
||||
OR (c.relreplident = 'd' AND EXISTS (
|
||||
SELECT 1 FROM pg_index i WHERE i.indrelid = c.oid AND i.indisprimary
|
||||
))
|
||||
)
|
||||
ORDER BY 1",
|
||||
)
|
||||
.fetch_all(&db)
|
||||
.await?;
|
||||
|
||||
assert!(
|
||||
offenders.is_empty(),
|
||||
"logical replication will reject UPDATE and DELETE on these tables, because \
|
||||
they carry no replica identity it can use: {}. \
|
||||
Give each one a primary key -- a natural composite key where every column \
|
||||
is NOT NULL, otherwise a surrogate `BIGINT GENERATED ALWAYS AS IDENTITY`.",
|
||||
offenders.join(", ")
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -17,6 +17,7 @@ use windmill_api_auth::{
|
||||
};
|
||||
use windmill_common::db::DB;
|
||||
use windmill_common::per_minute_counter::PerMinuteCounter;
|
||||
use windmill_common::ssrf::{private_git_host_allowed, private_git_host_hint, GitRemoteCaller};
|
||||
use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult};
|
||||
|
||||
use crate::secret_backend_ext::rename_vault_secret;
|
||||
@@ -3429,8 +3430,10 @@ fn git_url_userinfo(url: &str) -> Option<&str> {
|
||||
git_url_userinfo_range(url).map(|r| &url[r])
|
||||
}
|
||||
|
||||
/// Validates a git URL to prevent option injection, SSRF, and local file read.
|
||||
async fn validate_git_url(url: &str) -> Result<()> {
|
||||
/// Validates a git URL to prevent option injection, SSRF, and local file read. The
|
||||
/// syntax and scheme checks apply to every caller; the private-host refusal only
|
||||
/// where [`private_git_host_allowed`] refuses `caller`.
|
||||
async fn validate_git_url(url: &str, caller: GitRemoteCaller) -> Result<()> {
|
||||
let url = url.trim();
|
||||
if url.is_empty() {
|
||||
return Err(Error::BadRequest("Git URL cannot be empty".to_string()));
|
||||
@@ -3484,25 +3487,26 @@ async fn validate_git_url(url: &str) -> Result<()> {
|
||||
let host = extract_host_from_git_url(url)
|
||||
.ok_or_else(|| Error::BadRequest("Could not parse hostname from git URL".to_string()))?;
|
||||
|
||||
// CI/dev escape hatch: integration tests run their git remote (a Gitea
|
||||
// container) on localhost, which the network-target checks below reject.
|
||||
// Scheme and option-injection validation above still applies.
|
||||
if std::env::var("ALLOW_LOCAL_GIT_REMOTES").is_ok_and(|v| v == "true" || v == "1") {
|
||||
// Scheme and option-injection validation above applies to every caller.
|
||||
if private_git_host_allowed(caller) {
|
||||
return Ok(());
|
||||
}
|
||||
let hint = private_git_host_hint(caller)
|
||||
.map(|h| format!(" {h}"))
|
||||
.unwrap_or_default();
|
||||
|
||||
if host == "localhost" || host.ends_with(".local") || host == "[::1]" {
|
||||
return Err(Error::BadRequest(
|
||||
"Git URLs targeting localhost or local network are not allowed".to_string(),
|
||||
));
|
||||
return Err(Error::BadRequest(format!(
|
||||
"Git URLs targeting localhost or local network are not allowed.{hint}"
|
||||
)));
|
||||
}
|
||||
|
||||
// Check literal IP addresses
|
||||
if let Ok(ip) = host.parse::<IpAddr>() {
|
||||
if is_private_or_reserved_ip(&ip) {
|
||||
return Err(Error::BadRequest(
|
||||
"Git URLs targeting private or reserved IP addresses are not allowed".to_string(),
|
||||
));
|
||||
return Err(Error::BadRequest(format!(
|
||||
"Git URLs targeting private or reserved IP addresses are not allowed.{hint}"
|
||||
)));
|
||||
}
|
||||
} else {
|
||||
// Hostname — resolve via DNS and reject if any address is private. Fail
|
||||
@@ -3523,9 +3527,9 @@ async fn validate_git_url(url: &str) -> Result<()> {
|
||||
}
|
||||
for addr in addrs {
|
||||
if is_private_or_reserved_ip(&addr.ip()) {
|
||||
return Err(Error::BadRequest(
|
||||
"Git URL hostname resolves to a private or reserved IP address".to_string(),
|
||||
));
|
||||
return Err(Error::BadRequest(format!(
|
||||
"Git URL hostname resolves to a private or reserved IP address.{hint}"
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3627,8 +3631,14 @@ async fn get_git_commit_hash(
|
||||
.map_err(|e| {
|
||||
Error::BadRequest(format!("Invalid git repository resource format: {}", e))
|
||||
})?;
|
||||
let caller = if authed.is_admin {
|
||||
GitRemoteCaller::AdminOrSystem
|
||||
} else {
|
||||
GitRemoteCaller::NonAdmin
|
||||
};
|
||||
git_resource.url =
|
||||
resolve_azure_devops_url(&db_with_opt_authed, &w_id, &git_resource.url, false).await?;
|
||||
resolve_azure_devops_url(&db_with_opt_authed, &w_id, &git_resource.url, false, caller)
|
||||
.await?;
|
||||
// A credential is stored under the repository it was issued for, so a
|
||||
// resource repointed elsewhere finds none. Which credential can be attached
|
||||
// is bounded by that; who may use it is bounded here, on the same terms as
|
||||
@@ -3658,7 +3668,7 @@ async fn get_git_commit_hash(
|
||||
let (git_ssh_cmd, filenames) =
|
||||
get_git_ssh_cmd(&authed, &user_db, &db, &w_id, identities).await?;
|
||||
|
||||
let commit_hash = get_repo_latest_commit_hash(&git_resource, git_ssh_cmd).await;
|
||||
let commit_hash = get_repo_latest_commit_hash(&git_resource, git_ssh_cmd, caller).await;
|
||||
|
||||
delete_paths(&filenames).await;
|
||||
|
||||
@@ -3762,12 +3772,17 @@ async fn get_git_ssh_cmd(
|
||||
const GIT_PROBE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
|
||||
|
||||
/// `git` command for a remote probe, with HTTP redirects disabled. `validate_git_url`
|
||||
/// only vets the host in the URL; git's default (`http.followRedirects=initial`)
|
||||
/// would let a validated public remote 302 the probe onto a private or link-local
|
||||
/// address that no check ever sees. Build every probe through this.
|
||||
/// checks the host in the URL, never one a redirect names; git's default
|
||||
/// (`http.followRedirects=initial`) would let a public remote 302 the probe of a
|
||||
/// caller refused private hosts onto one. Build every probe through this.
|
||||
///
|
||||
/// The transports are pinned too: an SCP-shaped remote-helper string such as
|
||||
/// `ext::<command>@host:path` passes the URL check for a caller allowed private
|
||||
/// hosts, and only git's own config would stop it from running the command.
|
||||
fn git_probe_command() -> Command {
|
||||
let mut git_cmd = Command::new("git");
|
||||
git_cmd.args(["-c", "http.followRedirects=false"]);
|
||||
git_cmd.env("GIT_ALLOW_PROTOCOL", "http:https:ssh:git");
|
||||
git_cmd
|
||||
}
|
||||
|
||||
@@ -3827,8 +3842,8 @@ fn dot_git_url(url: &str) -> Option<String> {
|
||||
}
|
||||
|
||||
/// Run a remote probe, retrying against [`dot_git_url`] if the remote answered the
|
||||
/// URL as given with a redirect. Extending the path keeps the retry on the host
|
||||
/// `validate_git_url` already cleared, which is exactly what following the redirect
|
||||
/// URL as given with a redirect. Extending the path keeps the retry on the host of
|
||||
/// the URL `validate_git_url` checked, which is exactly what following the redirect
|
||||
/// would not guarantee. `build` must produce the probe for the URL it is handed.
|
||||
///
|
||||
/// A retry that also fails reports the *original* failure, so the caller's message
|
||||
@@ -3968,6 +3983,7 @@ async fn resolve_azure_devops_url(
|
||||
w_id: &str,
|
||||
url: &str,
|
||||
allow_cache: bool,
|
||||
caller: GitRemoteCaller,
|
||||
) -> Result<String> {
|
||||
// Trim first: the http(s) gates the callers apply trim too, so a stored URL with
|
||||
// leading whitespace must not reach the scheme check here as a non-http one.
|
||||
@@ -3980,7 +3996,7 @@ async fn resolve_azure_devops_url(
|
||||
// cost a live credential (nor cache one), and whoever can edit the URL would
|
||||
// otherwise drive a token mint per poll tick.
|
||||
let probe_url = url.replace(placeholder, "windmill");
|
||||
validate_git_url(&probe_url).await?;
|
||||
validate_git_url(&probe_url, caller).await?;
|
||||
|
||||
// The background poller reads the referenced resource under the system identity,
|
||||
// which bypasses RLS. Confining the destination is what keeps that from becoming an
|
||||
@@ -4180,9 +4196,10 @@ fn git_sync_system_dba(db: &DB) -> DbWithOptAuthed<'static, ApiAuthed> {
|
||||
async fn get_repo_latest_commit_hash(
|
||||
git_resource: &GitRepositoryResource,
|
||||
git_ssh_command: Option<String>,
|
||||
caller: GitRemoteCaller,
|
||||
) -> Result<String> {
|
||||
// Validate URL and branch to prevent option injection and SSRF attacks
|
||||
validate_git_url(&git_resource.url).await?;
|
||||
validate_git_url(&git_resource.url, caller).await?;
|
||||
|
||||
let ref_spec = git_resource
|
||||
.branch
|
||||
@@ -4315,8 +4332,14 @@ pub async fn get_git_repo_head_for_autopull(
|
||||
"Automatic pull can't authenticate an SSH git remote in the background. Use an HTTPS URL with an embedded token, or connect the repository through the GitHub App.".to_string(),
|
||||
));
|
||||
}
|
||||
git_resource.url =
|
||||
resolve_azure_devops_url(&git_sync_system_dba(db), w_id, &git_resource.url, true).await?;
|
||||
git_resource.url = resolve_azure_devops_url(
|
||||
&git_sync_system_dba(db),
|
||||
w_id,
|
||||
&git_resource.url,
|
||||
true,
|
||||
GitRemoteCaller::AdminOrSystem,
|
||||
)
|
||||
.await?;
|
||||
// A repo whose credential Windmill holds carries none in its URL, so the
|
||||
// poller has to attach it here or every probe would be unauthenticated.
|
||||
git_resource.url =
|
||||
@@ -4324,14 +4347,15 @@ pub async fn get_git_repo_head_for_autopull(
|
||||
|
||||
if let Some(branch) = git_resource.branch.as_deref().filter(|s| !s.is_empty()) {
|
||||
let branch = branch.to_string();
|
||||
let sha = get_repo_latest_commit_hash(&git_resource, None).await?;
|
||||
let sha = get_repo_latest_commit_hash(&git_resource, None, GitRemoteCaller::AdminOrSystem)
|
||||
.await?;
|
||||
return Ok(Some((branch, sha)));
|
||||
}
|
||||
|
||||
// No explicit branch: resolve the remote's default-branch NAME along with
|
||||
// its head in one call. Fork sync needs the concrete name to scope
|
||||
// `wm-fork/<branch>/*`, so a bare "HEAD" ref would silently disable it.
|
||||
validate_git_url(&git_resource.url).await?;
|
||||
validate_git_url(&git_resource.url, GitRemoteCaller::AdminOrSystem).await?;
|
||||
let output = run_git_probe_for_url(&git_resource.url, "ls-remote --symref HEAD", |url| {
|
||||
let mut git_cmd = git_probe_command();
|
||||
git_cmd.args(["ls-remote", "--symref", url, "HEAD"]);
|
||||
@@ -4426,13 +4450,20 @@ pub async fn get_git_repo_fork_heads_for_autopull(
|
||||
"Automatic pull can't authenticate an SSH git remote in the background. Use an HTTPS URL with an embedded token, or connect the repository through the GitHub App.".to_string(),
|
||||
));
|
||||
}
|
||||
git_resource.url = resolve_azure_devops_url(&dba, w_id, &git_resource.url, true).await?;
|
||||
git_resource.url = resolve_azure_devops_url(
|
||||
&dba,
|
||||
w_id,
|
||||
&git_resource.url,
|
||||
true,
|
||||
GitRemoteCaller::AdminOrSystem,
|
||||
)
|
||||
.await?;
|
||||
// Same reason as the head probe above: a repository whose credential Windmill
|
||||
// holds carries none in its URL, and listing the fork branches is the half of
|
||||
// polling that would otherwise go out unauthenticated.
|
||||
git_resource.url =
|
||||
windmill_common::git_sync_oss::with_stored_credential(db, w_id, git_resource.url).await?;
|
||||
validate_git_url(&git_resource.url).await?;
|
||||
validate_git_url(&git_resource.url, GitRemoteCaller::AdminOrSystem).await?;
|
||||
validate_git_ref(base_branch)?;
|
||||
|
||||
for r in extra_refs {
|
||||
@@ -4818,46 +4849,52 @@ mod tests {
|
||||
));
|
||||
}
|
||||
|
||||
// A caller let through to private hosts must still hit the scheme check.
|
||||
#[tokio::test]
|
||||
async fn test_validate_git_url_blocks_file_scheme() {
|
||||
let result = validate_git_url("file:///etc/passwd").await;
|
||||
let result = validate_git_url("file:///etc/passwd", GitRemoteCaller::AdminOrSystem).await;
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().to_string().contains("https://"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_validate_git_url_blocks_private_ips() {
|
||||
assert!(validate_git_url("http://127.0.0.1/repo.git").await.is_err());
|
||||
assert!(validate_git_url("http://169.254.169.254/latest/meta-data/")
|
||||
.await
|
||||
.is_err());
|
||||
assert!(validate_git_url("http://10.0.0.1/repo.git").await.is_err());
|
||||
assert!(validate_git_url("http://172.16.0.1/repo.git")
|
||||
.await
|
||||
.is_err());
|
||||
assert!(validate_git_url("http://192.168.1.1/repo.git")
|
||||
.await
|
||||
.is_err());
|
||||
assert!(validate_git_url("git://0.0.0.0/repo.git").await.is_err());
|
||||
let v = |url: &'static str| validate_git_url(url, GitRemoteCaller::NonAdmin);
|
||||
assert!(v("http://127.0.0.1/repo.git").await.is_err());
|
||||
assert!(v("http://169.254.169.254/latest/meta-data/").await.is_err());
|
||||
let err = v("http://10.0.0.1/repo.git").await.unwrap_err();
|
||||
assert!(err.to_string().contains("ALLOW_LOCAL_GIT_REMOTES"), "{err}");
|
||||
assert!(v("http://172.16.0.1/repo.git").await.is_err());
|
||||
assert!(v("http://192.168.1.1/repo.git").await.is_err());
|
||||
assert!(v("git://0.0.0.0/repo.git").await.is_err());
|
||||
// IPv6 loopback, unique-local, and link-local literals
|
||||
assert!(validate_git_url("git://[::1]/repo.git").await.is_err());
|
||||
assert!(validate_git_url("git://[fd00::1]/repo.git").await.is_err());
|
||||
assert!(validate_git_url("git://[fe80::1]/repo.git").await.is_err());
|
||||
assert!(v("git://[::1]/repo.git").await.is_err());
|
||||
assert!(v("git://[fd00::1]/repo.git").await.is_err());
|
||||
assert!(v("git://[fe80::1]/repo.git").await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_validate_git_url_lets_admins_reach_private_hosts() {
|
||||
assert!(
|
||||
validate_git_url("http://10.0.0.1/repo.git", GitRemoteCaller::AdminOrSystem)
|
||||
.await
|
||||
.is_ok()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_validate_git_url_blocks_localhost() {
|
||||
assert!(validate_git_url("http://localhost/repo.git").await.is_err());
|
||||
assert!(validate_git_url("http://myhost.local/repo.git")
|
||||
.await
|
||||
.is_err());
|
||||
let v = |url: &'static str| validate_git_url(url, GitRemoteCaller::NonAdmin);
|
||||
assert!(v("http://localhost/repo.git").await.is_err());
|
||||
assert!(v("http://myhost.local/repo.git").await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_validate_git_url_blocks_local_paths() {
|
||||
assert!(validate_git_url("/etc/passwd").await.is_err());
|
||||
assert!(validate_git_url("../relative/path").await.is_err());
|
||||
assert!(validate_git_url("./local/repo").await.is_err());
|
||||
let v = |url: &'static str| validate_git_url(url, GitRemoteCaller::AdminOrSystem);
|
||||
assert!(v("/etc/passwd").await.is_err());
|
||||
assert!(v("../relative/path").await.is_err());
|
||||
assert!(v("./local/repo").await.is_err());
|
||||
}
|
||||
|
||||
/// Minimal loopback HTTP server: replies to every request with `response` and
|
||||
@@ -4987,7 +5024,11 @@ mod tests {
|
||||
async fn test_validate_git_url_fails_closed_on_unresolvable_host() {
|
||||
// `.invalid` never resolves (RFC 6761). The private-IP check is only
|
||||
// meaningful if a failed lookup rejects instead of falling through.
|
||||
let result = validate_git_url("https://this-host-does-not-exist.invalid/repo.git").await;
|
||||
let result = validate_git_url(
|
||||
"https://this-host-does-not-exist.invalid/repo.git",
|
||||
GitRemoteCaller::NonAdmin,
|
||||
)
|
||||
.await;
|
||||
assert!(
|
||||
result.is_err(),
|
||||
"an unresolvable host was allowed — does this resolver synthesize records for NXDOMAIN?"
|
||||
@@ -4998,21 +5039,33 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn test_validate_git_url_allows_valid_urls() {
|
||||
// Needs DNS: validation fails closed on a host it cannot resolve.
|
||||
assert!(validate_git_url("https://github.com/user/repo.git")
|
||||
let v = |url: &'static str| validate_git_url(url, GitRemoteCaller::NonAdmin);
|
||||
assert!(v("https://github.com/user/repo.git").await.is_ok());
|
||||
assert!(v("git@github.com:user/repo.git").await.is_ok());
|
||||
assert!(v("ssh://git@github.com/user/repo.git").await.is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_git_probe_refuses_remote_helpers() {
|
||||
// A caller allowed private hosts skips the DNS step that would reject this
|
||||
// SCP-shaped string, so the transport pin is what keeps git from running it.
|
||||
let output = git_probe_command()
|
||||
.args(["ls-remote", "testhelper::x@127.0.0.1:repo"])
|
||||
.output()
|
||||
.await
|
||||
.is_ok());
|
||||
assert!(validate_git_url("git@github.com:user/repo.git")
|
||||
.await
|
||||
.is_ok());
|
||||
assert!(validate_git_url("ssh://git@github.com/user/repo.git")
|
||||
.await
|
||||
.is_ok());
|
||||
.unwrap();
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
assert!(
|
||||
stderr.contains("transport 'testhelper' not allowed"),
|
||||
"{stderr}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_validate_git_url_blocks_option_injection() {
|
||||
assert!(validate_git_url("-evil").await.is_err());
|
||||
assert!(validate_git_url("--upload-pack=evil").await.is_err());
|
||||
let v = |url: &'static str| validate_git_url(url, GitRemoteCaller::AdminOrSystem);
|
||||
assert!(v("-evil").await.is_err());
|
||||
assert!(v("--upload-pack=evil").await.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -5108,24 +5161,21 @@ mod tests {
|
||||
// GHSA-p5cj-8cfh-mjv6: a loopback authority must stay blocked, and the
|
||||
// fragment/query `@public-host` bypasses of #8600 must be rejected so the
|
||||
// host git dials can never diverge from the validated host.
|
||||
assert!(validate_git_url("http://127.0.0.1:40173/repo.git")
|
||||
.await
|
||||
.is_err());
|
||||
assert!(validate_git_url(
|
||||
"http://127.0.0.1:40173/repo.git#@github.com/windmill-labs/windmill.git"
|
||||
)
|
||||
.await
|
||||
.is_err());
|
||||
assert!(validate_git_url(
|
||||
"http://127.0.0.1:40173/repo.git?@github.com/windmill-labs/windmill.git"
|
||||
)
|
||||
.await
|
||||
.is_err());
|
||||
// A legitimate public repo URL still validates.
|
||||
let v = |url: &'static str| validate_git_url(url, GitRemoteCaller::NonAdmin);
|
||||
assert!(v("http://127.0.0.1:40173/repo.git").await.is_err());
|
||||
assert!(
|
||||
validate_git_url("https://github.com/windmill-labs/windmill.git")
|
||||
v("http://127.0.0.1:40173/repo.git#@github.com/windmill-labs/windmill.git")
|
||||
.await
|
||||
.is_ok()
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
v("http://127.0.0.1:40173/repo.git?@github.com/windmill-labs/windmill.git")
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
// A legitimate public repo URL still validates.
|
||||
assert!(v("https://github.com/windmill-labs/windmill.git")
|
||||
.await
|
||||
.is_ok());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -538,6 +538,20 @@ pub struct Suspend {
|
||||
pub hide_cancel: Option<bool>,
|
||||
#[serde(skip_serializing_if = "false_or_empty")]
|
||||
pub continue_on_disapprove_timeout: Option<bool>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub skin: Option<ApprovalSkin>,
|
||||
}
|
||||
|
||||
/// How an approval request is presented, on the approval page and in Slack/Teams messages.
|
||||
#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum ApprovalSkin {
|
||||
Minimal,
|
||||
/// A skin this server does not know renders as the detailed one rather than failing to
|
||||
/// deserialize the whole flow, so a flow authored against a newer version still runs.
|
||||
#[default]
|
||||
#[serde(other)]
|
||||
Detailed,
|
||||
}
|
||||
|
||||
fn false_or_empty(v: &Option<bool>) -> bool {
|
||||
@@ -1365,6 +1379,23 @@ mod tests {
|
||||
assert_eq!(val.modules.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn suspend_skin_unknown_value_falls_back_to_detailed() {
|
||||
let skin_of = |skin: &str| {
|
||||
let val: FlowValue = serde_json::from_value(json!({
|
||||
"modules": [{
|
||||
"id": "a",
|
||||
"value": {"type": "identity"},
|
||||
"suspend": {"required_events": 1, "skin": skin}
|
||||
}]
|
||||
}))
|
||||
.unwrap();
|
||||
val.modules[0].suspend.as_ref().unwrap().skin
|
||||
};
|
||||
assert_eq!(skin_of("minimal"), Some(ApprovalSkin::Minimal));
|
||||
assert_eq!(skin_of("not_a_skin_yet"), Some(ApprovalSkin::Detailed));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_tool_keeps_description_through_locking() {
|
||||
// #10244: the dependency job rebuilds each tool from its locked FlowModule; the
|
||||
|
||||
@@ -105,9 +105,11 @@ const p = {
|
||||
const normalized = (isRelative ? join(dirname(file_path), pathNoExt) : pathNoExt.slice(1)).replace(/\\/g, "/");
|
||||
const hash = TEMP_SCRIPT_REFS?.[normalized];
|
||||
|
||||
// Lock generation substitutes `raw`: the dependency scan reads versions from the
|
||||
// `pkg@version` specifiers in imported scripts, which `raw_unpinned` strips.
|
||||
const url = (isRelative
|
||||
? `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${file_path}/../${args.path}${endExt}`
|
||||
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`
|
||||
? `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${file_path}/../${args.path}${endExt}`
|
||||
: `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${args.path}${endExt}`
|
||||
) + (hash ? `?temp_script_hash=${hash}` : "");
|
||||
const file = isRelative
|
||||
? resolve("./" + file_path + "/../" + args.path + ".url")
|
||||
|
||||
@@ -13,7 +13,7 @@ use itertools::Itertools;
|
||||
use serde_json::value::RawValue;
|
||||
|
||||
use uuid::Uuid;
|
||||
use windmill_parser_ts::remove_pinned_imports;
|
||||
use windmill_parser_ts::{remove_pinned_import_specifiers, remove_pinned_imports};
|
||||
|
||||
use windmill_queue::{append_logs, CanceledBy, MiniPulledJob, PrecomputedAgentInfo};
|
||||
|
||||
@@ -1784,8 +1784,19 @@ pub async fn handle_bun_job(
|
||||
if modules.as_ref().is_some_and(|m| !m.is_empty()) {
|
||||
let bundle_path = std::path::Path::new(job_dir).join("out").join("main.js");
|
||||
if bundle_path.exists() {
|
||||
// The lock-generation build kept every `pkg@version` specifier, and bun resolves
|
||||
// a pinned specifier outside node_modules, loading a second copy of the package.
|
||||
// The bundle holds the user's code too, so only the specifiers are rewritten, and
|
||||
// a bundle the parser rejects still runs as built, pins and all.
|
||||
let bundled = std::fs::read_to_string(&bundle_path)?;
|
||||
write_file(job_dir, "main.ts", &bundled)?;
|
||||
let unpinned = remove_pinned_import_specifiers(&bundled).unwrap_or_else(|e| {
|
||||
tracing::warn!(
|
||||
job_id = %job.id,
|
||||
"could not unpin the modules bundle, running it as built: {e:#}"
|
||||
);
|
||||
bundled
|
||||
});
|
||||
write_file(job_dir, "main.ts", &unpinned)?;
|
||||
}
|
||||
}
|
||||
"\n\n--- BUN CODE EXECUTION ---\n".to_string()
|
||||
@@ -1975,7 +1986,7 @@ async function run() {{
|
||||
return {{ type: "inline_checkpoint", key: dispatch.key, result: dispatch.result ?? null, started_at: dispatch.started_at, duration_ms: dispatch.duration_ms }};
|
||||
}}
|
||||
if (dispatch.mode === "approval") {{
|
||||
return {{ type: "approval", key: dispatch.key, timeout: dispatch.timeout, form: dispatch.form, self_approval_disabled: dispatch.self_approval_disabled }};
|
||||
return {{ type: "approval", key: dispatch.key, timeout: dispatch.timeout, form: dispatch.form, self_approval_disabled: dispatch.self_approval_disabled, skin: dispatch.skin, description: dispatch.description }};
|
||||
}}
|
||||
if (dispatch.mode === "sleep") {{
|
||||
return {{ type: "sleep", key: dispatch.key, seconds: dispatch.seconds }};
|
||||
@@ -3206,7 +3217,7 @@ pub async fn handle_wac_v2_output(
|
||||
job.id, num_steps
|
||||
)))
|
||||
}
|
||||
WacOutput::Approval { key, timeout, form, self_approval_disabled } => {
|
||||
WacOutput::Approval { key, timeout, form, self_approval_disabled, skin, description } => {
|
||||
let db = match conn {
|
||||
Connection::Sql(db) => db,
|
||||
_ => {
|
||||
@@ -3322,15 +3333,19 @@ pub async fn handle_wac_v2_output(
|
||||
};
|
||||
|
||||
// Store approval form metadata for the approval page endpoint
|
||||
let approval_meta = serde_json::json!({
|
||||
let mut approval_meta = serde_json::json!({
|
||||
"key": key,
|
||||
"form": form,
|
||||
"timeout": timeout_secs as u32,
|
||||
"self_approval_disabled": sad,
|
||||
"skin": skin.unwrap_or_default(),
|
||||
"resume": resume_url,
|
||||
"cancel": cancel_url,
|
||||
"approvalPage": approval_page_url,
|
||||
});
|
||||
if let Some(description) = description.filter(|d| !d.is_null()) {
|
||||
approval_meta["description"] = description;
|
||||
}
|
||||
sqlx::query(
|
||||
"UPDATE v2_job_status SET workflow_as_code_status = jsonb_set(
|
||||
COALESCE(workflow_as_code_status, '{}'::jsonb),
|
||||
|
||||
@@ -2271,7 +2271,11 @@ pub(crate) fn dbt_command(p: &PreparedProject, args: &[&str]) -> Command {
|
||||
.envs(PROXY_ENVS.clone())
|
||||
.env("PATH", PATH_ENV.as_str())
|
||||
.env("TZ", TZ_ENV.as_str())
|
||||
.env("GIT_PATH", GIT_PATH.as_str());
|
||||
.env("GIT_PATH", GIT_PATH.as_str())
|
||||
// dbt reports anonymous usage to dbt Labs from every invocation unless told
|
||||
// not to, and a project's `flags:` block cannot override the variable. Set
|
||||
// before the project's environment so a descriptor can still opt back in.
|
||||
.env("DBT_SEND_ANONYMOUS_USAGE_STATS", "false");
|
||||
// Both environments belong to the child. Under a sandbox they reach it through
|
||||
// the jail profile instead: set here, they would reach the dynamic loader that
|
||||
// execs nsjail itself, so an `LD_PRELOAD` from the project would run as the
|
||||
|
||||
@@ -37,8 +37,9 @@ use windmill_common::{
|
||||
scripts::ScriptLang,
|
||||
utils::calculate_hash,
|
||||
worker::{
|
||||
copy_dir_recursively, is_allowed_file_location, pad_string, split_python_requirements,
|
||||
write_file, Connection, PyVAlias, PythonAnnotations, WORKER_CONFIG,
|
||||
copy_dir_recursively, is_allowed_file_location, lockfile_line_has_continuation, pad_string,
|
||||
requirement_from_lockfile_line, split_python_requirements, write_file, Connection,
|
||||
PyVAlias, PythonAnnotations, WORKER_CONFIG,
|
||||
},
|
||||
};
|
||||
|
||||
@@ -227,9 +228,9 @@ fn filter_pip_local_dependencies(lines: Vec<String>) -> (Vec<String>, Vec<String
|
||||
/// `(kept, ignored)`. A line is ignored when it is not a `#` comment and matches any of
|
||||
/// `compiled_deps`. Kept separate from config/regex loading so it can be unit-tested.
|
||||
fn filter_lines_by_deps(lines: Vec<String>, compiled_deps: &[Regex]) -> (Vec<String>, Vec<String>) {
|
||||
let (ignored, kept): (Vec<String>, Vec<String>) = lines
|
||||
.into_iter()
|
||||
.partition(|s| !s.starts_with('#') && compiled_deps.iter().any(|dep| dep.is_match(s)));
|
||||
let (ignored, kept): (Vec<String>, Vec<String>) = lines.into_iter().partition(|s| {
|
||||
!s.trim_start().starts_with('#') && compiled_deps.iter().any(|dep| dep.is_match(s))
|
||||
});
|
||||
|
||||
(kept, ignored)
|
||||
}
|
||||
@@ -323,6 +324,11 @@ pub async fn uv_pip_compile(
|
||||
"compile",
|
||||
"-q",
|
||||
"--no-header",
|
||||
// The `#`-line filter applied to the output below only catches whole-line
|
||||
// annotations, and uv's annotation style is configurable: `[pip]
|
||||
// annotation-style = "line"` in the worker HOME's uv.toml emits them inline
|
||||
// ("anyio==4.15.1 # via httpx"), which that filter keeps.
|
||||
"--no-annotate",
|
||||
file,
|
||||
"--strip-extras",
|
||||
"-o",
|
||||
@@ -2520,11 +2526,23 @@ pub async fn handle_python_reqs(
|
||||
// Find out if there is already cached dependencies
|
||||
// If so, skip them
|
||||
let mut in_cache = vec![];
|
||||
if requirements
|
||||
.iter()
|
||||
.any(|r| lockfile_line_has_continuation(r))
|
||||
{
|
||||
tracing::warn!(workspace_id = %w_id, job_id = %job_id, "lockfile continues entries across lines; the continued lines are dropped");
|
||||
append_logs(
|
||||
job_id,
|
||||
w_id,
|
||||
"\n[!] lockfile continues entries across lines and the continued lines are dropped: `--hash=` pins, extras and markers written that way do not apply\n".to_string(),
|
||||
conn,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
for req in &requirements {
|
||||
// Ignore python version annotation backed into lockfile
|
||||
if req.starts_with('#') || req.starts_with('-') || req.trim().is_empty() {
|
||||
let Some(req) = requirement_from_lockfile_line(req) else {
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let py_prefix = &py_version.to_cache_dir(false);
|
||||
|
||||
let venv_p = format!(
|
||||
|
||||
@@ -46,6 +46,10 @@ pub enum WacOutput {
|
||||
form: Option<Value>,
|
||||
#[serde(default)]
|
||||
self_approval_disabled: Option<bool>,
|
||||
#[serde(default)]
|
||||
skin: Option<windmill_common::flows::ApprovalSkin>,
|
||||
#[serde(default)]
|
||||
description: Option<Value>,
|
||||
},
|
||||
/// Server-side sleep — suspend the workflow for a duration without holding a worker.
|
||||
#[serde(rename = "sleep")]
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
|
||||
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
|
||||
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
|
||||
|
||||
export const VERSION = "v1.805.0";
|
||||
export const VERSION = "v1.809.0";
|
||||
|
||||
export async function login(email: string, password: string): Promise<string> {
|
||||
return await windmill.UserService.login({
|
||||
|
||||
@@ -35,6 +35,28 @@ binary and starts a shared backend instance.
|
||||
|
||||
Examples: `sync_pull_push`, `dev_server`, `standalone_commands`
|
||||
|
||||
## Module mocks
|
||||
|
||||
`mock.module` replaces a module for the **whole process**, and it does reach modules that
|
||||
were already imported — a stub one file installs lands on a consumer an earlier file
|
||||
loaded.
|
||||
|
||||
Handing the module back in `afterAll` is not a reliable undo. Files do run one at a time
|
||||
(a root-level `afterAll` completes before the next file's body evaluates), so it looks
|
||||
like it should be — but stubbing `bundle.ts` and restoring it that way still left
|
||||
`raw_app_svelte_plugin_unit.test.ts` asserting against an empty bundle, green on Linux
|
||||
and red on Windows, where the `readdir` file order differs. Treat a stub as permanent for
|
||||
the run.
|
||||
|
||||
So the rule is about what you stub, not how you clean up: **stub only a module no other
|
||||
in-process suite imports.** Check with `grep -rl "<exported fn>" test/` before reaching
|
||||
for one. A suite that drives the CLI through a spawned process is out of reach of a
|
||||
module mock and doesn't count.
|
||||
|
||||
`raw_app_push_policy_unit.test.ts` is the worked example: it stubs `gen/services.gen.ts`,
|
||||
which passes the rule because nothing else in `test/` imports the three API functions it
|
||||
replaces, and deliberately does not stub `bundle.ts`, which failed it.
|
||||
|
||||
## AI Benchmark Caveats
|
||||
|
||||
The repo-level benchmark CLI lives under `ai_evals/`, but it currently depends on
|
||||
|
||||
+106
-29
@@ -119,9 +119,10 @@ export function isExecutionModeAnonymous(app: any) {
|
||||
export function isExecutionModeGuest(app: any) {
|
||||
return app?.["policy"]?.["execution_mode"] == "guest";
|
||||
}
|
||||
export type AppExecutionMode = "anonymous" | "guest" | "publisher";
|
||||
export type AppExecutionMode = "anonymous" | "guest" | "publisher" | "viewer";
|
||||
/** The access mode is the one policy field a tracked app keeps, as `public` (anonymous)
|
||||
* or `guests` (guest); the rest of the policy is regenerated on push. */
|
||||
* or `guests` (guest); the rest of the policy is preserved from the deployed app on
|
||||
* push (see `generatingPolicy`). */
|
||||
export function markAccessFromPolicy(app: any) {
|
||||
if (isExecutionModeAnonymous(app)) {
|
||||
app.public = true;
|
||||
@@ -129,14 +130,35 @@ export function markAccessFromPolicy(app: any) {
|
||||
app.guests = true;
|
||||
}
|
||||
}
|
||||
export function executionModeFromAppFile(app: any): AppExecutionMode {
|
||||
/** The mode the tracked file states, or `undefined` when it states none — the
|
||||
* normal case, since a pull writes only the two open-access markers. `viewer`
|
||||
* and `publisher` have no marker of their own, so a file can only name them
|
||||
* through a policy block it was hand-written with. */
|
||||
function statedExecutionMode(app: any): AppExecutionMode | undefined {
|
||||
if (app?.["public"] ?? isExecutionModeAnonymous(app)) {
|
||||
return "anonymous";
|
||||
}
|
||||
if (app?.["guests"] ?? isExecutionModeGuest(app)) {
|
||||
return "guest";
|
||||
}
|
||||
return "publisher";
|
||||
const mode = app?.["policy"]?.["execution_mode"];
|
||||
return mode === "viewer" || mode === "publisher" ? mode : undefined;
|
||||
}
|
||||
|
||||
/** The mode this push deploys under. A file that states one is authoritative, in
|
||||
* both directions. Otherwise the two open-access markers are all it says, so
|
||||
* their absence closes a deployed `anonymous`/`guest` app back down to
|
||||
* `publisher` — while a deployed `viewer` is not a grant those markers revoke,
|
||||
* so it carries over rather than widening to `publisher`. */
|
||||
export function executionModeForPush(
|
||||
localApp: any,
|
||||
deployedPolicy: Policy | undefined,
|
||||
): AppExecutionMode {
|
||||
const stated = statedExecutionMode(localApp);
|
||||
if (stated) {
|
||||
return stated;
|
||||
}
|
||||
return deployedPolicy?.execution_mode === "viewer" ? "viewer" : "publisher";
|
||||
}
|
||||
export async function pushApp(
|
||||
workspace: string,
|
||||
@@ -161,12 +183,9 @@ export async function pushApp(
|
||||
//ignore
|
||||
}
|
||||
|
||||
let remoteOnBehalfOf: string | undefined;
|
||||
let remoteOnBehalfOfEmail: string | undefined;
|
||||
if (app?.policy) {
|
||||
remoteOnBehalfOf = app.policy.on_behalf_of;
|
||||
remoteOnBehalfOfEmail = app.policy.on_behalf_of_email;
|
||||
}
|
||||
// `app.policy` is cleared a few lines down, so capture it first: it is the
|
||||
// base the regenerated policy is built on.
|
||||
const deployedPolicy: Policy | undefined = app?.policy;
|
||||
|
||||
markAccessFromPolicy(app);
|
||||
// console.log(app);
|
||||
@@ -181,20 +200,18 @@ export async function pushApp(
|
||||
const localApp = (await yamlParseFile(path)) as AppFile;
|
||||
|
||||
replaceInlineScripts(localApp.value, localPath, true);
|
||||
await generatingPolicy(localApp, remotePath, executionModeFromAppFile(localApp));
|
||||
|
||||
const preserveFields: { preserve_on_behalf_of?: boolean } = {};
|
||||
if (permissionedAsContext?.userIsAdminOrDeployer) {
|
||||
if (app) {
|
||||
if (localApp.policy && remoteOnBehalfOf) {
|
||||
(localApp.policy as any).on_behalf_of = remoteOnBehalfOf;
|
||||
(localApp.policy as any).on_behalf_of_email = remoteOnBehalfOfEmail;
|
||||
preserveFields.preserve_on_behalf_of = true;
|
||||
log.info(`Preserving ${remoteOnBehalfOfEmail ?? remoteOnBehalfOf} as permissioned_as for app ${remotePath}`);
|
||||
}
|
||||
}
|
||||
// On create: backend applies folder defaults
|
||||
}
|
||||
// On create the backend applies folder defaults, so there is nothing to preserve.
|
||||
const preserveFields = preserveOnBehalfOfFields(
|
||||
remotePath,
|
||||
deployedPolicy,
|
||||
permissionedAsContext
|
||||
);
|
||||
await generatingPolicy(
|
||||
localApp,
|
||||
remotePath,
|
||||
executionModeForPush(localApp, deployedPolicy),
|
||||
basePolicy(localApp, deployedPolicy, !!preserveFields.preserve_on_behalf_of)
|
||||
);
|
||||
|
||||
// extra_perms goes through /acls/* — strip from the body so a perms-only
|
||||
// edit never bumps the app version (see applyExtraPermsDiff for details).
|
||||
@@ -251,18 +268,76 @@ export async function pushApp(
|
||||
export async function generatingPolicy(
|
||||
app: any,
|
||||
path: string,
|
||||
executionMode: AppExecutionMode
|
||||
executionMode: AppExecutionMode,
|
||||
base: Policy | undefined
|
||||
) {
|
||||
log.info(colors.gray(`Generating fresh policy for app ${path}...`));
|
||||
try {
|
||||
app.policy = await windmillUtils.updatePolicy(app.value, undefined);
|
||||
app.policy.execution_mode = executionMode;
|
||||
app.policy = await windmillUtils.updatePolicy(app.value, base);
|
||||
finalizeDerivedPolicy(app.policy, executionMode);
|
||||
} catch (e) {
|
||||
log.error(colors.red(`Error generating policy for app ${path}: ${e}`));
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
/** What the regenerated policy starts from: the deployed one, so a push keeps
|
||||
* settings the tracked file doesn't record; on a first push, whatever the file
|
||||
* states. The run identity rides along only when `claimsOnBehalfOf` — never
|
||||
* from the file, never from a pusher who may not preserve one, since `wmill`
|
||||
* is regularly pointed at servers older than the rewrite that would fix it. */
|
||||
export function basePolicy(
|
||||
localApp: any,
|
||||
deployedPolicy: Policy | undefined,
|
||||
claimsOnBehalfOf: boolean
|
||||
): Policy | undefined {
|
||||
const stated = deployedPolicy ?? (localApp?.policy as Policy | undefined);
|
||||
if (!stated || claimsOnBehalfOf) {
|
||||
return stated;
|
||||
}
|
||||
const base: Policy = { ...stated };
|
||||
delete base.on_behalf_of;
|
||||
delete base.on_behalf_of_email;
|
||||
return base;
|
||||
}
|
||||
|
||||
/** Claim the run-as identity the regenerated policy carries over from the
|
||||
* deployed app. Only a deployed identity may be claimed, never one the tracked
|
||||
* file states — a repo doesn't get to pick who an app runs as. Without the flag
|
||||
* the backend rewrites `on_behalf_of` to whoever is pushing, and it only honors
|
||||
* the flag for an admin or a `wm_deployers` member, so a caller who is neither
|
||||
* doesn't get to claim it here either. */
|
||||
export function preserveOnBehalfOfFields(
|
||||
remotePath: string,
|
||||
deployedPolicy: Policy | undefined,
|
||||
permissionedAsContext: PermissionedAsContext | undefined
|
||||
): { preserve_on_behalf_of?: boolean } {
|
||||
const onBehalfOf = deployedPolicy?.on_behalf_of;
|
||||
if (!permissionedAsContext?.userIsAdminOrDeployer || !onBehalfOf) {
|
||||
return {};
|
||||
}
|
||||
log.info(
|
||||
`Preserving ${deployedPolicy?.on_behalf_of_email ?? onBehalfOf} as permissioned_as for app ${remotePath}`
|
||||
);
|
||||
return { preserve_on_behalf_of: true };
|
||||
}
|
||||
|
||||
/** The policy is written wholesale by the deploy, so the fields it does not
|
||||
* derive from the tracked sources have to survive the trip. The policy builder
|
||||
* has already recomputed what it can — the triggerables on both paths, plus the
|
||||
* S3 rules on the low-code one, which `updateRawAppPolicy` has no equivalent of
|
||||
* and so carries over. This sets the two left: the access mode, and the legacy
|
||||
* `triggerables`, which still grant execution (the backend folds them into
|
||||
* `triggerables_v2` at run time) and so are dropped rather than carried, or a
|
||||
* deployed app would keep being able to run runnables this push removed. */
|
||||
export function finalizeDerivedPolicy(
|
||||
policy: Policy,
|
||||
executionMode: AppExecutionMode
|
||||
) {
|
||||
policy.triggerables = undefined;
|
||||
policy.execution_mode = executionMode;
|
||||
}
|
||||
|
||||
async function list(opts: GlobalOptions & { includeDraftOnly?: boolean; json?: boolean }) {
|
||||
const workspace = await resolveWorkspace(opts);
|
||||
await requireLogin(opts);
|
||||
@@ -425,14 +500,16 @@ async function push(
|
||||
if (isRawAppByName || hasRawAppYaml) {
|
||||
const { pushRawApp } = await import("./raw_apps.ts");
|
||||
const merged = await mergeConfigWithConfigFile(opts);
|
||||
// Raw-app ownership preservation is not implemented on either push
|
||||
// path: sync push hands pushRawApp no context either.
|
||||
await pushRawApp(
|
||||
workspace.workspaceId,
|
||||
remotePath,
|
||||
absoluteFilePath,
|
||||
undefined,
|
||||
merged.defaultTs,
|
||||
await buildPermissionedAsContext(
|
||||
workspace.workspaceId,
|
||||
await readEffectiveSyncBehavior(opts, workspace),
|
||||
),
|
||||
);
|
||||
log.info(colors.bold.underline.green("Raw app pushed"));
|
||||
} else {
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
import { requireLogin } from "../../core/auth.ts";
|
||||
import { resolveWorkspace, validatePath } from "../../core/context.ts";
|
||||
import { mergeConfigWithConfigFile } from "../../core/conf.ts";
|
||||
import {
|
||||
mergeConfigWithConfigFile,
|
||||
readEffectiveSyncBehavior,
|
||||
} from "../../core/conf.ts";
|
||||
import { colors } from "@cliffy/ansi/colors";
|
||||
import * as log from "../../core/log.ts";
|
||||
import { sep as SEP } from "node:path";
|
||||
@@ -17,13 +20,22 @@ import { deepEqual, readTextFile } from "../../utils/utils.ts";
|
||||
|
||||
import {
|
||||
type AppExecutionMode,
|
||||
executionModeFromAppFile,
|
||||
basePolicy,
|
||||
executionModeForPush,
|
||||
finalizeDerivedPolicy,
|
||||
markAccessFromPolicy,
|
||||
preserveOnBehalfOfFields,
|
||||
replaceInlineScripts,
|
||||
repopulateFields,
|
||||
} from "./app.ts";
|
||||
import type { PermissionedAsContext } from "../../core/permissioned_as.ts";
|
||||
import { buildPermissionedAsContext } from "../../core/permissioned_as.ts";
|
||||
import { createBundle, detectFrameworks } from "./bundle.ts";
|
||||
import { APP_BACKEND_FOLDER, RECORDINGS_FOLDER } from "./app_metadata.ts";
|
||||
import {
|
||||
NEVER_DEPLOYED_DIRS,
|
||||
NEVER_DEPLOYED_FILES,
|
||||
} from "../../utils/app_files.ts";
|
||||
import { writeIfChanged } from "../../utils/utils.ts";
|
||||
import { yamlOptions } from "../sync/sync.ts";
|
||||
import { applyExtraPermsDiff } from "../../core/extra_perms.ts";
|
||||
@@ -316,13 +328,11 @@ async function collectAppFiles(
|
||||
const relativePath = basePath + entry.name;
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
// Skip the runnables, node_modules, and sql_to_apply subfolders
|
||||
// The backend folder deploys as `value.runnables`, not as a bundled
|
||||
// file; the rest reach the server through no channel at all.
|
||||
if (
|
||||
entry.name === APP_BACKEND_FOLDER ||
|
||||
entry.name === "node_modules" ||
|
||||
entry.name === "dist" ||
|
||||
entry.name === ".claude" ||
|
||||
entry.name === "sql_to_apply"
|
||||
NEVER_DEPLOYED_DIRS.has(entry.name)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
@@ -334,13 +344,11 @@ async function collectAppFiles(
|
||||
}
|
||||
await readDirRecursive(fullPath + SEP, relativePath + "/");
|
||||
} else if (entry.isFile()) {
|
||||
// Skip generated/metadata files that shouldn't be part of the app
|
||||
// `raw_app.yaml` deploys as the request's metadata rather than as a
|
||||
// bundled file; the rest reach the server through no channel at all.
|
||||
if (
|
||||
entry.name === "raw_app.yaml" ||
|
||||
entry.name === "package-lock.json" ||
|
||||
entry.name === "DATATABLES.md" ||
|
||||
entry.name === "AGENTS.md" ||
|
||||
entry.name === "wmill.d.ts"
|
||||
NEVER_DEPLOYED_FILES.has(entry.name)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
@@ -360,6 +368,7 @@ export async function pushRawApp(
|
||||
localPath: string,
|
||||
message?: string,
|
||||
defaultTs: "bun" | "deno" = "bun",
|
||||
permissionedAsContext?: PermissionedAsContext,
|
||||
): Promise<void> {
|
||||
if (alreadySynced.includes(localPath)) {
|
||||
return;
|
||||
@@ -376,6 +385,10 @@ export async function pushRawApp(
|
||||
} catch {
|
||||
//ignore
|
||||
}
|
||||
// `app.policy` is cleared a few lines down, so capture it first. `raw_app.yaml`
|
||||
// records none of the policy, so anything the deploy drawer set is only here.
|
||||
const deployedPolicy: Policy | undefined = app?.policy;
|
||||
|
||||
markAccessFromPolicy(app);
|
||||
// console.log(app);
|
||||
if (app) {
|
||||
@@ -424,10 +437,21 @@ export async function pushRawApp(
|
||||
|
||||
// Create a temporary app object for policy generation
|
||||
const appForPolicy = { ...localApp, runnables };
|
||||
// On create the backend applies folder defaults, so there is nothing to preserve.
|
||||
const preserveFields = preserveOnBehalfOfFields(
|
||||
remotePath,
|
||||
deployedPolicy,
|
||||
permissionedAsContext,
|
||||
);
|
||||
await generatingPolicy(
|
||||
appForPolicy,
|
||||
remotePath,
|
||||
executionModeFromAppFile(localApp),
|
||||
executionModeForPush(localApp, deployedPolicy),
|
||||
basePolicy(
|
||||
localApp,
|
||||
deployedPolicy,
|
||||
!!preserveFields.preserve_on_behalf_of,
|
||||
),
|
||||
);
|
||||
|
||||
const files = await collectAppFiles(localPath);
|
||||
@@ -482,6 +506,7 @@ export async function pushRawApp(
|
||||
path: remotePath,
|
||||
summary: localApp.summary,
|
||||
policy: appForPolicy.policy,
|
||||
...preserveFields,
|
||||
deployment_message: message,
|
||||
// Preserve any user draft at this path (see backend skip_draft_deletion).
|
||||
skip_draft_deletion: true,
|
||||
@@ -532,14 +557,12 @@ export async function generatingPolicy(
|
||||
app: any,
|
||||
path: string,
|
||||
executionMode: AppExecutionMode,
|
||||
base: Policy | undefined,
|
||||
) {
|
||||
log.info(colors.gray(`Generating fresh policy for app ${path}...`));
|
||||
try {
|
||||
app.policy = await windmillUtils.updateRawAppPolicy(
|
||||
app.runnables,
|
||||
app.policy,
|
||||
);
|
||||
app.policy.execution_mode = executionMode;
|
||||
app.policy = await windmillUtils.updateRawAppPolicy(app.runnables, base);
|
||||
finalizeDerivedPolicy(app.policy, executionMode);
|
||||
} catch (e) {
|
||||
log.error(colors.red(`Error generating policy for app ${path}: ${e}`));
|
||||
throw e;
|
||||
@@ -564,6 +587,10 @@ async function pushRawAppCommand(
|
||||
filePath,
|
||||
undefined,
|
||||
merged.defaultTs,
|
||||
await buildPermissionedAsContext(
|
||||
workspace.workspaceId,
|
||||
await readEffectiveSyncBehavior(opts, workspace),
|
||||
),
|
||||
);
|
||||
log.info(colors.bold.underline.green("Raw app pushed"));
|
||||
}
|
||||
|
||||
@@ -31,10 +31,18 @@ import {
|
||||
import {
|
||||
isFlowInlineScriptPath,
|
||||
isAppInlineScriptPath,
|
||||
isRawAppPath,
|
||||
isFolderResourcePathAnyFormat,
|
||||
getFolderSuffix,
|
||||
getScriptBasePathFromModulePath,
|
||||
} from "../../utils/resource_folders.ts";
|
||||
import { exts } from "../script/script.ts";
|
||||
import { isFilesetResource } from "../../utils/utils.ts";
|
||||
import {
|
||||
exts,
|
||||
findContentFile,
|
||||
hasScriptExt,
|
||||
isModuleEntryMetadata,
|
||||
UnresolvableScriptContentFileError,
|
||||
} from "../script/script.ts";
|
||||
|
||||
interface LintOptions extends GlobalOptions {
|
||||
json?: boolean;
|
||||
@@ -67,6 +75,9 @@ export interface LintReport {
|
||||
|
||||
const YAML_FILE_REGEX = /\.ya?ml$/i;
|
||||
const NATIVE_TRIGGER_REGEX = /\.[^.]+_native_trigger\.ya?ml$/i;
|
||||
// The metadata suffixes `findContentFile` resolves a flat script from. `.yml` is
|
||||
// deliberately absent, since the push does not accept it there either.
|
||||
const FLAT_SCRIPT_METADATA_REGEX = /\.script\.(yaml|json|lock)$/;
|
||||
|
||||
function normalizePath(p: string): string {
|
||||
return p.replaceAll(SEP, "/");
|
||||
@@ -643,6 +654,83 @@ export async function checkMissingLocks(
|
||||
return issues;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a path is a script's own metadata, as opposed to metadata the push
|
||||
* deploys through some parent: a folder resource's inline scripts, a fileset's
|
||||
* children (arbitrarily named, so one may be spelled exactly like a script's
|
||||
* metadata) and the files of a module or dbt bundle all belong to that parent.
|
||||
*
|
||||
* Takes the path as the SYNC ROOT spells it, like the push. Relative to the
|
||||
* lint target the enclosing folder is gone whenever the target IS that folder;
|
||||
* absolute, the classifiers match their suffixes ANYWHERE in the string, so a
|
||||
* checkout under `acme.app` reads as one app and nothing is ever reported.
|
||||
*/
|
||||
function isStandaloneScriptMetadata(rootedPath: string): boolean {
|
||||
// Both suffix formats, because the dotted/non-dotted setting is read from the
|
||||
// invocation directory and an explicit lint target may not share it.
|
||||
if (
|
||||
isFolderResourcePathAnyFormat(rootedPath) ||
|
||||
isFilesetResource(rootedPath)
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
// A module folder keeps its metadata inside itself (`<base>__mod/script.yaml`),
|
||||
// which is standalone even though every other path under `__mod/` is not.
|
||||
if (isModuleEntryMetadata(rootedPath)) return true;
|
||||
if (getScriptBasePathFromModulePath(rootedPath) !== undefined) return false;
|
||||
return FLAT_SCRIPT_METADATA_REGEX.test(rootedPath);
|
||||
}
|
||||
|
||||
/**
|
||||
* `findContentFile` quotes the paths it was given back in its errors, so the
|
||||
* lint target's own prefix comes off them again. Anchored at a path start: a
|
||||
* plain substring replace of `f/` also eats the one inside `conf/`, mangling
|
||||
* the very filename the message is telling the reader to delete.
|
||||
*/
|
||||
function relativizeMessage(message: string, prefix: string): string {
|
||||
if (!prefix) return message;
|
||||
const escaped = prefix.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
return message.replaceAll(new RegExp(`(^|[\\s(])${escaped}/`, "g"), "$1");
|
||||
}
|
||||
|
||||
/**
|
||||
* Script metadata files that cannot be paired with exactly one content file:
|
||||
* the push refuses those, and no metadata format makes them deployable, so the
|
||||
* inactive twin of a format switch (`foo.script.json` in a yaml repo) is dead
|
||||
* weight worth reporting even though the push skips it rather than refusing it.
|
||||
*
|
||||
* Resolved through `findContentFile` rather than by probing `exts` directly, so
|
||||
* lint and push agree on what counts as paired: a dbt project's descriptor is
|
||||
* optional and its absence is not an orphan, while two content files beside one
|
||||
* metadata file is just as undeployable as none. It classifies what it is given
|
||||
* and looks under `syncRoot`, so where the command was invoked from is not part
|
||||
* of the answer.
|
||||
*/
|
||||
async function checkOrphanScriptMetadata(
|
||||
syncRoot: string,
|
||||
prefix: string,
|
||||
metadataPaths: string[],
|
||||
): Promise<FileIssue[]> {
|
||||
const issues: FileIssue[] = [];
|
||||
for (const metadataPath of metadataPaths) {
|
||||
const rootedPath = prefix ? `${prefix}/${metadataPath}` : metadataPath;
|
||||
try {
|
||||
await findContentFile(rootedPath, syncRoot);
|
||||
} catch (e) {
|
||||
if (!(e instanceof UnresolvableScriptContentFileError)) {
|
||||
log.debug(`Failed to resolve content file for ${rootedPath}: ${e}`);
|
||||
continue;
|
||||
}
|
||||
issues.push({
|
||||
path: metadataPath,
|
||||
target: "script",
|
||||
errors: [relativizeMessage(e.message, prefix)],
|
||||
});
|
||||
}
|
||||
}
|
||||
return issues;
|
||||
}
|
||||
|
||||
export async function runLint(
|
||||
opts: LintOptions,
|
||||
directory?: string,
|
||||
@@ -674,8 +762,16 @@ export async function runLint(
|
||||
const root = await FSFSElement(targetDirectory, [], false);
|
||||
const validator = new WindmillYamlValidator();
|
||||
|
||||
// Walked paths are relative to the lint target; this puts them back the way
|
||||
// the sync root spells them, which is what the two below are written against.
|
||||
const syncRoot = await findSyncRoot(targetDirectory);
|
||||
const metadataPrefix = normalizePath(
|
||||
path.relative(syncRoot, targetDirectory),
|
||||
);
|
||||
|
||||
const warnings: LintWarning[] = [];
|
||||
const issues: FileIssue[] = [];
|
||||
const scriptMetadataPaths: string[] = [];
|
||||
let scannedFiles = 0;
|
||||
let validatedFiles = 0;
|
||||
let validFiles = 0;
|
||||
@@ -689,6 +785,17 @@ export async function runLint(
|
||||
const normalizedPath = normalizePath(entry.path);
|
||||
|
||||
scannedFiles += 1;
|
||||
|
||||
// Collected before the YAML filter below: `.script.lock` and `.script.json`
|
||||
// are metadata too, and both fail the push when nothing pairs with them.
|
||||
if (
|
||||
isStandaloneScriptMetadata(
|
||||
metadataPrefix ? `${metadataPrefix}/${normalizedPath}` : normalizedPath,
|
||||
)
|
||||
) {
|
||||
scriptMetadataPaths.push(normalizedPath);
|
||||
}
|
||||
|
||||
if (!YAML_FILE_REGEX.test(normalizedPath)) {
|
||||
continue;
|
||||
}
|
||||
@@ -727,6 +834,16 @@ export async function runLint(
|
||||
}
|
||||
}
|
||||
|
||||
// Unconditional: unlike a missing lock, metadata with no content file fails
|
||||
// every push, so there is no mode in which it is acceptable.
|
||||
issues.push(
|
||||
...(await checkOrphanScriptMetadata(
|
||||
syncRoot,
|
||||
metadataPrefix,
|
||||
scriptMetadataPaths,
|
||||
)),
|
||||
);
|
||||
|
||||
// Check for missing locks if --locks-required is set
|
||||
if (opts.locksRequired) {
|
||||
const lockIssues = await checkMissingLocks(opts, explicitTargetDirectory);
|
||||
@@ -820,6 +937,15 @@ async function lint(opts: LintOptions & { watch?: boolean }, directory?: string)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a changed file can change what a lint run reports: metadata in any of
|
||||
* its formats, and the content files whose presence is what keeps that metadata
|
||||
* from being an orphan.
|
||||
*/
|
||||
function affectsLint(filename: string): boolean {
|
||||
return /\.(ya?ml|json|lock)$/i.test(filename) || hasScriptExt(filename);
|
||||
}
|
||||
|
||||
async function lintWatch(opts: LintOptions, directory?: string) {
|
||||
const { watch } = await import("node:fs");
|
||||
const targetDir = directory ? path.resolve(process.cwd(), directory) : process.cwd();
|
||||
@@ -842,7 +968,7 @@ async function lintWatch(opts: LintOptions, directory?: string) {
|
||||
|
||||
let debounce: ReturnType<typeof setTimeout> | null = null;
|
||||
watch(targetDir, { recursive: true }, (_event, filename) => {
|
||||
if (!filename || !filename.toString().endsWith(".yaml") && !filename.toString().endsWith(".yml")) return;
|
||||
if (!filename || !affectsLint(filename.toString())) return;
|
||||
if (debounce) clearTimeout(debounce);
|
||||
debounce = setTimeout(runAndReport, 300);
|
||||
});
|
||||
@@ -853,7 +979,7 @@ async function lintWatch(opts: LintOptions, directory?: string) {
|
||||
|
||||
const command = new Command()
|
||||
.description(
|
||||
"Validate Windmill flow, schedule, and trigger YAML files in a directory",
|
||||
"Validate Windmill flow, schedule, and trigger YAML files in a directory, and report script metadata that has no deployable content file",
|
||||
)
|
||||
.arguments("[directory:string]")
|
||||
.option("--json", "Output results in JSON format")
|
||||
|
||||
@@ -1075,10 +1075,12 @@ export class DbtPathCollisionError extends UnresolvableScriptContentFileError {}
|
||||
* guard on one of them leaves the other silently overwriting.
|
||||
*/
|
||||
export async function collidingDbtProject(
|
||||
basePath: string
|
||||
basePath: string,
|
||||
baseDir?: string
|
||||
): Promise<string | undefined> {
|
||||
const project = basePath + "__dbt/dbt_project.yml";
|
||||
return (await stat(project).then(() => true).catch(() => false))
|
||||
const onDisk = baseDir ? path.join(baseDir, project) : project;
|
||||
return (await stat(onDisk).then(() => true).catch(() => false))
|
||||
? project
|
||||
: undefined;
|
||||
}
|
||||
@@ -1139,7 +1141,14 @@ async function readScriptContent(filePath: string): Promise<string> {
|
||||
}
|
||||
}
|
||||
|
||||
export async function findContentFile(filePath: string) {
|
||||
/**
|
||||
* The script file `filePath`'s metadata belongs to. `baseDir`, when given, is
|
||||
* where the disk lookups happen, leaving `filePath` classified as written: the
|
||||
* layout helpers below match their suffixes ANYWHERE in a path, so a caller
|
||||
* that prefixed a checkout named `repo__mod` would have it read as the module.
|
||||
*/
|
||||
export async function findContentFile(filePath: string, baseDir?: string) {
|
||||
const onDisk = (p: string) => (baseDir ? path.join(baseDir, p) : p);
|
||||
// Folder layout: __mod/script.yaml -> __mod/script.ts
|
||||
const isModuleFolderMeta = isModuleEntryMetadata(filePath);
|
||||
const toCandidate = (ext: string) =>
|
||||
@@ -1163,7 +1172,7 @@ export async function findContentFile(filePath: string) {
|
||||
const validCandidates = (
|
||||
await Promise.all(
|
||||
candidates.map((x) => {
|
||||
return stat(x)
|
||||
return stat(onDisk(x))
|
||||
.catch(() => undefined)
|
||||
.then((x) => x?.isFile())
|
||||
.then((e) => {
|
||||
@@ -1183,6 +1192,7 @@ export async function findContentFile(filePath: string) {
|
||||
const dbtCandidate = toCandidate("__dbt/" + DBT_DESCRIPTOR_NAME);
|
||||
const dbtProject = await collidingDbtProject(
|
||||
dbtCandidate.slice(0, -("__dbt/" + DBT_DESCRIPTOR_NAME).length),
|
||||
baseDir,
|
||||
);
|
||||
const nonDbtCandidates = validCandidates.filter((c) => c !== dbtCandidate);
|
||||
if (dbtProject && nonDbtCandidates.length > 0) {
|
||||
|
||||
@@ -152,6 +152,7 @@ import {
|
||||
generateAppLocksInternal,
|
||||
RECORDINGS_FOLDER,
|
||||
} from "../app/app_metadata.ts";
|
||||
import { deploysWithRawApp } from "../../utils/app_files.ts";
|
||||
import {
|
||||
isFlowPath,
|
||||
isAppPath,
|
||||
@@ -2018,20 +2019,18 @@ export async function elementsToMap(
|
||||
}
|
||||
|
||||
if (isRawAppFile(path)) {
|
||||
// FSFSElement builds paths with the platform separator, while the checks
|
||||
// below are written with "/": without normalizing, none of them match on
|
||||
// Windows and the push collector's own exclusions become perpetual diffs.
|
||||
// FSFSElement builds paths with the platform separator, while
|
||||
// `deploysWithRawApp` is written with "/": without normalizing it matches
|
||||
// nothing on Windows and the push collector's own exclusions become
|
||||
// perpetual diffs.
|
||||
const suffix = path
|
||||
.split(getFolderSuffix("raw_app") + SEP)
|
||||
.pop()
|
||||
?.replaceAll(SEP, "/");
|
||||
if (
|
||||
suffix?.startsWith("dist/") ||
|
||||
suffix?.startsWith(RECORDINGS_FOLDER + "/") ||
|
||||
suffix == "wmill.d.ts" ||
|
||||
suffix == "package-lock.json" ||
|
||||
suffix == "DATATABLES.md"
|
||||
) {
|
||||
// A file no push sends is not a change to track. Listing it leaves it
|
||||
// pending forever — nothing ever uploads it — and pushing it redeploys
|
||||
// the whole app, reassigning its run-as user, to ship nothing.
|
||||
if (suffix && !deploysWithRawApp(suffix)) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
@@ -6142,7 +6141,7 @@ export async function push(
|
||||
undefined,
|
||||
opts.plainSecrets ?? false,
|
||||
alreadySynced,
|
||||
{ message: opts.message },
|
||||
{ message: opts.message, permissionedAsContext },
|
||||
);
|
||||
} else {
|
||||
// Flow folder doesn't exist locally — delete on server
|
||||
@@ -6187,7 +6186,7 @@ export async function push(
|
||||
undefined,
|
||||
opts.plainSecrets ?? false,
|
||||
alreadySynced,
|
||||
{ message: opts.message },
|
||||
{ message: opts.message, permissionedAsContext },
|
||||
);
|
||||
} else {
|
||||
// App folder doesn't exist locally — delete on server
|
||||
@@ -6233,7 +6232,11 @@ export async function push(
|
||||
undefined,
|
||||
opts.plainSecrets ?? false,
|
||||
alreadySynced,
|
||||
{ message: opts.message, defaultTs: opts.defaultTs },
|
||||
{
|
||||
message: opts.message,
|
||||
defaultTs: opts.defaultTs,
|
||||
permissionedAsContext,
|
||||
},
|
||||
);
|
||||
} else {
|
||||
// The entire raw app folder was deleted locally,
|
||||
|
||||
@@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork";
|
||||
// (e.g. utils.ts) can read it without importing main.ts and creating a circular
|
||||
// dependency (main → workspace → utils → main) that triggers a TDZ.
|
||||
// Re-exported from main.ts for backwards compatibility.
|
||||
export const VERSION = "1.805.0";
|
||||
export const VERSION = "1.809.0";
|
||||
|
||||
@@ -3,6 +3,12 @@ import * as log from "./log.ts";
|
||||
import { colors } from "@cliffy/ansi/colors";
|
||||
import { Confirm } from "@cliffy/prompt/confirm";
|
||||
import { getTypeStrFromPath } from "../types.ts";
|
||||
import {
|
||||
extractFolderPath,
|
||||
isAppFolderMetadataFile,
|
||||
isRawAppFolderMetadataFile,
|
||||
} from "../utils/resource_folders.ts";
|
||||
import { deploysWithRawApp } from "../utils/app_files.ts";
|
||||
import { parseSyncBehavior } from "./conf.ts";
|
||||
|
||||
export interface PermissionedAsContext {
|
||||
@@ -89,6 +95,45 @@ function contentHasOnBehalfOf(content: string, typeStr: string): boolean {
|
||||
return false;
|
||||
}
|
||||
|
||||
type AppTypeStr = "app" | "raw_app";
|
||||
|
||||
function isAppTypeStr(typeStr: string): typeStr is AppTypeStr {
|
||||
return typeStr === "app" || typeStr === "raw_app";
|
||||
}
|
||||
|
||||
/** The app folder a file belongs to. `isAppFolderMetadataFile` and its raw twin
|
||||
* match a literal `/`, unlike `extractFolderPath` — so normalize before either,
|
||||
* or a Windows path takes a different branch from the same file on Linux. */
|
||||
function appFolderOf(path: string, typeStr: AppTypeStr): string {
|
||||
return extractFolderPath(path, typeStr) ?? path;
|
||||
}
|
||||
|
||||
function toPosix(path: string): string {
|
||||
return path.replaceAll("\\", "/");
|
||||
}
|
||||
|
||||
/** App folders whose own metadata file is being added or deleted, which is how a
|
||||
* whole app arrives or goes rather than being redeployed. Neither takes an owner
|
||||
* over: a create has none yet, and a delete leaves none behind. */
|
||||
function appsArrivingOrLeaving(changes: Change[]): Set<string> {
|
||||
const folders = new Set<string>();
|
||||
for (const change of changes) {
|
||||
if (change.name === "edited") continue;
|
||||
const path = toPosix(change.path);
|
||||
if (!isAppFolderMetadataFile(path) && !isRawAppFolderMetadataFile(path)) {
|
||||
continue;
|
||||
}
|
||||
let typeStr: string;
|
||||
try {
|
||||
typeStr = getTypeStrFromPath(path);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (isAppTypeStr(typeStr)) folders.add(appFolderOf(path, typeStr));
|
||||
}
|
||||
return folders;
|
||||
}
|
||||
|
||||
export async function preCheckPermissionedAs(
|
||||
changes: Change[],
|
||||
userEmail: string,
|
||||
@@ -101,6 +146,12 @@ export async function preCheckPermissionedAs(
|
||||
if (userIsAdminOrDeployer) return;
|
||||
|
||||
const wouldChangeItems: { path: string; currentOwner: string }[] = [];
|
||||
const addItem = (item: { path: string; currentOwner: string }) => {
|
||||
if (!wouldChangeItems.some((i) => i.path === item.path)) {
|
||||
wouldChangeItems.push(item);
|
||||
}
|
||||
};
|
||||
const arrivingOrLeaving = appsArrivingOrLeaving(changes);
|
||||
|
||||
for (const change of changes) {
|
||||
let typeStr: string;
|
||||
@@ -110,6 +161,22 @@ export async function preCheckPermissionedAs(
|
||||
continue;
|
||||
}
|
||||
|
||||
// An app is redeployed whole by any change to any of the files it actually
|
||||
// sends — added, edited or deleted alike — so its policy is rewritten
|
||||
// regardless of what the file holds. Settled here, before the content the
|
||||
// other kinds parse to find their owner, which an app has none of to parse.
|
||||
if (isAppTypeStr(typeStr)) {
|
||||
const path = toPosix(change.path);
|
||||
const folder = appFolderOf(path, typeStr);
|
||||
if (
|
||||
!arrivingOrLeaving.has(folder) &&
|
||||
(typeStr === "app" || deploysWithRawApp(path.slice(folder.length)))
|
||||
) {
|
||||
addItem({ path: folder, currentOwner: "(app policy owner)" });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (change.name === "added") {
|
||||
const content = change.content;
|
||||
if (!content) continue;
|
||||
@@ -130,11 +197,6 @@ export async function preCheckPermissionedAs(
|
||||
const label =
|
||||
typeStr === "script" ? "(script owner)" : "(flow owner)";
|
||||
wouldChangeItems.push({ path: change.path, currentOwner: label });
|
||||
} else if (typeStr === "app") {
|
||||
wouldChangeItems.push({
|
||||
path: change.path,
|
||||
currentOwner: "(app policy owner)",
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
@@ -177,12 +239,6 @@ export async function preCheckPermissionedAs(
|
||||
}
|
||||
}
|
||||
continue;
|
||||
} else if (typeStr === "app") {
|
||||
wouldChangeItems.push({
|
||||
path: change.path,
|
||||
currentOwner: "(app policy owner)",
|
||||
});
|
||||
continue;
|
||||
} else if (typeStr === "schedule") {
|
||||
const match = beforeContent.match(
|
||||
/email:\s*["']?([^\s"']+)["']?/
|
||||
|
||||
Generated
+32
-8
File diff suppressed because one or more lines are too long
+1
-1
@@ -234,7 +234,7 @@ export async function pushObj(
|
||||
if (!rawAppName) {
|
||||
throw new Error(`Could not extract raw app name from path: ${p}`);
|
||||
}
|
||||
await pushRawApp(workspace, rawAppName, buildFolderPath(rawAppName, "raw_app"), message, defaultTs);
|
||||
await pushRawApp(workspace, rawAppName, buildFolderPath(rawAppName, "raw_app"), message, defaultTs, permissionedAsContext);
|
||||
} else if (typeEnding === "folder") {
|
||||
await pushFolder(workspace, p, befObj, newObj);
|
||||
} else if (typeEnding === "variable") {
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
import {
|
||||
APP_BACKEND_FOLDER,
|
||||
RECORDINGS_FOLDER,
|
||||
} from "../commands/app/app_metadata.ts";
|
||||
|
||||
/** Directories under a raw app that no push sends. */
|
||||
const NEVER_DEPLOYED_DIRS = new Set([
|
||||
"node_modules",
|
||||
"dist",
|
||||
".claude",
|
||||
"sql_to_apply",
|
||||
]);
|
||||
|
||||
/** Files under a raw app that no push sends. */
|
||||
const NEVER_DEPLOYED_FILES = new Set([
|
||||
"package-lock.json",
|
||||
"DATATABLES.md",
|
||||
"AGENTS.md",
|
||||
"wmill.d.ts",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Whether an app-root-relative path (`/` separators, leading slash optional)
|
||||
* reaches the server through any of a push's three channels: `raw_app.yaml` as
|
||||
* metadata, the backend folder as runnables, the rest bundled by
|
||||
* `collectAppFiles`. A path this rejects deploys nothing, so changing it is not
|
||||
* a change to the app however much the sync diff lists it. `collectAppFiles`
|
||||
* must not drift from this — it reads the same two sets.
|
||||
*/
|
||||
export function deploysWithRawApp(relativePath: string): boolean {
|
||||
const segments = relativePath.split("/").filter(Boolean);
|
||||
if (segments.length === 0) return false;
|
||||
const name = segments[segments.length - 1];
|
||||
const dirs = segments.slice(0, -1);
|
||||
// The sets below describe the bundle, which never walks into the backend
|
||||
// folder — applying them there would strip a runnable whose file shares a
|
||||
// name (`backend/wmill.d.ts` is the runnable `wmill.d`). Depth 1 because
|
||||
// `loadRunnablesFromBackend` reads that folder's top level only.
|
||||
if (dirs[0] === APP_BACKEND_FOLDER) return dirs.length === 1;
|
||||
if (NEVER_DEPLOYED_FILES.has(name)) return false;
|
||||
if (dirs.some((d) => NEVER_DEPLOYED_DIRS.has(d))) return false;
|
||||
// Session recordings are written at the app root only, so an app with a
|
||||
// `recordings/` component folder of its own still ships it.
|
||||
if (dirs[0] === RECORDINGS_FOLDER) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
export { NEVER_DEPLOYED_DIRS, NEVER_DEPLOYED_FILES };
|
||||
@@ -1,6 +1,6 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import {
|
||||
executionModeFromAppFile,
|
||||
executionModeForPush,
|
||||
generatingPolicy,
|
||||
markAccessFromPolicy,
|
||||
} from "../src/commands/app/app.ts";
|
||||
@@ -13,16 +13,39 @@ test("the access mode survives the app.yaml round trip", async () => {
|
||||
guest.policy = undefined;
|
||||
expect(guest.guests).toBe(true);
|
||||
expect(guest.public).toBeUndefined();
|
||||
expect(executionModeFromAppFile(guest)).toBe("guest");
|
||||
await generatingPolicy(guest, "u/test/app", executionModeFromAppFile(guest));
|
||||
expect(executionModeForPush(guest, undefined)).toBe("guest");
|
||||
await generatingPolicy(
|
||||
guest,
|
||||
"u/test/app",
|
||||
executionModeForPush(guest, undefined),
|
||||
undefined,
|
||||
);
|
||||
expect(guest.policy.execution_mode).toBe("guest");
|
||||
|
||||
const anonymous: any = { policy: { execution_mode: "anonymous" }, value: {} };
|
||||
markAccessFromPolicy(anonymous);
|
||||
anonymous.policy = undefined;
|
||||
expect(anonymous.public).toBe(true);
|
||||
expect(executionModeFromAppFile(anonymous)).toBe("anonymous");
|
||||
expect(executionModeForPush(anonymous, undefined)).toBe("anonymous");
|
||||
|
||||
expect(executionModeFromAppFile({ policy: { execution_mode: "publisher" } })).toBe("publisher");
|
||||
expect(executionModeFromAppFile({})).toBe("publisher");
|
||||
expect(executionModeForPush({ policy: { execution_mode: "publisher" } }, undefined)).toBe("publisher");
|
||||
expect(executionModeForPush({}, undefined)).toBe("publisher");
|
||||
});
|
||||
|
||||
// `viewer` is the narrowest mode — each runnable runs as the viewer, not as the
|
||||
// app's identity — and the only one with no marker in the file, so both ways it
|
||||
// can reach a push must survive rather than widen to `publisher`.
|
||||
test("viewer is never widened to publisher by a push", () => {
|
||||
// Carried over from the deployed app: a pull writes no marker for it.
|
||||
expect(executionModeForPush({}, { execution_mode: "viewer" })).toBe("viewer");
|
||||
// Stated by the file, which is all a first push has to go on.
|
||||
expect(executionModeForPush({ policy: { execution_mode: "viewer" } }, undefined)).toBe("viewer");
|
||||
// The open-access markers still win, in either direction.
|
||||
expect(executionModeForPush({ public: true }, { execution_mode: "viewer" })).toBe("anonymous");
|
||||
expect(executionModeForPush({}, { execution_mode: "anonymous" })).toBe("publisher");
|
||||
// A stated mode is authoritative both ways: the carry-over is for a file that
|
||||
// says nothing, so it must not pin a deployed app to `viewer` forever.
|
||||
expect(
|
||||
executionModeForPush({ policy: { execution_mode: "publisher" } }, { execution_mode: "viewer" })
|
||||
).toBe("publisher");
|
||||
});
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
import { expect, test, describe } from "bun:test";
|
||||
import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import * as path from "node:path";
|
||||
import { runLint } from "../src/commands/lint/lint.ts";
|
||||
|
||||
const WMILL_YAML = "defaultTs: bun\nincludes:\n - f/**\nexcludes: []\n";
|
||||
const METADATA = "summary: test\nlock: ''\nschema:\n properties: {}\n";
|
||||
|
||||
async function write(dir: string, rel: string, content: string) {
|
||||
const full = path.join(dir, rel);
|
||||
await mkdir(path.dirname(full), { recursive: true });
|
||||
await writeFile(full, content, "utf-8");
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs `fn` with a sync root at `<temp>/<rootName>`, from which lint resolves
|
||||
* every walked path. The name is a parameter because it is load-bearing: the
|
||||
* folder suffixes lint classifies by (`.app`, `__mod`, …) are matched anywhere
|
||||
* in a path, so a root carrying one must not change what lint reports.
|
||||
*/
|
||||
async function withSyncRoot(
|
||||
rootName: string,
|
||||
fn: (syncRoot: string) => Promise<void>,
|
||||
opts: { runFromParent?: boolean } = {},
|
||||
): Promise<void> {
|
||||
const tempDir = await mkdtemp(path.join(os.tmpdir(), "wmill_lint_orphan_"));
|
||||
const syncRoot = path.join(tempDir, rootName);
|
||||
const originalCwd = process.cwd();
|
||||
try {
|
||||
await write(syncRoot, "wmill.yaml", WMILL_YAML);
|
||||
process.chdir(opts.runFromParent ? tempDir : syncRoot);
|
||||
await fn(syncRoot);
|
||||
} finally {
|
||||
process.chdir(originalCwd);
|
||||
await rm(tempDir, { recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
describe("orphan script metadata", () => {
|
||||
test("reports metadata with no content file, with locks not required", async () => {
|
||||
await withSyncRoot("repo", async (syncRoot) => {
|
||||
await write(syncRoot, "f/paired.py", "def main():\n pass\n");
|
||||
await write(syncRoot, "f/paired.script.yaml", METADATA);
|
||||
await write(syncRoot, "f/orphan.script.yaml", METADATA);
|
||||
await write(syncRoot, "f/orphan_json.script.json", "{}\n");
|
||||
await write(syncRoot, "f/orphan_lock.script.lock", "some-dep==1.0.0\n");
|
||||
|
||||
const report = await runLint({} as any, syncRoot);
|
||||
|
||||
expect(report.exitCode).toBe(1);
|
||||
expect(report.issues.map((i) => i.path).sort()).toEqual([
|
||||
"f/orphan.script.yaml",
|
||||
"f/orphan_json.script.json",
|
||||
"f/orphan_lock.script.lock",
|
||||
]);
|
||||
expect(report.issues[0].target).toBe("script");
|
||||
expect(report.issues[0].errors[0]).toContain("No script file found next to");
|
||||
});
|
||||
});
|
||||
|
||||
test("reports a module folder's own metadata with no content file", async () => {
|
||||
await withSyncRoot("repo", async (syncRoot) => {
|
||||
await write(syncRoot, "f/orphan__mod/script.yaml", METADATA);
|
||||
|
||||
const report = await runLint({} as any, syncRoot);
|
||||
|
||||
expect(report.issues.map((i) => i.path)).toEqual([
|
||||
"f/orphan__mod/script.yaml",
|
||||
]);
|
||||
|
||||
// Linting the module folder itself: the walked paths no longer carry the
|
||||
// `__mod/` boundary that says this is a module's metadata.
|
||||
const inFolder = await runLint(
|
||||
{} as any,
|
||||
path.join(syncRoot, "f/orphan__mod"),
|
||||
);
|
||||
|
||||
expect(inFolder.issues.map((i) => i.path)).toEqual(["script.yaml"]);
|
||||
});
|
||||
});
|
||||
|
||||
test("reports orphans under a sync root named like a resource folder", async () => {
|
||||
await withSyncRoot("acme.app", async (syncRoot) => {
|
||||
await write(syncRoot, "f/orphan.script.yaml", METADATA);
|
||||
|
||||
const report = await runLint({} as any, syncRoot);
|
||||
|
||||
expect(report.issues.map((i) => i.path)).toEqual(["f/orphan.script.yaml"]);
|
||||
});
|
||||
});
|
||||
|
||||
test("does not report a paired module under a sync root named like a module folder", async () => {
|
||||
// Run from OUTSIDE the checkout, the one invocation whose paths carry the
|
||||
// root's own name: nothing above the sync root may be classified.
|
||||
await withSyncRoot(
|
||||
"repo__mod",
|
||||
async (syncRoot) => {
|
||||
await write(syncRoot, "f/example__mod/script.yaml", METADATA);
|
||||
await write(
|
||||
syncRoot,
|
||||
"f/example__mod/script.ts",
|
||||
"export function main() {}\n",
|
||||
);
|
||||
|
||||
const report = await runLint({} as any, syncRoot);
|
||||
|
||||
expect(report.issues).toEqual([]);
|
||||
},
|
||||
{ runFromParent: true },
|
||||
);
|
||||
});
|
||||
|
||||
test("does not report a non-dotted folder resource's child", async () => {
|
||||
// The dotted/non-dotted setting is read from the invocation directory, so
|
||||
// an explicit target configured the other way must still be recognized.
|
||||
await withSyncRoot(
|
||||
"repo",
|
||||
async (syncRoot) => {
|
||||
await write(syncRoot, "f/a__raw_app/raw_app.yaml", "value: {}\n");
|
||||
await write(syncRoot, "f/a__raw_app/backend/config.script.lock", "x\n");
|
||||
|
||||
const report = await runLint({} as any, syncRoot);
|
||||
|
||||
expect(report.issues).toEqual([]);
|
||||
},
|
||||
{ runFromParent: true },
|
||||
);
|
||||
});
|
||||
|
||||
test("keeps the reported path whole when the lint target is a path segment", async () => {
|
||||
await withSyncRoot("repo", async (syncRoot) => {
|
||||
await write(syncRoot, "f/conf/orphan.script.yaml", METADATA);
|
||||
|
||||
const report = await runLint({} as any, path.join(syncRoot, "f"));
|
||||
|
||||
expect(report.issues.map((i) => i.path)).toEqual([
|
||||
"conf/orphan.script.yaml",
|
||||
]);
|
||||
expect(report.issues[0].errors[0]).toContain("conf/orphan.script.yaml");
|
||||
});
|
||||
});
|
||||
|
||||
test("does not report a fileset child spelled like script metadata", async () => {
|
||||
await withSyncRoot("repo", async (syncRoot) => {
|
||||
await write(syncRoot, "f/data.resource.yaml", "value: {}\n");
|
||||
await write(syncRoot, "f/data.fileset/config.script.yaml", "a: 1\n");
|
||||
|
||||
const report = await runLint({} as any, syncRoot);
|
||||
|
||||
expect(report.issues).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
test("does not report a dbt project whose optional descriptor is absent", async () => {
|
||||
await withSyncRoot("repo", async (syncRoot) => {
|
||||
await write(syncRoot, "f/proj.script.yaml", METADATA);
|
||||
await write(syncRoot, "f/proj__dbt/dbt_project.yml", "name: proj\n");
|
||||
await write(syncRoot, "f/proj__dbt/models/a.sql", "select 1\n");
|
||||
|
||||
const report = await runLint({} as any, syncRoot);
|
||||
|
||||
expect(report.issues).toEqual([]);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,147 @@
|
||||
/**
|
||||
* The pre-check is what stops a push from silently reassigning an item's run-as
|
||||
* user. Raw apps were missing from it, so the one kind whose file records no
|
||||
* policy at all was also the one that changed owner without a word.
|
||||
*/
|
||||
|
||||
import { expect, test } from "bun:test";
|
||||
import { preCheckPermissionedAs } from "../src/core/permissioned_as.ts";
|
||||
|
||||
/** Non-interactive and without the override flag, the pre-check exits rather
|
||||
* than reassigning silently — so a thrown exit is the signal it fired. */
|
||||
type Shape = "edited" | "added" | "deleted";
|
||||
|
||||
function change(path: string, name: Shape = "edited") {
|
||||
return { name, path, before: "summary: x\n", content: "summary: x\n" };
|
||||
}
|
||||
|
||||
async function precheck(
|
||||
changes: ReturnType<typeof change>[],
|
||||
): Promise<string | undefined> {
|
||||
const exit = process.exit;
|
||||
let code: number | undefined;
|
||||
(process as any).exit = (c?: number) => {
|
||||
code = c;
|
||||
throw new Error(`exit:${c}`);
|
||||
};
|
||||
const logged: string[] = [];
|
||||
const err = console.error;
|
||||
console.error = (...a: unknown[]) => void logged.push(a.join(" "));
|
||||
try {
|
||||
await preCheckPermissionedAs(changes, "pusher@corp", false, false, false);
|
||||
} catch (e) {
|
||||
if (!String(e).startsWith("Error: exit:")) throw e;
|
||||
} finally {
|
||||
(process as any).exit = exit;
|
||||
console.error = err;
|
||||
}
|
||||
return code === undefined ? undefined : logged.join("\n");
|
||||
}
|
||||
|
||||
test("a raw-app push warns the non-deployer it will take over the run-as user", async () => {
|
||||
const message = await precheck([change("f/test/myapp.raw_app/index.tsx")]);
|
||||
|
||||
expect(message).toBeDefined();
|
||||
expect(message).toContain("f/test/myapp.raw_app");
|
||||
expect(message).toContain("pusher@corp");
|
||||
});
|
||||
|
||||
// Deleting one file re-pushes the whole app rather than deleting it, so the
|
||||
// takeover happens there too.
|
||||
test("deleting one of an app's files warns like editing one", async () => {
|
||||
const message = await precheck([
|
||||
change("f/test/myapp.raw_app/gone.tsx", "deleted"),
|
||||
]);
|
||||
|
||||
expect(message).toContain("f/test/myapp.raw_app");
|
||||
});
|
||||
|
||||
// The metadata file going with it means the app itself is created or removed —
|
||||
// neither takes an owner over.
|
||||
test("an app arriving or leaving whole is not a takeover", async () => {
|
||||
const created = await precheck([
|
||||
change("f/test/new.raw_app/raw_app.yaml", "added"),
|
||||
change("f/test/new.raw_app/index.tsx", "added"),
|
||||
]);
|
||||
const removed = await precheck([
|
||||
change("f/test/old.raw_app/raw_app.yaml", "deleted"),
|
||||
change("f/test/old.raw_app/index.tsx", "deleted"),
|
||||
]);
|
||||
|
||||
expect(created).toBeUndefined();
|
||||
expect(removed).toBeUndefined();
|
||||
});
|
||||
|
||||
// An app carries no owner in its files, so nothing about it depends on their
|
||||
// content — an empty one redeploys it exactly like any other.
|
||||
test("an empty file still counts as a change to the app", async () => {
|
||||
const added = await precheck([
|
||||
{ name: "added", path: "f/test/myapp.raw_app/blank.ts", content: "" },
|
||||
]);
|
||||
const edited = await precheck([
|
||||
{ name: "edited", path: "f/test/myapp.raw_app/blank.ts", before: "" },
|
||||
]);
|
||||
|
||||
expect(added).toContain("f/test/myapp.raw_app");
|
||||
expect(edited).toContain("f/test/myapp.raw_app");
|
||||
});
|
||||
|
||||
// `extractFolderPath` normalizes separators but the metadata predicates match a
|
||||
// literal `/`, so a Windows path must not take a different branch.
|
||||
test("a Windows path classifies the same as its posix twin", async () => {
|
||||
const created = await precheck([
|
||||
change("f\\test\\new.raw_app\\raw_app.yaml", "added"),
|
||||
change("f\\test\\new.raw_app\\index.tsx", "added"),
|
||||
]);
|
||||
const edited = await precheck([
|
||||
change("f\\test\\myapp.raw_app\\index.tsx"),
|
||||
]);
|
||||
|
||||
expect(created).toBeUndefined();
|
||||
expect(edited).toContain("f/test/myapp.raw_app");
|
||||
});
|
||||
|
||||
// `collectAppFiles` never sends these, and the sync diff never stops listing
|
||||
// them (nothing uploads them, so they stay "added" forever) — so warning on one
|
||||
// would gate every push of a scaffolded app on the override flag.
|
||||
test("a file the push never sends is not a change to the app", async () => {
|
||||
const artifacts = await precheck([
|
||||
change("f/test/myapp.raw_app/AGENTS.md", "added"),
|
||||
change("f/test/myapp.raw_app/sql_to_apply/a.sql", "added"),
|
||||
change("f/test/myapp.raw_app/node_modules/dep/index.js", "added"),
|
||||
change("f/test/myapp.raw_app/recordings/r.json", "added"),
|
||||
change("f/test/myapp.raw_app/package-lock.json"),
|
||||
change("f/test/myapp.raw_app/wmill.d.ts"),
|
||||
// Only the backend folder's *top level* is a runnable; nothing reads deeper,
|
||||
// so the depth limit is what keeps a `backend/node_modules/` from becoming
|
||||
// the perpetual diff this predicate exists to remove.
|
||||
change("f/test/myapp.raw_app/backend/node_modules/dep/index.js", "added"),
|
||||
]);
|
||||
// The three channels a push does send through: bundled file, metadata, runnable.
|
||||
const sent = await precheck([change("f/test/myapp.raw_app/index.tsx")]);
|
||||
const meta = await precheck([change("f/test/myapp.raw_app/raw_app.yaml")]);
|
||||
const runnable = await precheck([change("f/test/myapp.raw_app/backend/a.ts")]);
|
||||
// The runnable channel is not the bundle: the bundle's name exclusions don't
|
||||
// reach into it, so a runnable file sharing one of those names still deploys.
|
||||
const namesake = await precheck([
|
||||
change("f/test/myapp.raw_app/backend/wmill.d.ts"),
|
||||
]);
|
||||
|
||||
expect(artifacts).toBeUndefined();
|
||||
expect(sent).toContain("f/test/myapp.raw_app");
|
||||
expect(meta).toContain("f/test/myapp.raw_app");
|
||||
expect(runnable).toContain("f/test/myapp.raw_app");
|
||||
expect(namesake).toContain("f/test/myapp.raw_app");
|
||||
});
|
||||
|
||||
test("an app is listed once however many of its files changed", async () => {
|
||||
const message = await precheck([
|
||||
change("f/test/myapp.raw_app/index.tsx"),
|
||||
change("f/test/myapp.raw_app/raw_app.yaml"),
|
||||
change("f/test/myapp.raw_app/backend/a.ts"),
|
||||
change("f/test/low.app/app.yaml"),
|
||||
change("f/test/low.app/inline.ts"),
|
||||
]);
|
||||
|
||||
expect(message).toContain("2 item(s)");
|
||||
});
|
||||
@@ -0,0 +1,142 @@
|
||||
/**
|
||||
* `raw_app.yaml` records none of the policy but the access-mode markers, so a
|
||||
* push that regenerated the whole policy reset the deploy drawer's settings —
|
||||
* run-as identity, sandbox isolation — to the pushing user's. Pin that the
|
||||
* deployed policy is carried over, that a first push still starts from what the
|
||||
* file states, and that the markers still close a deployed open app back down.
|
||||
*/
|
||||
|
||||
import { afterAll, beforeEach, expect, mock, test } from "bun:test";
|
||||
import { mkdtemp, symlink, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
let calls: any[] = [];
|
||||
let deployedPolicy: any;
|
||||
/** No app deployed at the path: `getAppByPath` 404s and the push creates one. */
|
||||
let deployed = true;
|
||||
|
||||
// Stub only what no other in-process suite imports, and treat a stub as
|
||||
// permanent for the run (see "Module mocks" in cli/TESTING.md). These three API
|
||||
// functions qualify — nothing else in `test/` imports them. `bundle.ts` did not:
|
||||
// stubbing it left `raw_app_svelte_plugin_unit.test.ts` asserting against an
|
||||
// empty bundle, which an `afterAll` hand-back did not prevent. So the real
|
||||
// bundler runs instead, on the app each push writes below.
|
||||
const realServices = await import("../gen/services.gen.ts");
|
||||
|
||||
mock.module("../gen/services.gen.ts", () => ({
|
||||
...realServices,
|
||||
getAppByPath: async () => {
|
||||
if (!deployed) throw new Error("not found");
|
||||
return {
|
||||
path: "f/test/raw",
|
||||
summary: "raw",
|
||||
value: { files: {}, runnables: {} },
|
||||
policy: deployedPolicy,
|
||||
};
|
||||
},
|
||||
updateAppRaw: async (a: unknown) => {
|
||||
calls.push(a);
|
||||
},
|
||||
createAppRaw: async (a: unknown) => {
|
||||
calls.push(a);
|
||||
},
|
||||
}));
|
||||
|
||||
// Belt and braces: nothing else in-process calls these, and a hand-back is not
|
||||
// what makes that safe.
|
||||
afterAll(() => {
|
||||
mock.module("../gen/services.gen.ts", () => realServices);
|
||||
});
|
||||
|
||||
const { pushRawApp } = await import("../src/commands/app/raw_apps.ts");
|
||||
|
||||
const ADMIN = {
|
||||
userCache: new Map(),
|
||||
userIsAdminOrDeployer: true,
|
||||
userEmail: "deployer@windmill.dev",
|
||||
};
|
||||
|
||||
async function push(yamlTail: string, admin = true): Promise<any> {
|
||||
calls = [];
|
||||
const dir = await mkdtemp(join(tmpdir(), "windmill_raw_push_"));
|
||||
await writeFile(
|
||||
join(dir, "raw_app.yaml"),
|
||||
`summary: raw\nrunnables: {}\n${yamlTail}`,
|
||||
"utf-8",
|
||||
);
|
||||
// Any file the remote doesn't have, so the push isn't short-circuited as
|
||||
// up to date. It is also the bundler's entry point, so it has to compile.
|
||||
await writeFile(join(dir, "index.tsx"), "export default 1\n", "utf-8");
|
||||
await writeFile(
|
||||
join(dir, "package.json"),
|
||||
JSON.stringify({ name: "app", private: true }),
|
||||
"utf-8",
|
||||
);
|
||||
// `ensureNodeModules` only checks the directory is there; borrowing the CLI's
|
||||
// own skips an npm install per push.
|
||||
await symlink(join(process.cwd(), "node_modules"), join(dir, "node_modules"));
|
||||
await pushRawApp("w", "f/test/raw", dir, undefined, "bun", admin ? ADMIN : undefined);
|
||||
expect(calls).toHaveLength(1);
|
||||
return calls[0].formData.app;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
deployed = true;
|
||||
deployedPolicy = {
|
||||
on_behalf_of: "u/svc",
|
||||
on_behalf_of_email: "svc@corp",
|
||||
sandbox: true,
|
||||
frontend_sdk_scopes: ["jobs:run"],
|
||||
execution_mode: "anonymous",
|
||||
// Legacy v1 grants: the backend folds them into v2 at run time, so keeping
|
||||
// them would keep granting runnables a push has removed.
|
||||
triggerables: { "script/f/test/gone": {} },
|
||||
triggerables_v2: { "a:script/f/test/gone": {} },
|
||||
};
|
||||
});
|
||||
|
||||
test("a raw-app push keeps the deployed run-as and sandbox settings", async () => {
|
||||
const body = await push("public: true\n");
|
||||
|
||||
expect(body.policy.on_behalf_of).toBe("u/svc");
|
||||
expect(body.policy.on_behalf_of_email).toBe("svc@corp");
|
||||
expect(body.preserve_on_behalf_of).toBe(true);
|
||||
expect(body.policy.sandbox).toBe(true);
|
||||
expect(body.policy.frontend_sdk_scopes).toEqual(["jobs:run"]);
|
||||
expect(body.policy.execution_mode).toBe("anonymous");
|
||||
expect(body.policy.triggerables).toBeUndefined();
|
||||
expect(body.policy.triggerables_v2).toEqual({});
|
||||
});
|
||||
|
||||
test("a raw-app push without the marker closes an anonymous app back down", async () => {
|
||||
const body = await push("");
|
||||
|
||||
expect(body.policy.execution_mode).toBe("publisher");
|
||||
});
|
||||
|
||||
test("a push that may not claim the deployed identity doesn't send it", async () => {
|
||||
const body = await push("", false);
|
||||
|
||||
expect(body.preserve_on_behalf_of).toBeUndefined();
|
||||
// Not just the flag: the identity itself stays off the wire, so no server can
|
||||
// deploy this push under it.
|
||||
expect(body.policy.on_behalf_of).toBeUndefined();
|
||||
expect(body.policy.on_behalf_of_email).toBeUndefined();
|
||||
// Everything the pusher is entitled to carry over still comes along.
|
||||
expect(body.policy.sandbox).toBe(true);
|
||||
});
|
||||
|
||||
test("a first raw-app push deploys the policy its file states", async () => {
|
||||
deployed = false;
|
||||
const body = await push(
|
||||
"policy:\n sandbox: true\n on_behalf_of: u/impostor\n on_behalf_of_email: impostor@corp\n",
|
||||
);
|
||||
|
||||
expect(body.policy.sandbox).toBe(true);
|
||||
// A repo doesn't get to pick who an app runs as: the identity never reaches
|
||||
// the wire, so no server can be talked into deploying under it.
|
||||
expect(body.policy.on_behalf_of).toBeUndefined();
|
||||
expect(body.policy.on_behalf_of_email).toBeUndefined();
|
||||
expect(body.preserve_on_behalf_of).toBeUndefined();
|
||||
});
|
||||
@@ -168,7 +168,7 @@ test(
|
||||
// Customer scenario: a barrel file (f/lib/errors/index.ts) re-exports from
|
||||
// siblings (./types.ts, ./WorkflowError.ts, ...). An importer in a different
|
||||
// folder imports from the barrel. On a fresh DB, the dep job for the importer
|
||||
// fetches index.ts via raw_unpinned + temp_script_hash, but bun's resolver
|
||||
// fetches index.ts via raw + temp_script_hash, but bun's resolver
|
||||
// then has to resolve the barrel's *sibling* imports — and those need to be
|
||||
// in TEMP_SCRIPT_REFS too.
|
||||
test(
|
||||
|
||||
@@ -1,390 +0,0 @@
|
||||
import * as wmillclient from "windmill-client";
|
||||
import { basename, join } from "node:path";
|
||||
import { existsSync, rmSync } from "fs";
|
||||
import process from "process";
|
||||
import { spawn } from 'child_process';
|
||||
import * as fs_async from 'fs/promises';
|
||||
import * as fs from 'node:fs';
|
||||
|
||||
const UPLOAD_CONCURRENCY = 16;
|
||||
const CLONE_MARKER_FILE = ".windmill_clone_complete";
|
||||
|
||||
type GitRepository = {
|
||||
url: string;
|
||||
branch: string;
|
||||
folder: string;
|
||||
gpg_key: any;
|
||||
is_github_app: boolean;
|
||||
};
|
||||
|
||||
export async function main(
|
||||
resource_path: string,
|
||||
workspace: string,
|
||||
git_ssh_identity?: string[],
|
||||
commit?: string
|
||||
) {
|
||||
let clonedRepoPath: string | undefined;
|
||||
|
||||
try {
|
||||
console.log("Starting git clone and Blob storage upload process");
|
||||
|
||||
// Get the git repository resource
|
||||
const repo_resource: GitRepository = await wmillclient.getResource(resource_path);
|
||||
|
||||
const cwd = process.cwd();
|
||||
|
||||
if (git_ssh_identity) {
|
||||
process.env.GIT_SSH_COMMAND = await get_git_ssh_cmd(cwd, git_ssh_identity)
|
||||
}
|
||||
|
||||
// Handle GitHub App authentication if needed
|
||||
if (repo_resource.is_github_app) {
|
||||
const token = await get_gh_app_token();
|
||||
repo_resource.url = prependTokenToGitHubUrl(repo_resource.url, token);
|
||||
}
|
||||
|
||||
process.env["HOME"] = ".";
|
||||
process.env.GIT_TERMINAL_PROMPT = "0";
|
||||
|
||||
// Clone the repository
|
||||
const { repo_name, commitHash } = await git_clone(cwd, repo_resource, commit);
|
||||
clonedRepoPath = join(cwd, repo_name);
|
||||
|
||||
// Remove .git directory to avoid uploading git history
|
||||
const gitDir = join(clonedRepoPath, ".git");
|
||||
if (existsSync(gitDir)) {
|
||||
rmSync(gitDir, { recursive: true, force: true });
|
||||
console.log("Removed .git directory");
|
||||
}
|
||||
|
||||
// Upload to S3
|
||||
const s3Path = `gitrepos/${workspace}/${resource_path}/${commitHash}`;
|
||||
const fileCount = await uploadDirectoryToS3(clonedRepoPath, s3Path, workspace);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
message: "Repository cloned and uploaded to S3 successfully",
|
||||
s3_path: s3Path,
|
||||
commit_hash: commitHash,
|
||||
file_count: fileCount,
|
||||
};
|
||||
|
||||
} catch (error) {
|
||||
console.error("Error in git clone and upload:", error);
|
||||
throw error;
|
||||
} finally {
|
||||
// Clean up cloned repository
|
||||
if (clonedRepoPath && existsSync(clonedRepoPath)) {
|
||||
rmSync(clonedRepoPath, { recursive: true, force: true });
|
||||
console.log("Cleaned up cloned repository");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function get_git_ssh_cmd(cwd: string, git_ssh_identity: string[]): Promise<string> {
|
||||
const sshIdFiles = await Promise.all(
|
||||
git_ssh_identity.map(async (varPath, i) => {
|
||||
const filePath = join(cwd, `./ssh_id_priv_${i}`);
|
||||
|
||||
try {
|
||||
// Get variable value using windmill
|
||||
let content = await wmillclient.getVariable(varPath);
|
||||
content += '\n';
|
||||
|
||||
// Write file with content
|
||||
await fs_async.writeFile(filePath, content, { encoding: 'utf8' });
|
||||
|
||||
// Set file permissions to 0o600 (read/write for owner only)
|
||||
await fs_async.chmod(filePath, 0o600);
|
||||
|
||||
// Escape single quotes for shell command
|
||||
const escapedPath = filePath.replace(/'/g, "'\\''");
|
||||
return ` -i '${escapedPath}'`;
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`Variable ${varPath} not found for git ssh identity: ${error}`
|
||||
);
|
||||
return '';
|
||||
}
|
||||
})
|
||||
);
|
||||
|
||||
const gitSshCmd = `ssh -o StrictHostKeyChecking=no${sshIdFiles.join('')}`;
|
||||
return gitSshCmd;
|
||||
}
|
||||
|
||||
async function git_clone(
|
||||
cwd: string,
|
||||
repo_resource: GitRepository,
|
||||
commit?: string,
|
||||
): Promise<{ repo_name: string; commitHash: string }> {
|
||||
if (commit) {
|
||||
return git_clone_at_commit(cwd, repo_resource, commit);
|
||||
} else {
|
||||
return git_clone_at_latest(cwd, repo_resource);
|
||||
}
|
||||
}
|
||||
|
||||
async function git_clone_at_commit(
|
||||
cwd: string,
|
||||
repo_resource: GitRepository,
|
||||
commit: string,
|
||||
): Promise<{ repo_name: string; commitHash: string }> {
|
||||
let repo_url = repo_resource.url;
|
||||
const subfolder = repo_resource.folder ?? "";
|
||||
let branch = repo_resource.branch ?? "";
|
||||
const repo_name = basename(repo_url, ".git");
|
||||
|
||||
const azureMatch = repo_url.match(/AZURE_DEVOPS_TOKEN\((?<url>.+)\)/);
|
||||
if (azureMatch) {
|
||||
console.log("Fetching Azure DevOps access token...");
|
||||
const azureResource = await wmillclient.getResource(azureMatch.groups.url);
|
||||
const response = await fetch(
|
||||
`https://login.microsoftonline.com/${azureResource.azureTenantId}/oauth2/token`,
|
||||
{
|
||||
method: "POST",
|
||||
body: new URLSearchParams({
|
||||
client_id: azureResource.azureClientId,
|
||||
client_secret: azureResource.azureClientSecret,
|
||||
grant_type: "client_credentials",
|
||||
resource: "499b84ac-1321-427f-aa17-267ca6975798/.default",
|
||||
}),
|
||||
}
|
||||
);
|
||||
const { access_token } = await response.json();
|
||||
repo_url = repo_url.replace(azureMatch[0], access_token);
|
||||
}
|
||||
|
||||
const repoPath = join(cwd, repo_name);
|
||||
await fs_async.mkdir(repoPath, { recursive: true });
|
||||
|
||||
process.chdir(repoPath);
|
||||
|
||||
let args = ['init', '--quiet']
|
||||
if (branch) {
|
||||
args.push(`--initial-branch=${branch}`)
|
||||
}
|
||||
await runCommand(undefined, 'git', ...args);
|
||||
|
||||
await runCommand(0, 'git', 'remote', 'add', 'origin', repo_url);
|
||||
|
||||
await runCommand(undefined, 'git', 'fetch', '--depth=1', '--quiet', 'origin', commit);
|
||||
|
||||
await runCommand(undefined, 'git', 'checkout', '--quiet', 'FETCH_HEAD');
|
||||
|
||||
const commitHash = (await runCommand(undefined, "git", "rev-parse", "HEAD")).trim();
|
||||
|
||||
// Return to original directory
|
||||
process.chdir(cwd);
|
||||
|
||||
return { repo_name, commitHash };
|
||||
}
|
||||
|
||||
async function git_clone_at_latest(
|
||||
cwd: string,
|
||||
repo_resource: GitRepository
|
||||
): Promise<{ repo_name: string; commitHash: string }> {
|
||||
let repo_url = repo_resource.url;
|
||||
const subfolder = repo_resource.folder ?? "";
|
||||
let branch = repo_resource.branch ?? "";
|
||||
const repo_name = basename(repo_url, ".git");
|
||||
|
||||
// Handle Azure DevOps token if needed
|
||||
const azureMatch = repo_url.match(/AZURE_DEVOPS_TOKEN\((?<url>.+)\)/);
|
||||
if (azureMatch) {
|
||||
console.log("Fetching Azure DevOps access token...");
|
||||
const azureResource = await wmillclient.getResource(azureMatch.groups.url);
|
||||
const response = await fetch(
|
||||
`https://login.microsoftonline.com/${azureResource.azureTenantId}/oauth2/token`,
|
||||
{
|
||||
method: "POST",
|
||||
body: new URLSearchParams({
|
||||
client_id: azureResource.azureClientId,
|
||||
client_secret: azureResource.azureClientSecret,
|
||||
grant_type: "client_credentials",
|
||||
resource: "499b84ac-1321-427f-aa17-267ca6975798/.default",
|
||||
}),
|
||||
}
|
||||
);
|
||||
const { access_token } = await response.json();
|
||||
repo_url = repo_url.replace(azureMatch[0], access_token);
|
||||
}
|
||||
|
||||
const args = ["clone", "--quiet", "--depth", "1"];
|
||||
if (subfolder !== "") args.push("--sparse");
|
||||
if (branch !== "") args.push("--branch", branch);
|
||||
args.push(repo_url, repo_name);
|
||||
|
||||
await runCommand(-1, "git", ...args);
|
||||
|
||||
const fullPath = join(cwd, repo_name);
|
||||
process.chdir(fullPath);
|
||||
|
||||
if (subfolder !== "") {
|
||||
await runCommand(undefined, "git", "sparse-checkout", "add", subfolder);
|
||||
const subfolderPath = join(fullPath, subfolder);
|
||||
|
||||
if (!existsSync(subfolderPath)) {
|
||||
throw new Error(`Subfolder ${subfolder} does not exist.`);
|
||||
}
|
||||
|
||||
process.chdir(subfolderPath);
|
||||
}
|
||||
|
||||
// Get the commit hash
|
||||
const commitHash = (await runCommand(undefined, "git", "rev-parse", "HEAD")).trim();
|
||||
|
||||
// Return to original directory
|
||||
process.chdir(cwd);
|
||||
|
||||
return { repo_name, commitHash };
|
||||
}
|
||||
|
||||
async function uploadDirectoryToS3(
|
||||
directoryPath: string,
|
||||
s3BasePath: string,
|
||||
workspace: string,
|
||||
): Promise<number> {
|
||||
console.log(`Uploading ${directoryPath} -> ${s3BasePath}`);
|
||||
|
||||
// Walk once into a flat task list so we can drive a bounded-concurrency pool.
|
||||
const tasks: { localPath: string; s3Key: string }[] = [];
|
||||
function walk(dir: string, s3Path: string) {
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const fullPath = join(dir, entry.name);
|
||||
const s3Key = s3Path ? `${s3Path}/${entry.name}` : entry.name;
|
||||
if (entry.isDirectory()) {
|
||||
walk(fullPath, s3Key);
|
||||
} else if (entry.isFile()) {
|
||||
tasks.push({ localPath: fullPath, s3Key });
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(directoryPath, s3BasePath);
|
||||
console.log(`Discovered ${tasks.length} files to upload`);
|
||||
|
||||
let nextIndex = 0;
|
||||
let uploaded = 0;
|
||||
let lastReport = 0;
|
||||
async function worker() {
|
||||
while (true) {
|
||||
const idx = nextIndex++;
|
||||
if (idx >= tasks.length) return;
|
||||
const { localPath, s3Key } = tasks[idx];
|
||||
const fileContent = fs.readFileSync(localPath);
|
||||
const blob = new Blob([fileContent], { type: 'application/octet-stream' });
|
||||
await wmillclient.HelpersService.gitRepoViewerFileUpload({
|
||||
workspace,
|
||||
fileKey: s3Key,
|
||||
requestBody: blob,
|
||||
});
|
||||
uploaded++;
|
||||
if (uploaded - lastReport >= 25 || uploaded === tasks.length) {
|
||||
lastReport = uploaded;
|
||||
console.log(`Uploaded ${uploaded} / ${tasks.length} files`);
|
||||
}
|
||||
}
|
||||
}
|
||||
await Promise.all(
|
||||
Array.from({ length: Math.min(UPLOAD_CONCURRENCY, tasks.length) }, () => worker())
|
||||
);
|
||||
|
||||
// Marker is the LAST write — its presence is what the viewer checks for.
|
||||
const markerKey = `${s3BasePath}/${CLONE_MARKER_FILE}`;
|
||||
const markerBody = JSON.stringify({
|
||||
completed_at: new Date().toISOString(),
|
||||
file_count: tasks.length,
|
||||
});
|
||||
await wmillclient.HelpersService.gitRepoViewerFileUpload({
|
||||
workspace,
|
||||
fileKey: markerKey,
|
||||
requestBody: new Blob([markerBody], { type: 'application/json' }),
|
||||
});
|
||||
console.log(`Wrote completion marker: ${markerKey}`);
|
||||
|
||||
return tasks.length;
|
||||
}
|
||||
|
||||
function runCommand(secret_position: number | undefined, cmd: string, ...args: string[]): Promise<string> {
|
||||
const nargs = secret_position != undefined ? args.slice() : args;
|
||||
if (secret_position && secret_position < 0)
|
||||
secret_position = nargs.length - 1 + secret_position;
|
||||
|
||||
let secret: string | undefined = undefined;
|
||||
if (secret_position != undefined) {
|
||||
nargs[secret_position] = "***";
|
||||
secret = args[secret_position];
|
||||
}
|
||||
console.log(`Running shell command: '${cmd} ${nargs.join(" ")} ...'`);
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const process = spawn(cmd, args);
|
||||
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
|
||||
process.stdout.on('data', (data) => {
|
||||
stdout += data.toString();
|
||||
});
|
||||
|
||||
process.stderr.on('data', (data) => {
|
||||
stderr += data.toString();
|
||||
});
|
||||
|
||||
process.on('error', (error) => {
|
||||
let errorString = error.toString();
|
||||
if (secret) errorString = errorString.replace(secret, "***");
|
||||
console.log(`Shell command FAILED: ${cmd}`, errorString);
|
||||
const e = new Error(
|
||||
`SH command '${cmd} ${nargs.join(" ")}' failed: ${errorString}`
|
||||
);
|
||||
reject(e);
|
||||
});
|
||||
|
||||
process.on('close', (code) => {
|
||||
if (stdout.length > 0) {
|
||||
console.log("Shell stdout:", stdout);
|
||||
}
|
||||
if (stderr.length > 0) {
|
||||
console.log("Shell stderr:", stderr);
|
||||
}
|
||||
if (code === 0) {
|
||||
console.log(`Shell command completed successfully: ${cmd}`);
|
||||
resolve(stdout);
|
||||
} else {
|
||||
reject(new Error(`Command failed with code ${code}: ${stderr}`));
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function get_gh_app_token() {
|
||||
const workspace = process.env["WM_WORKSPACE"];
|
||||
const jobToken = process.env["WM_TOKEN"];
|
||||
const baseUrl =
|
||||
process.env["BASE_INTERNAL_URL"] ??
|
||||
process.env["BASE_URL"] ??
|
||||
"http://localhost:8000";
|
||||
const url = `${baseUrl}/api/w/${workspace}/github_app/token`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${jobToken}`,
|
||||
},
|
||||
body: JSON.stringify({ job_token: jobToken }),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const errorBody = await response.text().catch(() => "");
|
||||
throw new Error(`GitHub App token error (${response.status}): ${errorBody || response.statusText}`);
|
||||
}
|
||||
const data = await response.json();
|
||||
return data.token;
|
||||
}
|
||||
|
||||
function prependTokenToGitHubUrl(gitHubUrl: string, installationToken: string) {
|
||||
const url = new URL(gitHubUrl);
|
||||
return `https://x-access-token:${installationToken}@${url.hostname}${url.pathname}`;
|
||||
}
|
||||
@@ -4,10 +4,10 @@
|
||||
anonymous usage-stats payload. It answers "does anyone use this, and which variant do they pick"
|
||||
without any identifying data leaving the instance.
|
||||
|
||||
It currently carries 42 registered actions across seventeen features (`ai_session`, `ai_chat`,
|
||||
It currently carries 49 registered actions across eighteen features (`ai_session`, `ai_chat`,
|
||||
`ai_fix`, `ai_agent`, `ai_agent_eval`, `app_sandbox`, `datatable`, `flow_editor`, `flow_run`,
|
||||
`flow_step`, `run_form`, `debugger`, `trigger`, `command_script`, `hub_script`, `usage_meter`,
|
||||
`sso_groups_claim`). Nearly all of the
|
||||
`flow_step`, `home`, `run_form`, `debugger`, `trigger`, `command_script`, `hub_script`,
|
||||
`usage_meter`, `sso_groups_claim`). Nearly all of the
|
||||
product is uninstrumented, so new user-facing work is the opportunity to change that.
|
||||
|
||||
## When to instrument
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
# Git repo viewer — hub script
|
||||
|
||||
The hub script `clone_repo_and_upload_to_instance_storage` is published from
|
||||
`windmill-integrations` and pinned in `frontend/src/lib/hubPaths.json` as
|
||||
The hub script `clone_repo_and_upload_to_instance_storage`
|
||||
([hub page](https://hub.windmill.dev/scripts/windmill/13968)) is published from
|
||||
`windmill-integrations`
|
||||
(`hub/windmill/scripts/action/13968_clone_repo_and_upload_to_instance_storage/script.ts`)
|
||||
and pinned in `frontend/src/lib/hubPaths.json` as
|
||||
`cloneRepoToS3forGitRepoViewer`. Hub paths are exact version pins, so editing
|
||||
the script means publishing a new version and repointing that entry.
|
||||
|
||||
@@ -23,6 +26,11 @@ The repo viewer in the Windmill app expects the hub script to:
|
||||
3. **Write a completion marker** as the very last action of a successful run,
|
||||
so the API and frontend can distinguish a fully-populated S3 directory from
|
||||
a partial / interrupted upload.
|
||||
4. **Follow symlinks that stay inside the checkout.** Both the git clone and
|
||||
the archive extraction keep a repository's symlinks as links, and
|
||||
`Dirent.isFile()` / `isDirectory()` are both false for a link, so a walk
|
||||
that only checks those drops every linked file and directory from the
|
||||
viewer. See [Symlinks](#symlinks).
|
||||
|
||||
The marker file the frontend looks for is `.windmill_clone_complete` at the
|
||||
root of the per-commit directory:
|
||||
@@ -43,28 +51,74 @@ after the walk completes:
|
||||
```ts
|
||||
const UPLOAD_CONCURRENCY = 16
|
||||
const CLONE_MARKER_FILE = ".windmill_clone_complete"
|
||||
const MAX_SYMLINKED_ENTRIES = 20_000
|
||||
const MAX_SYMLINKED_BYTES = 512 * 1024 * 1024
|
||||
|
||||
async function uploadDirectoryToS3(
|
||||
directoryPath: string,
|
||||
s3BasePath: string,
|
||||
workspace: string,
|
||||
) {
|
||||
): Promise<number> {
|
||||
console.log(`Uploading ${directoryPath} -> ${s3BasePath}`)
|
||||
|
||||
// Walk the directory once, producing a flat list of (localPath, s3Key) pairs.
|
||||
const tasks: { localPath: string; s3Key: string }[] = []
|
||||
function walk(dir: string, s3Path: string) {
|
||||
const root = fs.realpathSync(directoryPath)
|
||||
// Real paths of the directories being descended through.
|
||||
const ancestors = new Set<string>()
|
||||
// What entries reached through a link have cost so far; see Symlinks below.
|
||||
let symlinkedEntries = 0
|
||||
let symlinkedBytes = 0
|
||||
let symlinkBudgetSpent = false
|
||||
function chargeSymlinkBudget(relPath: string, entries: number, bytes: number): boolean {
|
||||
if (symlinkBudgetSpent) return false
|
||||
symlinkedEntries += entries
|
||||
symlinkedBytes += bytes
|
||||
if (symlinkedEntries <= MAX_SYMLINKED_ENTRIES && symlinkedBytes <= MAX_SYMLINKED_BYTES) {
|
||||
return true
|
||||
}
|
||||
symlinkBudgetSpent = true
|
||||
console.log(
|
||||
`Skipping ${relPath} and every symlinked entry after it: symlinks reach more than ` +
|
||||
`${MAX_SYMLINKED_ENTRIES} entries or ${MAX_SYMLINKED_BYTES / 2 ** 20} MiB`
|
||||
)
|
||||
return false
|
||||
}
|
||||
function walk(dir: string, relDir: string, viaLink: boolean) {
|
||||
ancestors.add(dir)
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const fullPath = join(dir, entry.name)
|
||||
const s3Key = s3Path ? `${s3Path}/${entry.name}` : entry.name
|
||||
if (entry.isDirectory()) {
|
||||
walk(fullPath, s3Key)
|
||||
} else if (entry.isFile()) {
|
||||
tasks.push({ localPath: fullPath, s3Key })
|
||||
const relPath = relDir ? `${relDir}/${entry.name}` : entry.name
|
||||
const linked = viaLink || entry.isSymbolicLink()
|
||||
if (linked && !chargeSymlinkBudget(relPath, 1, 0)) continue
|
||||
let localPath = join(dir, entry.name)
|
||||
if (entry.isSymbolicLink()) {
|
||||
const link = fs.readlinkSync(localPath)
|
||||
try {
|
||||
localPath = fs.realpathSync(localPath)
|
||||
} catch (e: any) {
|
||||
console.log(`Skipping symlink ${relPath} -> ${link}: cannot resolve target (${e.code})`)
|
||||
continue
|
||||
}
|
||||
if (localPath !== root && !localPath.startsWith(root + sep)) {
|
||||
console.log(`Skipping symlink ${relPath} -> ${link}: target is outside the repository`)
|
||||
continue
|
||||
}
|
||||
}
|
||||
const stat = fs.statSync(localPath)
|
||||
if (stat.isDirectory() && ancestors.has(localPath)) {
|
||||
console.log(`Skipping ${relPath}: links back to a directory it is inside`)
|
||||
continue
|
||||
}
|
||||
if (linked && stat.isFile() && !chargeSymlinkBudget(relPath, 0, stat.size)) continue
|
||||
if (stat.isDirectory()) {
|
||||
walk(localPath, relPath, linked)
|
||||
} else if (stat.isFile()) {
|
||||
tasks.push({ localPath, s3Key: `${s3BasePath}/${relPath}` })
|
||||
}
|
||||
}
|
||||
ancestors.delete(dir)
|
||||
}
|
||||
walk(directoryPath, s3BasePath)
|
||||
walk(root, "", false)
|
||||
|
||||
console.log(`Discovered ${tasks.length} files to upload`)
|
||||
|
||||
@@ -114,9 +168,40 @@ async function uploadDirectoryToS3(
|
||||
requestBody: new Blob([markerBody], { type: "application/json" }),
|
||||
})
|
||||
console.log(`Wrote completion marker: ${markerKey}`)
|
||||
|
||||
return tasks.length
|
||||
}
|
||||
```
|
||||
|
||||
## Symlinks
|
||||
|
||||
A link is resolved with `realpathSync` and followed only when its target lies
|
||||
inside the checkout's real path. A file target is uploaded under the link's own
|
||||
path; a directory target is walked as if it sat there, so
|
||||
`inventories/prod/group_vars -> ../../shared/group_vars` shows up in the viewer
|
||||
with its files. Everything else is skipped and logged:
|
||||
|
||||
- **A target outside the checkout.** The repository chooses the target, and the
|
||||
checkout sits in the job's working directory next to the ssh key
|
||||
`get_git_ssh_cmd` writes (`../ssh_id_priv_0`) and the job's `args.json`. A
|
||||
link to one of those, or to `/proc/self/environ` with the caller's
|
||||
`WM_TOKEN`, would put it in storage for every reader of the resource. This
|
||||
is why the walk does not follow links the way `aws s3 sync` does.
|
||||
- **A target that cannot be resolved**: a dangling link, or a link loop
|
||||
(`ELOOP`).
|
||||
- **A directory that is already being walked higher up** (`loop -> .`,
|
||||
`up -> ..`). The guard holds the real paths of the current descent only, as
|
||||
`find -L` does, not every directory seen so far: a directory reachable
|
||||
through two links is uploaded under both paths, as the checkout presents it.
|
||||
- **Anything reached through a link once the budget is spent.** Because a
|
||||
directory can be reached along many paths, two links to the next directory
|
||||
at each level double the tree, and a repository a few dozen links deep would
|
||||
expand past what the job can hold in memory. Every entry reached through a
|
||||
link counts against a budget of 20,000 entries and 512 MiB. It is charged
|
||||
before the link is resolved, so links that end up skipped count too, and
|
||||
neither their work nor their log lines can multiply. Past the budget, the rest
|
||||
are skipped with one log line. The checkout's own files are always uploaded.
|
||||
|
||||
## Notes for review
|
||||
|
||||
- **Concurrency level**: 16 is a starting point; tune based on instance
|
||||
@@ -127,6 +212,12 @@ async function uploadDirectoryToS3(
|
||||
paths on retry, so a partial upload + retry naturally heals. Old commit
|
||||
directories from before this patch are unreachable through the UI but still
|
||||
consume storage; an instance admin can prune them manually if desired.
|
||||
- **A new pin doesn't refresh commits already uploaded**: the viewer keys
|
||||
storage on the commit hash (`gitrepos/{workspace}/{resource_path}/{commit_hash}/`)
|
||||
and only checks that the marker exists. So a commit uploaded by an earlier
|
||||
script version keeps that version's tree (hub/28905's had no symlinks) until
|
||||
the repository's head moves to a new commit, or an admin deletes that
|
||||
commit's directory.
|
||||
- **Error propagation**: keep the existing `try/catch` in `main` so an upload
|
||||
failure surfaces in the job result and is shown in the new viewer error
|
||||
banner.
|
||||
|
||||
Generated
+5
-5
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@windmill-labs/components",
|
||||
"version": "1.805.0",
|
||||
"version": "1.809.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@windmill-labs/components",
|
||||
"version": "1.805.0",
|
||||
"version": "1.809.0",
|
||||
"hasInstallScript": true,
|
||||
"license": "AGPL-3.0",
|
||||
"dependencies": {
|
||||
@@ -5579,9 +5579,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/driver.js": {
|
||||
"version": "1.3.6",
|
||||
"resolved": "https://registry.npmjs.org/driver.js/-/driver.js-1.3.6.tgz",
|
||||
"integrity": "sha512-g2nNuu+tWmPpuoyk3ffpT9vKhjPz4NrJzq6mkRDZIwXCrFhrKdDJ9TX5tJOBpvCTBrBYjgRQ17XlcQB15q4gMg==",
|
||||
"version": "1.8.0",
|
||||
"resolved": "https://registry.npmjs.org/driver.js/-/driver.js-1.8.0.tgz",
|
||||
"integrity": "sha512-+8/IO7h1v14IzWh2GP60N7T3PFZweXwdn5e5POuxRSBoCYUojsBxzqawPeXh3YZIibRy7EehYNEyxe7slwwtdg==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/dts-bundle-generator": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@windmill-labs/components",
|
||||
"version": "1.805.0",
|
||||
"version": "1.809.0",
|
||||
"scripts": {
|
||||
"dev": "vite dev",
|
||||
"dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user