From adbf4720ae8f563ab857bf627566da36028d5912 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Fri, 3 Apr 2026 22:07:35 +0000 Subject: [PATCH] docs: add CONTEXT.md for benchmark framework continuation Comprehensive handoff document covering architecture, decisions, gotchas, and potential improvements for the competitor benchmark suite. Co-Authored-By: Claude Opus 4.6 (1M context) --- benchmarks/competitors/CONTEXT.md | 178 ++++++++++++++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100644 benchmarks/competitors/CONTEXT.md diff --git a/benchmarks/competitors/CONTEXT.md b/benchmarks/competitors/CONTEXT.md new file mode 100644 index 0000000000..3818741f22 --- /dev/null +++ b/benchmarks/competitors/CONTEXT.md @@ -0,0 +1,178 @@ +# Competitor Benchmark Framework — Context for Continuation + +## Goal + +Reproducible performance benchmarks comparing Windmill's workflow-as-code (WAC) against competitors, for SEO comparison pages. All platforms run an equivalent **3-step sequential workflow** with inline step execution (no subprocess/child job dispatch). This isolates orchestration overhead from actual compute. + +## Current Competitors (7) + +| Competitor | Version Tested | Adapter | Workflow Type | +|---|---|---|---| +| **Windmill** | CE v1.673.0 | `windmill/adapter.ts` | `step()` from `windmill-client` (inline, no child jobs) | +| **Temporal** | auto-setup:latest | `temporal/adapter.ts` | `proxyActivities()` with 3 sequential activities | +| **Inngest** | inngest:latest | `inngest/adapter.ts` | `step.run()` × 3 via Express app | +| **Restate** | restate:latest | `restate/adapter.ts` | `ctx.run()` × 3 via Node.js endpoint | +| **Kestra** | 1.3.7 | `kestra/adapter.ts` | `io.kestra.plugin.core.debug.Return` × 3 | +| **Prefect** | 3.6.25 | `prefect/adapter.ts` | `@task` × 3 with `.serve()` runner | +| **Airflow** | 2.10.5 | `airflow/adapter.ts` | `PythonOperator` × 3 with LocalExecutor | + +## Latest Results (local dev machine, 2026-04-03) + +| Competitor | Cold Start | Median Latency | P95 Latency | Throughput | Step Overhead | Workers | +|---|---|---|---|---|---|---| +| Restate | 15ms | 4.1ms | 5.0ms | 2,057/s | 1.4ms | event loop | +| **Windmill** | **106ms** | **103ms** | **105ms** | **152/s** | **34ms** | 10 | +| Temporal | 166ms | 119ms | 283ms | 39/s | 40ms | 10 | +| Kestra | 410ms | 177ms | 326ms | 68/s | 59ms | JVM threads | +| Inngest | 313ms | 260ms | 261ms | 90/s | 87ms | 10 | +| Airflow | 3,858ms | 3,792ms | 4,277ms | 7.4/s | 1,264ms | LocalExecutor | +| Prefect | 2,764ms | 9,679ms | 12,631ms | 1.3/s | 3,226ms | .serve() | + +## Architecture + +``` +benchmarks/competitors/ +├── types.ts # CompetitorAdapter interface + BenchmarkResult types +├── competitor_harness.ts # Main orchestrator: for each competitor → setup → deploy → test → teardown +├── competitor_suite.ts # CLI wrapper (cliffy): --competitors, --latency-samples, --throughput-batch, --output-dir +├── competitor_graphs.ts # SVG bar chart generation (D3 + JSDOM) +├── competitor_suite_config.json +├── competitor_graphs_config.json +├── lib/ +│ ├── docker.ts # composeUp(), composeDown(), waitForHealth(), composeLogs() +│ ├── timing.ts # measureLatency(), warmup(), collectLatencySamples(), computeStats() +│ └── results.ts # saveResult(), saveSummary(), getCpuCount(), getMachineId() +├── .gitignore # excludes node_modules/ and results/ +├── {competitor}/ +│ ├── adapter.ts # implements CompetitorAdapter +│ ├── docker-compose.yml # isolated container stack +│ ├── workflow.* # equivalent workflow definition +│ └── Dockerfile.* # (some competitors need custom app/worker images) +└── results/ # (gitignored) JSON output from benchmark runs +``` + +## CompetitorAdapter Interface + +```typescript +interface CompetitorAdapter { + readonly name: string; + readonly composeFile: string; + setup(): Promise; // docker compose up + health wait + deployWorkflow(): Promise; // register/create the 3-step workflow + triggerOne(): Promise<{ latencyMs: number; result: unknown }>; + triggerBatch(n: number): Promise<{ totalMs: number; results: unknown[] }>; + teardown(): Promise; // docker compose down -v + getVersion(): Promise; +} +``` + +## Harness Flow (per competitor) + +1. `setup()` — `docker compose up -d` + wait for health endpoint +2. `deployWorkflow()` — register the 3-step workflow via REST API / SQL / CLI +3. **Cold start** — `triggerOne()` immediately after deploy +4. **Warmup** — `triggerOne()` × N, discard results +5. **Single latency** — `triggerOne()` × N, collect `performance.now()` timings → compute median, P95, mean, stdev +6. **Throughput** — `triggerBatch(N)` (concurrent `Promise.all`), measure total wall-clock time +7. **Step overhead** — median_latency / 3 (trivial steps, so overhead ≈ latency) +8. `teardown()` — `docker compose down -v` +9. Save JSON results + +## How to Run + +```bash +cd benchmarks/competitors + +# Single competitor +deno run -A competitor_suite.ts --competitors windmill --latency-samples 50 --throughput-batch 100 --output-dir ./results + +# All competitors +deno run -A competitor_suite.ts --latency-samples 100 --throughput-batch 200 --warmup-count 10 --output-dir ./results + +# Generate SVG graphs +deno run -A --allow-import competitor_graphs.ts -c competitor_graphs_config.json --results-dir ./results +``` + +## Key Decisions & Gotchas + +### Windmill +- Uses `step()` (inline), NOT `task()` (child jobs). Other competitors don't create separate jobs per step either, so this is the fair comparison. +- `WORKER_GROUP=main` + `NUM_WORKERS=10` + `I_ACK_NUM_WORKERS_IS_UNSAFE=1` + `SLEEP_QUEUE=50`. Without `SLEEP_QUEUE=50`, the queue poll interval scales to `50ms × NUM_WORKERS / 2 = 250ms`, killing latency. See `backend/windmill-worker/src/worker.rs:318-329`. +- `WORKER_TAGS=bun,flow,dependency` — must include `bun` since WAC scripts deploy as bun language. +- Cannot use `WORKER_GROUP=native` or `NATIVE_MODE=true` because native mode forces `NUM_WORKERS=8` and doesn't support the `bun` tag needed for WAC scripts. + +### Temporal +- Needs a **separate worker container** (Dockerfile.worker) because the Temporal TypeScript SDK has native gRPC bindings that don't work in Deno. +- The adapter uses `node -e` subprocess calls to run Temporal client operations (connect, start workflow, get result). +- `npm install` in `temporal/` is needed before running (the adapter does this in `setup()`). +- `maxConcurrentWorkflowTaskExecution: 10` and `maxConcurrentActivityTaskExecution: 10` in worker.ts to match Windmill's 10 workers. +- Cold start is high (~6s) because `temporalio/auto-setup` does DB schema migration on first boot. Not representative of steady-state. + +### Inngest +- The dev server's event run status API (`/v1/events/{id}/runs`) **caches responses for 15 seconds**. Must add `?ts=${Date.now()}` cache-buster to polling requests, otherwise each step appears to take 5 seconds. +- `--queue-workers 10` caps concurrency to match other competitors. +- `--tick 10` (10ms) for fast queue polling, `--poll-interval 1` for fast app sync. +- The Express app needs `express.json()` middleware or Inngest SDK returns 500 "Missing body". +- `--no-discovery` is set because auto-discovery is unreliable in Docker networks. + +### Restate +- Extremely fast (4ms median) because it runs workflow steps **in-process** in the app's Node.js event loop. No queue hop, no network round-trip between steps. This is a fundamental architectural difference from Windmill/Temporal/Inngest. +- Uses `send` + `attach` pattern: POST `/benchmark/{id}/run/send` then GET `/restate/workflow/benchmark/{id}/attach`. +- The app must be registered with the Restate admin API: POST `http://admin:9070/deployments` with `{"uri": "http://app:9080"}`. +- Ports remapped to 8085/9075 to avoid conflict with port 8080 (often in use). + +### Kestra +- Kestra 1.3.x has **mandatory basic auth** that cannot be disabled via config. The `/api/v1/flows` endpoint always returns 401. +- Workaround: deploy the flow via **direct SQL insertion** into Kestra's Postgres `flows` table. Key format is `main_{namespace}_{id}_{revision}`. The `value` JSONB must include `tenantId: "main"`, `source`, `deleted: false`, etc. (see existing tutorial flows for format). +- Trigger via the **webhook endpoint** (`/api/v1/executions/webhook/{namespace}/{flowId}/{key}`) which is public. +- Poll execution status via SQL: `SELECT value->>'state' FROM executions WHERE key = '{executionId}'`. +- Uses `io.kestra.plugin.core.debug.Return` tasks (not shell tasks) to avoid subprocess overhead. + +### Prefect +- Very slow (~9.7s/workflow) because `.serve()` mode spawns a new subprocess per flow run. Prefect is designed for data pipeline tasks, not lightweight orchestration. +- The worker container runs `python flow.py` which calls `.serve(name="benchmark-deployment")` — this both registers the deployment and runs a worker loop. +- Deployment ID must be fetched via `GET /api/deployments/name/{flow_name}/{deployment_name}` before triggering. +- No auth required on self-hosted Prefect OSS server. + +### Airflow +- Slow (~3.8s/workflow) due to scheduler overhead and PythonOperator subprocess execution. +- DAG file is volume-mounted to `./dags/`. Scheduler detects it after `DAG_DIR_LIST_INTERVAL` (set to 5s). +- DAGs are paused by default — must PATCH `/api/v1/dags/{dag_id}` with `{"is_paused": false}`. +- All API calls require Basic Auth: `Authorization: Basic YWRtaW46YWRtaW4=` (admin:admin). +- `airflow-init` container runs DB migration + user creation, then exits. Webserver and scheduler depend on it via `service_completed_successfully`. +- Webserver port remapped to 8090 to avoid conflict with 8080. + +### Docker / Infrastructure +- All custom app/worker images must be **pre-built** before running (`docker build -t {name} -f Dockerfile ...`). Docker Compose's `build:` directive times out in some environments. +- Compose files use pre-built image references (e.g., `image: temporal-benchmark-worker`) not `build:` blocks. +- Docker Compose `--wait` flag was removed from `composeUp()` because it caused timeout failures. Each adapter handles its own readiness checking via `waitForHealth()` or custom polling. +- Competitors run sequentially with full `docker compose down -v` between runs to avoid resource contention and port conflicts. +- **Disk space**: Running all 7 competitors needs ~15GB for Docker images. Kestra (3.3GB) and Airflow (1.5GB) are the largest. Use `docker system prune -af` between runs if tight on space. + +### Graphs +- `competitor_graphs.ts` generates SVG bar charts using D3 + JSDOM (same stack as existing `benchmarks/graph.ts`). +- Reads from `results/competitor_comparison_benchmark.json` (flat summary format). +- Config in `competitor_graphs_config.json` defines 5 charts: cold start, median latency, P95, throughput, step overhead. +- D3 callback params need explicit `any` types to pass Deno type checking. + +## CI + +`.github/workflows/benchmark-competitors.yml`: +- Runs weekly (Monday 6AM UTC) + manual dispatch +- `ubicloud-standard-8` runner +- Pre-pulls all Docker images in parallel +- Sequential competitor runs +- Results committed to `benchmarks` branch + +## Potential Improvements + +1. **Dagster adapter** — Another popular Python DAG-based orchestrator (competitor to Airflow/Prefect) +2. **Hatchet adapter** — Rising Postgres-backed task orchestrator (YC W24) +3. **Multi-worker scaling test** — Run with 1, 4, 8, 16 workers to show scaling curves +4. **Windmill `task()` benchmark** — Add a separate test using `task()` (child jobs) for users who need per-step isolation +5. **Payload size test** — Steps that pass non-trivial data (1KB, 100KB, 1MB) to measure serialization overhead +6. **Error/retry test** — Steps that fail and retry to measure recovery overhead +7. **Long-running workflow test** — 100+ steps to measure checkpoint overhead at scale +8. **Graph improvements** — Add error bars, include version numbers in chart labels +9. **Prefect worker pool mode** — Test with `Process` work pool instead of `.serve()` for potentially better throughput +10. **Airflow CeleryExecutor** — Test with Celery + Redis for parallel task execution instead of LocalExecutor