docs: Example on how to monitor Windmill (#2938)

* docs: Example on how to track arbitrary job metrics * Add how to monitor Windmill servers and workers * Update README.md --------- Co-authored-by: gbouv <guillaume@windmill.dev>
2026-08-18 16:02:10 +00:00 · 2023-12-30 11:30:48 +09:00
parent 19246bcf3e
commit 329cf836bd
12 changed files with 1716 additions and 0 deletions
@@ -0,0 +1,124 @@
+Windmill monitoring with Prometheus
+===================================
+
+## Intro
+
+Windmill servers and workers both produces metrics Prometheus can scrap.
+
+Individual jobs producing their own metric can be difficult to achieve due to the ephemeral nature of a job. Prometheus might not have the time to scrape the metric if the job is too quick. Prometheus has developed the [Prometheus Gateway](https://prometheus.io/docs/practices/pushing/) exactly for this usecase, and we're going to go through the initial setup. By the end of this example, you'll have a Windmill stack running a job that produces its own metric, and a Grafana simple dashboard displaying it.
+
+Prometheus push gateway is hosted in [this repository](https://github.com/prometheus/pushgateway)
+
+## Setup
+
+First, we need to setup an entire stack composed of:
+- Windmill: a database, a server, and at least one worker
+- Prometheus: the server (if you don't already have one), and the gateway to which metrics will be pushed
+- Grafana to visualize the metrics (optional)
+
+We have assembled a docker compose with all those services. You can run it with a simple:
+```bash
+docker compose up -d
+```
+Once all the services are running, Windmill can be accessed at http://localhost:8000, Prometheus at http://localhost:9090, Grafana at http://localhost:3000 (credentials are set in [grafana.ini](./grafana/grafana.ini) config file).
+
+The Gateway exposes the metrics on http://localhost:9091/metrics and also has an interface at http://localhost:9091 to manage metric groups (see the Gateway documentation for what a metric group is). You can check that Prometheus can collect them directly on [Prometheus status page](http://localhost:9090/targets)
+
+## Monitoring Windmill servers and workers
+
+Metrics need to be enabled on Windmill server and workers. They can be enabled only on Windmill Enterprise Edition by setting the environment variable `METRICS_ADDR` to `1` on each container or toggle on "Expose Metrics" in instance settings -> core (additional "debug-level" metrics can be enabled in the instance settings > debug menu).
+
+Once metrics are enabled, Prometheus needs to discover Windmill service containers. Using docker-compose, it requires a few adjustments:
+- Each Windmill containers need to expose the metrics port (`8001` by default) so that Prometheus discovery knows which ports to scrap. This is done by adding the value `8001` to the `expose` block to both `windmill_server` and `windmill_worker`.
+- Each Windmill container needs to be labelled so that Prometheus service discovery filters out other services. Here we're using the `prometheus-job=windmill_server` and `prometheus-job=windmill_worker` in the docker compose to differentiate between server and worker
+- The Prometheus container needs to have access to the docker socket to discover other containers. To achieve this, Prometheus container needs to be run as root and the docker socket needs to be mounted on the container (see `user: root` and `/var/run/docker.sock:/var/run/docker.sock` in the `prometheus` block of docker-compose.yml) 
+- Finally, the following block should be added to `prometheus.yml` config file:
+
+```yaml
+scrape_configs:
+  - job_name: "windmill_server"
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+    relabel_configs:
+      - source_labels: [__meta_docker_container_label_prometheus_job]
+        regex: windmill_server
+        action: keep
+    scrape_interval: 1s
+
+  - job_name: "windmill_worker"
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+    relabel_configs:
+      - source_labels: [__meta_docker_container_label_prometheus_job]
+        regex: windmill_worker
+        action: keep
+    scrape_interval: 1s
+```
+
+With all this in place, Windmill servers and workers metrics will be scrapped by Prometheus and can be visualized in Grafana. We've added a simple dashboard in [grafana/dashboards/](./grafana/dashboards) showing the most interesting metrics.
+
+![Windmill monitoring dashboard](./static/windmill_monitoring_dashboard.png "Windmill monitoring dashboard")
+
+Note: If your Windmill containers are split onto multiple docker daemon, the above still works. Instead of pointing to the docker socket in Prometheus' `docker_sd_config` block, simply expose docker daemon to http or https and point to the remote docker daemon host: `http://remote_docker_daemon:2375`. Multiple remote docker daemons can be listed.
+
+## Producing metrics from individual Windmill jobs
+
+Here is an example of how a job can "push" its own metrics to Prometheus via Prometheus Gateway
+
+### Windmill script
+
+Now we will create a script in Windmill that pushes a value to the Prometheus Gateway. We will do it in Python using [prometheus-client](https://github.com/prometheus/client_python), but other clients are available for [Golang](https://github.com/prometheus/client_golang) or [Typescript](https://github.com/siimon/prom-client), and for bash metrics can be pushed via [simple CURL commands](https://github.com/prometheus/pushgateway?tab=readme-ov-file#command-line).
+
+In Windmill, create a new Python script (named `u/admin/random_number_metric_script` in this tutorial) with the following content:
+
+```python
+import os
+import random
+from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
+
+PROMETHEUS_GATEWAY_URL = "prometheus_gateway:9091"
+
+def main():
+    job_path = os.environ.get("WM_JOB_PATH")
+    registry = CollectorRegistry()
+    gauge = Gauge(
+        "job_records_processed",
+        "Number of records processed for {}".format(job_path),
+        registry=registry,
+    )
+    val = random.randint(0, 100)
+    print("Storing metrics value: ", val)
+    gauge.set(val)
+    push_to_gateway(PROMETHEUS_GATEWAY_URL, job=job_path, registry=registry)
+```
+
+It is quite simple. It generate a random number and pushed it via a Gauge metric named `job_records_processed` to the Prometheus Gateway. Note that we use the script path as the `job` label, so that all runs of this script will push to the same metric. Other labels (such as the job ID for example) can also be used via the `grouping_key` argument that `push_to_gateway` accepts.
+
+Once the script is created, you can create a schedule that runs is every 5 seconds for example, such that a new value is pushed regularly.
+
+### Visualizing the metric
+
+If you chose to not run Grafana, the raw metric value can be seen in Prometheus UI directly. Go to http://localhost:9090/graph and input the following in the Expression bar: 
+```promql
+job_records_processed{exported_job="u/admin/random_number_metric_script"}
+```
+
+![Metric visualization in Prometheus UI](./static/prometheus_metric_viz.png "Metric visualization in Prometheus UI")
+
+In Grafana, a similar visualization can be achieved. We've added a dashboard JSON in [grafana/dashboards](./grafana/dashboards/) and the easiest is to import it directly in your Grafana.
+
+The first panel is a simple representation of the `job_records_processed` metric. We've adjusted the `Min Step` to 5 seconds because we know our job runs every 5 seconds and we ideally want to display all the measurement points.
+
+![Job records processed raw metric](./static/job_records_processed_raw.png "Job records processed raw metric")
+
+The second panel is here to highlight a limitation to be aware of when using the Prometheus Gateway. It displays the instant value of the `job_records_processed` metric. The main issue with using the Gateway is that the value pushed to it will persist until a new value is pushed. Which means that if the schedule is somehow stopped, or if the script starts failing and stops pushing to Prometheus Gateway, the value will remain constant. It might not be ideal to spot any kind of undesired behavior. Instead what would be nice is to have the metric drop to a default value, like `0`. Thankfully the Gateway comes with a built-in metric called `push_time_seconds` which stores the timestamp of the last successful push for any given metric and label. You can combine the 2 metrics using basic functions to easily create an "instant" representation of signal. This is what we've done in the second panel:
+
+![Job records processed instant metric](./static/job_records_processed_instant.png "Job records processed instant metric")
+
+We simply multiply the `job_records_processed` with the `irate` of the `push_time_seconds` metric for the same label, and we set the `Min Step` to `1s` for the `irate` to properly work. 
+
+We obtain the following simple yet useful dashboard:
+
+![Grafana dashboard with the 2 panels](./static/grafana_dashboard.png "Grafana dashboard with the 2 panels")
+
+In between the 2 light blue vertical lines, we manually stopped the Windmill schedule to show that the raw metric (at the top) remains constant, while its instant representation drops to zero.
@@ -0,0 +1,100 @@
+version: "3.7"
+
+services:
+  db:
+    image: postgres:14
+    volumes:
+      - db_data:/var/lib/postgresql/data
+    environment:
+      POSTGRES_PASSWORD: changeme
+      POSTGRES_DB: windmill
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  windmill_server:
+    image: ghcr.io/windmill-labs/windmill-ee:main
+    deploy:
+      replicas: 1
+    labels:
+      - prometheus-job=windmill_server
+    expose:
+      - 8001
+    ports:
+      - 8000:8000
+    environment:
+      - DATABASE_URL=postgres://postgres:changeme@db/windmill?sslmode=disable
+      - MODE=server
+      - METRICS_ADDR=1
+    depends_on:
+      db:
+        condition: service_healthy
+
+  windmill_worker:
+    image: ghcr.io/windmill-labs/windmill-ee:main
+    pull_policy: always
+    deploy:
+      replicas: 3
+    labels:
+      - prometheus-job=windmill_worker
+    expose:
+      - 8001
+    restart: unless-stopped
+    environment:
+      - DATABASE_URL=postgres://postgres:changeme@db/windmill?sslmode=disable
+      - MODE=worker
+      - WORKER_GROUP=default
+      - METRICS_ADDR=1
+    depends_on:
+      db:
+        condition: service_healthy
+
+  prometheus:
+    image: prom/prometheus
+    user: root
+    volumes:
+      - prometheus_data:/prometheus
+      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
+      - /var/run/docker.sock:/var/run/docker.sock:ro # for service discovery
+    ports:
+      - 9090:9090
+    environment:
+      POSTGRES_PASSWORD: changeme
+      POSTGRES_DB: windmill
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--storage.tsdb.path=/prometheus"
+      - "--storage.tsdb.retention.time=365d"
+
+  prometheus_gateway:
+    image: prom/pushgateway
+    ports:
+      - 9091:9091
+    command:
+      - "--persistence.file=/data/persistence.dat"
+    volumes:
+      - prometheus_gateway_data:/data
+
+  grafana:
+    image: grafana/grafana
+    container_name: grafana
+    environment:
+      - GF_PATHS_CONFIG=/etc/grafana/grafana.ini
+    deploy:
+      replicas: 1
+    depends_on:
+      - prometheus
+    ports:
+      - 3030:3000
+    volumes:
+      - grafana_data:/var/lib/grafana
+      - ./grafana/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yml
+      - ./grafana/grafana.ini:/etc/grafana/grafana.ini
+
+volumes:
+  db_data: null
+  prometheus_data: null
+  prometheus_gateway_data: null
+  grafana_data: null
@@ -0,0 +1,319 @@
+{
+  "__inputs": [
+    {
+      "name": "DS_PROMETHEUS",
+      "label": "Prometheus",
+      "description": "",
+      "type": "datasource",
+      "pluginId": "prometheus",
+      "pluginName": "Prometheus"
+    }
+  ],
+  "__elements": {},
+  "__requires": [
+    {
+      "type": "grafana",
+      "id": "grafana",
+      "name": "Grafana",
+      "version": "10.2.3"
+    },
+    {
+      "type": "datasource",
+      "id": "prometheus",
+      "name": "Prometheus",
+      "version": "1.0.0"
+    },
+    {
+      "type": "panel",
+      "id": "timeseries",
+      "name": "Time series",
+      "version": ""
+    }
+  ],
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 7,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 3,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "disableTextWrap": false,
+          "editorMode": "builder",
+          "expr": "job_records_processed{exported_job=\"$script_path\"}",
+          "fullMetaSearch": false,
+          "includeNullMetadata": true,
+          "instant": false,
+          "interval": "5s",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A",
+          "useBackend": false
+        }
+      ],
+      "title": "Processed records (raw)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "description": "",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "bars",
+            "fillOpacity": 100,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 17,
+        "w": 24,
+        "x": 0,
+        "y": 7
+      },
+      "id": 2,
+      "interval": "15s",
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "disableTextWrap": false,
+          "editorMode": "code",
+          "expr": "irate(push_time_seconds{exported_job=\"$script_path\"}[$__interval]) * job_records_processed{exported_job=\"$script_path\"} / irate(push_time_seconds{exported_job=\"$script_path\"}[$__interval])",
+          "fullMetaSearch": false,
+          "hide": false,
+          "includeNullMetadata": true,
+          "instant": true,
+          "interval": "1s",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "B",
+          "useBackend": false
+        }
+      ],
+      "title": "Processed records (instant)",
+      "transformations": [],
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "5s",
+  "schemaVersion": 39,
+  "tags": [],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "PBFA97CFB590B2093"
+        },
+        "hide": 0,
+        "includeAll": false,
+        "multi": false,
+        "name": "prometheus",
+        "options": [],
+        "query": "prometheus",
+        "queryValue": "",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      },
+      {
+        "current": {},
+        "datasource": {
+          "type": "prometheus",
+          "uid": "${DS_PROMETHEUS}"
+        },
+        "definition": "label_values(exported_job)",
+        "hide": 1,
+        "includeAll": false,
+        "multi": false,
+        "name": "script_path",
+        "options": [],
+        "query": {
+          "qryType": 1,
+          "query": "label_values(exported_job)",
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 0,
+        "type": "query"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-15m",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "Job metrics Dashboard",
+  "uid": "a31424a8-5eaa-4d26-bded-19570145b7bd",
+  "version": 8,
+  "weekStart": ""
+}
@@ -0,0 +1,7 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
@@ -0,0 +1,3 @@
+[security]
+admin_user = windmill
+admin_password = changeme
@@ -0,0 +1,28 @@
+global:
+  scrape_interval: 1s
+  external_labels:
+    env: "infrastructure"
+
+scrape_configs:
+  - job_name: "windmill_server"
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+    relabel_configs:
+      - source_labels: [__meta_docker_container_label_prometheus_job]
+        regex: windmill_server
+        action: keep
+    scrape_interval: 1s
+
+  - job_name: "windmill_worker"
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+    relabel_configs:
+      - source_labels: [__meta_docker_container_label_prometheus_job]
+        regex: windmill_worker
+        action: keep
+    scrape_interval: 1s
+
+  - job_name: "prometheus_gateway"
+    static_configs:
+      - targets: ["prometheus_gateway:9091"]
+    scrape_interval: 1s