keep crawlers off the login page (#11057)

* fix(frontend): keep crawlers off the login page

Every page on the hub links to /user/login with itself in `rd`, so a crawler
sees one login URL per hub page — 4,311 of them in Search Console, all
rendering this same form and flagged as duplicates without a canonical. Nothing
about a login page belongs in an index, on any instance.

Mark the page noindex, as public_run already is, and ship a robots.txt that
keeps crawlers out of /user/ and /api/. The frontend is embedded as static
assets with an index.html fallback, which is why /robots.txt answered with the
app shell until now; a real file in static/ is served as itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(frontend): let crawlers fetch the login page so the noindex is seen

robots.txt disallowed /user/, which stopped a crawler fetching /user/login at
all — and a page that is never fetched never shows its noindex. The two halves
cancelled: the URLs would have moved from "duplicate" to "blocked" rather than
out of the index.

Drop the disallow, keeping /api/. And since the app is client-rendered, the
meta tag only exists after a render pass; send X-Robots-Tag on /user/* from
serve_path as well, which a crawler sees on the first fetch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

backend/windmill-api/src/static_assets.rs

@@ -107,6 +107,14 @@ fn serve_path(path: &str, original_path: &str, query: Option<&str>) -> Response<
                     .header("Cross-Origin-Resource-Policy", "cross-origin");
             }
 
+            // Login and its siblings carry a different `rd` on every page that links
+            // to them, so a crawler meets thousands of URLs for one form. The app is
+            // client-rendered, so a meta tag only exists after a render pass; the
+            // header is seen on the first fetch.
+            if original_path.starts_with("/user/") {
+                res = res.header("X-Robots-Tag", "noindex, nofollow");
+            }
+
             // Add Content-Security-Policy header for static assets when policy is set
             if !CSP_POLICY.is_empty() {
                 if let Ok(header_value) = HeaderValue::try_from(CSP_POLICY.as_str()) {

frontend/src/routes/(root)/(logged)/user/(user)/login/+page.svelte

@@ -141,6 +141,12 @@
 	}
 </script>
 
+<svelte:head>
+	<!-- Linked with a different `rd` from every page, so a crawler meets many URLs
+	     for this one form. Nothing about a login page belongs in an index. -->
+	<meta name="robots" content="noindex, nofollow" />
+</svelte:head>
+
 <!-- Anchored to the top, not centered: the card grows when the password form opens or an
 	error appears, and centering would slide the mark and the fields under the pointer. -->
 <div class="flex flex-col pt-24 pb-12 sm:px-6 lg:px-8 relative bg-surface-secondary min-h-screen">

frontend/static/robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /api/