* fix(frontend): keep crawlers off the login page
Every page on the hub links to /user/login with itself in `rd`, so a crawler
sees one login URL per hub page — 4,311 of them in Search Console, all
rendering this same form and flagged as duplicates without a canonical. Nothing
about a login page belongs in an index, on any instance.
Mark the page noindex, as public_run already is, and ship a robots.txt that
keeps crawlers out of /user/ and /api/. The frontend is embedded as static
assets with an index.html fallback, which is why /robots.txt answered with the
app shell until now; a real file in static/ is served as itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(frontend): let crawlers fetch the login page so the noindex is seen
robots.txt disallowed /user/, which stopped a crawler fetching /user/login at
all — and a page that is never fetched never shows its noindex. The two halves
cancelled: the URLs would have moved from "duplicate" to "blocked" rather than
out of the index.
Drop the disallow, keeping /api/. And since the app is client-rendered, the
meta tag only exists after a render pass; send X-Robots-Tag on /user/* from
serve_path as well, which a crawler sees on the first fetch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
