# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: * Allow: / # ⚠️ Sign-in and sign-up are deliberately crawlable, and deliberately indexable # (they carry no `noindex` — see the views). Sitelinks, the child links under a # brand result, are drawn only from indexed pages, so these two are eligible on # purpose. # # Behind the /users Disallow below they were *discovered but uncrawlable* — # linked from the nav and footer of every marketing page, so Google found them # however it was told not to. That is the one state that lists a bare, # description-less URL on a brand query, competing with the homepage on exactly # the query the homepage should own. Same rule as /s, which this file already # follows: a URL behind a Disallow is never fetched, so nothing in the page can # be read or obeyed. Longest-match wins, so these beat the Disallow. # # The rest of /users stays closed — account pages, and password-reset URLs that # carry single-use tokens. Allow: /users/sign-in Allow: /users/sign-up Disallow: /diagrams Disallow: /billing Disallow: /api-keys Disallow: /api_keys Disallow: /users Disallow: /view # Link-preview crawlers get the same rules *except* /view, so a shared guest # diagram unfurls with a preview instead of a bare URL. These bots build cards, # they don't index, so keeping /view out of search results is unaffected. # (A bot obeys only its own group, so each one repeats the private paths.) User-agent: Twitterbot Disallow: /diagrams Disallow: /billing Disallow: /api-keys Disallow: /api_keys Disallow: /users User-agent: facebookexternalhit Disallow: /diagrams Disallow: /billing Disallow: /api-keys Disallow: /api_keys Disallow: /users User-agent: LinkedInBot Disallow: /diagrams Disallow: /billing Disallow: /api-keys Disallow: /api_keys Disallow: /users User-agent: Slackbot Disallow: /diagrams Disallow: /billing Disallow: /api-keys Disallow: /api_keys Disallow: /users User-agent: WhatsApp Disallow: /diagrams Disallow: /billing Disallow: /api-keys Disallow: /api_keys Disallow: /users Sitemap: https://flostep.dev/sitemap.xml