Crawler access // GEO baseline

robots.txt + llms.txt

Generate a robots.txt that names the major AI crawlers and an llms.txt skeleton that points answer engines at your key pages. Defaults follow the rule of allowing named AI crawlers unless you have a reason to block them.

Inputs

Site base URL
AI crawlers to allow by name
Disallow paths (one per line)
llms.txt brand name
llms.txt one-line description
llms.txt summary paragraph

        
Place this at your site root as /robots.txt. The wildcard block sets site-wide rules, each named crawler is allowed explicitly, and the sitemap lines sit at the top.

      


      

        

        
Place this at your site root as /llms.txt. Replace the example links with your real pages and write each description specific to what that page answers.