Robots.txt Generator
Create a correct robots.txt file, with your sitemap and optional AI crawler blocking.
These don't affect Google Search. Blocking a search assistant (like OAI-SearchBot or PerplexityBot) means your site won't appear in its answers.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/
Save this as a plain text file named robots.txt in the main folder of your site, so it opens at yoursite.com/robots.txt.
Common mistakes
- Disallow does not remove a page from Google. To hide a page, use a noindex tag instead.
- Never block your CSS, JavaScript or image folders. Google needs them to understand your pages.
- robots.txt is public. Don't list secret folders here; protect them with a password.
You might also like
Related calculators and converters
Meta Tag & Share Preview
See how your page looks on Google, Facebook and WhatsApp, and get ready-made meta tags.
Local Business Schema Generator
Make LocalBusiness structured data for your shop or office in Nepal.
Website Hosting Checker
Find out who hosts any website, its IP address, network and DNS provider.
UTM Link Builder
Build tracked links for Facebook, TikTok, WhatsApp and email campaigns.
About the Robots.txt Generator
robots.txt is a small text file at the root of a website that tells search engine crawlers which parts of the site they may visit. A good robots.txt lets Google crawl your public pages, keeps crawlers out of private areas like admin and checkout pages, and lists your sitemap.
You can also choose whether AI crawlers may use your content. Blocking training crawlers such as GPTBot or Google-Extended doesn't change your Google Search rankings.
How to use the Robots.txt Generator
- Choose Allow (recommended) unless this is a test site.
- Pick folders to keep crawlers out of, such as /admin/, /cart/ or /checkout/, or type your own.
- Optionally tick AI crawlers you want to block.
- Add the full URL of your sitemap.
- Copy the result, save it as robots.txt and upload it to your site's main folder.
Tips
- Check it opens at yoursite.com/robots.txt after uploading.
- To keep a page out of Google, use a noindex tag — robots.txt alone doesn't remove pages.
Frequently asked questions
Where do I put the robots.txt file?
In the main (root) folder of your website, so it opens at yoursite.com/robots.txt. A robots.txt in any other folder is ignored.
Does robots.txt remove a page from Google?
No. It only asks crawlers not to visit. A blocked page can still appear in results if other sites link to it. To keep a page out of Google, use a noindex meta tag and let Google crawl it.
Should I block AI crawlers?
It's your choice. Blocking training crawlers keeps your content out of future AI models. Blocking search assistants such as OAI-SearchBot or PerplexityBot also removes your site from their answers, which can cost you visitors.
Can a wrong robots.txt hurt my site?
Yes. 'Disallow: /' under User-agent: * blocks your whole site. Only use 'Block everything' on test or staging websites.