What is a robots.txt file, and what should I put in it? I am worried about blocking pages Google needs by mistake.
A robots.txt file is a plain text file at the root of your site (yoursite.com/robots.txt) that tells search engine crawlers which parts of the site they may or may not crawl. It controls crawling, not indexing, so it is not a tool for hiding pages from Google.
For most small websites, a simple file is best:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xml
What to keep in mind:
- Never block CSS, JavaScript or image folders. Google needs them to render your pages properly.
- Do not use robots.txt to remove a page from search results. A blocked page can still be indexed if other sites link to it. Use a noindex meta tag instead, and leave the page crawlable so Google can see that tag.
- Block only low-value areas such as admin folders, internal search result pages or cart and checkout pages.
- Add your sitemap URL at the bottom so crawlers find it quickly.
After any change, open the robots.txt report in Google Search Console to check that Google fetched the new version, and use the URL Inspection tool on a few important pages to confirm they are not blocked. Google's own guide to robots.txt explains the full syntax if you need more advanced rules.