What robots.txt does
robots.txt is a text file in the root of your website. It tells crawlers which areas to skip and gives the address of your sitemap.
How it works
Before a crawler (a program that fetches pages for a search engine) visits your website, it reads https://your-domain/robots.txt. The file consists of groups. Each starts with User-agent: and names the crawler it applies to; * stands for all of them. Rules follow below:
User-agent: *
Disallow: /internal/
Allow: /
Sitemap: https://www.example.com/sitemap.xmlDisallow forbids a path, Allow permits it. A crawler follows only the group that names it most precisely. If Googlebot has a group of its own, the rules under * do not apply to it.
If there is no robots.txt (status 404), crawlers may fetch everything. That is fine. A file with a sitemap line still helps, because search engines then find your sitemap straight away.
What matters
- No
Disallow: /for everyone. This one line in theUser-agent: *group locks the whole website. It is often left over from development. bytereport rates it as a failure. - A readable text file. The file must answer with status 200 as text. If the server returns an HTML page instead, such as the start page or an error page, crawlers cannot read it. If it answers with a server error (5xx), Google temporarily treats the whole site as blocked.
- Blocking is not hiding. A blocked page is not fetched, but it can still be in the index if others link to it. To keep a page out of search results you need
noindex; see What determines indexability. - No secrets. The file is public. Blocking a path there also announces it.
- Sitemap with the full address. The
Sitemap:line needs the complete URL, includinghttps://.
AI crawlers read robots.txt too. How to allow or block them separately is covered in What AI crawlers may do.
How to set it up
The file sits in the root of the website, at /robots.txt on your domain. Many CMSs generate it themselves; in WordPress, for instance, you edit it through an SEO plugin. Otherwise upload it via FTP or your host's file manager. A good starting point:
User-agent: *
Allow: /
Sitemap: https://…/sitemap.xmlReplace the dots with your domain. Google Search Console shows under Settings whether Google can read the file.