What determines indexability

A page only appears in search results if the search engine may add it to its index. A single noindex in the code or in a server header prevents that.

How it works

Search engines work in two steps. First a crawler fetches the page. Then the search engine decides whether to add the page to its index, the database that search results are drawn from. Only what is in the index can be found.

The noindex directive says: please do not add this page. It can sit in two places:

<meta name="robots" content="noindex">
X-Robots-Tag: noindex

The first is a meta tag in the head of the page, the second an HTTP header the server sends along. Both have the same effect. The value none counts as well: it means noindex, nofollow.

bytereport checks both on the start page. If it carries noindex, that is almost always left over from development: while a site is being built it should not be found, and at launch someone forgets to lift the block. AI search services that build on a search index will not find the page either.

What matters

  • No noindex on pages that should be found. That applies to the meta tag and the header alike. Check both, because you cannot see the header in the page source.
  • noindex is not robots.txt. A block in robots.txt only forbids fetching. A blocked page can still end up in the index if others link to it. Conversely, a page has to be fetchable for the search engine to see its noindex at all.
  • Use it on purpose. For pages such as internal search results, thank-you pages or test environments, noindex is right. On the start page it almost never is.
  • Status 200. Only pages that answer with status 200 are indexed. Error pages and redirects are not.

How to set it up

In WordPress the block is under Settings · Reading as “Discourage search engines from indexing this site”. Untick it. Other CMSs and SEO plugins have a similar setting per page or for the whole site.

If the noindex comes as a header, it lives in the web server configuration, for example as add_header X-Robots-Tag "noindex"; in nginx or Header set X-Robots-Tag "noindex" in Apache. Whoever runs the server changes it: you, your host or your agency. Afterwards you can ask for the start page to be indexed again in Google Search Console.

More on Crawlability

Check your website now

The SEO check shows how search engines see your page.

Check website