Block search engines from indexing the head version

[mdjermanovic]

We want the pages under https://eslint.org/docs/latest/ to appear in search results, so we should probably block the pages under https://eslint.org/docs/head/ from indexing.

[mdjermanovic]

Per https://developers.google.com/search/docs/advanced/robots/intro robots.txt is not the place to do this. In fact, per https://developers.google.com/search/docs/advanced/crawling/block-indexing the pages should not be blocked from crawling.

We could set up X-Robots-Tag: noindex response header and/or add <meta name="robots" content="noindex"> tags.

I prepared eslint/eslint#16016 if we opt for the tag approach.

[nzakas]

I think a better approach might be to just set the canonical URL of pages in docs/head to be docs/latest. That way, Google will index the content but will consider it another version of the latest docs, which I think is the correct interpretation.

[mdjermanovic]

I tried googling SEO best practices regarding docs for multiple versions of same product, but couldn't find anything conclusive. I think of pages with same canonical URL as pages that provide essentially same information, either exact same or even reduced or enhanced. Docs for different versions of a software package could provide substantially different information.

Either way, regardless of my theoretical interpretation of canonical URLs, the solution you proposed seems to work well on https://sinonjs.org/ so I'm 👍 for trying that approach.

[mdjermanovic]

But, we'll have a situation where some pages exist only in the head version for a certain period of time. For example, when we merge a new rule. At that point, canonical URL that points to the latest version will return 404, which doesn't seem right.

[nzakas]

While that can happen, that’s an edge case, and one that will be resolved within two weeks each time it occurs. So I think it’s a safe way to start for now. We can always adjust things if we find they aren’t working correctly down the line.

[mdjermanovic]

As agreed in eslint/eslint#16016, this issue is fixed by adding noindex meta tags.