Remove generated pages (category and keyword pages) from search index

[adborden]

Generated pages like the category and keyword pages are indexed in search.gov, cluttering search results. If a user is using search, they're probably looking for content, I don't think a user would ever be searching for a specific category or keyword. If a user was looking for categories and keywords, they should be using the category index and keyword index pages and not search.

How to reproduce

1. Go to https://search.usa.gov/search?utf8=%E2%9C%93&affiliate=datagov-resources&query=how+to+get+data+on+data.gov

Expected result

I should see relevant content pages specific to my query.

Actual result

Generated, non-content pages should not be displayed in results.

You can see in the screenshot below, 12 of the 20 results are category or keyword pages.