Consider making more types internal

[martincostello]

Review the public API and see if there's any types that are public that don't need to be, such as SwaggerMiddleware. That gives us more agility to change implementation details, like adding new constructor overloads (see #2801).

It may be that some types don't need to be public and are only done so through oversight or to be accessible to unit tests. In the case of tests, it might be too tricky to do with the way that some assemblies are strong named and others aren't, as [InternalsVisibleTo] requires IVT'd assemblies to also be strong named, and all dependencies of a project need to be strong named for it to be, which creates issues where a test project depends on one project that is signed and another that isn't.

Context: #2418 (comment)