docs: Document best practices for parallel testing

[radeksimko]

Historically the testing framework didn't have any way to run acceptance tests in parallel. We used a separate tool to achieve this and we still use it today. This worked & works in context of official providers (github.com/terraform-providers/*), but not in others and most importantly not on developer's workstations.

This is one of the reasons @bflad introduced ParallelTest in hashicorp/terraform#18688

The goal is to document this with some examples from provider which already use this helper function.

There are clear benefits we can gain by parallel testing:

• It provides a way of ensuring the provider implementation & SDK & API works correctly under real load without having to wait until end users report misbehaviours to us.
• Terraform defaults to parallelism=10 which means that it will issue up to 10 requests in parallel by default. Parallel testing therefore reflects how users would use the provider in the wild.
• This saves users & machines costly time and is one of essential features of Terraform.

There are however some (side) effects of this:

• Many APIs aren't ready to process more than 1 (or N) requests at the same time, without documenting this, or even knowing this. Maintainers often learn this hard-way through duplicate IDs returned for different resources, HTTP 500 responses, or otherwise misbehaving APIs.
  • This then leads to maintainers reaching to API vendors to fix their APIs, and/or implement various workarounds in providers, e.g. mutexkv to practically prevent Terraform from sending parallel requests to the affected API.
• Many APIs implement some form of rate limiting, to prevent abuse.
• Many APIs have shared namespaces which expect unique resource names. Provider developers may use static names in tests, which then makes tests to fail as a result of name clash.