Hopefully fix flaky Consul fencing test

[banks]

I've seen this test fail in our Enterprise repo several times now with:

2023-09-25T15:10:52.9999552Z     consul_fencing_test.go:168:
2023-09-25T15:10:53.0000151Z         	Error Trace:	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/external_tests/consul_fencing_binary/consul_fencing_test.go:168
2023-09-25T15:10:53.0000414Z         	Error:      	Received unexpected error:
2023-09-25T15:10:53.0001126Z         	            	client 1 error: unable to perform patch: error performing merge patch to /test/data/data: Error making API request.
2023-09-25T15:10:53.0001258Z
2023-09-25T15:10:53.0001619Z         	            	URL: PATCH https://127.0.0.1:32875/v1/test/data/data
2023-09-25T15:10:53.0001841Z         	            	Code: 400. Errors:
2023-09-25T15:10:53.0001965Z
2023-09-25T15:10:53.0003027Z         	            	* Waiting for the primary to upgrade from non-versioned to versioned data. This backend will be unavailable for a brief period and will resume service when the primary is finished.
2023-09-25T15:10:53.0003275Z         	Test:       	TestConsulFencing_PartitionedLeaderCantWrite

This is due to it relying on the KV-v2 mount which has an asynchronous upgrade procedure on mount which must complete before writes are accepted.

This is true for Community Edition Vault too, however the process is usually quick (I've not been able to recreate the failure locally in either version). It fails more frequently in CI in Enterprise because in Enterprise the non-active nodes are performance standbys that not only have to wait for the primary to complete the upgrade, they also have to notice that by polling some state which increases the chances that the first write to one will error.

This fix should avoid that in either case by spending a few seconds ensuring that writes are available through a non-active node before we start the main body of the test where we treat errors more seriously.

Edit: Nick brought to my attention that we fixed mount upgrades to be synchronous for empty mounts last year which explains why this test never flakes in CE or on the active node in Ent. In Ent though the perf standbys are still async checking when the active node is done which gives the race between that and the first request.

[github-actions]

Build Results:
All builds succeeded! ✅

[github-actions]

CI Results:
All Go tests succeeded! ✅

[miagilepner]

it might be worth checking the error to verify that it is an upgrade error, and erroring out if it's something else.

[miagilepner]

also, typo for the word faile

[hghaf099]

Do we need this? Maybe we could just include an aggregate of all the error messages to the fatal message in ctx.Done case?

[banks]

I originally thought I wouldn't sniff the error message just to be more robust (both against other errors that are similarly transient during setup and against us changing the specific error message in this case), but I guess it might be nicer to fail fast?

If we catch some other non-transient error here we'll at least slow down on the retries and only output it a handful of times before we fail.

What do you think is preferable?

[miagilepner]

I don't feel strongly about it, I'm fine to merge it as is