reverseproxy: Implement retry count, alternative to try_duration

[francislavoie]

Partially addresses some of the remaining points in #4245

This makes it possible to configure an amount of retries without having a particular time limit.

If both are configured, then the first of the two limits to be reached will cause retries to stop. For example if duration is 10s and retries is 5, then if 5 fast retries happen before 10s, it will stop. If 3 slow retries happen and 10s is reached, it will stop.

I was mulling over whether we should default try_interval to 250ms as well if a retries count is configured, I think it's okay if we don't; it's unlikely "CPU will spin" if we use a retry count because the count should almost always be low enough that it won't last very long. The user can always configure the interval if they want it.

IMO this isn't a complete solution, there's still a usecase where you may want to actually continue retrying up to N amount of times even if you did configure a duration, in certain situations. I think for that we'd need to flesh out the retry_match stuff (which isn't currently supported in Caddyfile). We might also want a way to have handle_response trigger a retry according to a response matcher. And we may want a way to make try_interval do exponential backoff according to the number of tries (now that we've introduced that).

[francislavoie]

Decided to spend a bit of time adding support for lb_retry_match in the Caddyfile. Pretty much stole the code from MatchNot. Pretty simple.

[mholt]

This actually LGTM for the most part. So retries end based on a "whichever comes first" kind of policy, if I understand correctly.

[mholt]

Slick refactor. That's probably much better as its own function. Thank you!