Support mutliple cluster contact points on the standalone Gateway

[Zelldon]

Is your feature request related to a problem? Please describe.

Currently we support only one cluster contact point, which is problematic when this contact point is currently not available.
This can happen in multiple situations, directly on start but also on a network partition. Imagine the gateway was partitioned it removes all there nodes, because it can't connect to them. When the network partition is over but the contact point is not available it will not be able to connect to the rest of the cluster.

We handle this in k8s setup via dns, since we use as contact point a domain which can be resolved to multiple nodes. This is not easy possible on bare metal, docker setups or integration tests.

Especially on the last one I would like to have these kind of feature, because it makes it possible to write more different integration tests and make them also less flaky.

It is also a bit confusing why we support only one in the gateway and multiple at the broker side.

Note: The contact point is directly used as initial contact point in atomix.

Describe the solution you'd like

Similar to the initial contact points in the broker I would expect a list of contact points.

Describe alternatives you've considered
Idk how to fix that in it tests or docker compose setup?

[npepinpe]

Currently blocks us from testing the behaviour of a gateway after network partition.

[npepinpe]

I'm downgrading this to mid until we make a decision on whether or not standalone gateway will be the deployment for Cloud GA.

[Zelldon]

Would be interesting if we deploy standalone gateway in cloud @npepinpe

[npepinpe]

In cloud we use the broker's service URL, which already performs load balancing, so I'm not sure if having multiple contact points is all that useful for that particular set up. I think it's still mostly useful for deployments where there is nothing between the gateway and brokers, i.e. bare metal deployments.

[Zelldon]

Ah true you're right.

[aivinog1]

Hey @Zelldon @npepinpe!
We are currently deploying Zeebe on our bare metal environment and it would be nice if we could have load balancing support for this out of the box. I can contribute it if you are generally is okay with this feature in Zeebe :) I would also vote for the same in #9413.

[npepinpe]

Go for it. Keep in mind that there's no real load balancing happening here - this list is just a way for the gateway to join the cluster initially. Once it has joined, it will talk to the different brokers based on the topology of the cluster.

The only asterisk here is make sure everything works as is with the embedded gateway.