Authentication load in a multi-tenant cluster using RBAC

[kennydo]

I'd like to run a multi-tenant etcd cluster where we use RBAC to limit each tenant's access to a particular prefix. For example, we'd like tenant X to have a user that grants them access only to the /userX prefix. When we add a new tenant, we'd do a series of steps to setup the user. It would look like: create role roleX, grant the role access to the /userX prefix, create user userX, then grant roleX to userX.

If I understand correctly, each of those actions increases the auth store revision. Since auth tokens are checked against the current auth store revision, this means that:

• When we add user 100, users 1-99 would all have simultaneously have invalid auth tokens and need to re-authenticate.
• Because there are multiple auth store changes needed to setup user 100 and role 100, user 99 might need to re-authenticate several times. For example, if we create role role100, then user99 re-authenticates, then we grant that role access to /user100, then user99 would need to re-authenticate again, etc.
• The thundering herd of re-authentications would get larger as the number of users increases.

So in a worst case scenario, the timeline of events could look like:
T=0 users 1-99 authenticate
T=1 admin creates a new role
T=2 users 1-99 try to do an operation, but it fails due to the auth revision being old. They re-authenticate.
T=3 admin grants the new role some permissions
T=4 users 1-99 tries again to do an operation, but it fails due to the auth revision being old. They re-authenticate.
T=5 admin creates a new user
T=6 users 1-99 tries again to do an operation, but it fails due to the auth revision being old. They re-authenticate.
...and so on

Does this sound accurate? Do you have any advice on how to better control access in this scenario?

One potential workaround is lowering the bcrypt cost, which would decrease the server load of each authenticate call. But I think there's a problem with the pattern of invalidating every other user's auth token whenever a new user is set up.

[mitake]

hi @kennydo , if the cost of re-authenticate is too high, is it possible for you to try CN based auth? https://etcd.io/docs/v3.5/op-guide/authentication/rbac/#using-tls-common-name In this case you can skip Authenticate() so the cost from bcrypt can be avoided.

[kennydo]

We have a proxy in front of our etcd members that terminates TLS, so unfortunately the etcd members can't access the CN of the client certs.

We're not using Envoy as the proxy, but it looks like they have a pattern where they pass along client cert information via a HTTP Header (https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert). Do you think that might be an appropriate feature to add to etcd?

[ahrtr]

The thundering herd of re-authentications would get larger as the number of users increases

Do you have any performance test data to share? I don't expect the RBAC operation is frequent, so usually the re-authentication won't too frequent either.

Indeed JWT tokens need re-authencation when RBAC changes; simple tokens do not need re-authentication, but it isn't recommended for production use.

We're not using Envoy as the proxy, but it looks like they have a pattern where they pass along client cert information via a HTTP Header (https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert). Do you think that might be an appropriate feature to add to etcd?

No such plan (for grpc-proxy) yet. What proxy are you using? Have you tried Envoy, can it resolve this using x-forwarded-client-cert ?