GCP PubSub Authorisation Issue In Java Using Service Account

[shubhwip]

Describe the bug
I am facing below issue while publishing messages to various topics from one java service.

exception is com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\ncom.google.cloud.spring.pubsub.core.PubSubDeliveryException: Publishing to projects/testing-development/topics/testing-bacd-testing-1 topic failed.; nested exception is com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\n\tat com.google.cloud.spring.pubsub.core.publisher.PubSubPublisherTemplate$1.onFailure(PubSubPublisherTemplate.java:111)\n\tat com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n\tat com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)\n\tat com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)\n\tat com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:52)\n\tat com.google.cloud.pubsub.v1.Publisher$OutstandingBatch.onFailure(Publisher.java:545)\n\tat com.google.cloud.pubsub.v1.Publisher$OutstandingBatch.access$1600(Publisher.java:512)\n\tat com.google.cloud.pubsub.v1.Publisher$3.onFailure(Publisher.java:488)\n\tat com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n\tat com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200)\n\tat com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)\n\tat com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)\n\tat com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)\n\tat com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)\n\tat com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n\tat com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)\n\tat io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)\n\tat io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)\n\tat io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\n\tat com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)\n\tat com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)\n\tat com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)\n\t... 24 common frames omitted\nCaused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.\n\tat io.grpc.Status.asRuntimeException(Status.java:535)\n\t... 16 common frames omitted\n","context":"default"}

My Setup

We are trying to publish to many topics from one java service. We have created

topics1 => using a wrapper library over spring-cloud-gcp-pubsub and using it to publish to some of the topics
topics2 => using spring-cloud-gcp-pubsub directly to publish to other topics and this is behind a conditional property.
We are seeing above error for all the topics in topics1 when conditional property for topics2 is enabled.

### Attempts

• There was library version mismatch, fixed it.
• Tried to run it locally with minimal setup and it is working fine.
• Checked permissions of service account being used and it has all the required permissions.
• Enabled pubsub auditing as well to find out but pubsub doesn't log Publish API calls.

I would like to find out what is this user who is not authorised. Is there some access_token invalidation happening ?

Thanks in advance.

[burkedavison]

I would like to find out what is this user who is not authorised. Is there some access_token invalidation happening ?

You can enable logging for the auth library to see if it provides any additional clues as to what is happening -- see the logging section of the Google java client library TROUBLESHOOTING.md for an example of how to do that.

[shubhwip]

Thanks @burkedavison, I have a bearer token with me, can i decode this by any chance ?
I am suspecting if scopes are getting changed.

{"timestampSeconds":1712307807,"timestampNanos":830000000,"severity":"INFO","thread":"Gax-14","logger":"com.google.api.client.http.HttpTransport","message":"{\"access_token\":\"XXXXXXXXXXXX\",\"expires_in\":3598,\"token_type\":\"Bearer\"}","context":"default"}

Publish Request

{"timestampSeconds":1712307913,"timestampNanos":796000000,"severity":"DEBUG","thread":"grpc-default-worker-ELG-1-1","logger":"io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler","message":"[id: 0xe0a011e9, L:/10.70.5.54:37214 - R:pubsub.googleapis.com/108.177.15.95:443] OUTBOUND HEADERS: streamId\u003d9 headers\u003dGrpcHttp2OutboundHeaders[:authority: pubsub.googleapis.com:443, :path: /google.pubsub.v1.Publisher/Publish, :method: POST, :scheme: https, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.39.0, x-goog-api-client: gl-java/17.0.4 gapic/1.113.5 gax/1.66.0 grpc/1.39.0, x-goog-request-params: topic\u003dprojects/testing-development/topics/testing-bacd-testing-1, grpc-accept-encoding: gzip, authorization: Bearer XXXXXXXXXXXX, grpc-timeout: 4999700u] streamDependency\u003d0 weight\u003d16 exclusive\u003dfalse padding\u003d0 endStream\u003dfalse","context":"default"}

I tried this

curl -i "https://oauth2.googleapis.com/tokeninfo?access_token=$TOKEN"
HTTP/2 400 
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Fri, 05 Apr 2024 09:06:40 GMT
content-type: application/json; charset=UTF-8
vary: X-Origin
vary: Referer
vary: Origin,Accept-Encoding
server: ESF
x-xss-protection: 0
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
accept-ranges: none

{
  "error": "invalid_token",
  "error_description": "Invalid Value"
}

[meltsufin]

@westarle Can you please help with the question about decoding Bearer tokens?

[westarle]

@shubhwip Does the token look like ya29.... or does it look like a JWT-structured token? If it is the latter, you can use something like jwt.io to decode!