Spring Cloud GCP Pub/Sub reconnection : Subscriber doesn't receive message after subscription is delete and recreated with same name for same topic

[Vaageesh]

Describe the bug

• I had a topic and subscription created and running subscriber. Subscriber is not receiving the message when subscription is deleted and recreated subscription with same name. I waited for more than 5 minutes but subscriber didn't receive the messages.

Sample

1. Subscribe to some subscription.
2. Delete the subscription while subscriber is still active
3. Recreate subscription with same name
4. Publish message to topic.
   Expected that pubSub client must reconnect to same subscription after few mins.

I am wondering if any configuration available for this?

[burkedavison]

When I replicate the above steps, I am able to receive the message from the recreated subscription; however, when the client attempts to ACK, I get the following exceptions:

NOT_FOUND: Resource not found (resource=projects/.../subscriptions/my-subscription-123).

and

INVALID_ARGUMENT: You have passed a subscription that does not belong to the given ack ID (resource=projects/.../subscriptions/my-subscription-123).

Found the following thread that discusses this behavior: https://groups.google.com/g/cloud-pubsub-discuss/c/seAgY28sVc0

Regarding the "Resource not found (resource=)" error: are you deleting (and re-creating) subscriptions while using them (i.e,. while pull()'ing and acknowledge()'ing messages on them)? Subscriptions are persistent resources that shouldn't just "disappear".

Regarding "You have passed a subscription that does not belong to the given ack ID": when you pull a message from a subscription, you get an ack id along with the message, and that ack id is valid only for the subscription from which that message was pulled. If you send an acknowledge() request with that ack id and a different subscription, you will get this error.

So this does not appear to be a use case that is supported.

[Vaageesh]

ok. any plan to support in the near future? I get the below exception as soon as subscription is deleted.

`
2024-02-15T21:46:59.407+05:30 ERROR 73572 --- [test-app] [Subscriber-SE-1-3] c.g.c.p.v.StreamingSubscriberConnection : terminated streaming with exception

com.google.api.gax.rpc.NotFoundException: com.google.api.gax.rpc.NotFoundException: io.grpc.StatusRuntimeException: NOT_FOUND: Resource not found (resource=projects/project1/subscriptions/test-sub).
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:90)
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
at com.google.cloud.pubsub.v1.StreamingSubscriberConnection$1.onFailure(StreamingSubscriberConnection.java:339)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1127)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:92)
at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:74)
at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:51)
at com.google.cloud.pubsub.v1.StreamingSubscriberConnection$StreamingPullResponseObserver.onError(StreamingSubscriberConnection.java:261)
at com.google.api.gax.tracing.TracedResponseObserver.onError(TracedResponseObserver.java:104)
at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:84)
at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:84)
at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:148)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:570)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.google.api.gax.rpc.NotFoundException: io.grpc.StatusRuntimeException: NOT_FOUND: Resource not found (resource=projects/project1/subscriptions/test-sub).
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:90)
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
... 18 common frames omitted
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Resource not found (resource=projects/project1/subscriptions/test-sub).
at io.grpc.Status.asRuntimeException(Status.java:537)
... 17 common frames omitted

2024-02-15T21:46:59.410+05:30 WARN 73572 --- [test-app] [Subscriber-SE-1-3] c.g.c.p.v.StreamingSubscriberConnection : Setting response: SUCCESSFUL on outstanding messages
`

[burkedavison]

Given the linked thread is from 2017, this seems like very established behavior which is unlikely to be modified near-term.

@maitrimangal, could you verify?

@Vaageesh, consider submitting a support ticket @ https://cloud.google.com/support/. This will ensure your situation gets attention and may result in guidance on how to achieve your end goal.

[burkedavison]

Closing this ticket, as it is not Spring Cloud GCP specific. Please submit a ticket to https://cloud.google.com/support/ if this use case is something you wish the service to consider changing.