Add reconnect loop for GCP pub/sub subscriptions

[yordan-pavlov]

Description

Add a reconnection loop for GCP pub/sub subscriptions, very similar to the one that already exists for Azure Service Bus subscriptions here https://github.com/dapr/components-contrib/blob/master/pubsub/azure/servicebus/servicebus.go#L384

This is related to #1250 where the connection to a GCP pub/sub subscription could be lost due to a recoverable error. Even if the GCP pub/sub library does handle some recoverable errors internally, a reconnect loop can still be useful in a number of cases. For example:

• the GCP pub/sub library might not handle all recoverable errors (that's why Fix #1250 by upgrading cloud.google.com/go/pubsub to v1.12.2 #1285 exists to upgrade the GCP pub/sub library)
• a reconnect loop enables a scenario where the application has to wait for the subscription to be created
• a reconnect loop would also report errors such as a "not found" error thus making the investigation of issues with a GCP pub/sub subscription easier

Issue reference

Please reference the issue this PR will close: #1250

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Created issue in the https://github.com/dapr/docs/ repo: Add documentation for new GCP pubsub reconnect loop configuration docs#1970

[CodeMonkeyLeet]

Generally LGTM; per the PR template, can you make sure that there's an issue filed in https://github.com/dapr/docs for the new config metadata properties? It helps to make sure that the added features are appropriately documented with each release.

[CodeMonkeyLeet]

Thanks, LGTM!

Tests reveal break

[CodeMonkeyLeet]

Spoke too soon: Linter is complaining the code needs a gofmt

[yordan-pavlov]

yes, it must be the automated commit for the log message typo, I will fix it tomorrow morning when I am back at work

[artursouza]

@pkedy Can this use the retry logic from the kit library?

[pkedy]

@pkedy Can this use the retry logic from the kit library?

Yes. I think dapr/kit/retry probably makes sense here. No loop needed. I’ll paste some links to examples in other components. Usage is pretty simple and straightforward.

[pkedy]

Some suggestions for using the dapr/kit/retry package.

[yordan-pavlov]

@pkedy

Yes. I think dapr/kit/retry probably makes sense here. No loop needed.

I don't think the behavior implemented with the retry loop in this PR can be achieved by simply using dapr/kit/retry without a loop - retry up to max retries every time the subscription is disconnected (not just one time).

Even if the back-off library is used within a loop, the behavior still won't be the same - retry attempts are used and slowly refilled back up over time - the longer the subscription connection lasts, the more retry attempts accumulate up to max retries. Only if retry attempts are used up in quick succession will the retry loop stop retrying.

[pkedy]

@yordan-pavlov thanks for the explanation. I’ll take a closer look at this after a few days of Thanksgiving PTO.

[pkedy]

@yordan-pavlov sorry for the delay. I'm preparing for a conference talk with week.

[yordan-pavlov]

@yaron2 I have fixed the DCO issues and now all checks are passing; it would be great if this PR could be included in the 1.6 release.

[berndverst]

@pkedy can you give this another look to decide whether dapr/kit/retry works here? If not, any thoughts on what should be improved?

[Taction]

@yordan-pavlov can you please resolve conflicts?

[yordan-pavlov]

@Taction done - conflict resolved

[codecov]

Codecov Report

Merging #1293 (edfbb88) into master (c4aa200) will decrease coverage by 0.02%.
The diff coverage is 22.64%.

@@            Coverage Diff             @@
##           master    #1293      +/-   ##
==========================================
- Coverage   35.79%   35.77%   -0.03%     
==========================================
  Files         160      160              
  Lines       14637    14680      +43     
==========================================
+ Hits         5240     5252      +12     
- Misses       8829     8860      +31     
  Partials      568      568              

Impacted Files	Coverage Δ
pubsub/gcp/pubsub/pubsub.go	25.14% <22.64%> (+0.92%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4b0fdd...edfbb88. Read the comment docs.

dapr/kit retry doesn't seem to be a good fit here, besides pkedy does not have time to further review

[berndverst]

@yordan-pavlov I have merged this. Please make sure to complete the docs PR against the v1.7 branch of docs as soon as possible. dapr/docs#1970

[yordan-pavlov]

thanks @berndverst, docs pull request added here dapr/docs#2262