plugins/logs: Fix broken retry logic

[ashutosh-narkar]

We were incorrectly resetting the retry counter after
every error condition instead of using the incremented
value. As a result, retry delay would always be 0s.
This meant that if OPA encountered an error while
uploading decision logs it would immediately retry
instead of doing an exponential backoff.

Fixes: #4486

Signed-off-by: Ashutosh Narkar anarkar4387@gmail.com

[ashutosh-narkar]

Post fix: We can see the delay increasing after each unsuccessful attempt to upload decision logs

{"headers":{"Content-Encoding":["gzip"],"Content-Type":["application/json"],"User-Agent":["Open Policy Agent/0.39.0-dev (darwin, amd64)"]},"level":"debug","method":"POST","msg":"Sending request.","time":"2022-03-25T12:26:16-07:00","url":"http://localhost:8001/logs"}
{"level":"error","msg":"log upload failed: Post \"http://localhost:8001/logs\": dial tcp [::1]:8001: connect: connection refused.","plugin":"decision_logs","time":"2022-03-25T12:26:16-07:00"}
{"level":"debug","msg":"Waiting 0s before next upload/retry.","plugin":"decision_logs","time":"2022-03-25T12:26:16-07:00"}
{"headers":{"Content-Encoding":["gzip"],"Content-Type":["application/json"],"User-Agent":["Open Policy Agent/0.39.0-dev (darwin, amd64)"]},"level":"debug","method":"POST","msg":"Sending request.","time":"2022-03-25T12:26:16-07:00","url":"http://localhost:8001/logs"}
{"level":"error","msg":"log upload failed: Post \"http://localhost:8001/logs\": dial tcp [::1]:8001: connect: connection refused.","plugin":"decision_logs","time":"2022-03-25T12:26:16-07:00"}
{"level":"debug","msg":"Waiting 185.461446ms before next upload/retry.","plugin":"decision_logs","time":"2022-03-25T12:26:16-07:00"}
{"headers":{"Content-Encoding":["gzip"],"Content-Type":["application/json"],"User-Agent":["Open Policy Agent/0.39.0-dev (darwin, amd64)"]},"level":"debug","method":"POST","msg":"Sending request.","time":"2022-03-25T12:26:17-07:00","url":"http://localhost:8001/logs"}
{"level":"error","msg":"log upload failed: Post \"http://localhost:8001/logs\": dial tcp [::1]:8001: connect: connection refused.","plugin":"decision_logs","time":"2022-03-25T12:26:17-07:00"}
{"level":"debug","msg":"Waiting 224.251515ms before next upload/retry.","plugin":"decision_logs","time":"2022-03-25T12:26:17-07:00"}

[anderseknert]

The fix looks reasonable to me, but don't we have any tests on this? We should have one if not.

[ashutosh-narkar]

The fix looks reasonable to me, but don't we have any tests on this? We should have one if not.

I did an end-to-end test with the change and posted the results here. Not sure what a unit test would be here unless we move private counter on the plugin struct and check for it's value which does not seem necessary here.