fix request context cancellation is ignored when retryBackoff (#539)

[zhyu]

Fixes #539

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 15.37%. Comparing base (06a6dc8) to head (b85eff2).
Report is 25 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #540       +/-   ##
===========================================
- Coverage   57.29%   15.37%   -41.92%     
===========================================
  Files         315      316        +1     
  Lines        9823     7751     -2072     
===========================================
- Hits         5628     1192     -4436     
- Misses       2902     6476     +3574     
+ Partials     1293       83     -1210     

Flag	Coverage Δ
integration	?
unit	15.37% <100.00%> (+2.53%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
opensearchtransport/opensearchtransport.go	87.16% <100.00%> (+2.00%)	⬆️

... and 312 files with indirect coverage changes

[dblock]

Thanks.

I'd love some additions to the docs on retries and back offs, can be a separate topic in https://github.com/opensearch-project/opensearch-go/tree/main/guides or https://github.com/opensearch-project/opensearch-go/blob/main/USER_GUIDE.md, but something generic enough on configuring the client?

Since you reference Elastic in #539, can you please confirm that you're not bringing in any non-APLv2 code in this project?

[dblock]

To fix DCO, run git commit -s --amend. Thanks.

[zhyu]

I'd love some additions to the docs on retries and back offs

I added a very simple doc about configuring the client with retries and backoffs.

can you please confirm that you're not bringing in any non-APLv2 code in this project?

Yes, I can confirm that.

I also found the client will sleep when the last retry failed, which seems not expected. So I pushed another fix (8b2649c) for it. Please have a look. Thanks!

[dblock]

I also found the client will sleep when the last retry failed, which seems not expected. So I pushed another fix (8b2649c) for it.

Good find. This deserves its own test, too.

[dblock]

Looks great. @Jakob3xD should take a look / merge, I am not super familiar with ways to do this in Go, I might be missing something.

[Jakob3xD]

Only a small nit about the indent of the guide but good to go.

[dblock]

Merging, thanks!

I don't think we want an entire guide for retry and backoff. Maybe a "Configuration" doc that has a "Retries" section (and other configuration options)? @zhyu interested in (re)organizing things?

[dblock]

@zhyu Can you please take a quick look at the failing integration tests?

=== RUN   TestNodes/Stats/without_request
    api_nodes_test.go:182: 
        	Error Trace:	/home/runner/work/opensearch-go/opensearch-go/internal/test/helper.go:147
        	            				/home/runner/work/opensearch-go/opensearch-go/opensearchapi/api_nodes_test.go:182
        	Error:      	Should be empty, but was [{"op":"remove","path":"/nodes/QR8UfvZLSXmNJrK8ZCf0Pg/indices/search/search_idle_reactivate_count_total"}]
        	Test:       	TestNodes/Stats/without_request

I am not sure it's related, but could be. If it's a flake, check whether we already have an issue for it / open a new one.

[Jakob3xD]

I am not sure it's related, but could be. If it's a flake, check whether we already have an issue for it / open a new one.

The tests are failing because the new opensearch release adds a new field. Not related to the PR.