Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport of [VAULT-5003] Use net/http client in Sys().RaftSnapshotRestore into release/1.9.x #14837

Conversation

hc-github-team-secure-vault-core
Copy link
Collaborator

Backport

This PR is auto-generated from #14269 to be assessed for backporting due to the inclusion of the label backport/1.9.x.

WARNING automatic cherry-pick of commits failed. Commits will require human attention.

The below text is copied from the body of the original PR.


What does this PR do?

Modifies (*client).Sys().RaftSnapshotRestore function to use a net/http client to prevent attempting to read the entire body of a request (which is the case in our (*client).RawRequestWithContext() )
The go-retryablehttp package makes a copy of the request body on construction of a retry-able request to be able to re-play the request. When large snapshots are streamed to the server, this is problematic for the API client’s Sys().RaftSnapshotRestore function.

Any useful background information?

This likely hasn't been identified as a problem in our CLI tool because we use files that go-retryablehttp can handle without reading the full value into memory.
Our (*client).Sys().RaftSnapshot method currently implements a net/http client.

How has this been tested?

Tests have been updated to use helper functions.
Analyzed a memory profile of our current code when writing a "large" snapshot, which copies the entire request body, where the new code does not. Large writes have not been included in the updated tests as to not gobble up unnecessary resources.
"large" data object:

	data := map[string]interface{}{"target": strings.Repeat("a", 1000000)}
	_, err := leaderClient.Logical().Write("secret/big", data)

@vercel vercel bot temporarily deployed to Preview – vault April 1, 2022 18:08 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook April 1, 2022 18:08 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook April 1, 2022 18:12 Inactive
@vercel vercel bot temporarily deployed to Preview – vault April 1, 2022 18:12 Inactive
@vercel vercel bot temporarily deployed to Preview – vault April 1, 2022 18:23 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook April 1, 2022 18:23 Inactive
@VinnyHC VinnyHC added this to the 1.9.5 milestone Apr 1, 2022
Copy link
Contributor

@averche averche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

changelog/14269.txt Show resolved Hide resolved
@VinnyHC VinnyHC merged commit 22f310e into release/1.9.x Apr 1, 2022
@VinnyHC VinnyHC deleted the backport/vinnyhc/VAULT-5003_retry/seriously-harmless-fowl branch April 1, 2022 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants