Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large request cause OutOfMemoryException #331

Open
GeirGrusom opened this issue Oct 11, 2023 · 0 comments · May be fixed by #332
Open

Large request cause OutOfMemoryException #331

GeirGrusom opened this issue Oct 11, 2023 · 0 comments · May be fixed by #332

Comments

@GeirGrusom
Copy link

In our project VaultSharp consistently causes an OutOfMemoryException when doing a batch encryption. It's probably caused by our large request sizes.

We're using 1.13.1

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at 
   at System.Text.Json.JsonReaderHelper.TranscodeHelper(ReadOnlySpan`1 utf8Unescaped)
   at System.Text.Json.Utf8JsonReader.GetString()
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at VaultSharp.Core.Polymath.MakeRequestAsync[TResponse](String resourcePath, HttpMethod httpMethod, Object requestData, IDictionary`2 headers, Boolean rawResponse, Action`1 postResponseAction)
   at VaultSharp.Core.Polymath.MakeVaultApiRequest[TResponse](String resourcePath, HttpMethod httpMethod, Object requestData, Boolean rawResponse, Action`1 postResponseAction, String wrapTimeToLive, Boolean unauthenticated)
   at VaultSharp.Core.Polymath.MakeVaultApiRequest[TResponse](String mountPoint, String path, HttpMethod httpMethod, Object requestData, Boolean rawResponse, Action`1 postResponseAction, String wrapTimeToLive, Boolean unauthenticated)
   at VaultSharp.V1.SecretsEngines.Transit.TransitSecretsEngineProvider.EncryptAsync(String keyName, EncryptRequestOptions encryptRequestOptions, String mountPoint, String wrapTimeToLive)

So I think the reason for this is fairly straightforward, and I think has a simple fix. In Polymath.cs on line 203 it uses StringContent. Switching this with JsonContent (which requires System.Net.Http.Json in Framework and .net standard) should enable streaming the object graph rather than duplicating the JSON content twice in memory (one for the immediate UTF-16 version, and then once more for the actual UTF-8 JSON)

This should likely improve performance, and reduce memory usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant