Unnecessary read operations in the in-memory truncate implementation #4898
Comments
ashutosh-narkar
added a commit
to ashutosh-narkar/opa
that referenced
this issue
Jul 21, 2022
While writing data to the in-memory store via truncate op, OPA reads data at the current path before adding new data to the store to ensure the path exists. A bundle that contains data files at non-root locations will trigger a read on the store for each file and hence for a large bundle this can cause an increase in the bundle activation time and also resource usage. This change attempts to avoid multiple read ops by merging all the data in the bundle and performing a single write on the store. This fix was tested by observing the bundle activation time and cpu usage during bundle activation. The test bundle consisted of multiple data files at non-root locations. The bundle structure was something like: a/b/data.json, a/c/data.json etc. Improvements were seen in both cpu usage and activation time as compared to the older approach of doing a read while writing each file. This can be attributed to not reading all data under "a" in the test bundle for every write which was the case earlier. Fixes: open-policy-agent#4898 Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Short description
While writing data in the in-memory store via
truncate, OPA reads data at the current path before adding new data to the store. This is done so that OPA does not does write to a store path that does not exist. If the bundle contains multiple data files at non-root locations, these read operations will increase latency for bundle activation. Based on the bundle size, cpu and memory usage will increase as well.To avoid the multiple read calls, we could merge data passed via the bundle iterator and write that into the store in one call.
The text was updated successfully, but these errors were encountered: