New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
performance: creation of object with many key/val pairs #4625
Comments
I'm collecting some notes on how other programming languages have handled dictionary/hashtable data structures, and we can potentially sample from these to find ideas for different implementations: Python:
Lua:
Ruby: |
This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. |
I finally have bandwidth to pick this issue back up again! 😃 I'm going to tackle the benchmarking first, since that's necessary for us to quantify performance diffs. The closest existing test data generator in My current plan is to have 1-2 different tests, since different implementations could have performance differences between these cases:
The second case under most circumstances will be a specialized version of the first case, so it might not be worth adding. 🤷♂️ |
Interestingly, we do have an existing object insertion test under It only goes up to a size of 50k keys though, so I might add a 100k keys option, since the default benchtest results look like this on my T480 ThinkPad:
|
So, it looks like the one interface function that was relying on the existing slice-based key structure for This may be slightly trickier to replace than anticipated, as this site appears to strongly rely on the ordering of the keys, and replacing the iterator with a naive I'll keep investigating. 🤔 There's definitely a clean solution possible here. |
This commit delays the sorting of keys until just-before-use. This is a net win on asymptotics as Objects get larger, even with Quicksort as the sorting algorithm. This commit also adjusts the evaluator to use the new ObjectKeysIterator interface, instead of the raw keys array. Fixes open-policy-agent#4625. Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit delays the sorting of keys until just-before-use. This is a net win on asymptotics as Objects get larger, even with Quicksort as the sorting algorithm. This commit also adjusts the evaluator to use the new ObjectKeysIterator interface, instead of the raw keys array. Fixes #4625. Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
TODO:
The text was updated successfully, but these errors were encountered: