OPA HTTP server: Add cache control config options

[philipaconrad]

What is the underlying problem you're trying to solve?

The OPA HTTP server currently caches only the most recent 100 queries. This works great for many small workloads, but can result in degraded performance in more complex authz scenarios, such as when more than 100 common query types are sent to the same OPA instance in a short time frame.

The problem I'd like to see solved is the current lack of control over the request cache.

Describe the ideal solution

Ideally, I'd like to see a few configuration knobs exposed to the user, which could be provided as part of an OPA config.yaml file.

Properties that I'd expect might be useful to control:

• Number of cache entries ("cache size")
  • Currently: 100 entries, set as a package variable.
• Max memory?
  • There is precedent for a memory cap in the decision logger plugin, where the cache attempts to stay within a certain size limit.
• Cache eviction/replacement strategy
  • Currently: Least-Recently Used (LRU)
  • See the Wikipedia page on Cache Replacement Policies for more ideas!
  • TTLs for cache entries would be a bit more work, but could be a fantastic way to keep cache sizes down across time. (Infrequently-used entries would expire, and automatically get evicted instead of lingering around in memory.)

Describe a "Good Enough" solution

Control of the number of cache entries would be a great start, and having maybe 2-3 different cache eviction strategies to pick from would be incredible. 🙂

Additional Context

[johanfylling]

An additional eviction strategy that comes to mind would be to prioritize evicting quick-running queries 🤔; keeping the more expensive (time-wise) results.
This would of course need to be tempered by some other strategy, so we don't end up with a cache full of one-shots.