[Feature] Kyverno Performance Optimizations

[KhaledEmaraDev]

Problem Statement

Kyverno stands to gain from some performance optimizations. Currently when applying 2 mutate policies using Strategic Merge Patch we get the following performance:

avg=1.25s min=98.68ms med=996.53ms max=5.69s p(90)=2.6s p(95)=3.1s

After rewriting the policies to use JSON Patch:

avg=443.42ms min=96.83ms med=398.18ms max=1.1s p(90)=719.09ms p(95)=798.45ms

Also for validate in constrained resources scenarios we could improve a little. Using the profiler it seems that the two biggest offenders are:

1. Memory Allocation and Garbage Collection
2. Regex Used in:
   a. Variable and Reference substitution
   b. Anchor Processing

Also, time delay seems to grow linearly with the number of policies suggesting we could make use of parallelization.

Solution Description

For Validate we have three solutions:

• Optimize Away Regex feat(vars): use state machine instead of regexp #10229
• Manage Memory Better (Pooling, Pre-allocating slices, etc...)
• Pre-compute and cache computations that don't change often and store them on insertion to be used later.

For Mutate we already have our solution which is to use JSON Patch. However, other things should be considered like the user experience. Currently a non-issue as most Policy engines suffer from the same fate.

Finally we could parallelize policy application whether Mutate or Validate.

Alternatives

No response

Additional Context

No response

Slack discussion

No response

Research

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.