[State CosmosDb] Built-in compression of value

[olitomlinson]

This customer blog-post highlights a very simple and warranted need to compress state before it hits CosmosDb.

This simple act of compression reduced the document size from 300KB to 9KB, resulting in a roughly 97% reduction. Consequently, the RU consumption plummeted from 600K RU/s to an average of 20K RU/s in a 24 hour period.

The customer rolled their own client side compression solution for this, but I think the CosmosDb component should include its own turn-key compression implementation to achieve the same goal.

This behavior could be enabled via a flag on the spec enableValueCompression

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: my-state-store
spec:
  type: state.azure.cosmosdb
  version: v1
  metadata:
     ...
  - name: enableValueCompression
    value: true

Pros

• huge efficiency, resulting in massive savings for the customer, making the component more viable for high-throughput use-cases i.e. Actors & Workflows

Cons

• The value would no longer be a document but a string, meaning the query API capability would not be possible.
• Increased CPU for compression / decompression. (However, in reality this CPU increase is simply shifted to the sidecar away from the App, rather than net new cost)

---

Thinking out loud...

Given that Postgres V2 has dropped support for query API due to changing the way it stores the value data- I do not see this as a controversial move to make for CosmosDb. Maybe it makes sense to leave the CosmosDb v1 component alone, and implement this compression feature as the standard behavior for CosmosDb V2?

---

@KrylixZA Maybe you could provide some thoughts on if this would feature/change would be welcome to you and how a turnkey solution provided by the component would simplify your client-side concerns.

[KrylixZA]

Hey @olitomlinson

Thanks for raising this issue. We had plans to raise it ourselves but got a bit wrapped up in work and day-to-day.

We would absolutely welcome this change. We were going to raise under dapr/dapr against actors as the usual state management is already base64 encoded while actor state is raw, formatted JSON. Having it as a configurable flag on the component YAML actually feels like a neater solution than what we had planned. We were hoping for an overload on the actor state management API. With it on the YAML, the team will be able to set it once during project bootstrapping as a best practice and ensure costs don't ever grow wildly beyond their expected range. I like it!

Giving it more thought, it makes sense to be an option for all forms of state management and for any consumption-based database, not just CosmosDB. This same behaviour would apply neatly to DynamoDB as well, for example.

[Spharah]

Thanks @olitomlinson for detailed information on the request. I raised the issue under dapr/dapr yesterday.
[Proposed] Add compression option when saving state

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[olitomlinson]

Bump

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[olitomlinson]

Bump