zstandard compression for parameters and results

[guzzijones]

why

mitigate (as much as possible) 'document too large' mongo exceptions. 16 mb is the limit for mongodb document size.

done

1. zstandard compression for parameter and results
2. remove embedded liveaction doc in executions db and replace with liveaction id string. This change will potentially save 1/2 the time to write the liveaction to the database as the liveaction won't be written 2 times.
3. added traceback info to exception handling for action execution engine and workflow engine. This should make it easier to debug for someone when the workflow fails for some reason. invalid orjson or unterminated yaql strings cause these exceptions in my experience.
4. This will require a data migration for the executionDB model for the liveaction paramter as it is now a string as opposed to an embedded document
5. I added a config option to turn off zstandard compression for parameters and results in the execution db and the liveaction db

[guzzijones]

@amanda11 @cognifloyd would you please take a look at this and give me some feedback. I am interested in if the direction the code is going in is acceptable.

We are having numerous instances now of mongo documents being too large. I ran a check and one of the larger objects we have would compress down to .5 mb from 10 mb using zstandard.

[cognifloyd]

Minor nit.

I don't know why Circle CI is falling. I'm guessing something we really on is no longer supported or was upgraded.

[guzzijones]

@cognifloyd The circle ci is fixed. rabbitmq needs priv=true and redis needed to be pinned. pr. can you merge?

[guzzijones]

I merged the packaging fixes.

[Kami]

I think zstandard compression is a good change in case numbers across various types of datasets confirm it's indeed worth it / good trade off of CPU cycles used for compressing / decompressing those field values.

We decided to not enable compression by default as part of #4846 to keep things simple and reduce the scope of those changes. I'm fine with enabling the compression by default now, in case it numbers still back this up.

As far as this change goes:

remove embedded liveaction doc in executions db and replace with liveaction id string. This change will potentially save 1/2 the time to write the liveaction to the database as the liveaction won't be written 2 times.

This one I'm not too sure about it yet and I need to dig in and think about it some more.

I think it would be good if we split this PR into two - one for field compression changes and one for field value / foreign key change.

In the end, both changes are somewhat related, but they have vastly different implications - one was already benchmarked in the past, it's backward / forward compatible and the other one is not, it could have large implications and also requires a potentially expensive and breaking migration (which is not backward compatible).

[Kami]

Thanks for your contribution.

I think we are on the right track, but I added some comments on splitting this PR into two and potentially adding more micro benchmarks, etc.

[guzzijones]

Thanks @Kami for the review. I would be ok just leaving this a breaking change. I don't have a lot of cycles to split it up. Thanks.

[guzzijones]

FWIW we have forked over to this branch along with removing the deep copies in orquesta . We are seeing huge speed improvements. Also, no more Mongo document too large.