Extract Streamprocessor from engine module

[Zelldon]

Description

Split the engine module into two separate modules.

One called stream-platform (zeebe-stream-platform), the name is of course discussable, which contains everything related to stream processing. This module contains right now the API and implementation. It can and will be split further in the future.
There is a new package names io.camunda.stream.api/impl etc. Here we can also agree on a different name of course.

The other remaining module is still called engine, which contains the record processor implementation. It is the core or business logic of zeebe (the process engine). Right now it still depends on some impl classes like CopiedRecords, RecordValues mostly related to tests. This can hopefully be resolved in the future via iteration over the engine tests. #10004

I prepared some structure in the API and impl, but can also be discussed.

There are several classes which have been moved, which we might want
to move to a different place later.

For example:

• ZbColumnFamily
• CopiedRecords
• RecordValues
• etc.

Related issues

closes #10977
closes #10130
closes #9727
closes #9600

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Please refer to our review guidelines.

[Zelldon]

I guess it is best to just check the structure locally. 😁 Or via clicking . (dot) to open the web editor 😅

[github-actions]

Test Results

   954 files  +  1     954 suites  +1   1h 46m 39s ⏱️ + 6m 11s
7 514 tests +21  7 507 ✔️ +21  7 💤 ±0  0 ❌ ±0 
7 713 runs  +21  7 704 ✔️ +21  9 💤 ±0  0 ❌ ±0 

Results for commit 0ffe34d. ± Comparison against base commit a8317c7.

♻️ This comment has been updated with latest results.

[npepinpe]

+1,439 −995 ? Instant LGTM ;)

I'll try to have a look at it today assuming no medic stuff pops up 🙏

[Zelldon]

@npepinpe @oleschoenburg @oleschoenburg someone of you time for a review today? To avoid too much merge conflicts with other PR's 😅

[npepinpe]

👍

[npepinpe]

💭 Yes, definitely, this should be moved somewhere else 😄

[Zelldon]

Do you already have an idea?

[npepinpe]

OTOH, it wouldn't be that crazy for the whole DB persistence layer to have its own module, e.g. engine-state or something, then that could go in there. I guess not something we can unilaterally decide. Or it can go in the API module once we have one? Since that will be a dependency for both the stream platform impl and the engine?

I guess in the stream platform module we just need it for the processed position and such things? We don't really access the engine states, right? It would be cool if we could somehow share the DB but not know about the engine state/column families either...like a way to extend the state? We open the DB, create our own state/columns, then pass it on to the engine. They also don't need to know about our own columns/states. Not sure how feasible that is due to how we index though :(

[Zelldon]

Yeah state module is something I also wanted since a while but still this wouldn't solve it since stream platform doesn't want to know all these details.

What I thought first I could extend an enum, first define the two columns and then later extend in the engine with more columns, but unfortunately enums can't be extended since they are final 😅

Theoretically, we could do something similar via composition I guess, we have some default columns defined on base level which always need to be defined. Call the class BaseColumns these are used by stream-platform, and engine defines own columns and also uses BaseColumns. 🤷 But I felt this is too much for this PR. :)

[npepinpe]

Yeah that's fair. I think we should do it long term (both the state module and taking the composition approach, so each module doesn't need to know all columns).

[Zelldon]

bors r+

[zeebe-bors-camunda]

Build succeeded:

• Test summary