Export missing records to ES

[lzgabel]

Description

Currently, the Zeebe Elasticsearch exporter doesn't allow to export all records. The following records are missing:

• timer
• message_start_event_subscription
• process_event
• deployment_distribution

Related issues

closes #8337

Definition of Done

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• New content is added to the release announcement
• If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Please refer to our review guidelines.

[lzgabel]

Hi @remcowesterhoud. Please take a moment to check this out. 🙇

[github-actions]

Test Results

   939 files  ±  0     939 suites  ±0   1h 44m 56s ⏱️ +9s
7 495 tests +96  7 488 ✔️ +95  7 💤 +1  0 ❌ ±0 
7 685 runs  +96  7 676 ✔️ +95  9 💤 +1  0 ❌ ±0 

Results for commit fef28f6. ± Comparison against base commit 7c7a652.

♻️ This comment has been updated with latest results.

[remcowesterhoud]

@saig0 We had an issue before to export these records by default. We never got around to doing it for these records since they were more work than flipping a boolean. Do we still want to export these by default?

[saig0]

@remcowesterhoud good point. 👍

Yes, let's export these records by default. 🚀 Except, the process_event records. Reasoning:

• we write a lot of process_event records, at least one for each completed job
• I'm not sure how useful these records are for analyzing

[remcowesterhoud]

Thanks for another contribution @lzgabel 🏅

The indexes themselves look good. There is a few other things that require some attention though.

1. This wasn't in the issue so you couldn't have known, but please change exporting of the timer, message_start_event_subscription and deployment_distribution to true by default.
2. We'll have to add these records to the ElasticSearchExporter
3. We should modify the toString() method of the ElasticSearchExporterConfiguration.
4. We have some parametrised tests these tests use this TestSupport class. We will have to add the value types of these records to the setIndexingForValueType() method and remove them from the provideValueTypes() method.

If you will, please make these changes in a separate commit. This makes it easier for me as I'll only have to review the new commit 🙂

[lzgabel]

Hi @remcowesterhoud. Please check this out. 🙇

[remcowesterhoud]

You are lightning fast ⚡ I will have a look tomorrow!

[remcowesterhoud]

Awesome! 🏆

The issue is not marked as hacktoberfest, but I reckon it still deserves it so I'll add the hacktoberfest-accepted label to this PR. If you have 4 PRs within Camunda labelled with hacktoberfest-accepted you have completed the hacktoberfest challenge. In this case please submit the entry form here 🙂

bors merge

[lzgabel]

Hi @remcowesterhoud. Seems like bors is not working again? Do I need to rebase this PR on main again?

[remcowesterhoud]

@lzgabel Yes, please do a rebase

[remcowesterhoud]

bors cancel

[lzgabel]

Done.

[remcowesterhoud]

bors merge

[zeebe-bors-camunda]

Build succeeded:

• Test summary