Document how to write engine tests

[korthout]

Description

As part of our ongoing efforts to document Zeebe's internals and make it easier for developers to contribute, we should document how to write engine tests.

This documentation should probably include:

• what we value in tests
• how to set up a scenario and run it
• how to assert results
• how to make tests run fast

We should also go into some details related to time travel:

Hiya team 👋 Recently, we moved from repeated tasks to tasks that reschedule themselves with a fixed delay. Although the change makes sense, I’ve run into a drawback: time-traveling in tests doesn’t affect the later iterations of the task scheduling.

Example:
The time is 09:00 and I schedule a task to run every 10 min. At 09:10 it triggers. At 09:20 it triggers. At 09:25 I add 30 min to the clock, it does not trigger 3 times immediately, but instead triggers once immediately and then again at 09:35.

I ran into this, in the test I’m writing for the Delete existing PENDING_DEPLOYMENT bug, where I want to trigger the redistribution. The redistribution of deployments works by checking every 10 seconds whether there are any deployments that can be retried. They are skipped the first time to make sure we don’t retry immediately after our initial try (for example if the deployment is created just before the redistribution checks again).

So in the test I have to:

• deploy the process
• time travel to trigger the redistribution check
• wait for the redistribution check task to finish so that it is rescheduled again
• then and only then can I add more time to trigger the redistribution check again

This is cumbersome in tests, so I’m considering making the Deployment Redistributor available for access in the EngineRule, so I can trigger it directly. But I’d like to consider whether this is a unique case, or that we may have other such cases in the future.

• https://camunda.slack.com/archives/C037W9NMATG/p1660577026289419