Reject duplicate DeploymentDistribution Complete command

[korthout]

Description

There existed a special case that could lead to a ZeebeDbInconsistentException:

• a pending deployment is distributed multiple times to another partition by the DeploymentRedistributor.
• the other partition processes the distribution twice and both times sends a DeploymentDistribution:Complete command to the deployment partition (i.e. partitionId: 1).
• the deployment partition processes the first complete command, and writes DeploymentDistribution:Completed event, which is applied to the state.
• applying the completed event results in deleting the Pending Deployment for that partition.
• when it processes the second complete command, there could still be a pending deployment for another partition open for the same deployment, if so, the error happens.
• the second command is not rejected, because there is still a pending deployment for the deployment key, so another completed event is written and applied.
• applying fails this time, because the pending deployment no longer exists.

This PR changes the behavior. It makes sure the second command is rejected because the specific pending deployment no longer exists. In that case, we don't write the completed event a second time.

Related issues

closes #10064

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• New content is added to the release announcement
• If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Please refer to our review guidelines.

[github-actions]

Test Results

   844 files  +    1     844 suites  +1   1h 43m 13s ⏱️ + 8m 1s
6 226 tests  - 231  6 214 ✔️  - 232  11 💤 ±0  1 ❌ +1 
6 410 runs   - 231  6 398 ✔️  - 232  11 💤 ±0  1 ❌ +1 

For more details on these failures, see this check.

Results for commit 6db151e. ± Comparison against base commit 17630e8.

♻️ This comment has been updated with latest results.

[saig0]

@korthout LGTM 👍

[korthout]

bors merge

[zeebe-bors-camunda]

Build failed:

• Test summary

[korthout]

I missed that this failed to merge, but it seems to have happened due to Out of space in CodeCache 🤷

bors retry

[zeebe-bors-camunda]

Build succeeded:

• Go linting
• Test summary
• Java checks

[backport-action]

Backport failed for stable/1.3, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git fetch origin stable/1.3
git worktree add -d .worktree/backport-10074-to-stable/1.3 origin/stable/1.3
cd .worktree/backport-10074-to-stable/1.3
git checkout -b backport-10074-to-stable/1.3
ancref=$(git merge-base 17630e8cc3f1d81c5dc9523208196147d17114ed 6db151ecf7578ff97c6aa7b590614fc81d666293)
git cherry-pick -x $ancref..6db151ecf7578ff97c6aa7b590614fc81d666293

[backport-action]

Successfully created backport PR #10117 for stable/8.0.

[korthout]

Deployment redistribution works completely differently on 8.0 and 1.3. For example, the complete command is already written when the distribute command is written to the other partition. So it won't ever write it twice on the other partition. This is totally different from 8.1 where the command can be written multiple times to the other partition.

@saig0 I think we shouldn't backport this, so I'll close the opened backport PR. Let me know if you feel otherwise.