Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accept Process Instance cancelling and new deployment when backpressure is high #10388

Merged
merged 4 commits into from
Sep 19, 2022

Conversation

remcowesterhoud
Copy link
Contributor

@remcowesterhoud remcowesterhoud commented Sep 16, 2022

Description

Allow cancelling of process instances and deployments of new processes when the back pressure is high.

It could occur that a cluster has a high back pressure because of a faulty process. This could be resolved by cancelling the instance, or by creating a new deployment. When these commands get rejected because of this high backpressure it becomes unrecoverable.

Related issues

closes #9283

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

  • The changes are backwards compatibility with previous versions
  • If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

  • There are unit/integration tests that verify all acceptance criterias of the issue
  • New tests are written to ensure backwards compatibility with further versions
  • The behavior is tested manually
  • The change has been verified by a QA run
  • The impact of the changes is verified by a benchmark

Documentation:

  • The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
  • New content is added to the release announcement
  • If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Please refer to our review guidelines.

Allows the cancelling of a process and deploying new versions of a process when backpressure is high. There are situations where a faulty process is the cause of the high backpressure. When this happens a way to resolve the issue would be to cancel the process instance. This is not possible when the cancel command gets rejected because of the backpressure.
In a similar manner it could occur that a process is faulty and causing a fork-bomb with call activities, resulting in high backpressure. Redeploying a new version of the process could resolve this situation.
This assertThat never actually verified anything. The loop should remain as this is what increases the in flight requests to the limit.
This test verifies that the following action can be still done even when the backpressure is high:

1. Complete and Fail jobs
2. Cancel process instances
3. Perform new deployments
@remcowesterhoud remcowesterhoud force-pushed the 9283_whitelist_deploy_and_cancel_command branch from 409d5ec to 5b68e05 Compare September 16, 2022 15:13
@github-actions
Copy link
Contributor

Test Results

   859 files  ±    0     859 suites  ±0   1h 50m 38s ⏱️ - 1m 22s
6 673 tests  - 153  6 662 ✔️  - 153  11 💤 ±0  0 ±0 
6 857 runs   - 153  6 846 ✔️  - 153  11 💤 ±0  0 ±0 

Results for commit 5b68e05. ± Comparison against base commit 9649e4d.

Copy link
Member

@saig0 saig0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@remcowesterhoud LGTM 👍

@saig0
Copy link
Member

saig0 commented Sep 19, 2022

bors merge

@zeebe-bors-camunda
Copy link
Contributor

Build succeeded:

@zeebe-bors-camunda zeebe-bors-camunda bot merged commit 1ead10a into main Sep 19, 2022
@zeebe-bors-camunda zeebe-bors-camunda bot deleted the 9283_whitelist_deploy_and_cancel_command branch September 19, 2022 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants