IllegalStateException: Not expected to have an active sequence flow count lower then zero! #6778
Comments
saig0
added
kind/bug
|
It is expected that this issue will be solved when the migration is done because it replaces the whole reprocessing by event applying. |
|
Seems to happen again with https://console.cloud.google.com/errors/detail/CJG2u-KVh-_LigE;service=zeebe;version=8.0.2;time=P7D?project=camunda-cloud-240911 I will try to get more data |
|
It seems that the error was produced by an model where the gateway was taken multiple times (at least twice). Since this is confidential data I put it into our gdrive https://drive.google.com/drive/u/0/folders/1NIgMxfWjZ5T16mDQ9DEdHMpr1SVg0w9F |
|
I have added a version of the process to the drive that doesn't require any variables, for easier debugging. \cc @saig0 |
|
I have also just added a none_event_repro file. Creating an instance of this will result in the exception. |
|
Hypothesis It all has to do with when we reset the number of taken sequence flows. This happens in the completed applier. This in combination with activating the parallel gateway through a command causes this behaviour. When starting the process the following happens:
|
|
@remcowesterhoud great analysis 👍 I can confirm the behavior ✔️ |
|
In my opinion, this bug is critical (at least high severity), there is no work-around and the process instances are stuck in this case. Plus: As a SaaS user it is hard to detect what went wrong. |
|
I would argue there is a workaround, which is fixing the model. Strictly looking at the labels that means mid would fit the best: "Marks a bug as having a noticeable impact but with a known workaround". I'm still happy to fix it right now as I don't think it'll be a difficult thing. |
Well but this doesn't help you if your instances are stuck, like real payments or transactions right? |
remcowesterhoud
added
severity/high
9759: Reject duplicate parallel gateway activate command r=remcowesterhoud a=remcowesterhoud ## Description <!-- Please explain the changes you made here. --> Parallel gateways get activated by taking a sequence flow and checking if the number of taken flows is greater or equal to the number of incoming sequence flows. If this is the case an activate command is sent. The number of taken sequence flows get rest upon activation of the parallel gateway. This proves troublesome when a "bad" model causes one of the incoming sequence flows to be taken twice. This could result in the activation command being sent twice. Imagine there is a parallel gateway with 2 incoming flows. What would happen is: 1. First flow is taken 2. Second flow is taken. Incoming flows == taken flows so an activate command is sent. 3. Second flow is taken again. The first activate command has not been processed yet. The number of taken flows has not been reset. As a result incoming flows < taken flows. A second activate command is sent. This is solved by always sending an activate command when a sequence flow is taken. Once the `BpmnStreamProcessor` tries to process the record it will check if the state is valid. Here a check has been added to verify that when we receive an activate command for a parallel gateway we will first check if all the incoming flows have been taken. If this is not the case we will reject the command. ## Related issues <!-- Which issues are closed by this PR or are related --> closes #6778 Co-authored-by: Remco Westerhoud <remco@westerhoud.nl>
Zelldon
added
the
version:8.1.0
Describe the bug
On reprocessing, the stream processor ran in an endless loop.
Followed by
To Reproduce
Not clear.
It seems to be related to the termination of an (event-) subprocess.
Expected behavior
This issue is probably caused by the current work in the workflow engine that applies event sourcing to the stream processors.
Log/Stacktrace
Full Stacktrace
Follow-up Exception
Error Reporting:
Log: https://console.cloud.google.com/logs/query;query=%0AlogName:%22stdout%22%0Aresource.type%3D%22k8s_container%22%0Aresource.labels.cluster_name%3D%22ultratest%22%0Aresource.labels.namespace_name%3D%2299936d0e-e21b-493c-8b96-6d79ac11c40d-zeebe%22%0Aresource.labels.container_name%3D%22zeebe%22;timeRange=2021-04-12T10:06:14.712Z%2F2021-04-12T11:06:14.712Z;pinnedLogId=2021-04-12T10:35:15.759906Z%2F32bhcwg42mromx;cursorTimestamp=2021-04-12T10:35:15.789511Z?project=camunda-cloud-240911
Environment:
The text was updated successfully, but these errors were encountered: