[EPIC] Start Process Instance Anywhere

[korthout]

Description

This issue represents the epic of the feature to start a process instance at an arbitrary point in the process.

There are a few use cases for this feature:

• A user might want to migrate instances from one version to another one, also known as the process instance migration feature. Allowing users to start a process anywhere is giving them the first tool to do this until we have a better UX for the migration available in Operate.

• Additionally, a user might want to modify an existing process instance. It may happen by accident that the process instance ends up in the wrong state. Until the process instance modification feature is available, users can cancel their process instance and start a new one in the corrected state.

• From a testing perspective, a User might want to test specific parts of the process without running through the process. This reduces the boilerplate and setup needed to pass through all steps of the process.

Concept

In order to start a process instance at an arbitrary point in the process, we need to create a process instance and set it in a state that allows us to continue the process execution from the chosen point.

When starting a process instance in its usual point (the root none-start event), we create the process instance by writing the ACTIVATE_ELEMENT command for the PROCESS. When processed, the processor of this command writes events that create the element instance for the PROCESS (the process instance), set the variables on the process instance, and open subscriptions to relevant events (i.e. events for the event-sub process). It also writes the follow-up command to activate the start event.

When starting a process instance at a specified point, we can do mostly the same things:

• write the same events that would otherwise be produced when activating the process;
• write a command for the element that we want to start the process execution at.

We'll also need to deal with element scopes and event scopes. Consider the following example.

To start at the purple user task, we only need to activate the process and activate the blue embedded sub-process. We can do that by writing the respective ELEMENT_ACTIVATING and ELEMENT_ACTIVATED events for these elements, in that order. We also have to subscribe to the relevant events. In this case, we only have to open a message subscription for the message boundary event.

To start at the green user task, we need to activate the process, activate the blue embedded sub-process, subscribe to the message boundary event, activate the orange embedded sub-process, and finally also subscribe to the timer boundary event, in that order.

Spike

The team did a spike to research the implementation of this concept. We had to change the CreateInstanceProcessor to make it start an instance in the right place (i.e. before a specified element). We also had to change the API to define the elementId at which place the instance would be started. In the spike, this was just a single element, but the feature should allow starting at multiple elements to accommodate concurrent and parallel flows. We also created the process instance with Process Variables, which worked out of the box. If we would have continued further with the feature during the spike, we'd also had to change the API to define local variables. However, in our opinion, local variables don't have to be part of the initial version of this feature.

Looking at the processor, it has to create the relevant element scopes by writing ELEMENT_ACTIVATING and ELEMENT_ACTIVATED events. The relevant scopes can be determined by traversing the parents of the target element in the process model recursively until we reach the root process. We need to be careful to duplicate these when creating multiple tokens. The processor also needs to start the process execution at the specific element by writing the ACTIVATE_ELEMENT command. Effectively, we start the process execution before the target element.

In addition, event subscriptions have to be opened for the created element scopes. The element that activates will already have its event subscriptions opened by the engine as part of the ACTIVATE_ELEMENT command processing (e.g. boundary events on the target element).

IO mappings of the target element are applied as usual by the command processing. Initially, we do not want to apply the input mappings of element scopes, but we might add this later if requested. We do not want to apply the output mappings of the root none-start event at all when the instance is created in a different element, this feels like unexpected behavior. This matters, because these are usually applied before the root processes event subscriptions are opened.

Task Breakdown

Broker

• Extend ProcessInstanceCreation record with start instructions #9388
• Start instance at specified elements #9390
• Subscribe to events of created element scopes #9391
• Reject ProcessInstanceCreation command when scope construction fails #9421
• Reject ProcessInstanceCreation command when failing to subscribe to events #9422
• Reject ProcessInstanceCreation when unable to start at target element #9392
• Reject ProcessInstanceCreation command targeting element inside multi-instance #9394
• Reject ProcessInstanceCreation command targeting unsupported element #9408
• Flaky Test – io.camunda.zeebe.engine.processing.processinstance.CreateProcessInstanceAnywhereTest#shouldSubscribeToScopeEvents #9557
• Flaky test - CreateProcessInstanceAnywhereTest#shouldSubscribeToEventsOnlyOnce #9589
• Replace active sequence count with a different solution #9528 (not required)

Gateway API

• Extend the gateway protocol to create a process instance with start instructions #9396
• Extend gateway's create process instance request mapping with start instructions #9397
• Extend the Java Client's CreateProcessInstance API with start instructions #9398
• Extend the Go Client's CreateProcessInstance API with start instructions #9399

Exporters/Metrics

• Expand Elastic Search exporter for ProcessInstanceCreation record #9405
• Update metrics exporter for start process instance anywhere #9406
• Collect metrics about started instances with start instructions through Zeebe Analytics #9423
• Print out start instructions in compact logger #9555
• Add Grafana visualization for start process instance anywhere #9622
• Add MixPanel visualization for start process instance anywhere

Testing

• Enhance randomized process tests with start process instance anywhere ability #9407
• Test start instance anywhere for supported element types #9621
• End-2-End test from java/go client to ElasticSearch, Grafana and MixPanel

Zeebe Process Test

• Extend ZPT gateway with start instructions zeebe-process-test#410 Extend ZPT gateway with start instructions
• Extend record logger with start instructions zeebe-process-test#411 Extend record logger

Operate

• https://github.com/camunda/operate/issues/2956

Documentation

• Update GRPC API with start instructions for CreateProcessInstance RPC camunda-docs#943
• Document "Start Process Instance from Anywhere" feature camunda-docs#944

Out of scope of MVP

These tasks are defined as out of scope for the initial version of this feature.

• Defining local variables
• Skip/Apply Input Mappings of element scopes
• Start after the element, instead of starting before the element
• Start instance inside nested call activity
• Start instance inside nested multi-instance
• Start instance anywhere with result
• ZPT: add assertions to check that the process instance was started at a specific element
• Extend zbctl create process instance with start instructions
• Support logical transactions for command processing #9420 (originally considered necessary, but it was blocked by a large issue and a workaround is available)
• Reject create process instance command using logical transaction #9644 (replacement of workaround with better solution)

Additional links

• C7 Docs
• Kickoff (internal doc)
• C7 Feature Exploration (internal doc)
• Product Hub Epic

[aivinog1]

Hey @korthout, we want to see this feature so much, maybe I can help you with this somehow? :)

[korthout]

Thanks for offering to help @aivinog1. We really appreciate it. 🙇

However, I think your help would be more useful on other topics. This particular topic is highly complex and some unforeseen challenges may arise. The team can clarify relevant questions faster through internal communication than would be possible through GitHub. So, we think that it makes more sense for the team to work on this.

We acknowledge the importance of this feature. And it's great to see that you're interested in it as well. Currently, the team is shifting its focus toward this feature and active development is about to start. We expect to release this feature as part of the upcoming 8.1 release.

[aivinog1]

Thanks @korthout :)

[sarialem]

This is one of the crucial needs for us to use Camunda 8.x, a few questions:

• will it provide Process modification capabilities similar to what was in Camunda 7?
• Is this getting to 8.1? (Can't see a reference to it in the 8.1.0-alpha1 release notes)

[saig0]

will it provide Process modification capabilities similar to what was in Camunda 7?

This epic is not about process instance modification. It is just about starting a new process instance at any point. I will create a new epic for process instance modification soon. We will also work on this topic.

Is this getting to 8.1?

Yes. The start process instance anywhere feature will be in version 8.1. We plan to ship the process instance modification topic in the same version.

[shahamit]

I will create a new epic for process instance modification soon. We will also work on this topic.

@saig0 - could you please share the id for process instance modification? It is a very critical feature for us. I have a couple of questions about it too that I can ask on the epic.

Thanks.

[dave-apex]

Will there be a separate epic for the "Start instance inside nested call activity"? We encounter this a lot in migrating process instances in another BPM tool.

[saig0]

@dknapp47 currently, we don't plan to support this case. However, you can create a new issue and describe your use case. We will triage the issue with the stakeholders and decide on the priority.

[saig0]

@shahamit I created a new issue for the process instance modification topic: #9570
I'll add more details and issues in the following days.

[dave-apex]

@dknapp47 currently, we don't plan to support this case. However, you can create a new issue and describe your use case. We will triage the issue with the stakeholders and decide on the priority.

Thanks @saig0 . I have created another issue to cover this aspect. Let me know if there is anything that I can help with in defining the details of the feature. #9583

[korthout]

🚀 This feature is scheduled to be released in 8.1.0-alpha3, closing this epic.

[skayliu]

Start instance inside nested call activity

Hi, @korthout, I see there is already had a test case CreateProcessInstanceSupportedElementTest for callActity related to #9583

[aleksander-dytko]

Hi @sarialem, @shahamit,

We have released Process Instance Modification with Camunda Platfom 8.1 - #9570.

Did you have a chance to try this feature yet? I would be curious about your feedback 😄
Thanks!

[amir169]

Hi,
I'm going to use this cool feature but realized when I start an instance at a given point, all previous tasks are executed again.
For example in the following picture, choosing the User Task as start element would cause the Check White And Blacklist service task to run again.

1. Is this the way it's supposed to be?
2. If so, any plans to extend the feature to make us able to skip previous tasks?

[korthout]

Thanks for reporting @amir169, and sorry for my late reply. It looks like your case was resolved by using a newer version of Zeebe (8.2.0 instead of 8.0.5). Is that correct?

[amir169]

Thanks for reporting @amir169, and sorry for my late reply. It looks like your case was resolved by using a newer version of Zeebe (8.2.0 instead of 8.0.5). Is that correct?

Yes.
That's resolved by using a newer version.