Large number of rolled back transactions on JPA/JDBC TokenStore

[abuijze]

With a large number of nodes running a certain tracking processor, each node will have 1 WorkerLauncher threads checking for segments to claim. It will iterate of each of the segments and do an attempt to claim that segment. Under normal circumstances, this process will fail most of the time, as segments are already claimed.

Problem:
The WorkerLauncher initiates a transaction within a loop to attempt to claim each available segment individually. Because segments are claimed, the TokenStore throws a UnableToClaimTokenException, causing the transaction to be rolled back. While the processor deals with the exception transparently, it causes a large number of transactions to start, and roll back.

Suggestions:
The TokenStore should expose a method that allows for more efficient implementations of the "attempt to claim a token" process. This implementation should check within the scope of a processor name, if any unclaimed or abandoned segments are available. If not, it simply returns an empty optional. If there is a segment available, it will claim and return the token for that segment.

This can be implemented as a default method, which does the iteration internally. An extra "tryFetchToken" method can be added that returns an Optional instead of throwing an exception when a claim fails.

[heatzync]

@smcvb: Why was this feature removed from the Release 4.5 milestone? Is it still a valid feature that may be included in the future?

[smcvb]

It was removed from 4.5 in favour of other more pressing feature's we've been working on.
So, not out of disinterest in the feature, more from the perspective of manpower and other obligations.

Added, the perspective of this issue is mainly with pain points in respect of the TrackingEventProcessor.
In Axon Framework 4.5 we have introduced a new Event Processor implementation, called the PooledStreamingEventProcessor.
Obviously, this processor took quite some time to implement, as such minimizing the chance to add other features or enhancements, but we also feel it provides a better, faster and simpler replacement of the TrackingEventProcessor.

Concluding, the fact the milestone was removed from this issue (or from any issue in fact) does not state we no longer have interest in the issue.
If this would be the case, we'd close off the issue with a clear description of why it has become obsolete or why we won't fix it anymore.

Hoping this clarifies our stance on the matter @heatzync.
Furthermore, hoping you'd like to give the new PooledStreamingEventProcessor instead of the TrackingEventProcessor which contains the above described potential perdicement.

[heatzync]

Thanks for the quick reply @smcvb. We are experiencing strange behaviour with v3.4.3 and the feature described above seems like something that might stop the strange behaviour. I don't want to pollute this issue with the strange behaviour we are experiencing. I will ask a question on StackOverflow with the details of our scenario.

[smcvb]

We deem that the efforts that we took in issue #1967, and its subsequent pull request #1997, are an acceptable solution for the time being.
Due to that, I will close this issue as 'resolved.'

We do have further improvements in mind, but none of thus are backward compatible by any means.
As such, they will not be part of Axon 4.x, but rather an adjustment to be made for a future major release.