ZeebeDbInconsistentException in ColumnFamily DMN_DECISION_REQUIREMENTS

[pihme]

Describe the bug
Found in error logs
https://console.cloud.google.com/errors/detail/CPDM9-CV9Nvk3wE;service=zeebe;time=P7D?project=camunda-cloud-240911
https://console.cloud.google.com/errors/detail/CLWTn7vY7pS04QE;service=zeebe;time=P7D?project=camunda-cloud-240911

Expected behavior

Log/Stacktrace

Full Stacktrace

 io.camunda.zeebe.db.ZeebeDbInconsistentException: Key DbLong{2251799813685350} in ColumnFamily DMN_DECISION_REQUIREMENTS already exists
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.assertKeyDoesNotExist(TransactionalColumnFamily.java:273) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.lambda$insert$0(TransactionalColumnFamily.java:81) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.lambda$ensureInOpenTransaction$17(TransactionalColumnFamily.java:301) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInTransaction(DefaultTransactionContext.java:33) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.ensureInOpenTransaction(TransactionalColumnFamily.java:300) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.insert(TransactionalColumnFamily.java:76) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.state.deployment.DbDecisionState.storeDecisionRequirements(DbDecisionState.java:163) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.state.appliers.DeploymentDistributedApplier.lambda$putDmnResourcesInState$0(DeploymentDistributedApplier.java:50) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at java.lang.Iterable.forEach(Unknown Source) ~[?:?]
	at io.camunda.zeebe.engine.state.appliers.DeploymentDistributedApplier.putDmnResourcesInState(DeploymentDistributedApplier.java:45) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.state.appliers.DeploymentDistributedApplier.applyState(DeploymentDistributedApplier.java:39) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.state.appliers.DeploymentDistributedApplier.applyState(DeploymentDistributedApplier.java:23) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.state.appliers.EventAppliers.applyState(EventAppliers.java:239) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.writers.EventApplyingStateWriter.appendFollowUpEvent(EventApplyingStateWriter.java:36) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.deployment.distribute.DeploymentDistributeProcessor.processRecord(DeploymentDistributeProcessor.java:58) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.lambda$processInTransaction$3(ProcessingStateMachine.java:300) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:84) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.processInTransaction(ProcessingStateMachine.java:290) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.processCommand(ProcessingStateMachine.java:253) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.tryToReadNextRecord(ProcessingStateMachine.java:213) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.readNextRecord(ProcessingStateMachine.java:189) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:79) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:44) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:97) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:80) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:189) ~[zeebe-util-8.0.0.jar:8.0.0] 

Environment:

• Zeebe Version: 8.0.0

[pihme]

@oleschoenburg / @npepinpe Please have a look at this one. Both in terms of the bug and whether or not the consistency checks should be enabled

[pihme]

@npepinpe Which area would you use for data integrity?

[pihme]

BTW First occurrence was for a paying customer on what looks like a test system.

[npepinpe]

Hm, the checks should be enabled only on the trial plan 🤔

[pihme]

It is a trial plan

[npepinpe]

Then yes, all trial users should get the checks as a means to test it via progressive roll out. The goal is to make sure there's no false positives and no big performance impact before rolling it out to everyone.

[npepinpe]

Regarding label, each of them has a description, so pick which one seems to fit best (e.g. area/reliability). However, like I mentioned last night, if you think it makes more sense to have a more specific label, go for it - while I want to avoid having tons of labels and nobody using the same ones or understanding what others are doing, it makes sense to let it grow organically, and I'll just keep an eye on them and do a pass to consolidate them every once in a while (so I might have some questions I guess).

[oleschoenburg]

Looks like someone deployed DMN resources but the decision requirements key already exists. If this were valid, that would mean that DbDecisionState#storeDecisionRequirements should upsert, not insert into DMN_DECISION_REQUIREMENTS.
However, looking at tests such as DeploymentDmnTest#shouldDeployDuplicate, redeploying the same DMN twice seems to not re-use the same decision requirements key. I think I'd need some help from @saig0 or @korthout to understand if upsert or insert is correct here.

zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/deployment/DbDecisionState.java

Lines 159 to 166 in 75eeff0

	@Override
	public void storeDecisionRequirements(final DecisionRequirementsRecord record) {
	dbDecisionRequirementsKey.wrapLong(record.getDecisionRequirementsKey());
	dbPersistedDecisionRequirements.wrap(record);
	decisionRequirementsByKey.insert(dbDecisionRequirementsKey, dbPersistedDecisionRequirements);
	updateLatestDecisionRequirementsVersion(record);
	}

[saig0]

It is interesting that the error only happens on partition 2 but not on partition 1. I assume that it is related to the distribution of deployments.

Having a quick look at DeploymentDistributedApplier, it seems that we don't check for duplicated DRG or decisions. As a result, we try to insert the DRG again.

[oleschoenburg]

Based on the logs it looks like maybe the distribution of the deployment timed out, was retried and then fails:

Distribute deployment 2251799813685349 to partition 2
Deployment DISTRIBUTE command for deployment 2251799813685349 was written on partition 2
Failed to receive deployment response for partition 2 (on topic 'deployment-response-2251799813685349-2'). Retrying
Deployment DISTRIBUTE command for deployment 2251799813685349 was written on partition 2
io.camunda.zeebe.db.ZeebeDbInconsistentException: Key DbLong{2251799813685347} in ColumnFamily DMN_DECISION_REQUIREMENTS already exists

[oleschoenburg]

I think we could temporarily disable the checks for this one cluster, restart the brokers, wait for the deployment distribution to finish and then re-enable the checks. This would make this cluster healthy again (and not corrupt any data) but not solve the underlying issue so we might see the same exception again soon.

[npepinpe]

Cool! Let's do a patch release tomorrow 👍

[oleschoenburg]

Apparently versions 8.0.1 and 8.0.2 show the same symptoms. At least https://console.cloud.google.com/errors/detail/CPDM9-CV9Nvk3wE;service=zeebe;time=P7D?project=camunda-cloud-240911 shows that the same exception is thrown on newer versions.
I'll re-open this issue. Maybe @remcowesterhoud could take another look?

[oleschoenburg]

I saved the state from a 8.0.2 cluster where this has happened again in case it's useful for root-causing: https://drive.google.com/file/d/1EO6a_zBeTR5bJvc-GXYRYasY972_9z7B/view?usp=sharing

[remcowesterhoud]

This is most likely related to #9337

It’s interesting this is happening in Zeebe now as I couldn’t find how to reproduce it. But I have found a most likely root cause and will fix it when I’m back from holiday.

[oleschoenburg]

Reappeared on 8.0.4: https://console.cloud.google.com/errors/detail/CPDM9-CV9Nvk3wE;service=zeebe;version=8.0.4?project=camunda-cloud-240911

[oleschoenburg]

fixed by #9887