Acquiring lock for Azure CosmosDB using Cassandra API not working

[marshall-lamb]

For Azure's CosmosDB emulating Cassandra APIs, there is some odd, as-of-yet unexplained behavior where updates to a table are not immediately available to query, potentially having to do with quorum data consistency. This manifests itself as failure for liquibase to acquire the lock in the DATABASECHANGELOGLOCK table even though queries of the table clearly show the lock was acquired by the liquibase process.

In the 4.23.1 version of this library, in src/main/java/liquibase/ext/cassandra/lockservice/LockServiceCassandra.java, a method was added (isLockedbyCurrentInstance()) that adds a check to see if the lock is held by the current process (by hostname and IP address) that requested it. This seems to have been added to address the behavior observed. This almost works for Azure CosmosDB, however the very next call to acquireLock() from the next changeset operation fails because isLocked() returns true.

To address the problem, I added a call to isLockedbyCurrentInstance() to method isLocked() so that on each iteration of the check for the presence of the lock, it also checks to see if the lock is held by the current process. This resolved the issue.

A separate PR will be created and linked to this issue for review of the actual change.

[marshall-lamb]

Would you please add my ID to authorization to create pull requests? I'm getting denied for marshall-lamb.

[dinchand]

I encounter the same issue. @marshall-lamb could you may be share your PR ?

[marshall-lamb]

As an optimization, I'm removing the return call to executor.queryForInt() as the call to isLockedByCurrentInstance() should be sufficient to indicate whether or not the current process holds the lock.

[marshall-lamb]

@dinchand I'm trying to submit a PR, but I'm getting a 403 access denied error on the push. So far nothing I've tried fixes that.

@r2-lf Can you help me with this? Do I need permissions to push added for my ID?

[giejay]

@marshall-lamb You are not able to create a PR from your forked repo to this one?

[marshall-lamb]

@giejay I got it working. Thanks.

@dinchand Here is the PR: #229

[dinchand]

@marshall-lamb Do you have liquibase working with cosmos(Cassandra) ? I tried but still facing issues...If you have it could you may be share an example ?

[giejay]

I'm using the 4.25 with your fix, and after spending some time on fixing SSL errors, I'm now getting these errors:

Unexpected error running Liquibase: Error executing SQL SELECT TABLE_NAME, KEYSPACE_NAME, COMMENT FROM system_schema.tables WHERE keyspace_name = 'mykeyspace' AND TABLE_NAME = 'databasechangelog': com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query

@marshall-lamb You did not face this issue? There is another blocking issue in the 4.25 release: #242, but I fixed that class cast exception locally.

[maximevw]

Hello @giejay,
I discussed a similar issue with @dinchand here: ing-bank/cassandra-jdbc-wrapper#37 (reply in thread)
The NoNodeAvailableException can have various origins but one of most common is a missing or invalid localdatacenter property when the default load balancing policy is used.