Data Loss During Redis master slave transition

[Datta19]

console.log

java version 1.8
jedis version 4.4.0-rc1

I am facing issue while writing 100 demo objects into the Redis database with transaction with a delay of Thread.sleep(1800L) after each write. During this write operation, I deleted the master instance of Redis for a high availability test, but I observed data loss during this write operation.out of 100, 98 objects written in to redis db.

The log file shows that the commit result for Key19 and Key69 is OK, but I do not see these keys in the Redis database. The Redis master-slave environment is deployed on Kubernetes pods, with Redis Sentinel for automatic failover-handling for high availability.

I am attaching the log file and java files
Transaction-Issue.zip
for your reference. Please let me know what is going wrong here.

Java Code which writes Demo objects

`package com.test.ha;

public class TestHa {

static {
	try {
		File logFile = new File("c:\\temp\\console.log");

        // Create a FileOutputStream to write to the file
        FileOutputStream fos = new FileOutputStream(logFile);

        // Create a PrintStream that writes to the FileOutputStream
        PrintStream ps = new PrintStream(fos);

        // Redirect System.out and System.err to the PrintStream
        System.setOut(ps);
        System.setErr(ps);
        
		System.setProperty("com.magicsoftware.xpi.ImmHost", "xpiqa.factoryeye.net:6379");

	} catch (Exception e) {
		e.printStackTrace();
	}
}

public void demowrite()
{
	IMMTransactionClient iMMtransactionClient=null;
	try {
		for(int i=0;i<100;i++) {
			Demo demo =new Demo("Value"+i);
			iMMtransactionClient = new IMMTransactionClient();
			Transaction transaction = iMMtransactionClient.beginTransaction();
			IMMClient.writeTransactional("Key"+i, demo, transaction);
			iMMtransactionClient.commitTransaction();
			System.out.println("Written object with key : "+"Key"+i);
			Thread.sleep(1800L);
		}	
	}
	catch (JedisConnectionException e) {
		iMMtransactionClient.rollbackTransaction();
		System.err.println(" Exception during writing"+e.getMessage());
		//System.setProperty("com.magicsoftware.xpi.ImmHost", "xpiqa.factoryeye.net:6379");
	}
	catch (Exception e) {
		iMMtransactionClient.rollbackTransaction();
		System.err.println(" Exception during writing"+e.getMessage());
	}
	
}

public static void main(String args[])
{
	System.out.println("Started....");
	TestHa testHa =new TestHa();
	
	Thread thread1 =new Thread(new Runnable() {
		@Override
		public void run() {
			System.out.println("writting thread started");
			testHa.demowrite();
		}
	});
	thread1.start();
}

}
class Demo
{
String value;
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
public Demo(String value) {
super();
this.value = value;
}

}`

Thank you in advance for your help.

[sazzad16]

I can only assume that the write updates from master didn't reach the replica(s).
You can try WAIT (waitReplias method) like commands.

[Datta19]

@sazzad16 Thank you for your quick response. I will try your suggestion and get back to you with the results.
I was wondering, is there any setting related to Redis replication and failover that we might have missed? What do you think?

[Datta19]

@sazzad16 I have tried with transaction.waitReplicas(2, 10000) but it not solved the issue, still 2 or 3 objects were lost during failover of master slave transition.

[chenshi5012]

The replication of M => R is asynchronous.
Therefore, M may break down before some data is replicated to R. As a result, the data is lost.

you can try this congfigure.

min-slaves-max-lag

If the slave node replicates data and the ACK delay is too long, the master node considers that too much data may be lost after the master node breaks down and rejects the write request.
In this way, the data loss caused by some data not being synchronized to the slave node when the master node breaks down can be reduced to a controllable range.

[chenshi5012]

for example

min-slaves-to-write 3
min-slaves-max-lag 10

If the connection delay of three or more slave Redis is longer than 10 seconds, the master Redis does not accept external write requests. If either of the preceding two configurations is set to 0, this feature is disabled. By default, min-slaves-to-write is 0 and min-slaves-max-lag is 10

[Datta19]

@chenshi5012 Thanks for the detailed reply, I will try with your suggestions and let you know the result.