No exception thrown from Jedis when AWS Elasticache instance is out of memory

[kalimatas]

Hi!

I was recently investigating a bug on production and discovered a weird behavior with JedisPool and AWS Elasticache. Basically, when Elasticache doesn't have enough memory left, the next write requests from Jedis are just... canceled. I cannot find another word for it :) No errors or anything, the program just continues to execute.

Our code looks like this:

Iterators.partition(someiterator(), batchSize)
    .forEachRemaining(batch -> {
        try (var jedis = jedisPool.getResource()) {
            var pipe = jedis.pipelined();
            batch.forEach(entry -> {   pipe.set(someKey, someValue);  });
            pipe.sync();
         }
    });

// After indexing, we swap the Redis databases
try (Jedis resource = jedisPool.getResource()) {
    resource.swapDB(dbIindexTmp, dbIndex);
}

What happens is that during the iteration the Redis database becomes full, the iteration just stops without errors, and the code continues to swap the databases with only partially applied changes.

Has anybody met this behavior? Is it something I have to tweak in Jedis or rather in AWS Elasticache? I tried to find similar issues but without luck.

Any clue is appreciated. Thanks!

[sazzad16]

@kalimatas Did you check the Response?

[kalimatas]

@sazzad16 No, I was expecting an exception in case of any errors. Is it possible that there is no exception but Response contains some kind of an error? From the code of Response, I see that the only way to understand if there is a problem is to call get() method, and it might throw an exception.

I will try to check Response and see what we can get out of it. Thanks for the clue.

[kalimatas]

Hi,

Thanks for the hint: indeed, there are failed responses which we can extract and check. Here how it works:

Iterators.partition(someiterator(), batchSize)
    .forEachRemaining(batch -> {
        try (var jedis = jedisPool.getResource()) {
            var pipe = jedis.pipelined();
            batch.forEach(entry -> {   pipe.set(someKey, someValue);  });
            failOnResponseError(pipe.syncAndReturnAll());
         }
    });

static void failOnResponseError(final List<Object> redisResponses) {
        for (var response : redisResponses) {
            if (response instanceof JedisException ex) {
                throw ex;
            }
        }
    }