UnifiedJedis' jsonSetLegacy consumes nearly 8 times more java heap memory to execute a write operation for a 63 MB JSON data.

[Datta19]

I am writing a 63.4 MB data file into Redis using UnifiedJedis' jsonSetLegacy. I observe that the data size in Redis remains the same, but during this operation, it consumes approximately 499,356 KB (almost 500 MB) of heap memory. Please let me know what will be the correct way to perform this large operations. I am using jedis 4.4.0-rc1.

public static void main(String a[]) {
		HostAndPort config = new HostAndPort("localhost", 6379);
		PooledConnectionProvider pooledConnectionProvider = new PooledConnectionProvider(config);
		try(UnifiedJedis	jedisClient = new UnifiedJedis(pooledConnectionProvider, 10, Duration.ofSeconds(120))) {
			String key = "exampleKey" + System.currentTimeMillis();
			byte[] encoded = java.nio.file.Files.readAllBytes(Paths.get("C:\\tmp\\flowdata1.txt"));
			jedisClient.jsonSetLegacy(key, new String(encoded));
			System.out.println("Data written to Redis successfully!");
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
		}
	}

[sazzad16]

The amount of heap memory is too much, but can't anything for sure.

For now, if encoded is a properly formatted json object (it seems it is),
instead of jedisClient.jsonSetLegacy(key, new String(encoded));,
try jedisClient.jsonSet(key, encoded); which will avoid all internal object alteration.

[Datta19]

Hi @sazzad16 ,

Thank you for the prompt response. However, we have attempted the suggested solution, and unfortunately, we observed that it still consumes the same amount of heap memory

[sazzad16]

May be it's just your regular application's heap size?

1. What if you simply do jedisClient.set(SafeEncoder.encode(key), encoded);?
2. What if you don't do any of such operations?

[Datta19]

@sazzad16

As per your previous comment, if 'encoded' is a properly formatted JSON object, then jedisClient.jsonSet(key, encoded); takes only the required memory. It takes near about 100 MB for a 63 MB file (with jsonSetLegacy, it was taking 500 MB). However, I am still facing an issue when reading the same amount of data with jedisClient.jsonGet; it takes 500 MB of heap memory for a 63 MB JSON data

[sazzad16]

It's likely due to JSON processing library.

Check jsonGetAsPlainString method.