ujson uses 30% memory compared to json, even after load complete

[hyperknot]

In an extremely simple test of loading a 140 MB JSON file, ujson ends up with 30% higher memory usage even after loading has happened.

Both built-in json and simplejson uses exactly the same amount of memory: 992 MB
ujson on the other hand uses: 1314 MB

Test is super simple, like:

import json
import ujson
import time

with open('test.json') as infile:
    data = ujson.load(infile)

time.sleep(60)

And I check in htop's RES column for the memory usage. It's the same as macOS's Activity Monitor's memory column.

[ncopa]

This may be related to #254 (comment)

[ramanguptativo]

This is a memory leak: explosion/srsly#4

[hugovk]

Note: explosion/srsly#4 was fixed by PR explosion/srsly#5 which is a port of PR #270, not yet fully merged.

[hugovk]

Although it would be good to retest with 2.0.0 to see if the merged subsets of #270 have fixed this particular issue.

[magiceses]

ujson version 2.0.3 maybe still have the problem of memory leak.

I test ujson version 2.0.3 ,here is script in container gnocchi-api:

()[root@gnocchi-api-69c75646dc-lzg9l /]# ls -lh test.json
-rw-r--r-- 1 root root 65M Jul  1 10:17 test.json

import ujson

print ujson.__version__ # 2.0.3

while True:
    with open('test.json') as infile:
        data = ujson.load(infile)

And i found memory leak maybe stall exist;

the memory usage of gnocchi-api before test
kubectl top po -n openstack gnocchi-api-69c75646dc-lzg9l
NAME CPU(cores) MEMORY(bytes)
gnocchi-api-69c75646dc-lzg9l 1m 951Mi

when testing
kubectl top po -n openstack gnocchi-api-69c75646dc-lzg9l
NAME CPU(cores) MEMORY(bytes)
gnocchi-api-69c75646dc-lzg9l 965m 1685Mi

Interrupt program;
And waiting for 20min
kubectl top po -n openstack gnocchi-api-69c75646dc-lzg9l
NAME CPU(cores) MEMORY(bytes)
gnocchi-api-69c75646dc-lzg9l 1m 1019Mi

see that there are still 68M of memory that have not been released; but when i use json to test the script , it did not happen.

Is there anything wrong with my test method? Any body help?

[JustAnotherArchivist]

#270 and colleagues addressed leaks during encoding, not decoding. I don't see any relevant commits that might have changed anything on decoding.

[shortland]

Still relevant for ujson 5.7.0 (at least in my case, with respect to decoding)

While I don't have an exact number, it is around 20-30% as the original OP commented.