Huge Memory usage from Serializer

[PolSpock]

Hi,

I'm currently developing with Django Rest Framework an interface where users have to upload huge JSON files. So basically a GET request for getting the forms, and a POST request for uploading files.
These JSON files will be merged into one big JSON Array cause they depend on each other.

Then, i use rest_framework.serializers.ModelSerializer to transform my JSON Array to an OrderedDict.
Currently, this OrderedDict will be used for treatment without saving in a database.

However, i'm encountering several memory issues. As i found on these issues ;

• Memory usage optimization on Serializer class #7250
• DRF does not release memory after large JSON/XML request data is read or send through response #8472
• request.data has a memory leak #5146

Django Rest Framework seems to not release memory automatically for Serialized data after treatment.
So if users upload several files in a short time, my server is running out of memory cause of big Serializer memory usage

So how to force memory from Serializer to be released ? Or does anyone have a better logic to implement for my use case ? (Upload JSON files -> merging into one big JSON array -> validation -> casting to OrdererdDict -> treatment)

Regards

[foarsitter]

Huge doesn't mean anything so can you tell how many ram you processes are actually using? I'm saying processes since you are probably running in production with multiple worker processes.

Python does not necessarily release the memory back to the Operating System.

So in the end all your workers have become greedy and you are running out of memory. Lowering the amount of workers to 1 will probably solve it if your machine has enough memory to host at least 1 greedy process.

Gunicorn has the max_requests config and Celery comes with the worker_max_tasks_per_child to mitigate this problem by stopping the old process and start a new one.

[PolSpock]

Hi, thx for your answer

I have only one process. It's a monolithic application. I'm currently using a test VPS server with 2go RAM on Debian 11. I have Apache2 with a simple virtual proxy to the python manage.py runserver running process.

Yeah "huge" is not the correct word: this memory consummation is quite normal, i'm manipulating several JSON Files sized at ~1Go, causing huge memory usage. But i would like to release my Serializer instance after treatment, that's all.

Also thanks for your link, very interesting for Python memory usage. I've tried some del and gc.collect() but my Serializer is still alive so won't be deleted now.

Soo may i have to use a better server? I have to continue some tests some gc func. And i will also try to don't use Serializer :/

[foarsitter]

More ram is an easy (and mostly cheap) solution. Your baseline would be 1gb ram (in example) for other services (Apache2, OS), doubling the amount of ram to 4gb would triple the amount available for django.