ArrayMemoryError after triggering the weight computation

[VKTB]

I am running the service in a Docker container and trying to test it with some data. After populating the service with ~30K documents, I tried triggering the weight computation by sending a POST request to the /compute endpoint but the API threw an ArrayMemoryError exception.

docker-pss-api-1  | ERROR:    Exception in ASGI application
docker-pss-api-1  | Traceback (most recent call last):
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
docker-pss-api-1  |     result = await app(self.scope, self.receive, self.send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
docker-pss-api-1  |     return await self.app(scope, receive, send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
docker-pss-api-1  |     await super().__call__(scope, receive, send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
docker-pss-api-1  |     await self.middleware_stack(scope, receive, send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
docker-pss-api-1  |     raise exc from None
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
docker-pss-api-1  |     await self.app(scope, receive, _send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
docker-pss-api-1  |     raise exc from None
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
docker-pss-api-1  |     await self.app(scope, receive, sender)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 580, in __call__
docker-pss-api-1  |     await route.handle(scope, receive, send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 241, in handle
docker-pss-api-1  |     await self.app(scope, receive, send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 55, in app
docker-pss-api-1  |     await response(scope, receive, send)
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/responses.py", line 146, in __call__
docker-pss-api-1  |     await self.background()
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/background.py", line 35, in __call__
docker-pss-api-1  |     await task()
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/background.py", line 18, in __call__
docker-pss-api-1  |     await self.func(*self.args, **self.kwargs)
docker-pss-api-1  |   File "/app/app/routers/compute.py", line 97, in run_background_weight_computation
docker-pss-api-1  |     wc = await WC.runWorkflow(
docker-pss-api-1  |   File "/app/app/ml/weightscomputation.py", line 279, in runWorkflow
docker-pss-api-1  |     await wc.compute()
docker-pss-api-1  |   File "/app/app/ml/weightscomputation.py", line 200, in compute
docker-pss-api-1  |     (self._weights,self._weights_rows,self._weights_cols) = tf_iduf.TF_IDF(self._items)
docker-pss-api-1  |   File "/app/app/ml/tf_iduf.py", line 120, in TF_IDF
docker-pss-api-1  |     (
docker-pss-api-1  |   File "/usr/local/lib/python3.9/site-packages/scipy/sparse/_csr.py", line 177, in tocsc
docker-pss-api-1  |     data = np.empty(self.nnz, dtype=upcast(self.dtype))
docker-pss-api-1  | numpy.core._exceptions._ArrayMemoryError: Unable to allocate 6.52 GiB for an array with shape (874816512,) and data type float64

Based on the error message and from what I read online, it looks as though the application is trying to copy all of the data in memory. I increased my VM's RAM to 16GB (from 8GB) this morning as the container was exiting due to Docker not having enough RAM to compute the weights of ~9.7K documents, so maybe the application will not throw this error if I increase the RAM again. However, this will not solve the actual problem because the more data a facility has, the more RAM they will need. For example, we have ~165K documents in our dev environment (and definitely more on prod) so we will need a lot of RAM to run this service.

If the above error does in fact occur as a result of the application attempting to copy all of the data in memory then I believe that this will need to be re-implemented differently.

[nitrosx]

Dear VTKB

I am aware of this issue and I'm planning to work on it soon.

Thank you so much for reporting your finding.

[VKTB]

Thank you @nitrosx! I also wanted to check whether all of the documents that are already in the service need to be cleared and then added all over again each time new documents need to be added? Or can the new documents just be added and the weight computation then triggered?

[nitrosx]

dear @VKTB, you should be able to add a new document or update an existing one and trigger the weight computation without clearing all the documents.
I usually clear all the documents if the number of updates are too many and it is easier to delete and create new ones.

[VKTB]

That makes sense, thank you @nitrosx

[agbeltran]

@nitrosx I'm following up on this issue to check with you if you have a timeline on when it will be addressed? thanks!