Falcon app container memory pile up

[mmdmsb]

Hi

I have written a POST api which involves some multiprocessing, when monitoring the docker container running this falcon app I notice a memory build up is happening and NOT relieved over the time. Falcon route attaching to a single class instance may be reason behind this ? this is common scenario so everyone should see their memory building up or I am doing something basically wrong in setting up the falcon app.

Note: I m a beginner in the world of python.

TIA !

[CaselIT]

Hi,

Without a code snipped or at least some more details of what you are doing all I can say is that it's not a common issue, and not something does by its own.

[mmdmsb]

@CaselIT Sorry for the delay, below is the related class

class DecryptFieldsResource:
    def on_post(self, request, response):
        #mp -> multiprocessing
        manager = mp.Manager()
        request_data = json.loads(request.bounded_stream.read().decode())

    validate(instance=request_data, schema=constant.DECRYPT_FIELDS_SCHEMA)

    if len(request_data["values"]) > constant.MAX_NUMBER_OF_ITEMS:
        raise TooManyItemsException

    self.num_cpu_cores = math.ceil(mp.cpu_count()//2) #half the number of cpus available or min 1 cpu on machine is used
    self.result_dict = manager.dict() #dictionary that will be sent in reponse
    self.keys = request_data["keys"]

    values_for_each_process = np.array_split(request_data["values"], self.num_cpu_cores)

    pool = mp.Pool(self.num_cpu_cores)
    pool.map(self._decrypt_fields_per_process, values_for_each_process)
    pool.close()
    pool.join()

    response.media = dict(self.result_dict)
    response.status = falcon.HTTP_200

def _decrypt_fields_per_process(self, items):
    MAX_CACHE_SIZE = 10
    MAX_CACHE_AGE = 300.0

    client = aws_encryption_sdk.EncryptionSDKClient(
    )

    provider = aws_encryption_sdk.StrictAwsKmsMasterKeyProvider(key_ids=self.keys)

    #data key cache
    cache = aws_encryption_sdk.LocalCryptoMaterialsCache(MAX_CACHE_SIZE)

    #cache manager
    caching_cmm = CachingCryptoMaterialsManager(
        master_key_provider=provider,
        cache=cache,
        max_age=MAX_CACHE_AGE,
    )

    for item in items:
        decrypted_value_bytes, _ = client.decrypt(
            source=decoded_value, materials_manager=caching_cmm
        )
        self.result_dict[item["id"]] = decrypted_value_bytes

[mmdmsb]

thanks, the data returned in response (self.result_dict) is like half mb while the memory stashed during the API call is around 56mb.

In my docker network I/O is 23.7 MB / 530.1 kB, memory jumped from base image 207.8mb to 263.4mb after this API call.

Any other additional loop hole in my code might be contributing to this memory spike ?

@CaselIT

[CaselIT]

well, since it seems to be doing quite a bit of things, that change in memory would not be bad by itself. Does calling the api again has the same use of memory, or it stabilizes after the first couple of runs?

[mmdmsb]

Memory keeps pilling steadily up to 20-30 calls close to 2GB, not sure why the objects are not reclaimed via python's GC.
@CaselIT

[CaselIT]

That seems to be leaking somewere. I think you should avoid using the shared dict and do instead something like this:

class DecryptFieldsResource:
    def __init__(self):
        self.num_cpu_cores = math.ceil(mp.cpu_count()//2) #half the number of cpus available or min 1 cpu on machine is used
        self.pool = mp.Pool(self.num_cpu_cores)
    def on_post(self, request, response):
        #mp -> multiprocessing
        request_data = json.loads(request.bounded_stream.read().decode())

        validate(instance=request_data, schema=constant.DECRYPT_FIELDS_SCHEMA)

        if len(request_data["values"]) > constant.MAX_NUMBER_OF_ITEMS:
            raise TooManyItemsException

        self.keys = request_data["keys"]

        values_for_each_process = np.array_split(request_data["values"], self.num_cpu_cores)

        complete = {}
        parts = self.pool.map(self._decrypt_fields_per_process, values_for_each_process)
        for part in parts:
            complete.update(part)

        response.media = complete
        response.status = falcon.HTTP_200

    def _decrypt_fields_per_process(self, items):
        MAX_CACHE_SIZE = 10
        MAX_CACHE_AGE = 300.0

        client = aws_encryption_sdk.EncryptionSDKClient(
        )

        provider = aws_encryption_sdk.StrictAwsKmsMasterKeyProvider(key_ids=self.keys)

        #data key cache
        cache = aws_encryption_sdk.LocalCryptoMaterialsCache(MAX_CACHE_SIZE)

        #cache manager
        caching_cmm = CachingCryptoMaterialsManager(
            master_key_provider=provider,
            cache=cache,
            max_age=MAX_CACHE_AGE,
        )

        result_dict = {}
        for item in items:
            decrypted_value_bytes, _ = client.decrypt(
                source=decoded_value, materials_manager=caching_cmm
            )
            result_dict[item["id"]] = decrypted_value_bytes

        return result_dict

I've moved the pool to be shared among processes, but it should be ok to also keep in the on get. just remember to close it once done

[mmdmsb]

thanks ! will check this out