boto3 session client memory

[romiLightlytics]

hi,
I am using the last version of boto3 in order to get information about all the AWS resources that exist in an environment.
The resources are in different accounts and regions.

I am using boto3 session for each account and a client for each resource type and I have noticed that the memory is getting bigger each client creation and does not get released at the end of the function.
what eventually causes the process to terminate.

I read that there is some memory leak in boto client.. but could not get a solution
My question is how can I overcome this problem?
what is the best practice for doing this task?
Is there a way to clear the cache? or a way to change the client configuration in order to avoid creating multiple?

Thank you!

[aBurmeseDev]

Hi @romiLightlytics, thank you for reaching out.

We have seen a few similar memory leak issues in the past which I'm going to list them below for your reference. The possible root cause can be varied from different use cases and workflows. With that said, is it possible for you to provide me a code sample so that i can try reproduce the issue? Without looking at the code it is a little difficult for me to find out the exact cause.

Here are a few similar issues:
#3088
#3054
#1670

Hope it helps!

Best,
John

[romiLightlytics]

Hi @aBurmeseDev
Thank you for the answer

this is my use case:

import boto3
import os, psutil
map_account_to_region = {"124242" : ["us-east-1", "us-east-2"]}
#   example: service -> ec2
functions_in_service = {"ec2": ["describe_instances", "describe_internet_gateways", "describe_launch_templates"]}

def scan_aws_resources():
    result = list()
    for account in map_account_to_region.keys():
        account_session = boto3.session.Session()
        aws_services = account_session.get_available_services()
        for region in map_account_to_region.get(account):
            for service in aws_services:
                client = account_session.client(service, region_name=region)
                for describe_function in functions_in_service.get(service):
                    func = getattr(client, describe_function)
                    res = func()
                    result.append(res)
    return result

if __name__ == "__main__":
    process = psutil.Process(os.getpid())
    print(process.memory_info().rss) # low memory
    result = scan_aws_resources()
    print(process.memory_info().rss) # high memory

after this code, the memory persists high and does not get relesed
it is meaningful when scanning all the resource types in all the accounts and regions
what do you think?
thanks!

[aBurmeseDev]

Thank you for your patience @romiLightlytics. Upon further investigation, I think the possible root cause of memory leak, in this case, could be multithreading where single session is being used by multiple thread. Here's the documentation that mentions multithreading:

Resource instances are not thread safe and should not be shared across threads or processes. These special classes contain additional meta data that cannot be shared. It's recommended to create a new Resource for each thread or process...

The idea is to create one session per thread. For your reference, I also found this code example from other user and they mentioned how it resolved the memory leak issue.

Hope it helps!

[romiLightlytics]

@aBurmeseDev Thanks! I'll check it