psycopg.AsyncConnection.connect leaks memory on Mac

[mkanoor]

When using the psycopg.AsyncConnection.connect we have observed that it leaks memory, attached is a sample code.
Using Version: 3.1.12 of psycopg

import asyncio
import json
import logging

import psycopg
from psycopg import OperationalError

logger = logging.getLogger(__name__)

class PGNotify:
    def __init__(self, event: dict):
        self.event = event
        self.channel = "myevents"

    async def run(self):
        try:
            async with await psycopg.AsyncConnection.connect(
                host="localhost",
                dbname="db",
                user="postgres",
                password="password",
                autocommit=True,
            ) as conn:
                async with psycopg.AsyncClientCursor(connection=conn) as cursor:
                    payload = json.dumps(self.event)
                    await cursor.execute(f"NOTIFY {self.channel}, " f"'{payload}';")
        except OperationalError as e:
            logger.error(f"PG Notify operational error {e}")


async def main():
    for i in range(500000):
        event = {"data": "v" * 200, "sequence": i}
        await PGNotify(event).run()

asyncio.run(main())

[dvarrazzo]

I can't reproduce the issue. Running your script, top shows stable memory usage.

What Python version/OS are you using? My test is with Python 3.10 on Linux

Which libpq implementation are you using (see psycopg.pq.__impl__). In my test neither the C implementation nor the Python implementation seems to leak.

Can you find which object is leaking? see this comment and following ones for some hint about how to find the type of object leaking.

[mkanoor]

I am using Mac OS 14.1.1 (23B81)
Python 3.10.13
impl is binary
Attached is a file with the gc objects after I have called gc.collect
debug-04.txt

[dvarrazzo]

I'm afraid only one GC file doesn't help. What matters is to see the difference in counts between two iterations.

Please use this comment as reference about how to snipe an eventual memory leak.

[mkanoor]

I tested on Linux there is no leak it's happening only on Mac. On Mac the RSS keeps growing but on Linux it's steady. I tried your suggestions and can't see any difference in the gc.get_objects(). I will keep trying on Mac.

[dvarrazzo]

Out of trying random things, maybe you can try to connect with sslmode=disable to check if it's anything related to the ssl support in libpq.

Something else interesting to check is to try using PSYCOPG_IMPL=python to verify if the leak is in the C wait function.

Thank you for your help to look into it!

[dvarrazzo]

This is probably caused by a memory leak in krb5, see psycopg/psycopg2#1662.

Once a fixed krb5 version is released I can release new binary packages including it.

[dvarrazzo]

Still waiting for 1.21.3 to close this.

https://github.com/krb5/krb5/tags

[dvarrazzo]

@greghudson do you think it would be possible to make a release of krb5 including this bugfix? It is affecting psycopg macOS users.