PostgreSQL container is automatically shut down

[sz3lbi]

Hi,
I am really new to Podman. I wanted to try an immutable distro, Fedora Kinoite in my case, and because of that I need to use containers for some software like PostgreSQL if I want to install it.
So I pulled the docker.io/library/postgres image of PostgreSQL and ran it. When I did some configuration, such as creating new users and databases, everything was fine. However, after leaving the container idle for a while, it automatically shut down and I could see the status EXITED in podman.
On StackOverflow I've found information that "received smart shutdown request is the result of sending SIGTERM to postgres process, so you might want to investigate why such signal is sent".
Could you please help me to find out what is causing such a behaviour? Below I attach how I run the container and what terminal output I could see after the container was shut down.

Container run:
podman run -dt --name postgres-xxx --userns=keep-id -e POSTGRES_USER=username -e POSTGRES_PASSWORD=password -v "/home/my_user/postgres-xxx:/var/lib/postgresql/data:Z" -p 5432:5432 postgres

Container terminal output:

2023-10-06 16:31:32.691 UTC [180] STATEMENT:  CREATE TABLE public.test_table (
                column1 varchar NULL
        )
2023-10-06 16:35:59.355 UTC [9] LOG:  checkpoint starting: time
2023-10-06 16:37:32.868 UTC [9] LOG:  checkpoint complete: wrote 932 buffers (5.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=93.347 s, sync=0.139 s, total=93.513 s; sync files=306, longest=0.009 s, average=0.001 s; distance=4282 kB, estimate=4282 kB; lsn=0/2E57728, redo lsn=0/2E576F0
2023-10-06 16:56:00.268 UTC [9] LOG:  checkpoint starting: time
2023-10-06 16:56:01.328 UTC [9] LOG:  checkpoint complete: wrote 11 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.007 s, sync=0.036 s, total=1.061 s; sync files=9, longest=0.014 s, average=0.004 s; distance=46 kB, estimate=3858 kB; lsn=0/2E62F30, redo lsn=0/2E62EF8
2023-10-06 17:20:11.022 UTC [1] LOG:  received smart shutdown request
2023-10-06 17:20:11.030 UTC [1] LOG:  background worker "logical replication launcher" (PID 14) exited with exit code 1
2023-10-06 17:20:11.030 UTC [1] LOG:  server process (PID 18) was terminated by signal 1: Hangup
2023-10-06 17:20:11.030 UTC [1] LOG:  terminating any other active server processes
2023-10-06 17:20:11.036 UTC [1] LOG:  abnormal database system shutdown
2023-10-06 17:20:11.069 UTC [1] LOG:  database system is shut down

[kishaningithub]

Getting the same error..

[dkragen]

Me too.

[rhatdan]

Are you running rootless containers and logging out? If so you need to enable linger in systemd for the users or systemd will shut down processes launched by the user.

[dkragen]

I enabled lingerd, and yes, I am running rootless. No unexpected shutdowns for the past 3 days. Keeping my fingers crossed and the logs running.

On Mon, 13 May 2024 13:44:40 -0700, Daniel J Walsh wrote: Are you running rootless containers and logging out? If so you need to

enable linger in systemd for the users or systemd will shut down processes launched by the user.

-- Reply to this email directly or view it on GitHub:

#20294 (comment)

…

You are receiving this because you commented. Message ID:

[Phelsong]

I am also getting this error.
After much troubleshooting, seems like its possibly whenever postgres goes to write to disk. It crashes every container on the system its running on and they will not restart.

rootless with linger and podman-restart
rhel 9, podman version 4.6.1

[rhatdan]

What error is it throwing a corrupted DB?

[Phelsong]

there are no errors I can find. Nothing in postgres log, nothing in host journal, nothing in podman. The service just kills itself, taking some/all the containers with it.

2024-05-16 17:06:38.555 UTC [22] LOG:  checkpoint starting: time
2024-05-16 17:06:39.564 UTC [22] LOG:  checkpoint complete: wrote 13 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.004 s, sync=0.002 s, total=1.009 s; sync files=10, longest=0.001 s, average=0.001 s; distance=8 kB, estimate=8 kB; lsn=0/452E8D8, redo lsn=0/452E8A0
2024-05-16 17:21:38.836 UTC [22] LOG:  checkpoint starting: time
2024-05-16 17:21:39.644 UTC [22] LOG:  checkpoint complete: wrote 9 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.803 s, sync=0.002 s, total=0.808 s; sync files=7, longest=0.001 s, average=0.001 s; distance=2 kB, estimate=8 kB; lsn=0/452F158, redo lsn=0/452F120
2024-05-16 17:30:02.019 UTC [1] LOG:  received fast shutdown request
2024-05-16 17:30:02.020 UTC [1] LOG:  aborting any active transactions
2024-05-16 17:30:02.020 UTC [110] FATAL:  terminating connection due to administrator command
2024-05-16 17:30:02.020 UTC [750] FATAL:  terminating connection due to administrator command
2024-05-16 17:30:02.022 UTC [1] LOG:  background worker "logical replication launcher" (PID 27) exited with exit code 1
2024-05-16 17:30:02.024 UTC [22] LOG:  shutting down
2024-05-16 17:30:02.025 UTC [22] LOG:  checkpoint starting: shutdown immediate
2024-05-16 17:30:02.029 UTC [22] LOG:  checkpoint complete: wrote 4 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.005 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=7 kB; lsn=0/452F208, redo lsn=0/452F208
2024-05-16 17:30:02.038 UTC [1] LOG:  database system is shut down

PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-05-16 17:30:25.401 UTC [1] LOG:  starting PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-05-16 17:30:25.401 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-05-16 17:30:25.401 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-05-16 17:30:25.403 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-05-16 17:30:25.406 UTC [24] LOG:  database system was shut down at 2024-05-16 17:30:02 UTC
2024-05-16 17:30:25.416 UTC [1] LOG:  database system is ready to accept connections
2024-05-16 17:35:25.459 UTC [22] LOG:  checkpoint starting: time
2024-05-16 17:35:25.566 UTC [22] LOG:  checkpoint complete: wrote 4 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.102 s, sync=0.001 s, total=0.107 s; sync files=3, longest=0.001 s, average=0.001 s; distance=5 kB, estimate=5 kB; lsn=0/4530A30, redo lsn=0/45309F8
2024-05-16 17:40:25.664 UTC [22] LOG:  checkpoint starting: time
2024-05-16 17:40:26.070 UTC [22] LOG:  checkpoint complete: wrote 5 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.402 s, sync=0.002 s, total=0.407 s; sync files=4, longest=0.001 s, average=0.001 s; distance=23 kB, estimate=23 kB; lsn=0/45366D8, redo lsn=0/45366A0
2024-05-16 18:00:01.667 UTC [1] LOG:  received fast shutdown request
2024-05-16 18:00:01.669 UTC [1] LOG:  aborting any active transactions
2024-05-16 18:00:01.671 UTC [1] LOG:  background worker "logical replication launcher" (PID 27) exited with exit code 1
2024-05-16 18:00:01.671 UTC [22] LOG:  shutting down
2024-05-16 18:00:01.673 UTC [22] LOG:  checkpoint starting: shutdown immediate
2024-05-16 18:00:01.677 UTC [22] LOG:  checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.007 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=20 kB; lsn=0/4536788, redo lsn=0/4536788
2024-05-16 18:00:01.690 UTC [1] LOG:  database system is shut down

PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-05-16 18:09:33.540 UTC [1] LOG:  starting PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-05-16 18:09:33.541 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-05-16 18:09:33.541 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-05-16 18:09:33.542 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-05-16 18:09:33.545 UTC [24] LOG:  database system was shut down at 2024-05-16 18:00:01 UTC
2024-05-16 18:09:33.549 UTC [1] LOG:  database system is ready to accept connections
2024-05-16 18:14:33.588 UTC [22] LOG:  checkpoint starting: time
2024-05-16 18:14:33.593 UTC [22] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.002 s, total=0.005 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/4536870, redo lsn=0/4536838

This has been happening for days, just grabbed an example