database.yml is in the wrong location for docker containers

[JasperE84]

Used Zammad Version

6.3.1

Environment

• Installation method: docker-compose

Actual behaviour

Scheduler container's log shows errors causing scheduled jobs to fail if using default docker-compose setup:
ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use.

Runtime pool size of 5 isn't equal to value of 50 configured in default rails database.yml:

$ docker compose run --rm zammad-scheduler rails r "p ActiveRecord::Base.connection_pool.stat"
[+] Creating 3/0
 ✔ Container docker_zammad-zammad-memcached-1   Running                                                                                            0.0s
 ✔ Container docker_zammad-zammad-redis-1       Running                                                                                            0.0s
 ✔ Container docker_zammad-zammad-postgresql-1  Running                                                                                            0.0s
I, [2024-05-17T08:31:11.773981 #1]  INFO -- : ActionCable is using the redis instance at redis://zammad-redis:6379.
I, [2024-05-17T08:31:11.824991#1-6060]  INFO -- : Using memcached as Rails cache store.
I, [2024-05-17T08:31:11.825266#1-6060]  INFO -- : Using the Redis back end for Zammad's web socket session store.
{:size=>5, :connections=>0, :busy=>0, :dead=>0, :idle=>0, :waiting=>0, :checkout_timeout=>5.0}

When copying the default rails database.yml and mounting it in docker as such, problem is solved:

x-shared:
  zammad-service: &zammad-service
    environment: &zammad-environment
      [..snip..]
    volumes:
      - ./tempdb.yml:/opt/zammad/config/database.yml

I think the fix should be something like contrib/docker/docker-entrypoint.sh checking for existing ~/config/database.yml symlink to ~/config/database/database.yml and creating it if it doesnt exist. Or move database.yml from its subdirectory to /config/ directly.

More info here:
https://community.zammad.org/t/scheduler-fails-with-activerecord-could-not-obtain-a-connection-from-the-pool-within-5-000-seconds-waited-5-000-seconds-all-pooled-connections-were-in-use/14320/7

Expected behaviour

Scheduled jobs finish without problems

Steps to reproduce the behaviour

$ docker-compose up -d

Support Ticket

No response

I'm sure this is a bug and no feature request or a general question.

yes

[JasperE84]

So basically this function seems to be missing in docker entry point:

zammad/contrib/packager.io/functions

Line 172 in 0a0935d

function update_database_yml () {

[Keridos]

Can confirm that this happened on our 6.3.1 deployment, too. Adding a bind mount with a correct database.yml file to /opt/zammad/config/database.yml fixes the symptoms.

[mgruner]

Please update your .env instead to the latest state from upstream. There is a new variable POSTGRESQL_OPTIONS=?pool=50 which is required now.

https://github.com/zammad/zammad-docker-compose/blob/master/.env

[matbgn]

Just as info, only putting the variable POSTGRESQL_OPTIONS=?pool=50 didn't work in our case

[trstn70]

Same here.

[mgruner]

@matbgn @trstn70 can you provide some more details?

[matbgn]

Unfortunately not, only that we ended with an ugly cron every 5 mins... such a shame I know 😞

Just for others as a temporary workaround:

docker compose run zammad-scheduler rails r "Channel.fetch"

[mgruner]

What's the effective pool size on the server? You can use the commandline from @JasperE84 to determine that:
docker compose run --rm zammad-scheduler rails r "p ActiveRecord::Base.connection_pool.stat" Did you try increasing this value?

[matbgn]

50 as expected, and that's why I find it bizarre...

[mgruner]

Can you try increasing the pool value via ENV variable?

[matbgn]

I already did it via .env but was concerned about the practical limit of 50 that I read on various forums. What would you consider a reasonable test. Sorry for my concern and therefore my dummy question but our Zammad instance is productive and a bit critical too.

[mgruner]

Good question. It will probably depend on your server's max connection limit. Looks like the default is 100, where a pool size of 50 might already be too high, if all connections are actually used.

It will probably not hurt to increase it in small steps like 10 at a time, and try it with that. Maybe also changing the max connections of the server might be interesting. Could be this is something we should consider for this stack.

[matbgn]

Ok, here is a final feedback

I increased the value 10 by 10 up to 100 without seeing any effect. After exploring github and the community forum I finally stumbled upon a hint that made me realise that after the initial crash our system was stuck on previous errors.

So in the end, setting pools back to 50 and just resetting errors for a clean check under System -> Monitoring did the trick.

I will keep a close eye on whether 50 is enough in our case and increase it if necessary, noting here what level gave satisfaction if relevant for the community.

Many, many thanks @mgruner