Container Placement Strategies not working

[ChrisJBurns]

Summary

We have set our container replacement strategies to the following:

concourse:
    web:
        checkContainerPlacementStrategies:
        - limit-active-containers
        - fewest-build-containers
        containerPlacementStrategies:
        - limit-active-containers
        - fewest-build-containers
        noInputContainerPlacementStrategies:
        - limit-active-containers
        - fewest-build-containers

Expected results

We expect containers to be spread across the workers.

Actual results

We get workers with an disproportionate amount of containers. Often resulting in our builds failing because they've hit the container limit even if there are other workers that have plenty of room.

Additional context

The default max limit is 250.

Triaging info

• Concourse version: 7.10.0
• Browser (if applicable): Chrome

[xtremerui]

Just want to make sure if CONCOURSE_MAX_ACTIVE_CONTAINERS_PER_WORKER is set

[ChrisJBurns]

This is on the web deployment:

This is on the stateful set for the workers:

We are adding this value in the Helm Chart: concourse.web.limitActiveContainers: 200

So it doesn't seem to be honouring the max container limit since the screenshot in the original post had workers with over 200 containers

[mhlic]

We are also seeing this behavior with the containers not being spread out, and the limit-active-containers not being respected

[ChrisJBurns]

@xtremerui Any updates on the above?

[taylorsilva]

So I have a guess as to why people run into issues like this still. I think it's because FindOrSelectWorker() can be called any number of times in parallel:

concourse/atc/worker/pool.go

Line 45 in 86386eb

func (pool Pool) FindOrSelectWorker(

This function is how each step selects a worker and is the main entry point to container placement. We can see where it's getting called:

So if there's no limit on how many get/put/task steps are being run at any given time, I think it's highly likely that you can end up in a situation where there are a bunch of steps trying to find workers at the same time. They all look at the current data available to them and make the same choice which then goes over the limits set by operators and your worker falls over.

My current theory is that putting a rate limit on the number of calls to FindOrSelectWorker() would help with this general "workers failing" issue. It would likely mean steps take a bit longer to initialize, but I'd wager that situation is better than "my entire job failed because the worker fell over".