No LocalWorkers started when scheduler is launched in daemon mode.

[sorenarchibald]

Apache Airflow version

2.9.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

LocalExecutor workers aren't started when launching the airflow scheduler in daemon mode. I get this when running apache/airflow:latest docker image. Additionally, the Web UI says the scheduler isn't running and Jobs aren't started.

What you think should happen instead?

I would expect the same scheduler behavior whether launched as a deamon or not.

How to reproduce

compose.yml.txt

1. Run the attached docker compose file.
2. login to UI and note the message that says the scheduler isn't running.
3. login to web container, as root.
4. run apt-get update && apt-get install -y procps
5. run ps -ef and note that none of the LocalExecutor workers are running.

I can workaround this issue by starting the webserver first and the scheduler last -- not in daemon mode.

Operating System

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"

Versions of Apache Airflow Providers

N/A

Deployment

Other Docker-based deployment

Deployment details

Docker Compose version v2.26.1-desktop.1
Also observed run AWS Elastic Container Service.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[RNHTTR]

The documented pattern for running Airflow in Docker can be found in the Airflow guide Running Airflow in Docker. The docker compose file is a bit more extensive than the one you've shared.

Can you try following that guide? You can run everything in detached mode with docker compose up -d.

[sorenarchibald]

Thanks Ryan. I've worked around this issue by starting the scheduler last. I'm actually using AWS ECS and only made the compose file to debug the issue. I thought I would submit a bug as it did waste a couple hours of my time... and I thought it was time I try and get more involved with an open source project. Again: thanks for your help.

[RNHTTR]

Happy to help :) I converted this to a discussion because I don't think there's actually a bug here. Again, I'd like to highlight the docs on Running Airflow in Docker. Keep in mind that this is still not recommended for production.

[Taragolis]

Run Airflow services in daemon it is useful only on case if there is not possible for some reason to use external daemons (e.g. systemd) or containers.

In case of containers it's best practice to separate areas of concern by using one service per container.
It is much easier to maintain and restart due to the failure, e.g. if webserver goes down you don't have to restarts scheduler and vice versa.

I'm actually using AWS ECS and only made the compose file to debug the issue.

Even if you use cloud based container service the principal pretty the same, 1 service = 1 task.