Airflow webserver UI unable to access worker node logs

[devashish-parmar]

Apache Airflow version

2.3.4

What happened

We have set up a multi node Airflow configuration. We have the below setup.

• 4 nodes
• node 1: webserver, rabbitmq, flower
• node 2: meta db
• node 3: worker 1
• node 4: worker 2

Although the setup is working fine and DAGs are getting distributed among the worker nodes, however the logs generated in the worker nodes are not reflecting in the webserver UI in the master node.
Below is the error:
***** Log file does not exist: /home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** Fetching from: http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log'
For more information check: https://httpstatuses.com/403**

We have checked all the folder permission, secret_key, timezone but still the problem persist.

can anyone help here?

What you think should happen instead

No response

How to reproduce

No response

Operating System

Linux Red Hat

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[nsAstro]

Can you see if there are any files in the directory '/home/dvafadm/airflow/logs/' and if so, can you find the ones corresponding the the above DAG and list the file/directory structure? I'm curious if this is a name mismatch or if the logs just don't exist.

[devashish-parmar]

yes, the directory/file does exist. the below command runs fine from the worker node

less "/home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log"

[jledoux-sonergia]

We've ran into the same issue, worker's gunicorn log webserver does not start properly on 2.3.4, downgrading to 2.3.3 fix this issue.

I think it's related to this PR #24846
And saimon46's comment explains it: #24846 (comment)

[potiuk]

@jledoux-sonergia - The 2.4.0 is out today and It should have the IPV6 problem fixed. @devashish-parmar - if the request is forbidden, you should have looked at the webeserver logs and it should tell you the exact reason for the problem you have

We have checked all the folder permission, secret_key, timezone but still the problem persist.

Timezone is not an issue. You should check if TIME is synchronized on all your machines.

Converting that into a discussion as either some of the problems mentioned here (IPV6) have been fixed in 2.4.0 or they need more looking at the deployment side and checking the webserver logs.

[orihomie]

I ran into the same issue and I can see this in web UI:

*** Log file does not exist: /opt/airflow/logs/dag_id=new_users_daily/run_id=manual__2023-04-17T13:26:48.645829+00:00/task_id=get_new_users/attempt=2.log
*** Fetching from: http://airflow-worker-57bccbbc9c-w54qh:8793/log/dag_id=new_users_daily/run_id=manual__2023-04-17T13:26:48.645829+00:00/task_id=get_new_users/attempt=2.log
*** Failed to fetch log file from worker. [Errno -2] Name or service not known

It seems to me that webserver is trying to fetch files by the worker's pod name, not service name.
I've tried to curl from my webserver name by the worker's service name and it gave me 403.

Also, as I can see, logs are stored in /opt/airflow/logs folder.

I've tried to ping / health worker svc from webserver pod and all requests gave me 403.

[kobethuwis]

I elaborated on this in a different ticket,ver here, might work for you guys too

[potiuk]

Yes. Did you configure hostname_callable properly @orihomie ? https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable