Airflow issue: Failed to fetch log file from worker. 403 Client Error

[maleksan]

Airflow issue: Failed to fetch log file from worker. 403 Client Error.

Can't fetch logs via web from worker node, there is the following error:

*** Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://***.log

Airflow running in the following configuration:

Helm chart version 10.2.8
airflow:2.1.2-debian-10-r18
airflow-scheduler:2.1.2-debian-10-r17
airflow-worker:2.1.2-debian-10-r16
executor: "CeleryExecutor"

Can access the logs on the worker pods directly.

Downgrading and running airflow-worker to 2.1.1 version (for example 2.1.1-debian-10-r9), all works fine and there is possibility to fetch logs from worker node via web.

[javsalgar]

Hi,

It seems weird to me that this functionality stops working from one patch version to the other, specially taking into account that we did not perform any substantial changes in the logic. Did you report this to the Airflow devs? Maybe there is an issue on their side.

[maleksan]

Did you report this to the Airflow devs?

Not yet. Why I have to report to the Airflow devs when I'm using Bitnami Helm chart?
Or do you suggest me to switch to original Apache Airflow Helm chart?

[javsalgar]

If you see the issue in the upstream Airflow chart using version 2.1.1, then it would be clear to me that it is an upstream issue.

[maleksan]

To be honest, no idea where the issue is.
And it is not entirely clear how I'll address current issue to the Airflow devs.
Don't you test your Helm charts?

[javsalgar]

Let me clarify this better. While we package the Bitnami Airflow helm chart, we are not Airflow developers, so if an issue is not related how we package and configure Airflow but related to an upstream bug (which, according to what you mention, it seems highly likely), then this should be reported to the Airflow developers so they release a new version we can package.

We do test our charts in all major Kubernetes distributions, and we perform a basic test suite that ensures that it is configured correctly, such as login/logout, create users, run a DAG, check that the configuration is not exposed... But we cannot perform a bigger test suite than the Airflow devs themselves, as we assume most of the functionalities have been tested upstream. In this sense, we do not have a test that checks if it is possible to fetch a log file from the worker. Looking at what happened here, maybe it makes sense to add it.

In any case, today I have more bandwidth so I will try checking if it is an upstream issue or something related to our configuration. I will keep you posted.

[javsalgar]

I tried with the upstream chart version 2.1.2 and the logs fetch works. Because of this, I confirm that there must be something incorrect with how we configure the Airflow Worker logs server. I will open a task for investigation.

Thank you so much for reporting

[austin-ps]

I tried with the upstream chart version 2.1.2 and the logs fetch works. Because of this, I confirm that there must be something incorrect with how we configure the Airflow Worker logs server. I will open a task for investigation.

Thank you so much for reporting

Thanks for looking into this.
You are probably aware of this by now but If you look at the changelog for 2.1.2 https://airflow.apache.org/docs/apache-airflow/stable/changelog.html
You will see that they have modified how only the web server can request from the worker log server apache/airflow#16754 may be that affected something in your chart?

We really need the other bugfix that they have solved in 2.1.2 - 'Fix “Invalid JSON configuration, must be a dict” bug'
So downgrading to 2.1.1 is not looking like an option.
Do you have a timeline for a fix for this please?
Let me know if I can help.

[austin-ps]

Perhaps this one apache/airflow#17260 will also help.

[maleksan]

Great, this solves current issue.
Just need to add in Helm values.yaml:

...
extraEnvVars:
  - name: AIRFLOW__WEBSERVER__SECRET_KEY
    value: "<your_secret>"
...

Reference: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key

After this the logs fetch via web works.
Thanks a lot for advise.

[randradas]

Since this is an important parameter needed to get a fully working Airflow deployment I added support in the latest airflow containers and chart for secret_key so no need to user an "extraEnvVars" and you can set the values.yaml to set the secret_key, otherwise a random one is configured. The same secret_key is set for the workers, webservers and the scheduler. For this feature just use the latest version of Airflow containers or the chart.

[maleksan]

@randradas ,
Thanks a lot for your efforts in this matter.

you can set the values.yaml to set the secret_key, otherwise a random one is configured
For this feature just use the latest version of Airflow containers or the chart.

Currently don't see this parameter "secret_key" in the latest helm chart values.yaml:
https://github.com/bitnami/charts/blob/master/bitnami/airflow/values.yaml

Please explain, how and where need to add this "secret_key"?

[randradas]

'secret_key' is the name used by Airflow's conf file.

In the values.yaml you can find it with the name 'secretKey'.