Not able to read or write remote logs on MinIO

[xisberto]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.8.3

What happened?

I have Airflow 2.8.3 on Kubernetes, deployed through the Helm chart. On the same namespace, I have a MinIO tenant.

My DAGs can write output files to the MinIO using a connection and the S3Hook.

I'm trying to use remote_logging to get the logs to a bucket on the same MinIO tenant, but it's not working.

When I try to read the DAGs logs, the airflow-webserver logs say:

[2024-04-23T16:34:41.070+0000] {base.py:83} INFO - Using connection ID 'conn_logs' for task execution.
[2024-04-23T16:34:41.072+0000] {connection_wrapper.py:387} INFO - AWS Connection (conn_id='conn_logs', conn_type='aws') credentials retrieved from extra.
/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py:1061 InsecureRequestWarning: Unverified HTTPS request is being made to host 'airflow-storage-hl'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
[2024-04-23T16:34:42.721+0000] {file_task_handler.py:583} ERROR - Could not read served logs
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/backends/sync.py", line 94, in connect_tcp
    sock = socket.create_connection(
  File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
    raise exc
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
    response = connection.handle_request(request)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 86, in handle_request
    raise exc
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 63, in handle_request
    stream = self._connect(request)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 111, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/backends/sync.py", line 94, in connect_tcp
    sock = socket.create_connection(
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 561, in _read_from_logs_server
    response = _fetch_logs_from_service(url, rel_path)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 92, in _fetch_logs_from_service
    response = httpx.get(
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_api.py", line 189, in get
    return request(
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_api.py", line 100, in request
    return client.request(
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", line 821, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", line 908, in send
    response = self._send_handling_auth(
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", line 936, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", line 973, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_client.py", line 1009, in _send_single_request
    response = transport.handle_request(request)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/airflow/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno -3] Temporary failure in name resolution
10.244.250.0 - - [23/Apr/2024:16:34:42 +0000] "GET /api/v1/dags/extract-entradas-saidas/dagRuns/manual__2024-04-23T13:01:22.569932+00:00/taskInstances/extract_and_combine_task/logs/2?full_content=false HTTP/1.1" 200 286 "https://airflow.mpce.mp.br/dags/extract-entradas-saidas/grid?dag_run_id=manual__2024-04-23T13%3A01%3A22.569932%2B00%3A00&task_id=extract_and_combine_task&tab=logs" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

This is the relevant part of the values.yaml:

config:
  logging:
    remote_logging: 'True'
    logging_level: 'INFO'
    remote_base_log_folder: 's3://logs'
    remote_log_conn_id: 'conn_logs'
    delete_worker_pods: 'False'
    encrypt_s3_logs: 'False'
    colored_console_log: 'True'

This is the conn_logs connection extras currently:

{
  "aws_access_key_id": "XXXXXXXXXXXXXX",
  "aws_secret_access_key": "XXXXXXXXXXXXXXXXXXXXX",
  "endpoint_url": "https://airflow-storage-hl:9000",
  "verify": false
}

What I've tried to do:

• Use the complete URL for the endpoint_url as in https://airflow-storage-hl.airflow.svc.cluster.local:9000
• Use the cluster's CA file as the verify value (/var/run/secrets/kubernetes.io/serviceaccount/ca.crt)

Before setting the verify option in the connection, I was getting errors related to the certificate validation. This also was the error when setting the value to 'False'.

I have also successfully read from the bucket from the webserver pod:

kubectl -n airflow exec -it airflow-webserver-xxxxxxx-xxxx -c webserver -- curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://airflow-storage-hl.airflow.svc.cluster.local:9000/landing

kubectl -n airflow exec -it airflow-webserver-xxxxxxx-xxxx -c webserver -- curl -v -k https://airflow-storage-hl:9000/landing

What you think should happen instead?

The logs should be written to the MinIO 'logs' bucket the same way the DAGs write files to the other buckets in the same MinIO tenant.

How to reproduce

1. Deploy MinIO via helm chart
2. Create a MinIO tenant airflow-storage. Create a Access Key to use in the Airflow Connection
3. Deploy Airflow via helm chart, configure the Airflow Connection
4. Configure remote logging in the Airflow's values.yaml file, redeploy it

Operating System

Rocky Linux 9.3

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.19.0
apache-airflow-providers-celery==3.6.1
apache-airflow-providers-cncf-kubernetes==8.0.1
apache-airflow-providers-common-io==1.3.0
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-docker==3.9.2
apache-airflow-providers-elasticsearch==5.3.3
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.16.0
apache-airflow-providers-grpc==3.4.1
apache-airflow-providers-hashicorp==3.6.4
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==9.0.1
apache-airflow-providers-microsoft-mssql==3.5.0
apache-airflow-providers-mysql==5.5.4
apache-airflow-providers-odbc==4.4.1
apache-airflow-providers-openlineage==1.6.0
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-redis==3.6.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.9.0
apache-airflow-providers-slack==8.6.1
apache-airflow-providers-smtp==1.6.1
apache-airflow-providers-snowflake==5.3.1
apache-airflow-providers-sqlite==3.7.1
apache-airflow-providers-ssh==3.10.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

I'm deploying Airflow using the official Helm Chart using ArgoCD. The image is modified as follows:

FROM apache/airflow:2.8.3
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir pip install apache-airflow-providers-microsoft-mssql==3.5.0

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

Support of MinIO through Amazon Provider not officially supported/tested.

[xisberto]

Ok, I understand. Is there another Provider that can be used for MinIO connections?