Python operator getting failed: Task exited with return code Negsignal.SIGABRT

[ajaymalikbol]

Apache Airflow version

2.2.2

What happened

We recently upgraded the airflow version from 2.0.2 to 2.2.2

In one of the dag, we are getting the issue in the python operator. Error: Task exited with return code Negsignal.SIGABRT
The dag consists of multiple tasks and out of those tasks one particular task (python operator) is showing the above error consistently.

we are seeing the log Started process 535 to run the task and then it is failing with the above error. We are also seeing file core.535 getting generated in the disk which indicates that there is some issue with the application. Though it is working fine in local but failing in AWS ECS cluster

*** Reading local file: /opt/airflow/logs/HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0/update_redshift_table_for_cluster_index_10_0/2022-03-14T08:01:00+00:00/1.log
[2022-03-14, 09:05:15 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0.update_redshift_table_for_cluster_index_10_0 scheduled__2022-03-14T08:01:00+00:00 [queued]>
[2022-03-14, 09:05:15 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0.update_redshift_table_for_cluster_index_10_0 scheduled__2022-03-14T08:01:00+00:00 [queued]>
[2022-03-14, 09:05:15 UTC] {taskinstance.py:1241} INFO - 
--------------------------------------------------------------------------------
[2022-03-14, 09:05:15 UTC] {taskinstance.py:1242} INFO - Starting attempt 1 of 1
[2022-03-14, 09:05:15 UTC] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2022-03-14, 09:05:15 UTC] {taskinstance.py:1262} INFO - Executing <Task(PythonOperator): update_redshift_table_for_cluster_index_10_0> on 2022-03-14 08:01:00+00:00
[2022-03-14, 09:05:15 UTC] {standard_task_runner.py:52} INFO - Started process 535 to run task
[2022-03-14, 09:05:15 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0', 'update_redshift_table_for_cluster_index_10_0', 'scheduled__2022-03-14T08:01:00+00:00', '--job-id', '11', '--raw', '--subdir', 'DAGS_FOLDER/hoodie_etl_pipeline_scheduler.py', '--cfg-path', '/tmp/tmpaqj_dwlc', '--error-file', '/tmp/tmpe9dznvdr']
[2022-03-14, 09:05:15 UTC] {standard_task_runner.py:77} INFO - Job 11: Subtask update_redshift_table_for_cluster_index_10_0
[2022-03-14, 09:05:15 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0.update_redshift_table_for_cluster_index_10_0 scheduled__2022-03-14T08:01:00+00:00 [running]> on host 66bd98e63fe3
[2022-03-14, 09:05:16 UTC] {taskinstance.py:1427} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0
AIRFLOW_CTX_TASK_ID=update_redshift_table_for_cluster_index_10_0
AIRFLOW_CTX_EXECUTION_DATE=2022-03-14T08:01:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-03-14T08:01:00+00:00
[2022-03-14, 09:05:16 UTC] {logging_mixin.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py:1941 DeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'logical_date' or 'data_interval_start' instead.
[2022-03-14, 09:05:16 UTC] {logging_mixin.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py:1941 DeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use 'data_interval_end | ds' instead.
[2022-03-14, 09:05:17 UTC] {local_task_job.py:154} INFO - Task exited with return code Negsignal.SIGABRT
[2022-03-14, 09:05:17 UTC] {taskinstance.py:1270} INFO - Marking task as FAILED. dag_id=HUDI_ETL_PIPELINE_TENANTS_CLUSTER_0, task_id=update_redshift_table_for_cluster_index_10_0, execution_date=20220314T080100, start_date=20220314T090515, end_date=20220314T090517
[2022-03-14, 09:05:17 UTC] {local_task_job.py:264} INFO - 1 downstream tasks scheduled from follow-on schedule check

What you think should happen instead

Python operator should have worked

How to reproduce

No response

Operating System

PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

Operator log file:

log_file (1).log

Started process 535 to run task
core.535 process id is same in python operator logs as in core dump file.

[potiuk]

SIGABRT is generated by your task. you need to debug what your task is doing and why it aborts. This has nothing to do with Airflow. Likely you are using some "c-level" library that fails (SIGABRT is usually when the task uses low-level c library and fails)).

There is nothing anyone can help with here - whoever wrote the Python task must debug and analyze itl

[potiuk]

Since you have the "core" file - you should be able to open it with gdb or whatever debugger you have and analyze it - but again - this is far beyond Airlfow, it's your code and dependencies that create a problem.,

[ajaymalikbol]

@potiuk Thanks for replying.

In the very first line of the task, I have put one log message which is not showing in the operator log files. So I don't think it was related to the actual dag code.

[potiuk]

It could be an import or something else. It definitely is your code.

[hovikkhachatryan]

I get the "Task exited with return code Negsignal.SIGABRT" error and found that this is happening when inside the PythonOperator trying to log the context object like this. I removed the logging and the issue is solved.

`def _prepare_env(**context):
logging.info(context)

prepare_env = PythonOperator(
task_id='prepare_env',
python_callable=_prepare_env,
provide_context=True,
dag=dag)`

[potiuk]

Possibly your logging is badly configured or you have not enough resources. SIGABRT is usually called when some library (usually in C) decides that it got in the state that it should abort immediately. The fact that this is logging, indicates one of two things:

1. your logging library/handler/logger etc. is misconfigured and aborts
2. your logging causes allocation of extra resources (for example memory) that your deployment is not abe to give it and it gets into immediate abort state

In both cases you will not see extra information in logs, precisely because something decided to abort immediately and likely that something had not good configuration or not enough resources to even "tell you" what's wrong.

Look in logging configuration and resources.

[kacpermuda]

In case someone ends up here when Googling SIGBART or SIGSEGV error in airflow. You might find it useful, at least i would, before i wasted some time trying to find the source of the problem.

We were struggling with Task exited with return code Negsignal.SIGABRT and Task exited with return code Negsignal.SIGSEGV error on our company's Apple Sillicon Macs (M1 and M2), when running airflow task. It seemed random at first. The problem in our case was the grpcio issue (described here). It was fixed in 1.51.3, so simple update fixed it for us now. Before that version was released, we had to reinstall grpcio with the following variables exported. We were installing it using poetry, but then we reinstalled it after poetry finished, using updated pip (i don't know why, but exporting those variables before poetry install command, did not work).
Full snippet

    # poetry install
    export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
    export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
    pip install -U pip setuptools
    pip install grpcio --force-reinstall

It is described in more details in the grpcio issue linked above.

Apart from those, as mentioned here and here , You may also have to export this into Your shell: export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES. Even on Airflow 2.X, yes.

[mattvonrocketstein]

anyone seeing this on MWAA >=2.2.2, and not on local/osx? i'm seeing this, and the "abort" part seems to hint that my workers are not beefy enough. but in my case workers are not doing any heavy lifting .. literally just NOOPs while i'm trying to test the DAG's task topology.

[henriqueblobato]

Try decrease the number of worker_concurrency and increase worker memory, in my case Im using helm, so:

config:
  celery:
    worker_concurrency: 4

workers:
  resources:
    limits:
     cpu: 100m
     memory: 8Gi
    requests:
     cpu: 100m
     memory: 8Gi

[bekzodbuyukov]

Adding export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES to .zshenv file helped me. Read more about how to do it here.