Getting "No module named 'airflow.compat'"

[smunveruddin-mwb]

Hi Team,

While trying to test EMRServerless operator using the github link in requirements file, I am getting the following error

"Broken DAG: [/usr/local/airflow/dags/EMR_Serverless.py] Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/emr_serverless/operators/emr.py", line 22, in
from emr_serverless.hooks.emr import EmrServerlessHook
File "/usr/local/lib/python3.7/site-packages/emr_serverless/hooks/emr.py", line 23, in
from airflow.compat.functools import cached_property
ModuleNotFoundError: No module named 'airflow.compat' "

Versions used in Docker container:

• Airflow = 2.1.0
• boto3 = 1.23.9
• apache-airflow-providers-amazon==3.0.0
• Python 3.7 Slim-buster

It would be great if someone can suggest me workarounds.

Thanks

[dacort]

Hi @smunveruddin-mwb - I've only tested this with Airflow 2.2.2. I believe the airflow.compat package was added in 2.2.0 so that could be the reason for the exception here.

Can you upgrade to 2.2.x? Or are you pinned on 2.1.0?

[smunveruddin-mwb]

Hi @dacort, Right now we are not looking at Airflow upgrade. I have made a workaround to add the repo to plugins folder and import cached_property from cached_property package, since it works the same way i believe. The issue has been resolved upon trying this workaround and I am able to submit jobs on EMR Serverless Application.

Thanks for responding to the issue. Really appreciate it.

[dacort]

Sounds good and glad you got it working! For now I'll close this as I don't plan to add support for Airflow <2.2.