[BUG] Pydantic issue with airflow operator

[itaise]

Hi,
We are using GreatExpectationsOperator, and when trying to upgrade great expectations version from 0.15.32 to 0.15.33, Airflow Dags are unable to load, and we are getting an ImportError. this also happen in every version up to the latest (0.15.36).

Broken DAG: [/Users/AAA/my_dag.py] Traceback (most recent call last):
  File "pydantic/__init__.py", line 2, in init pydantic.__init__
  File "pydantic/dataclasses.py", line 52, in init pydantic.dataclasses
ImportError: cannot import name dataclass_transform

Trying to investigate - the row where it fails is this one in pydantic:
https://github.com/pydantic/pydantic/blob/main/pydantic/dataclasses.py#L52

Can you please help? thanks a lot!

Expected behavior
Dag should load properly

Environment
airflow-provider-great-expectations 0.2.2
great_expectations 0.15.36
pydantic - 1.10.2

[austiezr]

Hey @itaise! Thanks for raising this. We'll discuss this in the coming week. 🙇

[Kilo59]

@itaise
What version of python are you using?
What version of typing_extensions?

Does the issue go away if you update typing_extensions to the latest?

[itaise]

Hi,
I am using python 3.7.12
And typing-extensions 4.4.0, which is the latest.
Just guessing, maybe some circular dependency?

[itaise]

@talagluck @Kilo59 I also copy and pasted the example dag from the provider package and got the same error.
https://github.com/astronomer/airflow-provider-great-expectations/tree/main/great_expectations_provider/example_dags

[itaise]

And our Airflow version is - apache-airflow==2.2.3

[Kilo59]

@itaise
typing_extensions is the only dependency of pydantic.

This is the pydantic PR that introduced that breaking line.

pydantic/pydantic#4241

What happens if you use an earlier version of pydantic that doesn't have that line?

Maybe 1.9.0?
https://docs.pydantic.dev/changelog/#v190-2021-12-31

---

I'm a bit confused as to what's happening. I would expect to see more people complaining about this.

Maybe double-check that you are using the intended virtual environment 🤷 ?

Here's a related issue but this is because their typing_extensions version was out of date.
And the PR that should have fixed it

[itaise]

Hi @Kilo59, thanks.
I tried downgrading to pydantic - 1.9.0 and it indeed make the error dissapear and the dag load fine.
pydantic - 1.9.0
GX - 0.15.36
typing-extensions - 4.4.0

[itaise]

Should the requirements file be more strict?

[Kilo59]

@itaise
You could limit the upper bound of pydantic on the airflow operator package.
But until we can reproduce this issue on other python versions I would opt to keep the pydantic version on core great_expectations flexible.

This seems like it might be limited to python 3.7

I will try to re-create the error locally and raise the issue with the pydantic team.

[Kilo59]

I've created a gist that contains an error.py file that attempts to reproduce the error.
The gist also contains a Pipfile + Pipfile.lock for creating a reproducible virtual environment.

https://gist.github.com/Kilo59/b3ddf0135ab2f30cd83ac5daac657e59

With pipenv installed you can attempt to recreate this error with these 2 commands.

1. pipenv sync - To create a virtual environment and install all dependencies.
2. pipenv run error - To execute the error.py script inside the newly created environment.

Note
I was unable to reproduce the desired error.

@itaise can you take a look at the gist, and let me know if this is a faithful reproduction of the error you ran into? If not how could I update it?
Does setting up the environment and running the script produce the desired error for you?

[Kilo59]

@itaise with today's release we will rely on pydantic v10.0.4 which should resolve the issue so you'll no longer need to downgrade pydantic to workaround the issue.

https://docs.pydantic.dev/changelog/#v1104-2022-12-30

[itaise]

Ok, Thanks a lot!