Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slack provider does not install HTTP provider dependency #17475

Closed
cdabella opened this issue Aug 6, 2021 · 2 comments
Closed

Slack provider does not install HTTP provider dependency #17475

cdabella opened this issue Aug 6, 2021 · 2 comments
Labels
kind:bug This is a clearly a bug

Comments

@cdabella
Copy link

cdabella commented Aug 6, 2021

Apache Airflow version: 2.1.1+astro.2

Kubernetes version: N/A

Environment: astro CLI local dev environment

Host

  • Hardware configuration: AMD Ryzen 9 3900X (12-core, 3.8GHz), 32GB RAM, SSD
  • OS: Windows 10 Pro 20H2
  • Driver: Docker 20.10.7

Container

  • OS: Debian 10 (buster)
  • Kernel: Linux 971928acbe4c 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 GNU/Linux
  • Install tools: astro CLI v0.25.4 (https://github.com/astronomer/astro-cli) deployed using astro dev start
  • Others: Modify requirements.txt to include apache-airflow-providers-slack

What happened:
When importing the Slack Webhook operator, python throws a ModuleNotFoundError exception for the airflow.providers.http module.

$ python -c "import airflow.providers.slack.operators.slack_webhook"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/airflow/providers/slack/operators/slack_webhook.py", line 21, in <module>
    from airflow.providers.http.operators.http import SimpleHttpOperator
ModuleNotFoundError: No module named 'airflow.providers.http'

What you expected to happen:
The module should import correctly, potentially with a DeprecationWarning

$ python -c "import airflow.providers.slack.operators.slack_webhook"
/usr/local/lib/python3.7/site-packages/airflow/providers/http/operators/http.py:26 DeprecationWarning: This decorator is deprecated.

In previous versions, all subclasses of BaseOperator must use apply_default decorator for the`default_args` feature to work properly.

In current version, it is optional. The decorator is applied automatically using the metaclass.

It appears that the module does not include apache-airflow-providers-http as a dependency.

$ pipdeptree -p apache-airflow-providers-slack</summary>
Warning!! Cyclic dependencies found:
* astronomer-certified => astronomer-airflow-version-check => astronomer-certified
* astronomer-airflow-version-check => astronomer-certified => astronomer-airflow-version-check
------------------------------------------------------------------------
apache-airflow-providers-slack==1!4.0.0
  - astronomer-certified [required: >=2.1.0, installed: 2.1.1.post2]
    [...]
  - slack-sdk [required: >=3.0.0,<4.0.0, installed: 3.7.0]

The missing dependency is also missing from the provider.yaml file.

How to reproduce it:

Non-functional SlackWebhookOperator

  1. Add apache-airflow-providers-slack to the requirements.txt
  2. Start the astro local dev environment with astro dev start
  3. Add example-slack-dag.py file to the dags folder with the following code
from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime

from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator

with DAG('slackwebhookoperator_import',
         start_date=datetime(2019, 1, 1),
         max_active_runs=3,
         schedule_interval=None,  # https://airflow.apache.org/docs/stable/scheduler.html#dag-runs
         # catchup=False # enable if you don't want historical dag runs to run
         ) as dag:

    t0 = DummyOperator(
        task_id='start'
    )
  1. Observe DAG Import Error
Broken DAG: [/usr/local/airflow/dags/example-slack-dag.py] Traceback (most recent call last):
  File "/usr/local/airflow/dags/example-slack-dag.py", line 1, in <module>
    from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
  File "/usr/local/lib/python3.7/site-packages/airflow/providers/slack/operators/slack_webhook.py", line 21, in <module>
    from airflow.providers.http.operators.http import SimpleHttpOperator
ModuleNotFoundError: No module named 'airflow.providers.http'

Functional SlackWebhookOperator

  1. Add apache-airflow-providers-slack and apache-airflow-providers-http to the requirements.txt
  2. Stop and start the local airflow environment with astro dev stop && astro dev start
  3. Repeat step 3 from above, and observe no DAG Import Error.

Anything else we need to know: N/A

@cdabella cdabella added the kind:bug This is a clearly a bug label Aug 6, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Aug 6, 2021

Thanks for opening your first issue here! Be sure to follow the issue template!

@potiuk
Copy link
Member

potiuk commented Aug 6, 2021

Thanks for reporting it.

Http provider used to be preinstalled in 2.0 and it had been disabled due.to licensing issues. It has been restored as preinstalled provider and will be back in 2.1.3.

See #16974

@potiuk potiuk closed this as completed Aug 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:bug This is a clearly a bug
Projects
None yet
Development

No branches or pull requests

2 participants