The documentation is confusing because of different versions

[giftkugel]

Hi there,

I think the documentation is a bit confusing.

For example:

https://airflow.apache.org/docs/stable/installation.html#extra-packages says that I should use pip install 'apache-airflow[hdfs]' for HDFS hooks and operators, but https://github.com/apache/airflow/blob/master/docs/extra-packages-ref.rst says that I should use pip install 'apache-airflow[apache.hdfs]'

Which one is the correct one?

And the documentation at https://airflow.apache.org/docs/stable/installation.html#extra-packages shows much less possibilities then https://github.com/apache/airflow/blob/master/docs/extra-packages-ref.rst. For example spark is missing on https://airflow.apache.org/docs/stable/installation.html#extra-packages

Which documentation should I use?

Regards
Simon

[potiuk]

The stable documenttion is for released Arflow version (1.10.12 and very soon 1.10..13 will replace it). You can always select the stable version on the left side of the docs.

The "master" documentation is for the upcoming version of airflow (2.0.0). we are are 2.0.0b3 released last week and we are planning to release 2.0.0 version mid-December, then the documentation from master will be promoted to stable.

[giftkugel]

Thank you for the fast reply.

But why spark is missing at https://airflow.apache.org/docs/stable/installation.html#extra-packages ?

Which branch was used to generate the current documentation which is used on the website?

https://github.com/apache/airflow/blob/v1-10-stable/docs/installation.rst is also missing spark and says pip install 'apache-airflow[apache.hdfs]' not pip install 'apache-airflow[hdfs]'

[potiuk]

[potiuk]

And yes. There is no 'spark' extra in 1.10. It's only available in the upcoming Airflow 2.0. You can however use backported apache-spark provider and install it on top of 1.10. We have implemented backport providers that make it easier to run Airflow 2.0 operators on top of Airflow 1.10: https://pypi.org/project/apache-airflow-backport-providers-apache-spark/ - and it contains all the dependencies needed, you do not need to install "[spark]" extra.

You can read more about the backports in the "master" documentation: s.apache.org/airflow-docs (we are still working on those docse before 2.0 release so you might expect some changes). Currently provider information is here: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow-providers/index.html