Adding extra requirements for build and runtime of the PROD image.

[potiuk]

This PR adds capability of adding extra requirements to PROD image:

1. During the build by placing requirements.txt in the
   docker-context-files folder

2. During execution of the container - by passing
   _PIP_ADDITIONAL_REQUIREMENTS variable

The second case is only useful durint quick test/development and
should not be used in production.

Also updated documentation to contain all development/test
variables for docker compose and clarifying that the options
starting with _ are ment to be only used for quick testing.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[kaxil]

hmm .. build docs is failing

[potiuk]

fixed

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

I wonder if we should add new sections to the Helm Chart documentation to better promote this feature. What do you think to create a new page, e.g. FAQ and add the answer to the question "How to install extra pip packages?" ? I mainly think of users migrating from alternative Helm Charts as airflow-helm/airflow or bitnami/airflow has such a section.

Good point @mik-laj.

But rather than creating a new section, I extended two sections in the helm chart docs:

• production deployment - I mentioned typical scenarios when you need custom image and referred to "Build Images" for details

• quick-start with kind - I copied a few typical examples (adding apt/PypI packages) next to "Adding DAGs" with step-by-step instructions on how to build the images. This would be a gentle introduction to image building by users who do not know how to do it or even do not know that they could build and use their own image easily (plus see-more reference to "Build Images").

I think that hits the sweet-spot between copy/pasting some parts of documentation where users might need it and having common source of examples.

[potiuk]

Last comments please :)

[tanujdhiman]

Hello,
How can I use this variable?

_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- pydub dropbox}

I'm using this but didn't get good results. I want to install pydub, dropbox through docker compose file. Is it right syntax or what's wrong with it ?
Thanks

[potiuk]

https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#environment-variables-supported-by-docker-compose

[tanujdhiman]

Thanks @potiuk for a quick reply. But I saw this page actually but umm, okay.

Can you write down a command which I put in compose file for installing pydub, dropbox, patool ??
Thanks

[potiuk]

Just set env variables. Here is the first googled result about them https://opensource.com/article/19/8/what-are-environment-variables#:~:text=Environment%20variables%20contain%20information%20about,during%20installation%20or%20user%20creation.

To be perfectly honest - don't even try docker-compose if you find it difficult to set environment variable when you run a command. this is really basic thing. You should try to undrstand what you do and not ask someone to copy you some solutions to most basic things.

[tanujdhiman]

Hmm Actually I tried this command too ::

_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-pydub dropbox patool}
But got an error of invalid argument in server..

[tanujdhiman]

Thanks @potiuk