python-api-ref.rst

Python API Reference

DAGs

The DAG is Airflow's core model that represents a recurring workflow. Check out :class:`~airflow.models.dag.DAG` for details.

Operators

Operators allow for generation of certain types of tasks that become nodes in the DAG when instantiated. All operators derive from :class:`~airflow.models.BaseOperator` and inherit many attributes and methods that way.

There are 3 main types of operators:

• Operators that performs an action, or tell another system to perform an action
• Transfer operators move data from one system to another
• Sensors are a certain type of operator that will keep running until a certain criterion is met. Examples include a specific file landing in HDFS or S3, a partition appearing in Hive, or a specific time of the day. Sensors are derived from :class:`~airflow.sensors.base.BaseSensorOperator` and run a poke method at a specified :attr:`~airflow.sensors.base.BaseSensorOperator.poke_interval` until it returns True.

BaseOperator

All operators are derived from :class:`~airflow.models.BaseOperator` and acquire much functionality through inheritance. Since this is the core of the engine, it's worth taking the time to understand the parameters of :class:`~airflow.models.BaseOperator` to understand the primitive features that can be leveraged in your DAGs.

BaseSensorOperator

All sensors are derived from :class:`~airflow.sensors.base.BaseSensorOperator`. All sensors inherit the :attr:`~airflow.sensors.base.BaseSensorOperator.timeout` and :attr:`~airflow.sensors.base.BaseSensorOperator.poke_interval` on top of the :class:`~airflow.models.BaseOperator` attributes.

Operators packages

All operators are in the following packages:

.. toctree::
  :includehidden:
  :glob:
  :maxdepth: 1

  _api/airflow/operators/index

  _api/airflow/sensors/index

Hooks

Hooks are interfaces to external platforms and databases, implementing a common interface when possible and acting as building blocks for operators. All hooks are derived from :class:`~airflow.hooks.base.BaseHook`.

Hooks packages

All hooks are in the following packages:

.. toctree::
  :includehidden:
  :glob:
  :maxdepth: 1

  _api/airflow/hooks/index

Executors

Executors are the mechanism by which task instances get run. All executors are derived from :class:`~airflow.executors.base_executor.BaseExecutor`.

Executors packages

All executors are in the following packages:

.. toctree::
  :includehidden:
  :glob:
  :maxdepth: 1

  _api/airflow/executors/index

Models

Models are built on top of the SQLAlchemy ORM Base class, and instances are persisted in the database.

.. toctree::
  :includehidden:
  :glob:
  :maxdepth: 1

  _api/airflow/models/index

Exceptions

.. toctree::
  :includehidden:
  :glob:
  :maxdepth: 1

  _api/airflow/exceptions/index

Secrets Backends

Airflow relies on secrets backends to retrieve :class:`~airflow.models.connection.Connection` objects. All secrets backends derive from :class:`~airflow.secrets.BaseSecretsBackend`.

.. toctree::
  :includehidden:
  :glob:
  :maxdepth: 1

  _api/airflow/secrets/index

Timetables

Custom timetable implementations provide Airflow's scheduler additional logic to schedule DAG runs in ways not possible with built-in schedule expressions.

.. toctree::
  :includehidden:
  :maxdepth: 1

  _api/airflow/timetables/index