Skip to content

Commit

Permalink
Convert emr_eks example dag to system test (#26723)
Browse files Browse the repository at this point in the history
  • Loading branch information
vincbeck committed Oct 10, 2022
1 parent 869c9ce commit 62d5bab
Show file tree
Hide file tree
Showing 4 changed files with 288 additions and 95 deletions.
91 changes: 0 additions & 91 deletions airflow/providers/amazon/aws/example_dags/example_emr_eks.py

This file was deleted.

9 changes: 5 additions & 4 deletions docs/apache-airflow-providers-amazon/operators/emr_eks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ the eks cluster that you would like to use , and an eks namespace.
Refer to the `EMR on EKS Development guide <https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/virtual-cluster.html>`__
for more details.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_emr_eks.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_emr_eks.py
:language: python
:start-after: [START howto_operator_emr_eks_create_cluster]
:end-before: [END howto_operator_emr_eks_create_cluster]
Expand Down Expand Up @@ -81,8 +81,9 @@ and ``monitoringConfiguration`` to send logs to the ``/aws/emr-eks-spark`` log g
Refer to the `EMR on EKS guide <https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks-jobs-CLI.html#emr-eks-jobs-parameters>`__
for more details on job configuration.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_emr_eks.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_emr_eks.py
:language: python
:dedent: 4
:start-after: [START howto_operator_emr_eks_config]
:end-before: [END howto_operator_emr_eks_config]

Expand All @@ -91,7 +92,7 @@ can store them in a connection or provide them in the DAG. Your AWS region shoul
in the ``aws_default`` connection as ``{"region_name": "us-east-1"}`` or a custom connection name
that gets passed to the operator with the ``aws_conn_id`` parameter. The operator returns the Job ID of the job run.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_emr_eks.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_emr_eks.py
:language: python
:dedent: 4
:start-after: [START howto_operator_emr_container]
Expand All @@ -109,7 +110,7 @@ Wait on an Amazon EMR virtual cluster job
To wait on the status of an Amazon EMR virtual cluster job to reach a terminal state, you can use
:class:`~airflow.providers.amazon.aws.sensors.emr.EmrContainerSensor`

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_emr_eks.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_emr_eks.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_emr_container]
Expand Down
1 change: 1 addition & 0 deletions docs/spelling_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -956,6 +956,7 @@ nosasl
NotFound
notificationChannels
npm
ns
ntlm
ntpd
Nullable
Expand Down

0 comments on commit 62d5bab

Please sign in to comment.