GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. continution of #32404 #38854

don1uppa · 2024-04-08T19:06:05Z

don1uppa
Apr 8, 2024

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

Version: v2.8.1 current mwaa release

What happened?

[2024-04-08, 18:58:44 UTC] {{glue.py:189}} ERROR - Failed to run aws glue job, error: Parameter validation failed:
Invalid type for parameter GlueVersion, value: 4.0, type: <class 'float'>, valid types: <class 'str'>

Here is the full stack trace:

ip-10-0-20-19.ec2.internal
*** Reading remote log from Cloudwatch log_group: airflow-test-mwaa-Task log_stream: dag_id=glue_operator_failure/run_id=manual__2024-04-08T18_56_04.206370+00_00/task_id=submit_glue_job/attempt=2.log.
[2024-04-08, 18:58:43 UTC] {{taskinstance.py:1956}} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: glue_operator_failure.submit_glue_job manual__2024-04-08T18:56:04.206370+00:00 [queued]>
[2024-04-08, 18:58:43 UTC] {{taskinstance.py:1956}} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: glue_operator_failure.submit_glue_job manual__2024-04-08T18:56:04.206370+00:00 [queued]>
[2024-04-08, 18:58:43 UTC] {{taskinstance.py:2170}} INFO - Starting attempt 2 of 2
[2024-04-08, 18:58:43 UTC] {{taskinstance.py:2191}} INFO - Executing <Task(GlueJobOperator): submit_glue_job> on 2024-04-08 18:56:04.206370+00:00
[2024-04-08, 18:58:43 UTC] {{standard_task_runner.py:60}} INFO - Started process 18070 to run task
[2024-04-08, 18:58:43 UTC] {{standard_task_runner.py:87}} INFO - Running: ['airflow', 'tasks', 'run', 'glue_operator_failure', 'submit_glue_job', 'manual__2024-04-08T18:56:04.206370+00:00', '--job-id', '5118', '--raw', '--subdir', 'DAGS_FOLDER/test_dags/glue_operator_failure.py', '--cfg-path', '/tmp/tmpbdmm1cz0']
[2024-04-08, 18:58:43 UTC] {{standard_task_runner.py:88}} INFO - Job 5118: Subtask submit_glue_job
[2024-04-08, 18:58:43 UTC] {{task_command.py:423}} INFO - Running <TaskInstance: glue_operator_failure.submit_glue_job manual__2024-04-08T18:56:04.206370+00:00 [running]> on host ip-10-0-20-19.ec2.internal
[2024-04-08, 18:58:43 UTC] {{taskinstance.py:2480}} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='glue_operator_failure' AIRFLOW_CTX_TASK_ID='submit_glue_job' AIRFLOW_CTX_EXECUTION_DATE='2024-04-08T18:56:04.206370+00:00' AIRFLOW_CTX_TRY_NUMBER='2' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-04-08T18:56:04.206370+00:00'
[2024-04-08, 18:58:43 UTC] {{glue.py:172}} INFO - Initializing AWS Glue Job: simple ls. Wait for completion: True
[2024-04-08, 18:58:43 UTC] {{base.py:83}} INFO - Using connection ID 'aws_default' for task execution.
[2024-04-08, 18:58:43 UTC] {{glue.py:161}} INFO - Iam Role Name: test-glue-jobs
[2024-04-08, 18:58:43 UTC] {{glue.py:356}} INFO - Checking if job already exists: simple ls
[2024-04-08, 18:58:44 UTC] {{glue.py:421}} INFO - Creating job: simple ls
[2024-04-08, 18:58:44 UTC] {{glue.py:189}} ERROR - Failed to run aws glue job, error: Parameter validation failed:
Invalid type for parameter GlueVersion, value: 4.0, type: <class 'float'>, valid types: <class 'str'>
[2024-04-08, 18:58:44 UTC] {{taskinstance.py:2698}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 433, in _execute_task
    result = execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 177, in execute
    glue_job_run = self.glue_job_hook.initialize_job(self.script_args, self.run_job_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 183, in initialize_job
    job_name = self.create_or_update_glue_job()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 422, in create_or_update_glue_job
    self.conn.create_job(**config)
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/client.py", line 962, in _make_api_call
    request_dict = self._convert_to_request_dict(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/client.py", line 1036, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter GlueVersion, value: 4.0, type: <class 'float'>, valid types: <class 'str'>
[2024-04-08, 18:58:44 UTC] {{taskinstance.py:1138}} INFO - Marking task as FAILED. dag_id=glue_operator_failure, task_id=submit_glue_job, execution_date=20240408T185604, start_date=20240408T185843, end_date=20240408T185844
[2024-04-08, 18:58:44 UTC] {{standard_task_runner.py:107}} ERROR - Failed to execute job 5118 for task submit_glue_job (Parameter validation failed:
Invalid type for parameter GlueVersion, value: 4.0, type: <class 'float'>, valid types: <class 'str'>; 18070)
[2024-04-08, 18:58:44 UTC] {{local_task_job_runner.py:234}} INFO - Task exited with return code 1
[2024-04-08, 18:58:44 UTC] {{taskinstance.py:3280}} INFO - 0 downstream tasks scheduled from follow-on schedule check

What you think should happen instead?

The glue job should be created and run.

How to reproduce

Run this dag that has a glue operator.

from airflow import DAG
import pendulum
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator

script_bucket_name = "test-mwaa-20240303120246319700000001"
script_path = "hello_world.py"
script_location = f"s3://{script_bucket_name}/{script_path}"

with (DAG(
        dag_id=f'glue_operator_failure',
        schedule_interval=None,
        start_date=pendulum.datetime(2024, 1, 1),
        catchup=False,
        is_paused_upon_creation=True,
        max_active_tasks=1,
        max_active_runs=1,
        doc_md=__doc__,
        render_template_as_native_obj=True,
        tags=['glue_operator_failure']
) as my_dag):

    submit_glue_job = GlueJobOperator(
        task_id="submit_glue_job",
        job_name="simple ls",
        script_location=script_location,
        s3_bucket=script_bucket_name,
        concurrent_run_limit=1,
        update_config=True,
        iam_role_name="test-glue-jobs",
        create_job_kwargs={
            'WorkerType': 'G.2X',
            'NumberOfWorkers': '1',
            'DefaultArguments': {
                '--datalake-formats': 'iceberg',
                '--additional-python-modules': 'importlib',
            },
            'GlueVersion': '4.0'
        },
        wait_for_completion=True,
        deferrable=True,
        verbose=True
    )

upload this file to the script locationas : hello_world.py

print("Hello World!")

Operating System

mwaa

Versions of Apache Airflow Providers

[2024-04-08, 19:00:55 UTC] {{subprocess.py:75}} INFO - Running [2024-04-08, 19:00:55 UTC] {{subprocess.py:86}} INFO - Output:
[2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Package [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - ----------- [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - adal [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - adlfs [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - aenum [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - aiobotocore [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - aiofiles [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - aiohttp [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - aioitertools [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - aiosignal [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - alembic [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - amqp [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - annotated-types [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - anyio [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airflow [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apache-airf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - apispec [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - argcomplete [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - asgiref [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - asn1crypto [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - astronomer-cosmos [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - attrs [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-batch [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-common [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-core [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-cosmos [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-datalake-store [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-identity [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-keyvault-secrets [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-kusto-data [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt- [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt- [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt-core [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt-cosmosdb [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt-datafactory [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt- [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt- [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt-nspkg [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt-resource [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-mgmt-storage [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-nspkg [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-servicebus [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-storage-blob [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-stora [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-storage-file-share [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-synapse-artifacts [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - azure-synapse-spark [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Babel [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - backoff [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - beautifulsoup4 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - billiard [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - blinker [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - boto3 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - botocore [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - build [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - CacheControl [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cached-property [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cachelib [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cachetools [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cattrs [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - celery [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - certifi [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cffi [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - chardet [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - charset-normalizer [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cleo [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - click [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - click-didyoumean [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - click-plugins [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - click-repl [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - clickclick [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - colorama [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - colorlog [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - ConfigUpdater [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - connexion [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - crashtest [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cron-descriptor [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - croniter [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - cryptography [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - db-dtypes [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - decorator [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - delta-spark [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Deprecated [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - dill [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - distlib [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - dnspython [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - docutils [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - dulwich [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - email-validator [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Faker [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - fastjsonschema [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - filelock [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-AppBuilder [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-Babel [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-Caching [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-JWT-Extended [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-Limiter [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-Login [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-Session [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-SQLAlchemy [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Flask-WTF [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - flit_core [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - flower [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - frozenlist [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - fsspec [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - gcloud-aio-auth [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - gcloud-aio-bigquery [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - gcloud-aio-storage [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - gcsfs [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-ads [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-api-core [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-api-python-client [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-auth [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-auth-httplib2 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-auth-oauthlib [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-aiplatform [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-audit-log [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-automl [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-batch [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-bigquery [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-bigtable [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-build [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-compute [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-container [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-core [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-datacatalog [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-dataform [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-dataplex [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-dataproc [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-dlp [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-kms [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-language [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-logging [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-memcache [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-monitoring [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-os-login [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-pubsub [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-redis [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-run [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-spanner [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-speech [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-storage [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-tasks [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-translate [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-clou [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-vision [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-cloud-workflows [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-crc32c [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-re2 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - google-resumable-media [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - googleapis-common-protos [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - greenlet [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - grpc-google-iam-v1 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - grpcio [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - grpcio-gcp [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - grpcio-status [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - gunicorn [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - h11 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - hl7parser [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - httpcore [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - httplib2 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - httpx [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - humanize [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - idna [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - ijson [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - importlib-metadata [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - importlib-resources [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - inflection [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - iniconfig [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - installer [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - isodate [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - itsdangerous [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - jaraco.classes [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - jeepney [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Jinja2 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - jmespath [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - json-merge-patch [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - jsonpath-ng [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - jsonschema [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - jsonschema- [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - keyring [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - kombu [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - lazy-object-proxy [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - limits [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - linkify-it-py [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - lockfile [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - loguru [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - looker-sdk [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - lxml [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Mako [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Markdown [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - markdown-it-py [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - MarkupSafe [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - marshmallow [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - marshmallow-oneofschema [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - marshmallow-sqlalchemy [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - mdit-py-plugins [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - mdurl [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - more-itertools [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - msal [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - msal-extensions [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - msgpack [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - msrest [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - msrestazure [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - multidict [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - numpy [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - oauthlib [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemetry-api [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemet [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemet [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemet [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemet [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemetry-proto [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemetry-sdk [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - opentelemet [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - ordered-set [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - packaging [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pandas [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pandas-gbq [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pathspec [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pendulum [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pexpect [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pip [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pkginfo [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - platformdirs [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pluggy [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - ply [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - poetry [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - poetry-core [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - poetry-plugin-export [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - portalocker [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - prison [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - prometheus-client [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - prompt-toolkit [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - proto-plus [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - protobuf [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - psutil [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - psycopg2 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - psycopg2-binary [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - ptyprocess [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - py4j [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pyarrow [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pyasn1 [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pyasn1-modules [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pycparser [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pycurl [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pydantic [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pydantic_core [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pydata-google-auth [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - Pygments [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - PyJWT [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pyOpenSSL [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pyparsing [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pyproject_hooks [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pyspark [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - pytest [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - python-daemon [2024-04-08, 19:00:59 UTC] {{subprocess.py:93}} INFO - python-dateutil [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - python-dotenv [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - python-nvd3 [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - python-slugify [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - pytz [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - PyYAML [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - rapidfuzz [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - redshift-connector [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - referencing [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - requests [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - requests-oauthlib [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - requests-toolbelt [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - rfc3339-validator [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - rfc3986 [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - rich [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - rich-argparse [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - rpds-py [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - rsa [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - s3transfer [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - scramp [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - SecretStorage [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - setproctitle [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - setuptools [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - setuptools-scm [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - shapely [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - shellingham [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - six [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - slack_sdk [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - sniffio [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - soupsieve [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - SQLAlchemy [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - sqlalchemy-bigquery [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - SQLAlchemy-JSONField [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - sqlalchemy-redshift [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - sqlalchemy-spanner [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - SQLAlchemy-Utils [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - sqlparse [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - statsd [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - tabulate [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - template [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - tenacity [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - termcolor [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - test-utils [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - text-unidecode [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - time-machine [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - toml [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - tomlkit [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - tornado [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - trove-classifiers [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - typing_extensions [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - tzdata [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - uc-micro-py [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - unicodecsv [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - universal-pathlib [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - uritemplate [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - urllib3 [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - vine [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - virtualenv [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - watchtower [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - wcwidth [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - Werkzeug [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - wheel [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - wrapt [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - WTForms [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - yarl [2024-04-08, 19:01:00 UTC] {{subprocess.py:93}} INFO - zipp [2024-04-08, 19:01:01 UTC] {{subprocess.py:97}} INFO - Command [2024-04-08, 19:01:01 UTC] {{taskinstance.py:1138}} INFO [2024-04-08, 19:01:02 UTC] {{local_task_job_runner.py:234}} [2024-04-08, 19:01:02 UTC] {{taskinstance.py:3280}} INFO command: ['/usr/bin/bash', '-c', 'pip list']
Version
----------------------------- ---------------
1.2.7
2023.12.0
3.1.15
2.9.0
23.2.1
3.9.1
0.11.0
1.3.1
1.13.1
5.2.0
0.6.0
4.2.0
2.8.1
low-providers-amazon 8.16.0
low-providers-celery 3.5.1
low-providers-common-io 1.2.0
low-providers-common-sql 1.10.0
low-providers-ftp 3.7.0
low-providers-google 10.13.1
low-providers-http 4.8.0
low-providers-imap 3.5.0
low-providers-microsoft-azure 8.5.1
low-providers-postgres 5.10.0
low-providers-sqlite 3.7.0
6.4.0
3.2.1
3.7.2
1.5.1
1.3.2
23.2.0
14.1.0
1.1.28
1.29.6
4.5.1
0.0.53
1.15.0
4.7.0
4.3.1
containerinstance 10.1.0
containerregistry 10.3.0
1.4.0
9.4.0
4.0.0
datalake-nspkg 3.0.1
datalake-store 0.5.0
3.0.2
23.0.1
21.1.0
3.0.2
7.11.4
12.19.0
ge-file-datalake 12.14.0
12.15.0
0.18.0
0.7.0
2.14.0
2.2.1
4.12.2
4.2.0
1.7.0
1.33.13
1.33.13
1.2.1
0.14.0
1.5.2
0.9.0
5.3.2
23.2.3
5.3.1
2023.11.17
1.16.0
5.2.0
3.3.2
2.1.0
8.1.7
0.3.0
1.1.1
0.3.0
20.10.2
0.4.6
4.8.0
3.2
2.14.2
0.4.1
1.4.0
2.0.1
41.0.7
1.2.0
5.1.1
3.0.0
1.2.14
0.3.1.1
0.3.8
2.4.2
0.20.1
0.21.7
1.3.1
24.4.0
2.19.1
3.13.1
2.2.5
4.3.10
2.0.0
2.1.0
4.6.0
3.5.0
0.6.3
0.5.0
2.5.1
1.2.1
3.9.0
2.0.1
1.4.1
2023.12.2
4.2.3
7.0.0
9.0.0
2023.12.2.post1
22.1.0
2.15.0
2.113.0
2.26.2
0.2.0
1.2.0
1.39.0
d-appengine-logging 1.4.0
0.2.5
2.12.0
0.17.7
3.16.0
d-bigquery-datatransfer 3.13.0
d-bigquery-storage 2.24.0
2.22.0
3.22.0
1.15.0
2.37.0
2.4.1
3.17.2
d-dataflow-client 0.8.6
0.5.5
1.11.0
5.8.0
d-dataproc-metastore 1.14.0
3.14.0
2.20.0
2.12.0
3.9.0
1.8.0
2.18.0
d-orchestration-airflow 1.10.0
2.13.0
2.19.0
2.14.0
d-resource-manager 1.11.0
0.10.1
d-secret-manager 2.17.0
3.41.0
2.23.0
2.14.0
d-storage-transfer 1.10.0
2.15.0
d-texttospeech 2.15.1
3.14.0
d-videointelligence 2.12.0
3.5.0
1.13.0
1.5.0
1.1
2.7.0
1.62.0
3.0.3
0.13.0
1.60.0
0.2.2
1.60.0
21.2.0
0.14.0
0.7.4
0.16.3
0.22.0
0.23.3
4.9.0
3.6
3.2.3
6.11.0
6.1.1
0.5.1
2.0.0
0.7.0
0.6.1
2.1.2
3.3.0
0.8.0
3.1.3
0.10.0
0.2
1.6.1
4.20.0
specifications 2023.12.1
24.3.1
5.3.1
1.10.0
3.7.0
2.0.2
0.12.2
0.7.2
23.20.1
5.1.0
1.3.0
3.5.2
3.0.0
2.1.3
3.20.2
3.0.1
0.26.1
0.4.0
0.1.2
10.2.0
1.26.0
1.1.0
1.0.8
0.7.1
0.6.4
6.0.4
1.24.4
3.2.2
1.22.0
ry-exporter-otlp 1.22.0
ry-exporter-otlp-proto-common 1.22.0
ry-exporter-otlp-proto-grpc 1.22.0
ry-exporter-otlp-proto-http 1.22.0
1.22.0
1.22.0
ry-semantic-conventions 0.43b0
4.1.0
23.2
2.1.4
0.20.0
0.12.1
3.0.0
4.9.0
24.0
1.9.6
3.11.0
1.3.0
3.11
1.8.2
1.9.0
1.7.1
2.8.2
0.2.1
0.19.0
3.0.43
1.23.0
4.25.2
5.9.7
2.9.9
2.9.9
0.7.0
0.10.9.7
14.0.2
0.5.1
0.3.0
2.21
7.45.3
2.5.3
2.14.6
1.8.2
2.17.2
2.8.0
23.3.0
3.1.1
1.0.0
3.5.0
7.4.4
3.0.1
2.8.2
1.0.0
0.15.0
8.0.1
2023.3.post1
6.0.1
3.7.0
2.0.918
0.32.1
2.31.0
1.3.1
1.0.0
0.1.4
1.5.0
13.7.0
1.4.0
0.17.1
4.9
0.8.2
1.4.4
3.3.3
1.3.3
66.1.1
8.0.4
2.0.2
1.5.4
1.16.0
3.26.2
1.3.0
2.5
1.4.51
1.9.0
1.0.2
0.8.14
1.6.2
0.41.1
0.4.4
4.0.1
0.9.0
0.7.6
8.2.3
2.4.0
0.1.0
1.3
2.13.0
0.10.2
0.12.3
6.4
2024.1.8
4.9.0
2023.4
1.0.2
0.14.1
0.1.4
4.1.1
2.0.7
5.1.0
20.25.0
3.0.1
0.2.13
2.2.3
0.43.0
1.16.0
3.1.2
1.9.4
3.17.0
exited with return code 0
- Marking task as SUCCESS. dag_id=show_deployed_environment, task_id=show_packages, execution_date=20240408T190019, start_date=20240408T190054, end_date=20240408T190101
INFO - Task exited with return code 0
- 1 downstream tasks scheduled from follow-on schedule check

Deployment

Amazon (AWS) MWAA

Deployment details

deployed using mwaa - files uploaded to s3

Anything else?

Every time - hard error

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Taragolis · 2024-04-08T20:40:59Z

Taragolis
Apr 8, 2024
Collaborator

render_template_as_native_obj=True, will force to use jinja2.nativetypes.NativeEnvironment which are enforced to convert some to python types, see example:

from jinja2.sandbox import SandboxedEnvironment
from jinja2.nativetypes import NativeEnvironment

environments = {"sandboxed": SandboxedEnvironment(), "native": NativeEnvironment()}

template = "4.0"
for name, env in environments.items():
    print(f" Jinja2 Environment: {name} ".center(72, "="))
    rendered = env.from_string("4.0").render()
    print(f"Template: {template!r}")
    print(f"Rendered: {rendered!r}, Type: {type(rendered).__name__!r}")

==================== Jinja2 Environment: sandboxed =====================
Template: '4.0'
Rendered: '4.0', Type: 'str'
====================== Jinja2 Environment: native ======================
Template: '4.0'
Rendered: 4.0, Type: 'float'

There is two options here.

Option 1:
Do not use jinja2.nativetypes.NativeEnvironment, e.g. render_template_as_native_obj=True

Option 2 (Airflow 2.8+):
Use airflow.template.templater.LiteralValue (see: #35017) wrapper, which prevent template values, e.g.

+from airflow.template.templater import LiteralValue

...

        create_job_kwargs={
            'WorkerType': 'G.2X',
            'NumberOfWorkers': '1',
            'DefaultArguments': {
                '--datalake-formats': 'iceberg',
                '--additional-python-modules': 'importlib',
            },
-           'GlueVersion': '4.0'
+           'GlueVersion': LiteralValue('4.0')
        },

0 replies

don1uppa · 2024-04-08T23:45:35Z

don1uppa
Apr 8, 2024
Author

Thank you the literal option worked.

0 replies

Taragolis · 2024-04-09T07:27:11Z

Taragolis
Apr 9, 2024
Collaborator

I moved it into the discussion, because it it generic troubleshooting around templated fields in operators, which can't be easily resolve. But since we have LiteralValue, we could easily resolve some of popular troubles

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. continution of #32404 #38854

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments

{{title}}

{{title}}

{{title}}

Select a reply

GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. continution of #32404 #38854

don1uppa Apr 8, 2024

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Replies: 3 comments

Taragolis Apr 8, 2024 Collaborator

don1uppa Apr 8, 2024 Author

Taragolis Apr 9, 2024 Collaborator

don1uppa
Apr 8, 2024

Taragolis
Apr 8, 2024
Collaborator

don1uppa
Apr 8, 2024
Author

Taragolis
Apr 9, 2024
Collaborator