Skip to content

Latest commit

 

History

History
1066 lines (757 loc) · 40 KB

CHANGELOG.rst

File metadata and controls

1066 lines (757 loc) · 40 KB
Raw

Changelog

1.19.1 (2024-05-10)

Bug Fixes

  • Pass parameters to parent class in SnowflakeSensorAsync (#1519)

1.19.0 (2024-02-21)

  • Add ExternalDeploymentSensor (#1472)
  • Deprecate the following operators and sensors and proxy them to their upstream Apache Airflow providers' deferrable counterparts
    • EmrContainerSensorAsync
    • EmrStepSensorAsync
    • EmrJobFlowSensorAsync
    • RedshiftClusterSensorAsync
    • BatchSensorAsync
    • S3KeySensorAsync
    • S3KeysUnchangedSensorAsync
    • EmrContainerOperatorAsync
    • RedshiftDeleteClusterOperatorAsync
    • RedshiftResumeClusterOperatorAsync
    • RedshiftPauseClusterOperatorAsync
    • BatchOperatorAsync
    • SageMakerProcessingOperatorAsync
    • SageMakerTransformOperatorAsync
    • SageMakerTrainingOperatorAsync
    • RedshiftDataOperatorAsync
    • GCSObjectExistenceSensorAsync
    • GCSObjectsWithPrefixExistenceSensorAsync
    • GCSUploadSessionCompleteSensorAsync
    • GCSObjectUpdateSensorAsync
    • GKEStartPodOperatorAsync
    • BigQueryInsertJobOperatorAsync
    • BigQueryCheckOperatorAsync
    • BigQueryGetDataOperatorAsync
    • BigQueryIntervalCheckOperatorAsync
    • BigQueryValueCheckOperatorAsync
    • DataprocCreateClusterOperatorAsync
    • DataprocDeleteClusterOperatorAsync
    • DataprocSubmitJobOperatorAsync
    • DataprocUpdateClusterOperatorAsync
    • BigQueryTableExistenceSensorAsync
    • AzureDataFactoryPipelineRunStatusSensorAsync
    • WasbBlobSensorAsync
    • WasbPrefixSensorAsync
    • AzureDataFactoryRunPipelineOperatorAsync
    • FileSensorAsync
    • ExternalTaskSensorAsync
    • SnowflakeSqlApiOperatorAsync
    • DatabricksSubmitRunOperatorAsync
    • DatabricksRunNowOperatorAsync
    • KubernetesPodOperatorAsync
    • SFTPSensorAsync
    • HttpSensorAsync
    • DbtCloudJobRunSensorAsync
    • DbtCloudRunJobOperatorAsync
    • LivyOperatorAsync
  • Add deprecation notice docs for astronomer-providers (#1468)
  • Increase min airflow version to 2.6 (#1438)
  • Bump mininium python to 3.8 (#1442)

1.18.4 (2023-12-07)

Bug Fixes

  • DatabricksHookAsync: include 403 as retryable error (#1376)

1.18.3 (2023-11-29)

Misc

  • Revert "Pin max versions for CNCF Kubernetes and Google Airflow providers (#1359)" (#1368)

1.18.2 (2023-11-11)

Bug Fixes

  • Pin max versions for CNCF Kubernetes and Google Airflow providers (#1359)
  • Pin connexion==2.14.2 due to issue with latest release version3.0.0 (#1354)

1.18.1 (2023-11-01)

Bug Fixes

  • Handle ClientConnectorError to allow for retries (#1338)
  • Make resource_group_name and factory_name required in Azure DataFactory hook and trigger (#1341)

1.18.0 (2023-09-25)

Enhancements

  • Add keep_response parameter to HttpHookAsync (#1330)
  • Add AioRefreshableCredentials to AwsBaseHookAsync (#1304)

Bug Fixes

  • Fix ExternalDeploymentTaskSensorAsync by making simple http requests instead of aiohttp (#1306)
  • Fix impersonation failure on GKE start pod operator async (#1274)

Misc

  • Bring back hive provider for Python3.11 (#1317)

1.17.3 (2023-08-07)

Bug Fixes

  • Raise AirflowSkipException for sensors when soft_fail is set to True(#1276)
  • S3KeyTrigger: Call S3 sensor check_fn only with file size attribute (#1278)
  • HttpSensorAsync: Fix incorrect warning when poke_interval is passed (#1281)

Note

  • hive provider is not supported on Python 3.11 (#1237)

1.17.2 (2023-07-26)

Bug Fixes

  • RedshiftDataHook: remove snowflake dependencies and use DbApiHook.split_sql_string for parsing sql
  • DatabricksRunNowOperatorAsync: get job_id through Databricks API if job_name is passed

1.17.1 (2023-06-22)

Bug Fixes

  • Revert "feat(kubernetes): check state before deferring KubernetesPodOperatorAsync (#1104)" (#1209)

1.17.0 (2023-06-21)

Enhancements

  • Enhance S3KeySizeSensorAsync
    • Add use_regex param (#1172)
    • Handle soft fail (#1161)

Bug Fixes

  • Fixing the issue that check_fn was ignored in S3KeySensorAsync (#1171)

1.16.0 (2023-05-19)

Enhancements

  • Enhance SFTPSensorAsync by adding below features (#1072):
    • Remove the need for prefixing ssh- to host keys that don't have such prefix e.g. ecdsa type keys
    • Support validating host keys using a known_hosts file
    • Accept string values for newer_than field. e.g. passed via Jinja template
    • Use readdir for listing files instead of stat on each file to avoid throttling caused by multiple roundtrips to the server for each file

Bug Fixes

  • Mark DbtCloudRunJobOperatorAsync failed if cancelled by raising AirflowFailException (#1082)

1.15.5 (2023-04-24)

Bug Fixes

  • Support host key verification for SFTPSensorAsync (#963)
  • Make BigQuery & Google Cloud Storage async operators & sensors compatible with apache-airflow-providers-google>=10.0.0 (#981)
  • Make SageMakerProcessingOperatorAsync compatible with apache-airflow-providers-amazon>=8.0.0 (#979)
  • Make BatchOperatorAsync compatible with apache-airflow-providers-amazon>=8.0.0

1.15.4 (2023-04-19)

Bug Fixes

  • Fix backward compatibility issue with BigQuery Async Operators (#967)

1.15.3 (2023-04-17)

Bug Fixes

  • Allow and prefer non-prefixed extra fields for KubernetesHookAsync (#944)
  • Make GKEStartPodOperatorAsync compatible with apache-airflow-providers-google>=9.0.0. (#954)
  • BigQuery Async Operators accepts poll_interval as argument to override trigger's default poll_interval. (#953)
  • Fix S3 and GCS custom XCOM backend json loading issue (#961)
  • Pinning snowflake-sqlalchemy to greater than or equal to 1.4.4 (#962)

1.15.2 (2023-03-27)

Bug Fixes

  • Fix S3KeyTrigger to return only one trigger event when check_fn is set to none. (#925)
  • Handle unclosed connection errors in HttpTrigger . (#927)

1.15.1 (2023-03-09)

Bug Fixes

  • Allow and prefer non-prefixed extra fields for AzureDataFactoryHookAsync. (#899)
  • Fix HttpSensorAsync to use the correct connection id instead of a default connection id (#896)
  • Fix SFTPSensorAsync to succeed when there is at least one file newer than the provided date and set SFTPHookAsync default_known_hosts parameter to None. (#905)
  • Make SageMakerTransformOperatorAsync and SageMakerTrainingOperatorAsync compatible with apache-airflow-providers-amazon>=7.3.0. (#912)
  • Fix BigQueryInsertJobOperatorAsync to send location parameter to hook. (#866)
  • Mitigate race condition on deferrable sensors ExternalTaskSensorAsync and SnowflakeSensorAsync immediately getting successful, by making the changes to these sensors to check on the worker first, then defer if the condition is not met. (#908)

1.15.0 (2023-02-15)

New Operators

This release adds a new async sensor SnowflakeSensorAsync.

Sensor Class Import Path Example DAG
SnowflakeSensorAsync 
from astronomer.providers.snowflake.sensors.snowflake import SnowflakeSensorAsync
 Example DAG

Enhancements

  • Support tags param in EmrContainerOperatorAsync. (#836)
  • Enhance DbtCloudHookAsync to accept tenant domain name while making connection. (#855)

Bug Fixes

  • Ensure DataprocLink is visible for tracking the dataproc jobs status in DataprocSubmitJobOperatorAsync. (#835)
  • Handle SnowflakeHookAsync when a empty sql list is passed`. (#838)
  • Use timeout instead of execution_timeout with ExternalTaskSensor when sensing task_id`. (#858)

1.14.0 (2023-01-09)

Feature

  • Add custom XCom backend for S3. (#820)

Bug Fixes

  • Add poke_interval to ExternalTaskSensorAsync. (#823)
  • Support getting logs of pod, pod cleanup and reflect status of the pod in GKEStartPodOperatorAsync (#824)

1.13.0 (2022-12-16)

Feature

  • Enhance AwsBaseHookAsync to support assume role role_arn passed via extra_config in the airflow connection. (#804)
  • Support private key authentication for SFTPHookAsync via SFTP connection. (#749)

Bug Fixes

  • Fix KubernetesPodOperatorAsync to consider kubernetes connection id in the trigger. (#815)

1.12.0 (2022-12-02)

Feature

  • Add custom XCom backend for GCS. (#681)

Bug Fixes

  • Change return value of SnowflakeOperatorAsync to be same as that of SnowflakeOperator. (#781)
  • Add poke_interval to S3KeySensorAsync. (#782)
  • Use AwsConnectionWrapper to get connection details to create AWS async client. (#758)
  • Fix ADF Sensor broken docs. (#779)
  • Log warning message when response_check attribute is passed in HttpSensorAsync. (#780)

1.11.2 (2022-11-19)

Bug Fixes

  • Handle SFTPSensorAsync failure when file pattern is not passed. (#744)
  • Fix RedshiftDataHook to accept AWS access and secret keys from the connection object. (#746)
  • Make BigQueryHookAsync compatible with apache-airflow-providers-google>=8.5.0. (#751)
  • Make RedshiftSQLOperatorAsync compatible with apache-airflow-providers-amazon>=6.1.0. (#762)
  • Make SnowflakeOperatorAsync and SnowflakeSqlApiOperatorAsync compatible with apache-airflow-providers-snowflake>=4.0.0. (#767)

1.11.1 (2022-10-28)

Bug Fixes

  • Fix AwsBaseHookAsync to accept the AWS session token passed via extra_config in the airflow connection (#730)
  • Change return value of SageMakerTrainingOperatorAsync and SageMakerTransformOperatorAsync to be same as that of corresponding Sync operators (#737)

1.11.0 (2022-10-21)

New Operators

This release adds the following 3 new async operators:

Operator Class Import Path Example DAG
SageMakerProcessingOperatorAsync 
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerProcessingOperatorAsync
 Example DAG
SageMakerTrainingOperatorAsync 
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTrainingOperatorAsync
 Example DAG
SageMakerTransformOperatorAsync 
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTransformOperatorAsync
 Example DAG

1.10.0 (2022-09-30)

New Operators

This release adds the following 2 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
ExternalDeploymentTaskSensorAsync 
from astronomer.providers.core.sensors.external_task import ExternalDeploymentTaskSensorAsync
 Example DAG
SFTPSensorAsync 
from astronomer.providers.sftp.sensors.sftp import SFTPSensorAsync
 Example DAG

Bug Fixes

  • Make Dataproc operator compatible with apache-airflow-providers-google>=8.4.0 (#680)
  • Make EMR EKS operator compatible with apache-airflow-providers-amazon>=6.0.0 (#682)

Deprecation

  • Deprecate poll_interval and use poke_interval for all async sensors (#640)

1.9.0 (2022-09-13)

New Operators

This release adds the following 2 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
DbtCloudJobRunSensorAsync 
from astronomer.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensorAsync
 Example DAG
DbtCloudRunJobOperatorAsync 
from astronomer.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperatorAsync
 Example DAG

Bug Fixes

  • Include astronomer-providers in the Providers view within the Airflow UI (#626)

Enhancements

  • Implement OpenLineage custom extractor for Redshift Async Operators (#561)

1.8.1 (2022-09-01)

Bug Fixes

  • Fix timeout errors on AzureDataFactoryRunPipelineOperatorAsync (#602)
  • Remove werkzeug dep & limit protobuf to 3.20.0 (#615)
  • Raise exception in case of user error in async Databricks Operator (#612)

1.8.0 (2022-08-16)

Bug Fixes

  • Add poll interval to HttpSensorAsync (#554)
  • Replace execution_timeout with timeout in all the async sensors (#555)
  • Get default 'resource_group_name' and 'factory_name' for AzureDataFactoryPipelineRunStatusSensorAsync (#589)

Enhancements

  • Add elaborate documentation and use cases for SnowflakeOperatorAsync (#556)
  • Improve telemetry for Async Databricks Operators (#582)
  • Enhance S3KeySensorAsync to accept multiple keys and deprecate S3PrefixSensorAsync and S3KeySizeSensorAsync (#577)

1.7.1 (2022-07-25)

Bug Fixes

  • Bump up MarkupSafe version as per Airflow 2.3.3 constraints (#542)
  • Downgrade Werkzeug version below 2.2.0 as it causes ImportError: cannot import name 'parse_rule' from 'werkzeug.routing' (#551)

1.7.0 (2022-07-19)

New Operators

This release adds the following 4 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
BatchSensorAsync 
from astronomer.providers.amazon.aws.sensors.batch import BatchSensorAsync
 Example DAG
SnowflakeSqlApiOperatorAsync 
from astronomer.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperatorAsync
 Example DAG
WasbBlobSensorAsync 
from astronomer.providers.microsoft.azure.sensors.wasb import WasbBlobSensorAsync
 Example DAG
WasbPrefixSensorAsync 
from astronomer.providers.microsoft.azure.sensors.wasb import WasbPrefixSensorAsync
 Example DAG

Enhancements

  • Add copy button to code blocks in docs (#505)
  • Add custom Sphinx extension to list available operators & sensors (#504)
  • Add pre-commit hook to check for dead links in markdown files (#524)

1.6.0 (2022-06-28)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
DataprocCreateClusterOperatorAsync 
from astronomer.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperatorAsync
 Example DAG
DataprocDeleteClusterOperatorAsync 
from astronomer.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperatorAsync
 Example DAG
DataprocUpdateClusterOperatorAsync 
from astronomer.providers.google.cloud.operators.dataproc import DataprocUpdateClusterOperatorAsync
 Example DAG
RedshiftDataOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_data import RedshiftDataOperatorAsync
 Example DAG
RedshiftDeleteClusterOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftDeleteClusterOperatorAsync
 Example DAG

Enhancements

  • Implement OpenLineage custom extractor for BigQuery Async Operators (#429)
  • Add session specific query tag and OpenLineage Extractor for Snowflake Async operator (#437)
  • Handle DataprocCreateClusterOperatorAsync errors gracefully and add additional functionality with use_if_exists and delete_on_error parameters (#448)

Bug Fixes

  • Fix BigQueryInsertJobOperatorAsync failure after Google provider upgrade to 8.1.0 (#471)

1.5.0 (2022-06-15)

This release adds the following 2 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
BatchOperatorAsync 
from astronomer.providers.amazon.aws.operators.batch import BatchOperatorAsync
 Example DAG
GKEStartPodOperatorAsync 
from astronomer.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperatorAsync
 Example DAG

Improvements

  • Enhance KubernetesPodOperatorAsync to periodically resume the sync portion of the task to fetch and emit the latest logs before deferring again. (#139)
  • Fix a bug on the KubernetesPodOperatorAsync to not fail with ERROR - Unclosed client session (#394)

1.4.0 (2022-05-25)

Enhancements

  • Enable Kerberos Authentication in HivePartitionSensorAsync and NamedHivePartitionSensorAsync (#357)

Bug Fixes

  • Fix example Redshift DAGs to catch appropriate exception during cluster deletion (#348)
  • Move xcom_push call to execute method for all async operators (#371)

1.3.1 (2022-05-22)

Bug Fixes

  • Correct module name for DagStateTrigger which prevented use of ExternalTaskSensorAsync when external_task_id was not passed (#361)
  • Add template_fields to S3KeySensorAsync (#373)

Docs

  • Add missing Extras in README.rst and automate it (#329)

Misc

  • Improvements in Example DAGs (Hive, Livy) (#342, #348, #349)

1.3.0 (2022-05-09)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
AzureDataFactoryRunPipelineOperatorAsync 
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync
 Example DAG
AzureDataFactoryPipelineRunStatusSensorAsync 
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync
 Example DAG
EmrContainerOperatorAsync 
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync
 Example DAG
HivePartitionSensorAsync 
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync
 Example DAG
NamedHivePartitionSensorAsync 
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync
 Example DAG

Improvements

  • Improved example DAGs so that minimal resources are created during integration tests
  • Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)

1.2.0 (2022-04-12)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
DataprocSubmitJobOperatorAsync 
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync
 Example DAG
EmrContainerSensorAsync 
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync
 Example DAG
EmrStepSensorAsync 
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync
 Example DAG
EmrJobFlowSensorAsync 
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync
 Example DAG
LivyOperatorAsync 
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync
 Example DAG

Improvements

  • Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
  • Enhanced the Async Databricks Operator to persist run_id and run_page_url in XCom (#175)

1.1.0 (2022-03-23)

New Operators

This release adds the following 7 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
S3KeySizeSensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync
 Example DAG
S3KeysUnchangedSensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync
 Example DAG
S3PrefixSensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync
 Example DAG
GCSObjectsWithPrefixExistenceSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync
 Example DAG
GCSObjectUpdateSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync
 Example DAG
GCSUploadSessionCompleteSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync
 Example DAG
BigQueryTableExistenceSensorAsync 
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync
 Example DAG

Improvements

The dependencies for installing this repo are now split into multiple extras as follows (#113)

Extra Name Installation Command Dependencies
all pip install 'astronomer-providers[all]' All providers
amazon pip install 'astronomer-providers[amazon]' Amazon
cncf.kubernetes pip install 'astronomer-providers[cncf.kubernetes]' Kubernetes
databricks pip install 'astronomer-providers[databricks]' Databricks
google pip install 'astronomer-providers[google]' Google Cloud
http pip install 'astronomer-providers[http]' HTTP
snowflake pip install 'astronomer-providers[snowflake]' Snowflake

This will allow users to just install dependencies of a single provider. For example, if a user wants to just use KubernetesPodOperatorAsync, they should not need to install GCP, AWS or Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'.

Bug Fixes

  • Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)

1.0.0 (2022-03-01)

  • Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class Import Path Example DAG
RedshiftSQLOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync
 Example DAG
RedshiftPauseClusterOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync
 Example DAG
RedshiftResumeClusterOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync
 Example DAG
RedshiftClusterSensorAsync 
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync
 Example DAG
S3KeySensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync
 Example DAG
KubernetesPodOperatorAsync 
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync
 Example DAG
ExternalTaskSensorAsync 
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync
 Example DAG
FileSensorAsync 
from astronomer.providers.core.sensors.filesystem import FileSensorAsync
 Example DAG
DatabricksRunNowOperatorAsync 
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync
 Example DAG
DatabricksSubmitRunOperatorAsync 
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync
 Example DAG
BigQueryCheckOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync
 Example DAG
BigQueryGetDataOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync
 Example DAG
BigQueryInsertJobOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import  BigQueryInsertJobOperatorAsync
 Example DAG
BigQueryIntervalCheckOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync
 Example DAG
BigQueryValueCheckOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync
 Example DAG
GCSObjectExistenceSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync
 Example DAG
HttpSensorAsync 
from astronomer.providers.http.sensors.http import HttpSensorAsync
 Example DAG
SnowflakeOperatorAsync 
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync
 Example DAG