Status of testing Providers that were prepared on March 07, 2022

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 3.1.0rc1

• Added AWS RDS sensors (#21231): @kazanzhy
• Added AWS RDS operators (#20907): @kazanzhy
• Add RedshiftDataHook (#19137): @john-jac
• Feature: Add invoke lambda function operator (#21686): @schirag1993
• Add JSON output on SqlToS3Operator (#21779): @mariotaddeucci
• Implement a Sagemaker DeleteModelOperator and Delete model hook. (#21673): @hsrocks
• Added Hook for Amazon RDS (#20642): @kazanzhy
• Added SNS example DAG and rst (#21475): @ferruzzi
• retry on very specific eni provision failures (#22002): @zachliu
• Configurable AWS Session Factory (#21778): @ac1997
• S3KeySensor to use S3Hook url parser (#21500): @dstandish
• ECSOperator: Get log events after sleep to get all logs (#21574): @kanga333
• Use temporary file in GCSToS3Operator (#21295): @rafalh
• Fix the Type Hints in RedshiftSQLOperator (#21885): @kaxil
• Bug Fix - S3DeleteObjectsOperator will try and delete all keys (#21458): @njrs92
• Fix Amazon SES emailer signature (#21681): @potiuk
• Fix EcsOperatorError, so it can be loaded from a picklefile (#21441): @ngwallace
• [doc] Improve s3 operator example by adding task upload_keys (#21422): @zhongjiajie
• Add docs and sample dags for AWS Batch Operator/Sensor (#22010): @o-nikolas
• Cleanup RedshiftSQLOperator documentation (#21976): @vincbeck
• Move S3ToRedshiftOperator documentation to transfer dir (#21975): @vincbeck
• Rename 'S3' hook name to 'Amazon S3' (#21988): @treyyi
• Update ECS sample DAG and Docs to new standards (#21828): @ferruzzi
• Update EKS sample DAGs and docs (#21523): @ferruzzi

Provider apache.beam: 3.2.0rc1

• Add support for BeamGoPipelineOperator (#20386): @pierrejeambrun

Provider apache.hdfs: 2.2.1rc1

• Add how-to guide for WebHDFS operators (#21393): @Melodie97

Provider apache.hive: 2.3.0rc1

• Set larger limit get_partitions_by_filter in HiveMetastoreHook (#21504): @pingzh
• Add Python 3.9 support to Hive (#21893): @potiuk
• Fix key typo in template_fields_renderers for HiveOperator (#21525): @josh-fell
• add how-to guide for hive operator (#21590): @Melodie97

Provider apache.livy: 2.2.0rc1

• Added retries to LivyHook #19384 (#21550): @utkarsharma2
• add how-to guide for livy operator (#21529): @Melodie97

Provider apache.spark: 2.1.1rc1

• fix param rendering in docs of SparkSubmitHook (#21788): @eladkal

Provider cncf.kubernetes: 3.1.0rc1

• Add map_index label to mapped KubernetesPodOperator (#21916): @ashb
• Change KubePodOperator labels from exeuction_date to run_id (#21960): @ashb
• Fix Kubernetes example with wrong operator casing (#21898): @potiuk
• Remove types from KPO docstring (#21826): @jedcunningham

Provider databricks: 2.3.0rc1

• Add-showing-runtime-error-feature-to-DatabricksSubmitRunOperator (#21709): @akifcakir
• Databricks: add support for triggering jobs by name (#21663): @eskarimov
• Added template_ext = ('.json') to databricks operators #18925 (#21530): @utkarsharma2
• Databricks SQL operators (#21363): @alexott
• Updated Databricks docs for correct jobs 2.1 API and links (#21494): @tfayyaz

Provider docker: 2.5.0rc1

• added docker network_mode options in documentation (#21986): @Bowrna

Provider elasticsearch: 3.0.0rc1

• Change default log filename template to include map_index (#21495): @ashb

Provider ftp: 2.1.0rc1

• [Feature] Updates FTPHook provider to have test_connection (#21997): @frankcash

Provider google: 6.5.0rc1

• Add Looker PDT operators (#20882): @alekseiloginov
• Add autodetect arg to external table creation in GCS-to-BQ operator (#21944): @shahkshitij15
• Dataproc assets (#21756): @wojsamjan
• Create Auto ML operators for Vertex AI service (#21470): @MaksYermak
• [Part 2] Calendar to GCS Operator (#20769): @rsg17
• Make project_id argument optional in all dataproc operators (#21866): @rafalh
• Allow templates in more DataprocUpdateClusterOperator fields (#21865): @rafalh
• Dataflow Assets (#21639): @lwyszomi
• Extract ClientInfo to module level (#21554): @pierrejeambrun
• Datafusion assets (#21518): @wojsamjan
• Dataproc metastore assets (#21267): @wojsamjan
• Normalize *_conn_id parameters in BigQuery sensors (#21430): @rafalh
• Fix bigquery_dts parameter docstring typo (#21786): @totogo
• Fixed PostgresToGCSOperator fail on empty resultset for use_server_side_cursor=True (#21307): @kazanzhy
• Fix multi query scenario in bigquery example DAG (#21575): @phanikumv
• Unpin google-cloud-memcache (#21912): @kaxil
• Unpin pandas-gbq and remove unused code (#21915): @kaxil
• Remove cursor methods calls from GCSToBigQueryOperator (#20119): @kazanzhy

Provider hashicorp: 2.1.2rc1

• Fix Vault default conn name (#20792): @ac1997

Provider http: 2.1.0rc1

• Add 'method' to attributes in HttpSensor (#21831): @h2cone

Provider microsoft.azure: 3.7.0rc1

• Add test_connection method to AzureDataFactoryHook (#21924): @josh-fell : Note: It won't work until Update test connection functionality to use custom form fields #21330 is released.
• Add pre-commit check for docstring param types (#21398): @dstandish
• Make container creation configurable when uploading files via WasbHook (#20510): @josh-fell
• (AzureCosmosDBHook) Update to latest Cosmos API (#21514): @BenGatewood

Provider mysql: 2.2.1rc1

• Make dbapi inherit get_uri from connection (#21764): @bolkedebruin
• Task names should be consistent with variable names (drop vs create) in example dags (#21434): @vinooganesh

Provider oracle: 2.2.1rc1

• Fix oracle test connection (#21699): @hubert-pietron

Provider postgres: 4.0.0rc1

• Replaces the usage of postgres:// with postgresql:// (#21205): @potiuk
• Add more SQL template fields renderers (#21237): @josh-fell
• Add conditional template_fields_renderers check for new SQL lexers (#21403): @josh-fell

Provider presto: 2.1.0rc1

• Add GCSToPrestoOperator (#21084): @rsg17
• Quick Update GCS Presto (#21855): @rsg17
• Replaced hql references to sql in TrinoHook and PrestoHook #21597 (#21630): @utkarsharma2
• Fix K8S changelog to be PyPI-compatible (#20614): @potiuk
• Fix MyPy Errors for Presto provider (#20244): @kazanzhy

Provider sftp: 2.5.0rc1

• [Feature] Updates FTPHook provider to have test_connection (#21997): @frankcash

Provider trino: 2.1.0rc1

• Add GCSToTrinoOperator (#21704): @rsg17

Provider zendesk: 3.0.1rc1

• Update Zendesk example DAG to use TaskFlow API (#21411): @josh-fell

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[vinooganesh]

#21434 looks good (docs change)

[kazanzhy]

#21307 - tested before PR
#20119 - seems OK
#20244 - OK

#20642 - tested
#20907 - tested
#21231 - tested
Fix PR #22125

[treyyi]

#21988 Looks good (hook name change)

[zhongjiajie]

#21422 Looking good, and it just docs change BTW

[alexott]

Checked #21363 from the main branch, works.
#21494 looks good.

The rest for Databricks operator are also looks good from the source code perspective

[hsrocks]

#21673 Tested it. Worked fine

[h2cone]

#21831 Tested it, LGTM.

[akifcakir]

#21709 tested it , works fine.

[potiuk]

Thanks for the tests so far :) .

FYI. For those who struggle on how to test, here is a helpful guideline:

https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors

UPDATE: Actually the docs were a bit outdated. Better instructions here:

---

Installing in your local virtualenv

You have to make sure you have Airflow 2* installed in your PIP virtualenv
(the version you want to install providers with).

pip install apache-airflow-providers-<provider>==<VERSION>rc<X>

Installing with Breeze

./breeze start-airflow --use-airflow-version 2.2.4 --python 3.7 --backend postgres \
    --load-example-dags --load-default-connections

After you are in Breeze:

pip install apache-airflow-providers-<provider>==<VERSION>rc<X>

NOTE! You should Ctrl-C and restart the connections to restart airflow components and make sure new
provider packages is used.

---

I will add it to the issue template for next wave :)

[frankcash]

Confirmed #21997 via the UI utility and running a task that calls .test_connection() for both FTP and SFTP hooks

[schirag1993]

Tested #21686 - works as expected

[vincbeck]

#21976 LGTM. Only docs change

[vincbeck]

#21975 is good too. Docs change only

[alexchen8]

#21778 looks good on implementation side.
Found a small typo in doc, will submit a PR.
Thanks!

[vincbeck]

#19137 Found out some bugs while testing it. Will submit a PR soon

[alekseiloginov]

#20882 looks good, thanks!

[potiuk]

#19137 Found out some bugs while testing it. Will submit a PR soon

Regression or some new stuff added has errors @vincbeck ?

[vincbeck]

#19137 Found out some bugs while testing it. Will submit a PR soon

Regression or some new stuff added has errors @vincbeck ?

New stuff

[potiuk]

New stuff

Cool. Does not block AWS provider from releasing :) . We can release a new version right after.

[kanga333]

I tested #21574. It worked fine. LGTM!

[potiuk]

Cool. We have one more day and we are doing good with the new Providers. This one is great as will let us move to Python 3.10 finally :)

[eladkal]

@potiuk if we still can #21788 should be skipped from change log. Its a change that doesn't require testing
I guess the skip log label isn't used for providers?

[potiuk]

@potiuk if we still can #21788 should be skipped from change log. Its a change that doesn't require testing I guess the skip log label isn't used for providers?

Nope. It's not (and we might add it though I think with the coming towncrier changes we might not want to invest in it).

We can always remove the changelog entry from the docs (but not from the package README),

[zachliu]

i tested #22002. good news: the retry did kick in 🚀
however, i'd like to add reraise: True in the default_kwargs here

airflow/airflow/providers/amazon/aws/hooks/base_aws.py

Lines 588 to 594 in a4d514b

	default_kwargs = {
	'wait': tenacity.wait_exponential(multiplier=multiplier, max=max_limit, min=min_limit),
	'retry': tenacity.retry_if_exception(should_retry),
	'stop': tenacity.stop_after_delay(stop_after_delay),
	'before': tenacity_logger,
	'after': tenacity_logger,
	}

otherwise we get a very vague exception when we exhaust all retries

tenacity.RetryError: RetryError[<Future at 0x7f143d69a9a0 state=finished raised Exception>]

i changed my mind, keeping reraise=False by default is fine 😅

[eskarimov]

Tested functionality introduced with #21663, works as expected.

[potiuk]

@kazanzhy the #22125 is also "new" stuff fix right ? If so, we will release this RC tomorrow and i will run next Amazon provider release right after

[josh-fell]

#21525, #21237, #21403. and #21411 look good!

#21924 unfortunately won't work until #21330 is released (presumably in Airflow 2.3).

I should be able to get to testing #20510 later today.

@potiuk It's not part of this issue but when testing the new dbt Cloud provider a couple parameters in an operator don't work quite as intended, but there is a workaround to get the same functionality with existing operator as written though. I can work on getting a fix in tonight but I was wondering if it would be possible to release an rc2 for dbt Cloud? Even though there is a workaround, it would be nice to have v1.0.0 work as expected out of the gate. WDYT?

[potiuk]

@potiuk It's not part of this issue but when testing the new dbt Cloud provider a couple parameters in an operator don't work quite as intended, but there is a workaround to get the same functionality with existing operator as written though. I can work on getting a fix in tonight but I was wondering if it would be possible to release an rc2 for dbt Cloud? Even though there is a workaround, it would be nice to have v1.0.0 work as expected out of the gate. WDYT?

Sure. I can remove dbt from the list easily.

[potiuk]

#21924 unfortunately won't work until #21330 is released (presumably in Airflow 2.3).

Too bad. Not critical though.

[shahkshitij15]

#21944 LGTM, tested it!

[josh-fell]

@potiuk It's not part of this issue but when testing the new dbt Cloud provider a couple parameters in an operator don't work quite as intended, but there is a workaround to get the same functionality with existing operator as written though. I can work on getting a fix in tonight but I was wondering if it would be possible to release an rc2 for dbt Cloud? Even though there is a workaround, it would be nice to have v1.0.0 work as expected out of the gate. WDYT?

Sure. I can remove dbt from the list easily.

Thanks @potiuk!

[kazanzhy]

@potiuk
Yes. #22125 fixes this new RDS functionality that is not covered by moto tests.

[zachliu]

@potiuk oh 💩 there is a bug 😅 in #22002
fix coming right up...#22137

[josh-fell]

#20510 looks good too. Thanks for organizing the release as usual @potiuk

[lwyszomi]

#21756 #21639 #21518 and #21267 look good

[potiuk]

@potiuk oh there is a bug in #22002

Right. Seems like Amazon accumulated enough fixes to skip it from the release and run RC2 :)

[Bowrna]

#21986 looks good, documentation changes only :)

[alexott]

if we'll do second set of releases, then it would be nice to get another release for databricks provider - to include #22076

[potiuk]

if we'll do second set of releases, then it would be nice to get another release for databricks provider - to include #22076

I think unless we find some problems, I prefer to release only those that we have to as RC2. I think this is not very far for the next month release :).

[jedcunningham]

Checked and the change from #21826 is in the release.

[potiuk]

Thanks everyone! I will proceed with releasing those and will release amazon/dbt RC2 shortly.