[BEAM-11205] Update Libraries BOM dependencies to version 25.2.0 #17497

benEng · 2022-04-28T20:49:17Z

Updating Google Cloud Libraries BOM to the latest version 25.2.0 and associated dependencies to the compatible versions.

Updating dependencies listed in https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/25.2.0/artifact_details.html

Linkage check results
https://gist.github.com/benWize/37f0d34e157974f6c1ff5a8e53f94a9b

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

asf-ci · 2022-04-28T20:49:19Z

Can one of the admins verify this patch?

asf-ci · 2022-04-28T20:49:19Z

Can one of the admins verify this patch?

codecov · 2022-04-28T21:15:38Z

Codecov Report

Merging #17497 (0a6a07e) into master (bb53425) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #17497      +/-   ##
==========================================
- Coverage   73.83%   73.83%   -0.01%     
==========================================
  Files         690      690              
  Lines       90829    90843      +14     
==========================================
+ Hits        67066    67076      +10     
- Misses      22554    22558       +4     
  Partials     1209     1209

Flag	Coverage Δ
python	`83.65% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...eam/runners/portability/fn_api_runner/execution.py	`92.44% <0.00%> (-0.65%)`	⬇️
...eam/runners/interactive/interactive_environment.py	`90.18% <0.00%> (-0.31%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.51% <0.00%> (ø)`
.../python/apache_beam/ml/inference/sklearn_loader.py
...thon/apache_beam/ml/inference/sklearn_inference.py	`92.68% <0.00%> (ø)`
sdks/python/apache_beam/transforms/util.py	`96.06% <0.00%> (+0.08%)`	⬆️
...dks/python/apache_beam/examples/cookbook/coders.py	`63.15% <0.00%> (+0.99%)`	⬆️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (+2.43%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da6acf2...0a6a07e. Read the comment docs.

benEng · 2022-04-28T21:32:28Z

Run PythonLint PreCommit

benEng · 2022-04-28T21:32:58Z

Run Java PostCommit

benEng · 2022-04-28T21:33:10Z

Run PostCommit_Java_Hadoop_Versions

benEng · 2022-04-28T21:33:21Z

Run Dataflow ValidatesRunner

benEng · 2022-04-28T23:38:53Z

Run Java PostCommit

benEng · 2022-04-28T23:39:09Z

Run PostCommit_Java_Hadoop_Versions

benEng · 2022-04-28T23:39:20Z

Run Dataflow ValidatesRunner

benEng · 2022-04-29T15:17:33Z

Run Java PreCommit

benEng · 2022-04-29T16:13:21Z

R: @suztomo
Could you help me to review this?

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

suztomo

Can you also update datastore-v1-proto-client to 2.2.10?

suztomo · 2022-04-29T22:30:19Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

@@ -455,16 +455,16 @@ class BeamModulePlugin implements Plugin<Project> {
    def classgraph_version = "4.8.104"
    def errorprone_version = "2.10.0"
    // Try to keep gax_version consistent with gax-grpc version in google_cloud_platform_libraries_bom
-    def gax_version = "2.8.1"
+    def gax_version = "2.16.0"
    def google_clients_version = "1.32.1"
    def google_cloud_bigdataoss_version = "2.2.4"


Can you make it 2.2.6?

suztomo · 2022-04-29T22:30:40Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

-    def google_cloud_spanner_version = "6.20.0"
-    def google_code_gson_version = "2.8.9"
+    def google_cloud_spanner_version = "6.23.3"
+    def google_code_gson_version = "2.9.0"
    def google_oauth_clients_version = "1.32.1"


Can you update this to 1.33.3

suztomo · 2022-04-29T22:34:08Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

-    def google_cloud_spanner_version = "6.20.0"
-    def google_code_gson_version = "2.8.9"
+    def google_cloud_spanner_version = "6.23.3"
+    def google_code_gson_version = "2.9.0"
    def google_oauth_clients_version = "1.32.1"
    // Try to keep grpc_version consistent with gRPC version in google_cloud_platform_libraries_bom
    def grpc_version = "1.44.0"


Can you update this to 1.45.1?

Yes, updated

suztomo

@benWize Can you set datastore-v1-proto-client to 2.2.10 and override proto-google-cloud-datastore-v1 to 0.93.10 (currently no explicit version)? It's consistent with datastore-v1-proto-client to 2.2.10.

C.f. https://search.maven.org/artifact/com.google.cloud/google-cloud-datastore-bom/2.2.10/pom

CC: @kolea2

benEng · 2022-05-02T18:40:42Z

Run SQL PreCommit

benEng · 2022-05-02T18:40:55Z

Run Python_PVR_Flink PreCommit

benEng · 2022-05-02T18:41:03Z

Run Java PreCommit

suztomo · 2022-05-04T20:29:01Z

Run Java PreCommit

It failed org.apache.beam.sdk.io.pulsar.PulsarIOTest.classMethod

suztomo · 2022-05-04T20:35:56Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

@@ -661,10 +661,10 @@ class BeamModulePlugin implements Plugin<Project> {
        proto_google_cloud_bigtable_admin_v2        : "com.google.api.grpc:proto-google-cloud-bigtable-admin-v2", // google_cloud_platform_libraries_bom sets version
        proto_google_cloud_bigtable_v2              : "com.google.api.grpc:proto-google-cloud-bigtable-v2", // google_cloud_platform_libraries_bom sets version
        proto_google_cloud_datacatalog_v1beta1      : "com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1", // google_cloud_platform_libraries_bom sets version
-        proto_google_cloud_datastore_v1             : "com.google.api.grpc:proto-google-cloud-datastore-v1", // google_cloud_platform_libraries_bom sets version
+        proto_google_cloud_datastore_v1             : "com.google.api.grpc:proto-google-cloud-datastore-v1:0.93.10", // google_cloud_platform_libraries_bom sets version


My memo: This is in-line with com.google.cloud.datastore:datastore-v1-proto-client:2.2.10

(no action required)

@ansh0l For upcoming change by you, remove this version part and let google_cloud_platform_libraries_bom set the version.

This change suggests changes in the version for proto_google_cloud_datastore_v1. Can we(Spanner team) make this change or should we let the Datastore team make this change?

CC: @kileys

Can you (spanner team) take care of this?

Sure, we'll take care of this.

suztomo · 2022-05-04T20:36:23Z

@benWize The change looks good to me. Anything pending from your perspective?

suztomo · 2022-05-04T20:39:51Z

Run Java PostCommit

suztomo · 2022-05-04T20:40:02Z

Run PostCommit_Java_Dataflow

suztomo · 2022-05-04T20:40:08Z

Run PostCommit_Java_DataflowV2

suztomo · 2022-05-04T20:40:15Z

Run Spark ValidatesRunner

suztomo · 2022-05-04T20:40:21Z

Run SQL Postcommit

This failed with

18:05:31 FAILURE: Build failed with an exception.
18:05:31 
18:05:31 * What went wrong:
18:05:31 Execution failed for task ':sdks:java:core:compileTestJava'.
18:05:31 > Failed to store cache entry for task ':sdks:java:core:compileTestJava'

suztomo · 2022-05-04T20:42:30Z

Where are Jenkins builds...?

The change looks good now.

suztomo · 2022-05-04T22:09:09Z

Run Java Precommit

This failed at org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

suztomo · 2022-05-04T22:09:57Z

Run SQL Postcommit

This failed at org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT.testReportFailures

suztomo · 2022-05-05T02:07:06Z

Run SQL Postcommit

suztomo · 2022-05-05T02:07:16Z

Run Java Precommit

benEng · 2022-05-05T15:09:21Z

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

There is nothing pending from my perspective.

benEng · 2022-05-05T15:11:02Z

Run PostCommit_Java_Dataflow

suztomo · 2022-05-05T16:51:23Z

Nice. Let's merge when checks become green. I see "Dataflow Java Post Commit Tests" pending (running).

benEng · 2022-05-05T20:03:16Z

Nice. Let's merge when checks become green. I see "Dataflow Java Post Commit Tests" pending (running).

All checks are passing now! Thanks, @suztomo

suztomo · 2022-05-05T20:24:50Z

@benWize Thank you.

suztomo · 2022-08-29T15:34:12Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

@@ -661,10 +661,10 @@ class BeamModulePlugin implements Plugin<Project> {
        proto_google_cloud_bigtable_admin_v2        : "com.google.api.grpc:proto-google-cloud-bigtable-admin-v2", // google_cloud_platform_libraries_bom sets version
        proto_google_cloud_bigtable_v2              : "com.google.api.grpc:proto-google-cloud-bigtable-v2", // google_cloud_platform_libraries_bom sets version
        proto_google_cloud_datacatalog_v1beta1      : "com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1", // google_cloud_platform_libraries_bom sets version
-        proto_google_cloud_datastore_v1             : "com.google.api.grpc:proto-google-cloud-datastore-v1", // google_cloud_platform_libraries_bom sets version
+        proto_google_cloud_datastore_v1             : "com.google.api.grpc:proto-google-cloud-datastore-v1:0.93.10", // google_cloud_platform_libraries_bom sets version


@ansh0l For upcoming change by you, remove this version part and let google_cloud_platform_libraries_bom set the version.

suztomo · 2022-08-29T15:38:11Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

-    def protobuf_version = "3.19.3"
+    def protobuf_version = "3.19.4"


@ansh0l For the upcoming change by you, reference this:
https://github.com/googleapis/java-shared-dependencies/blob/v3.0.1/first-party-dependencies/pom.xml#L58

suztomo · 2022-08-31T11:46:38Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

-    def gax_version = "2.8.1"
+    def gax_version = "2.16.0"
    def google_clients_version = "1.32.1"
-    def google_cloud_bigdataoss_version = "2.2.4"
+    def google_cloud_bigdataoss_version = "2.2.6"
    // Try to keep google_cloud_spanner_version consistent with google_cloud_spanner_bom in google_cloud_platform_libraries_bom
-    def google_cloud_spanner_version = "6.20.0"
-    def google_code_gson_version = "2.8.9"
-    def google_oauth_clients_version = "1.32.1"
+    def google_cloud_spanner_version = "6.23.3"
+    def google_code_gson_version = "2.9.0"
+    def google_oauth_clients_version = "1.33.3"
    // Try to keep grpc_version consistent with gRPC version in google_cloud_platform_libraries_bom
-    def grpc_version = "1.44.0"
-    def guava_version = "31.0.1-jre"
+    def grpc_version = "1.45.1"
+    def guava_version = "31.1-jre"


@ansh0l For the upcoming change by you, reference this
https://github.com/googleapis/java-shared-dependencies/blob/v3.0.1/first-party-dependencies/pom.xml#L58 to update line 458 - line 467.

google_cloud_bigdataoss_version can be unaffected.

Should the google_code_gson_version be changed here? The above pom file doesn't have any reference to this dependency.

CC: @kileys

https://github.com/googleapis/java-shared-dependencies/blob/v3.0.1/third-party-dependencies/pom.xml has the gson version.

[BEAM-11205] Update bom version to 25.2.0

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

7382fb5

github-actions bot added build docker java labels Apr 28, 2022

Add package gax.rpc to GcpApiSurfaceTest

3d3d31c

github-actions bot added gcp io labels Apr 28, 2022

benEng marked this pull request as ready for review April 29, 2022 16:11

suztomo self-requested a review April 29, 2022 22:26

suztomo self-assigned this Apr 29, 2022

suztomo reviewed Apr 29, 2022

View reviewed changes

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy Show resolved Hide resolved

suztomo reviewed Apr 29, 2022

View reviewed changes

suztomo previously requested changes May 2, 2022

View reviewed changes

[BEAM-11205] Update dependencies BOM

0a6a07e

suztomo reviewed May 4, 2022

View reviewed changes

suztomo self-requested a review May 4, 2022 22:08

suztomo approved these changes May 5, 2022

View reviewed changes

suztomo merged commit b6565b9 into apache:master May 5, 2022

kellen mentioned this pull request Jul 5, 2022

Update beam to 2.40 spotify/scio#4433

Merged

suztomo reviewed Aug 29, 2022

View reviewed changes

suztomo reviewed Aug 31, 2022

View reviewed changes

		def protobuf_version = "3.19.3"
		def protobuf_version = "3.19.4"

[BEAM-11205] Update Libraries BOM dependencies to version 25.2.0 #17497

[BEAM-11205] Update Libraries BOM dependencies to version 25.2.0 #17497

Conversation

benEng commented Apr 28, 2022 • edited Loading

GitHub Actions Tests Status (on master branch)

asf-ci commented Apr 28, 2022

asf-ci commented Apr 28, 2022

codecov bot commented Apr 28, 2022 • edited Loading

Codecov Report

benEng commented Apr 28, 2022

benEng commented Apr 28, 2022

benEng commented Apr 28, 2022

benEng commented Apr 28, 2022

benEng commented Apr 28, 2022

benEng commented Apr 28, 2022

benEng commented Apr 28, 2022

benEng commented Apr 29, 2022

benEng commented Apr 29, 2022

suztomo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo left a comment

Choose a reason for hiding this comment

benEng commented May 2, 2022

benEng commented May 2, 2022

benEng commented May 2, 2022

suztomo commented May 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajatbhatta Sep 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo commented May 4, 2022 • edited Loading

suztomo commented May 4, 2022

suztomo commented May 4, 2022

suztomo commented May 4, 2022

suztomo commented May 4, 2022

suztomo commented May 4, 2022 • edited Loading

suztomo commented May 4, 2022

suztomo commented May 4, 2022 • edited Loading

suztomo commented May 4, 2022 • edited Loading

suztomo commented May 5, 2022

suztomo commented May 5, 2022

benEng commented May 5, 2022

benEng commented May 5, 2022

suztomo commented May 5, 2022

benEng commented May 5, 2022

suztomo commented May 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo Aug 31, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benEng commented Apr 28, 2022 •

edited

Loading

codecov bot commented Apr 28, 2022 •

edited

Loading

suztomo commented May 4, 2022 •

edited

Loading

rajatbhatta Sep 1, 2022 •

edited

Loading

suztomo commented May 4, 2022 •

edited

Loading

suztomo commented May 4, 2022 •

edited

Loading

suztomo commented May 4, 2022 •

edited

Loading

suztomo commented May 4, 2022 •

edited

Loading

suztomo Aug 31, 2022 •

edited

Loading