Upgrading Google auth library to 0.25.2 with dependency exclusion #8078

suztomo · 2021-04-12T15:11:13Z

Why dependency exclusion of httpcore?

Previous attempt failed (#8044) due to failOnVersionConflict, and there's no planned upcoming release for org.apache.httpcomponents:httpclient 4.5.X, which would fix the discrepancy.

suztomo-macbookpro44% ./gradlew     :grpc-all:dependencyInsight --configuration runtimeClasspath --dependency org.apache.httpcomponents:httpcore                                                                                 
*** Skipping the build of codegen and compilation of proto files because skipCodegen=true
  * Skipping the build of Android projects because skipAndroid=true

> Task :grpc-all:dependencyInsight
Dependency resolution failed because of conflict(s) on the following module(s):
   - org.apache.httpcomponents:httpcore between versions 4.4.14 and 4.4.13

org.apache.httpcomponents:httpcore:4.4.14
   variant "runtime" [
      org.gradle.status              = release (not requested)
      org.gradle.usage               = java-runtime
      org.gradle.libraryelements     = jar
      org.gradle.category            = library

      Requested attributes not found in the selected variant:
         org.gradle.dependency.bundling = external
         org.gradle.jvm.version         = 7
   ]
   Selection reasons:
      - By conflict resolution : between versions 4.4.14 and 4.4.13

org.apache.httpcomponents:httpcore:4.4.14
\--- com.google.http-client:google-http-client:1.39.1
     +--- com.google.auth:google-auth-library-oauth2-http:0.25.2
     |    \--- project :grpc-alts
     |         \--- project :grpc-xds
     |              \--- runtimeClasspath
     \--- com.google.http-client:google-http-client-gson:1.39.1
          \--- com.google.auth:google-auth-library-oauth2-http:0.25.2 (*)

org.apache.httpcomponents:httpcore:4.4.13 -> 4.4.14
\--- org.apache.httpcomponents:httpclient:4.5.13
     \--- com.google.http-client:google-http-client:1.39.1
          +--- com.google.auth:google-auth-library-oauth2-http:0.25.2
          |    \--- project :grpc-alts
          |         \--- project :grpc-xds
          |              \--- runtimeClasspath
          \--- com.google.http-client:google-http-client-gson:1.39.1
               \--- com.google.auth:google-auth-library-oauth2-http:0.25.2 (*)

Initially this PR tried to overwrite the httpcore dependency, but it does not have effect in the pom.xml consumed by gRPC users. Therefore this PR now excludes httpcore dependency and redeclare it as direct dependencies.

suztomo · 2021-04-12T15:52:22Z

build.gradle

@@ -155,6 +156,7 @@ subprojects {
            google_api_protos: 'com.google.api.grpc:proto-google-common-protos:2.0.1',
            google_auth_credentials: "com.google.auth:google-auth-library-credentials:${googleauthVersion}",
            google_auth_oauth2_http: "com.google.auth:google-auth-library-oauth2-http:${googleauthVersion}",
+            google_htt_client_jackson2: "com.google.http-client:google-http-client-jackson2:${googlehttpVersion}",


xds/src/main/java/io/grpc/xds/internal/sts/StsCredentials uses com.google.api.client.json.jackson2.JacksonFactory from this artifact.

suztomo · 2021-04-12T19:23:55Z

auth/src/test/java/io/grpc/auth/GoogleAuthLibraryCallCredentialsTest.java

-    assertEquals("https://example.com:123/a.service", payload.get("aud"));
+    assertEquals("https://example.com/", payload.get("aud"));


#8037 is about this change.

Looks like this is go/simpler-JWT-token-audience. I'm surprised to see the port removed since the port was not 443. @jiangtaoli2016, is this okay?

This does have the potential to break users. Technically the JWT logic was gRPC-specific, not googleapis.com-specific. I do wonder if there will be any blow-back from that. This right here is a service account which I don't believe is a standard format, so it may be fine. But I worry what other paths there are to create JWTs that aren't Google-specific that may be broken (Java for one thing; but other languages is also a concern).

It is the new standard of setting audience in JWT token
https://cloud.google.com/api-gateway/docs/authenticating-users-jwt

We also need to change some of the C++ grpc implementation.

@jiangtaoli2016, specifically, is stripping the port when it is not 443 "okay"? I see nothing regarding it in the doc you linked to.

To be pedantic, JWT is a standard, but this authority change appears to be Google-specific. With this change it would be easy to break the audience for anyone using JWT for their own services. For these specific Java APIs, the obvious ones to me seem to say "Google" prominently in the class name or javadoc. I don't know if that is true in other languages (e.g., C++).

IIUC, this auth library is only used for talking to Google APIs.

If a customer wants to use JWT to talk to its own services, he/she should use his/her own library rather than Google auth library.

@jiangtaoli2016, is stripping the port okay? I'd like an explicit answer. If the answer is "it is okay for Google properties" then that is still a useful answer.

IIUC, this auth library is only used for talking to Google APIs.

That was not the case for google-auth-library-java, at least for the Credentials and OAuth2Credentials classes. I expect most gRPC users doing authentication to use the google-auth-library-java Credentials class. The Google-specific credentials are primarily derived from GoogleCredentials. ServiceAccountJwtAccessCredentials does not extend GoogleCredentials, but the JSON service account stuff is fairly Google-specific.

The C++ API is nowhere near as clear. ServiceAccountJWTAccessCredentials doesn't have "Google" anywhere to be found. Yet AccessTokenCredentials, which should not be specific to Google, says "only send tokens to Google" (paraphrase), without actually saying they are Google tokens. I'm not trying to call out C++ saying it is "doing it wrong" or whatever; I'm trying to call out "we have to be careful" and "don't go around assuming things."

I found the PR with the change: googleapis/google-auth-library-java#572 . Interestingly, it is now auto-promoting ServiceAccountAccessCredentials to ServiceAccountJWTAccessCredentials. So that means we can delete all of JwtHelper that does that promotion.

I don't know if it is just me, but the release notes of google-auth-library-java seem really bad and useless. It seems I'm expected to read every PR to have any hope of understanding what changed. Deleting methods is considered bug fixes, even if only a single method is changed there's no mention of the class/method in the description, and completely-internal lint-level fixes are mixed in with everything else.

@ejona86

is stripping the port okay?

It is OK for google APIs. "API Gateway accepts all JWTs with the backend service name in the form of https://SERVICE_NAME in the aud claim."

https://somename:123/ is a different backend service name than https://somename/

The way that could be fine for Google APIs is if we said, "Google will never use a non-443 port, or if it does it will be considered equivalent."

I checked with Shin Fan who confirmed that we should stripe port. Don't know why we have port in the first place.
Also I think all Google APIs use 443 port.

suztomo · 2021-04-12T20:46:53Z

"Linux artifacts" and "Macos" didn't trigger job. (I guess some Kokoro issue)

suztomo · 2021-04-12T20:48:25Z

build.gradle

@@ -155,6 +156,7 @@ subprojects {
            google_api_protos: 'com.google.api.grpc:proto-google-common-protos:2.0.1',
            google_auth_credentials: "com.google.auth:google-auth-library-credentials:${googleauthVersion}",
            google_auth_oauth2_http: "com.google.auth:google-auth-library-oauth2-http:${googleauthVersion}",
+            google_http_client_jackson2: "com.google.http-client:google-http-client-jackson2:${googlehttpVersion}",


This is because grpc-xds uses a class from the artifact:

> Task :grpc-xds:compileJava FAILED /Users/suztomo/grpc-java/xds/src/main/java/io/grpc/xds/internal/sts/StsCredentials.java:29: error: package com.google.api.client.json.jackson2 does not exist import com.google.api.client.json.jackson2.JacksonFactory;

It was a used-but-undeclared dependency.

Fixes grpc#8037

suztomo · 2021-04-14T03:54:36Z

Now all checks are green.

@dapengzhang0 Would you review this?

dapengzhang0

LGTM

ejona86 · 2021-04-15T17:57:26Z

build.gradle

+                // https://github.com/grpc/grpc-java/issues/8037
+                configurations.all {
+                    resolutionStrategy {
+                        force 'org.apache.httpcomponents:httpcore:4.4.14'


I don't think this does what we need it to do. It may satisfy failOnVersionConflict(), but we have failOnVersionConflict() only to detect cases where Maven would interpret the dependencies incorrectly. Generally we have to exclude+redefine the dependency to force a specific transitive dependency version.

Good. I'll update so.

suztomo · 2021-04-15T21:30:08Z

Thank you for the review. I'll wait for @jiangtaoli2016 for the input.

jiangtaoli2016 · 2021-04-15T22:51:11Z

auth/src/test/java/io/grpc/auth/GoogleAuthLibraryCallCredentialsTest.java

-    assertEquals("https://example.com:123/a.service", payload.get("aud"));
+    assertEquals("https://example.com/", payload.get("aud"));


It is the new standard of setting audience in JWT token
https://cloud.google.com/api-gateway/docs/authenticating-users-jwt

We also need to change some of the C++ grpc implementation.

It's preferred to setup dependency exclusions and declare the problematic dependency as a direct dependency, because overwriting the dependency (in this case org.apache.httpcomponents:httpcore), while it may pass the checks in the build, does not have any effect in the produced pom.xml after the library is published to Maven Central.

ejona86 · 2021-04-16T19:05:33Z

auth/src/test/java/io/grpc/auth/GoogleAuthLibraryCallCredentialsTest.java

@@ -394,7 +394,7 @@ public void jwtAccessCredentialsInRequestMetadata() throws Exception {
    Map<?, ?> header = (Map<?, ?>) JsonParser.parse(jsonHeader);
    assertEquals("test-private-key-id", header.get("kid"));
    Map<?, ?> payload = (Map<?, ?>) JsonParser.parse(jsonPayload);
-    assertEquals("https://example.com:123/a.service", payload.get("aud"));
+    assertEquals("https://example.com/", payload.get("aud"));


This will require an atomic upgrade of the auth library inside Google as well. We don't want to require that. Change the logic here to allow either the old or the new audience, with a comment.

suztomo · 2021-04-16T19:44:44Z

Problem in ':grpc-all:dependencies' task over JUnit dependencies.

Execution failed for task ':grpc-all:dependencies'.
> Could not resolve all dependencies for configuration ':grpc-all:runtimeClasspath'.
   > Conflict(s) found for the following module(s):
       - junit:junit between versions 4.13.1 and 4.12
     Run with:
         --scan or
         :grpc-all:dependencyInsight --configuration runtimeClasspath --dependency junit:junit
     to get more insight on how to solve the conflict.

suztomo-macbookpro44% ./gradlew :grpc-all:dependencyInsight --configuration runtimeClasspath --dependency junit:junit 
*** Skipping the build of codegen and compilation of proto files because skipCodegen=true
  * Skipping the build of Android projects because skipAndroid=true

> Task :grpc-all:dependencyInsight
Dependency resolution failed because of conflict(s) on the following module(s):
   - junit:junit between versions 4.13.1 and 4.12

junit:junit:4.13.1
   variant "runtime" [
      org.gradle.status              = release (not requested)
      org.gradle.usage               = java-runtime
      org.gradle.libraryelements     = jar
      org.gradle.category            = library

      Requested attributes not found in the selected variant:
         org.gradle.dependency.bundling = external
         org.gradle.jvm.version         = 7
   ]
   Selection reasons:
      - By constraint
      - By conflict resolution : between versions 4.13.1 and 4.12

junit:junit:4.13.1
\--- com.fasterxml.jackson:jackson-bom:2.12.2
     \--- com.fasterxml.jackson.core:jackson-core:2.12.2
          +--- com.google.http-client:google-http-client-jackson2:1.39.1
          |    \--- project :grpc-xds
          |         \--- runtimeClasspath
          \--- com.fasterxml.jackson:jackson-bom:2.12.2 (*)

junit:junit:4.12 -> 4.13.1
\--- project :grpc-testing
     \--- runtimeClasspath

(*) - dependencies omitted (listed previously)

A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed

suztomo · 2021-04-16T19:53:18Z

com.fasterxml.jackson.core:jackson-core:2.12.2 declares jackson-base as its parent.

Jackson-base declares junit (with test scope appropriately)

  <dependencies>
    <dependency> <!-- all components use junit for testing -->
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

I'm not sure why the Gradle tool is showing com.fasterxml.jackson:jackson-bom as if it's declared as a dependency. But I think the intent is that the version of junit is set by jackson-bom

Having said that, I think this is a bug in Gradle. JUnit declared as test-scope at jackson-core (see the effective pom at jackson-core 2.12.2) should not affect any configuration (i.e. class path) of grpc-java project.

Google-http-client-jackson2 depends on jackson-core. Jackson-core depends on junit (with test scope). If we don't exclude the junit dependency, :grpc-all:dependencies (unexpectedly) fails to resolve dependencies. Details: grpc#8078 (comment)

ejona86 · 2021-04-16T20:18:38Z

jackson-parent has <dependencyManagement> on junit, and I think Gradle is just applying that globally instead of locally. I see evidence of similar behavior for jackson-core due to jackson-bom. The thing to notice is that junit is not listed in the details of grpc-xds until junit is pulled in elsewhere:

$ gradle :grpc-xds:dependencies --configuration runtimeClasspath
> Task :grpc-xds:dependencies

------------------------------------------------------------
Project :grpc-xds - gRPC: XDS plugin
------------------------------------------------------------

runtimeClasspath - Runtime classpath of source set 'main'.
+--- project :grpc-protobuf
...
+--- com.google.http-client:google-http-client-jackson2:1.39.1
|    +--- com.google.http-client:google-http-client:1.39.1 (*)
|    \--- com.fasterxml.jackson.core:jackson-core:2.12.2
|         \--- com.fasterxml.jackson:jackson-bom:2.12.2
|              \--- com.fasterxml.jackson.core:jackson-core:2.12.2 (c)
\--- org.apache.httpcomponents:httpcore:4.4.14

And this is if I disable failOnVersionConflict():

$ gradle :grpc-all:dependencies --configuration runtimeClasspath
> Task :grpc-all:dependencies

------------------------------------------------------------
Project :grpc-all - gRPC: All
------------------------------------------------------------

runtimeClasspath - Runtime classpath of source set 'main'.
+--- project :grpc-api
...
\--- project :grpc-xds
...
     +--- com.google.http-client:google-http-client-jackson2:1.39.1
     |    +--- com.google.http-client:google-http-client:1.39.1 (*)
     |    \--- com.fasterxml.jackson.core:jackson-core:2.12.2
     |         \--- com.fasterxml.jackson:jackson-bom:2.12.2
     |              +--- junit:junit:4.13.1 (c)
     |              \--- com.fasterxml.jackson.core:jackson-core:2.12.2 (c)
     \--- org.apache.httpcomponents:httpcore:4.4.14

suztomo · 2021-04-16T20:39:55Z

Android failed:


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':grpc-android-interop-testing:mergeDexDebug'.
> Could not resolve all files for configuration ':grpc-android-interop-testing:debugRuntimeClasspath'.
   > Failed to transform guava-30.1-android.jar (com.google.guava:guava:30.1-android) to match attributes {artifactType=android-dex, dexing-enable-desugaring=false, dexing-is-debuggable=true, dexing-min-sdk=16, org.gradle.category=library, org.gradle.libraryelements=jar, org.gradle.status=release, org.gradle.usage=java-runtime}.
      > Execution failed for DexingNoClasspathTransform: /tmpfs/tmp/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/30.1-android/1e904cca24cdf3dc84d5fe4f73e12f1186e04380/guava-30.1-android.jar.
         > Error while dexing.
           The dependency contains Java 8 bytecode. Please enable desugaring by adding the following to build.gradle
           android {
               compileOptions {
                   sourceCompatibility 1.8
                   targetCompatibility 1.8
               }
           }
           See https://developer.android.com/studio/write/java8-support.html for details. Alternatively, increase the minSdkVersion to 26 or above.

This PR does not touch Guava. I assume this is irrelevant?

ejona86 · 2021-04-16T21:12:47Z

This PR does not touch Guava. I assume this is irrelevant?

google-auth-library-java (specifically google-auth-library-oauth2-http) bumped the Guava version. https://github.com/googleapis/google-auth-library-java/releases/tag/v0.23.0

We didn't notice earlier because we exclude the version of guava (to upgrade it):

grpc-java/build.gradle

Lines 248 to 257 in b4fe07d

    
           googleOauth2Dependency = { configurationName -> 
        
               dependencies."$configurationName"(libraries.google_auth_oauth2_http) { 
        
                   exclude group: 'com.google.guava', module: 'guava' 
        
                   exclude group: 'io.grpc', module: 'grpc-context' 
        
                   exclude group: 'io.opencensus', module: 'opencensus-api' 
        
               } 
        
               dependencies.runtimeOnly project(':grpc-context') 
        
               censusApiDependency 'runtimeOnly' 
        
               guavaDependency 'runtimeOnly' 
        
           }

But now it is a downgrade.

suztomo · 2021-04-16T21:21:25Z

I see. The error is from the android-interop-testing, which uses libraries.google_auth_oauth2_http without excluding Guava:

implementation (libraries.google_auth_oauth2_http) {
    exclude group: 'org.apache.httpcomponents'
}

The error message suggests that guava-30.1-android is not compatible with Android. It's strange. Digging further.

ejona86 · 2021-04-16T21:21:58Z

For reference: google/guava#5358

Let's upgrade Guava as a separate PR before this one goes in. That's probably something better for grpc-team to manage, as it could potentially devolve.

google-auth-library-java:0.25.0 strips port and path parts in the audience claim ("aud"). Updating the test to pass in both old and new version of google-auth-library-java. This commit does not upgrade google-auth-library-java because it turned out that the upgrade involves the newer Guava version (google-auth-library-java's dependency) failing with DexingNoClasspathTransform. Details: grpc#8078 (comment) It's technically possible to exclude the newer Guava, but it's a good practice avoid excluding the newer version of a library.

ejona86 · 2021-04-16T22:07:21Z

We're splitting out two parts of this PR for the short-term. #8099 will update the test which allows upgrading google-auth-library-java anywhere the grpc tests are also run. That addresses the main issue @suztomo is suffering from. Additionally, #8100 upgrades to the newer Guava.

Since upgrading to the newer Guava is risky, and I don't get excited about the idea of flip-flopping on the JWT behavior of google-auth-library-java (if we have to revert the upgrade due to Guava), I think I'd like to hold off on the upgrade of google-auth-library-java for a release. Users are free to use a newer version, and the googleapis libraries can use a newer release.

google-auth-library-java:0.25.0 strips port and path parts in the audience claim ("aud"). Updating the test to pass in both old and new version of google-auth-library-java. This commit does not upgrade google-auth-library-java because it turned out that the upgrade involves the newer Guava version (google-auth-library-java's dependency) failing with DexingNoClasspathTransform. Details: #8078 (comment) It's technically possible to exclude the newer Guava, but it's a good practice avoid excluding the newer version of a library.

suztomo marked this pull request as draft April 12, 2021 15:24

suztomo commented Apr 12, 2021

View reviewed changes

suztomo force-pushed the google_auth_0.25.2 branch from 64e37f6 to b2aac49 Compare April 12, 2021 17:23

suztomo commented Apr 12, 2021

View reviewed changes

suztomo marked this pull request as ready for review April 12, 2021 20:47

suztomo commented Apr 12, 2021

View reviewed changes

suztomo added 3 commits April 12, 2021 19:36

Upgrading google-auth-library to 0.25.2

ec8d191

Fixes grpc#8037

Dependency override

c8e0264

google-http-java-client-jackson2 with dependency exclusions

dbe557f

suztomo force-pushed the google_auth_0.25.2 branch from b2aac49 to dbe557f Compare April 12, 2021 23:36

dapengzhang0 requested a review from ejona86 April 15, 2021 00:18

dapengzhang0 approved these changes Apr 15, 2021

View reviewed changes

ejona86 reviewed Apr 15, 2021

View reviewed changes

jiangtaoli2016 approved these changes Apr 15, 2021

View reviewed changes

suztomo changed the title ~~Upgrading Google auth library to 0.25.2 with dependency override~~ Upgrading Google auth library to 0.25.2 with dependency exclusion Apr 16, 2021

ejona86 approved these changes Apr 16, 2021

View reviewed changes

ejona86 added the kokoro:run Add this label to a PR to tell Kokoro the code is safe and tests can be run label Apr 16, 2021

grpc-kokoro removed the kokoro:run Add this label to a PR to tell Kokoro the code is safe and tests can be run label Apr 16, 2021

ejona86 requested changes Apr 16, 2021

View reviewed changes

Allow both old and new behavior from google-auth-library-java

b19fc94

ejona86 approved these changes Apr 16, 2021

View reviewed changes

ejona86 added the kokoro:run Add this label to a PR to tell Kokoro the code is safe and tests can be run label Apr 16, 2021

grpc-kokoro removed the kokoro:run Add this label to a PR to tell Kokoro the code is safe and tests can be run label Apr 16, 2021

ejona86 mentioned this pull request Apr 16, 2021

Upgrade to Guava 30.1, which warns on Java 7 #8100

Merged

suztomo closed this May 5, 2021

github-actions bot locked as resolved and limited conversation to collaborators Aug 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading Google auth library to 0.25.2 with dependency exclusion #8078

Upgrading Google auth library to 0.25.2 with dependency exclusion #8078

suztomo commented Apr 12, 2021 •

edited

suztomo Apr 12, 2021 •

edited

suztomo Apr 12, 2021

ejona86 Apr 15, 2021

jiangtaoli2016 Apr 15, 2021

ejona86 Apr 15, 2021

jiangtaoli2016 Apr 15, 2021

ejona86 Apr 16, 2021

jiangtaoli2016 Apr 16, 2021

ejona86 Apr 16, 2021

ejona86 Apr 16, 2021

jiangtaoli2016 Apr 16, 2021

suztomo commented Apr 12, 2021

suztomo Apr 12, 2021 •

edited

suztomo commented Apr 14, 2021

dapengzhang0 left a comment

ejona86 Apr 15, 2021

suztomo Apr 16, 2021

suztomo commented Apr 15, 2021

jiangtaoli2016 Apr 15, 2021

ejona86 Apr 16, 2021

suztomo commented Apr 16, 2021 •

edited

suztomo commented Apr 16, 2021 •

edited

ejona86 commented Apr 16, 2021

suztomo commented Apr 16, 2021

ejona86 commented Apr 16, 2021

suztomo commented Apr 16, 2021

ejona86 commented Apr 16, 2021

ejona86 commented Apr 16, 2021 •

edited

		assertEquals("https://example.com:123/a.service", payload.get("aud"));
		assertEquals("https://example.com/", payload.get("aud"));

Upgrading Google auth library to 0.25.2 with dependency exclusion #8078

Upgrading Google auth library to 0.25.2 with dependency exclusion #8078

Conversation

suztomo commented Apr 12, 2021 • edited

Why dependency exclusion of httpcore?

suztomo Apr 12, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo commented Apr 12, 2021

suztomo Apr 12, 2021 • edited

Choose a reason for hiding this comment

suztomo commented Apr 14, 2021

dapengzhang0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo commented Apr 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suztomo commented Apr 16, 2021 • edited

suztomo commented Apr 16, 2021 • edited

ejona86 commented Apr 16, 2021

suztomo commented Apr 16, 2021

ejona86 commented Apr 16, 2021

suztomo commented Apr 16, 2021

ejona86 commented Apr 16, 2021

ejona86 commented Apr 16, 2021 • edited

suztomo commented Apr 12, 2021 •

edited

suztomo Apr 12, 2021 •

edited

suztomo Apr 12, 2021 •

edited

suztomo commented Apr 16, 2021 •

edited

suztomo commented Apr 16, 2021 •

edited

ejona86 commented Apr 16, 2021 •

edited