Regression HTTPS verification javax.net.ssl.SSLPeerUnverifiedException on docker layer pull #3058
Labels
Milestone
Comments
|
@daxroc thanks for the report. But there's something that doesn't fit. According to the Apache httpcomponents:httpclient release notes, 4.5.12 should Apache HttpClient release notesAnd according to the google-http-java-client 1.35.0 release note, it upgraded the Apache httpclient to 4.5.12.
And I cannot reproduce the failure. With Jib 2.7.1 (using google-http-java-client 1.38.0), it has no problem interacting with logIt makes me suspect that |
|
Actually, what you see from |
|
@daxroc any update on this? We plan to make a release soon, and so far, it doesn't seem like we need to downgrade the Google HTTP Client with the new release. This looks like a project configuration issue in your project. |
|
Sorry for the delay, I can take another look tomorrow and get back to you.
It is odd local builds succeed, build envs don't unless we downgrade. They
both share the same upstream dependency cache but I'll double check that in
the morning.
…On Fri 19 Feb 2021, 22:44 Chanseok Oh, ***@***.***> wrote:
@daxroc <https://github.com/daxroc> any update on this? We plan to make a
release soon, and so far, it doesn't seem like we need to downgrade the
Google HTTP Client with the new release. This looks like a project
configuration issue in your project.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3058 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKYKWAG6PIFGX2YB5LRCV3S73SUTANCNFSM4XU5YPNQ>
.
|
|
Maybe you can even start with a simple hello world app on the build env to see if it's a project configuration issue.
UPDATE(8/20/2021): now I start to think a regional location may be a factor to this. |
|
Any update on this? |
|
Hi @chanseokoh, I can reproduce the problem on my build servers, just like @daxroc described, using 2.7.1 or 2.8.0. 2.7.0 works fine. This is the result of 2.7.1
|
|
Can you also share the output when it is 2.7.0? And is this a multi-module project? If so, can you share your |
settings.gradlepluginManagement {
repositories {
gradlePluginPortal()
mavenCentral()
}
}
rootProject.name="<redacted>"
include 'api'
include 'common'
include 'consumer'
include 'db'api/build.gradlemainClassName = '<redacted>'
ext {
dockerBasicImageName = "<redacted>"
}
sourceSets {
main {
proto.srcDir "${project.rootDir}/<redacted>/proto"
}
}
dependencies {
implementation project(":common")
implementation project(":db")
}
jib {
to.image = "${dockerBasicImageName}"
}build.gradleplugins {
id "com.github.johnrengelman.shadow" version "6.1.0" apply false
id "de.undercouch.download" version "4.1.1"
id 'com.google.cloud.tools.jib' version '2.8.0' apply false
id 'java'
id 'com.google.protobuf' version '0.8.12' apply false
id "com.diffplug.spotless" version "5.10.1" apply false
}
version = "0.1.0"
allprojects {
repositories {
...
}
}
ext {
raygunVersion = '[2.0.0,3.0.0)'
micronautProjects = ['api', 'consumer', 'common', 'db']
micronautCassandraProjects = ['api', 'consumer', 'db']
micronautGrpcProjects = ['api']
executableProjects = ['api', 'consumer']
glowrootVersion = "0.13.5"
glowrootUrl = "https://github.com/glowroot/glowroot/releases/download/v$glowrootVersion/glowroot-$glowrootVersion-dist.zip"
}
def getGitHash = { ->
def stdout = new ByteArrayOutputStream()
exec {
commandLine 'git', 'rev-parse', '--short', 'HEAD'
standardOutput = stdout
}
return stdout.toString().trim()
}
task downloadApmAgent(type: Download) {
src glowrootUrl
dest "${buildDir}/glowroot_agent.zip"
overwrite false
}
configure(subprojects.findAll{it.name in micronautProjects}) {
apply plugin: 'java'
apply plugin: 'com.github.johnrengelman.shadow'
apply plugin: 'com.diffplug.spotless'
sourceCompatibility = 11
targetCompatibility = 11
ext {
micronautVersion='2.3.3'
}
configurations {
testCompileOnly.extendsFrom compileOnly
testAnnotationProcessor.extendsFrom annotationProcessor
}
dependencies {
compileOnly "org.projectlombok:lombok"
annotationProcessor "org.projectlombok:lombok"
annotationProcessor platform("io.micronaut:micronaut-bom:$micronautVersion")
annotationProcessor "io.micronaut:micronaut-inject-java"
annotationProcessor "io.micronaut:micronaut-validation"
implementation platform("io.micronaut:micronaut-bom:$micronautVersion")
implementation "io.micronaut:micronaut-inject"
implementation "io.micronaut:micronaut-validation"
implementation "io.micronaut:micronaut-runtime"
implementation "io.micronaut:micronaut-management"
implementation "io.micronaut.micrometer:micronaut-micrometer-registry-prometheus"
implementation "com.mindscapehq:core:$raygunVersion"
implementation "com.mindscapehq:raygun4java:$raygunVersion"
<redacted>
runtimeOnly "ch.qos.logback:logback-classic:1.2.3"
testAnnotationProcessor platform("io.micronaut:micronaut-bom:$micronautVersion")
testAnnotationProcessor "io.micronaut:micronaut-inject-java"
testImplementation platform("io.micronaut:micronaut-bom:$micronautVersion")
testImplementation "org.junit.jupiter:junit-jupiter-api"
testImplementation "org.junit.jupiter:junit-jupiter-params"
testImplementation "io.micronaut.test:micronaut-test-junit5"
testImplementation "org.assertj:assertj-core"
testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine"
testImplementation "org.mockito:mockito-core"
testImplementation "org.mockito:mockito-junit-jupiter:3.6.28"
}
test {
useJUnitPlatform()
testLogging {
exceptionFormat = 'full'
}
}
spotless {
java {
googleJavaFormat()
targetExclude(fileTree("$buildDir/generated") { include("**/*.java") })
}
}
tasks.withType(JavaCompile) {
options.encoding = "UTF-8"
options.compilerArgs.add('-parameters')
}
shadowJar {
mergeServiceFiles()
}
}
configure(subprojects.findAll{it.name in executableProjects}) {
apply plugin: 'com.google.cloud.tools.jib'
apply plugin: 'application'
ext.getGitHash = getGitHash
dependencies {
implementation 'io.micronaut:micronaut-http-server-netty'
}
ext {
agentLocation = "/opt/glowroot"
jibExtraDirectory = "${buildDir}/jib-agents"
}
task extractApmAgent(dependsOn: downloadApmAgent, type: Copy) {
from zipTree(downloadApmAgent.dest)
into "${jibExtraDirectory}/${agentLocation}"
}
jib {
from.image = 'adoptopenjdk:11-jre-hotspot'
extraDirectories.paths = [file(jibExtraDirectory)]
container {
jvmFlags = [
'-XX:InitialRAMPercentage=60.0',
'-XX:MaxRAMPercentage=60.0',
'-javaagent:' + agentLocation + '/glowroot/glowroot.jar'
]
}
}
tasks.jib.dependsOn extractApmAgent
tasks.jibDockerBuild.dependsOn extractApmAgent
tasks.jibBuildTar.dependsOn extractApmAgent
}
configure(subprojects.findAll{it.name in micronautCassandraProjects}) {
ext {
scyllaDriverVersion = '4.9.0-scylla-0'
}
dependencies {
implementation group: 'com.scylladb', name: 'java-driver-core', version: scyllaDriverVersion
implementation group: 'com.scylladb', name: 'java-driver-query-builder', version: scyllaDriverVersion
implementation group: 'com.scylladb', name: 'java-driver-metrics-micrometer', version: scyllaDriverVersion
implementation 'ch.hsr:geohash:1.3.0'
implementation("io.micronaut.cassandra:micronaut-cassandra") {
exclude group: "com.datastax.oss", module: "java-driver-core"
exclude group: "com.datastax.oss", module: "java-driver-mapper-processor"
}
}
}
configure(subprojects.findAll{it.name in micronautGrpcProjects}) {
apply plugin: "com.google.protobuf"
ext {
protocVersion = "3.15.4"
grpcVersion = "1.33.1"
}
sourceSets {
main {
java {
srcDirs 'build/generated/source/proto/main/grpc'
srcDirs 'build/generated/source/proto/main/java'
}
}
}
dependencies {
implementation("com.google.protobuf:protobuf-java:${protocVersion}")
implementation("io.micronaut:micronaut-http-server-netty")
implementation("io.micronaut.grpc:micronaut-grpc-runtime")
implementation("me.dinowernli:java-grpc-prometheus:0.3.0") {
exclude group: "io.grpc", module: "grpc-rls"
exclude group: "io.grpc", module: "grpc-netty-shaded"
exclude group: "io.grpc", module: "grpc-xds"
}
}
configurations.testImplementation {
exclude group: "io.grpc", module: "grpc-okhttp"
}
protobuf {
protoc { artifact = "com.google.protobuf:protoc:${protocVersion}" }
plugins {
grpc { artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}" }
}
generateProtoTasks {
all()*.plugins { grpc {} }
}
}
}
java {
sourceCompatibility = JavaVersion.toVersion("11")
targetCompatibility = JavaVersion.toVersion("11")
}
wrapper {
gradleVersion = '6.9'
// waiting for https://github.com/gradle/gradle/pull/15333 to be released
distributionUrl = '<redacted>'
}consumer/build.gradleimport com.github.davidmc24.gradle.plugin.avro.GenerateAvroJavaTask
plugins {
id "com.github.davidmc24.gradle.plugin.avro" version "1.0.0"
}
mainClassName = '<redacted>'
sourceSets {
main {
java {
srcDir "${buildDir.absolutePath}/generated/sources/avro/main/java"
}
}
test {
java {
srcDir "${buildDir.absolutePath}/generated/sources/avro/main/java"
}
}
}
task generateAvro(type: GenerateAvroJavaTask) {
source("${project.rootDir}/<redacted>/avro/")
outputDir = file("${buildDir.absolutePath}/generated/sources/avro/main/java")
}
compileJava.source(generateAvro.outputs)
ext {
dockerBasicImageName = "<redacted>"
}
dependencies {
implementation project(":db")
implementation project(":common")
implementation("io.confluent:kafka-avro-serializer:5.4.1")
implementation("io.micronaut.kafka:micronaut-kafka")
implementation("org.apache.avro:avro:1.10.1")
implementation("com.inloco:kafka-auth-k8s:0.0.3")
testImplementation project(':db').sourceSets.test.output
}
jib {
to.image = "${dockerBasicImageName}"
}db/build.gradledependencies {
implementation project(":common")
}
2.7.0
|
|
@paulolieuthier thank you very much for the info. Out of an abundance of caution, I think we should revert upgrading Google HTTP Client. But I should ask this last question: is it technically possible that you can reproduce the error with this hello-world example in your environment? (I guess you can just edit And is it possible to share the hostname in question so that we can also try accessing it on our side? |
|
I just tested with the example project and the I'm pushing to AWS' ECR. |
|
Hmm... @paulolieuthier @daxroc @tburch there is an ultimate and indisputable workaround where you can force a particular dependency version used for the build script classpath. At the top of the root All of the 2.8.0, 2.7.1, and 2.7.0 dependencies should just work, so it's worth testing out all. (UPDATE: a user reported that only 2.7.0 dependencies worked.)
buildscript {
repositories {
mavenCentral()
...
}
dependencies {
// note force = true below
classpath('org.apache.httpcomponents:httpcore:4.4.12') { force = true }
classpath('org.apache.httpcomponents:httpclient:4.5.10') { force = true }
classpath('com.google.http-client:google-http-client:1.34.0') { force = true }
classpath('com.google.http-client:google-http-client-apache-v2:1.34.0') { force = true }
...
}
}
buildscript {
repositories {
mavenCentral()
...
}
dependencies {
// note force = true below
classpath('org.apache.httpcomponents:httpcore:4.4.13') { force = true }
classpath('org.apache.httpcomponents:httpclient:4.5.13') { force = true }
classpath('com.google.http-client:google-http-client:1.38.0') { force = true }
classpath('com.google.http-client:google-http-client-apache-v2:1.38.0') { force = true }
...
}
}
buildscript {
repositories {
mavenCentral()
...
}
dependencies {
// note force = true below
classpath('org.apache.httpcomponents:httpcore:4.4.14') { force = true }
classpath('org.apache.httpcomponents:httpclient:4.5.13') { force = true }
classpath('com.google.http-client:google-http-client:1.38.1') { force = true }
classpath('com.google.http-client:google-http-client-apache-v2:1.38.1') { force = true }
...
}
}Running |
|
@paulolieuthier @daxroc @tburch has anyone tried the workaround with all three version sets? It's a frequent Gradle issue that a Gradle build pulls in unintended dependency versions, and based on the observations so far, it still looks to me that it's a Gradle project configuration issue. |
|
Closing due to inactivity. Based on the evidence so far, this doesn't seem like a regression. I'm suspecting a Gradle project dependency resolution issue, which is not uncommon. For those who came here, the workaround in #3058 (comment) should work and allow you to move forward, |
|
I'm experiencing this with the buildscript {
repositories {
mavenCentral()
}
configurations.classpath {
resolutionStrategy {
force("org.apache.httpcomponents:httpcore:4.4.13")
force("org.apache.httpcomponents:httpclient:4.5.13")
force("com.google.http-client:google-http-client:1.38.0")
force("com.google.http-client:google-http-client-apache-v2:1.38.0")
}
}
}If it's helpful, without trying to force the versions the build environment dep graph is: buildEnvironment |
|
@mattupstate seems like you tried 2.7.1 dependencies. What about 2.7.0 dependencies? The issue reported here is that 2.7.0 had been working, but it stopped working since 2.7.1. |
|
@chanseokoh 🤦 thanks for that tip. That did the trick 🙇 |
|
Hmm... but that implies that Google HTTP Client dependencies above 2.7.0 do have a problem. I expected that BTW, for a multi-module Gradle project, the project should be set up in the way explained here to avoid using in unintended versions. That is, defining all of your plugins in the base project while applying them selectively in your subprojects as needed. |
|
Just to note that we have the same issue with the 3.1.1 release. Downgrading the Google HTTP Client dependencies to 2.7.0 as per above works for us as well. |
|
I'm using jib:3.1.2 with Kotlin and the code bellow solved for me. Thanks @mattupstate and @chanseokoh . |
|
@daxroc @paulolieuthier @tburch @adamthalhammer @mattupstate @edgarvonk @evandroabukamel FYI, we released Jib plugins 3.1.3 after upgrading Google HTTP Client to 1.39.2. However, 1.39.2 doesn't seem to change the Apache HTTP Client versions from what they where in Jib 2.8.0 (which used Google HTTP Client 1.38.1), so I'm not too optimistic. Nonetheless, I wonder if 1.39.2 resolves the issue. Can anyone test out Jib 3.1.3? Also test forcing the following versions by Jib 3.1.3. |
|
Tested with jib 3.1.3 without forcing dependencies, got the same problem. |
This was referenced Aug 12, 2021
|
Re-opening as the root cause is still unknown. |
|
@daxroc @racevedoo @paulolieuthier @tburch @adamthalhammer @mattupstate @edgarvonk @evandroabukamel So we decided to downgrade Google HTTP libraries to 1.34.0. We just released Jib plugins 3.1.4 with the downgraded versions, which should avoid the issue (unless the libraries are still overridden by your Gradle project setup). To signal that the latest Jib versions shouldn't hit this issue, I'll close this. #3416 is open for tracking. |
|
I'm still seeing this issue on 3.1.4 |
|
@bplotnick 3.1.4 definitely pulls in HTTP library versions of Jib 2.7.0, so it must be that your project setup pulls in newer HTTP versions in your build environment. If that is the case, then for now, unfortunately you'll have to continue forcing 2.7.0 dependencies as above. |
|
In my case, 3.1.4 worked fine. Thanks 😄 |
|
Thanks for the updates, folks. Downgrading HTTP libraries is only a temporary and incomplete solution (Gradle can still override old versions), so we think eventually this should be resolved. Is there anyone who can consistently reproduce this issue (preferably a small sample you can share) and is willing to spend a good amount of their time to help us debugging this issue? If so, please contact us through our Gitter channel: https://gitter.im/google/jib |
|
I suspect it is only Amazon that has this problem. Specifically, Amazon E3 ( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On upgrade to 2.7.1 a dependency change breaks host verification for docker pull
Change seen within the http request trace
This is possibly a host verification regression.
Environment:
Description of the issue:
Pulling of image fails on hostname verification
Plugin version and jib configuration required
plugins { id("com.google.cloud.tools.jib") version "2.7.1" // 2.7.0 works } jib { to.image = "${project.name}-lambda" from.image = "amazon/aws-lambda-java:11" }Log output:
Additional Information:
There was a similar verification issue encountered with apache-httpcomponents
The text was updated successfully, but these errors were encountered: