Docker image generation issues

[gclayburg]

Overview of the issue

I have been using jhipster for a while now and love the project. There are certainly top-notch integration techniques with many technologies here.

However, I noticed a few issues with the way docker images are generated. I have fixed these myself and I'm curious if this project might like to use some of these as a default. Here are the issues I ran into:

• The app running in docker cannot respond to OS signals like TERM or QUIT. The impact is that running a docker stop command will take a little longer to shutdown the app while it times out and sends a KILL signal. This means that your app will not get a chance to run any Spring @Predestroy Java beans and your app may not shutdown correctly.
• The entire app war file is added as a single docker layer. This can lead to slower build times and slower docker image pushes and pulls. This is especially true when you need to push or pull from a remote repository. If you change one line of code, the generated docker image layer will be dozens of MBs, even though most of it is unchanged from the previous build.
• The docker image does not allow command line options like this:

$ docker run synconsole:latest --spring.data.mongodb.uri=mongodb://syncuser:qwerty1234@yale:27017/synconsoledev

You can however pass environment variables with the docker --env flag.

• The docker image runs as the root user inside the container. Applications are somewhat more secure when running as a normal unprivileged user.
• The FROM line in the Dockerfile uses ':8-jre-alpine' as the version tag. Currently, this tag is the same as ':8u151-jre-alpine' on Docker Hub, but chances are it will change as it has in the past when new versions are released. This can lead to non-deterministic builds if your local docker installation already has an older ':8-jre-alpine'

Motivation for or Use Case

Reproduce the error

Related issues

Suggest a Fix

See follow-up below

JHipster Version(s)

4.14.4 using gradle

JHipster configuration

Entity configuration(s) entityName.json files generated in the .jhipster directory

Browsers and Operating System

• [x ] Checking this box is mandatory (this is just to show you read everything)

[gclayburg]

I have fixed this issues in my project with this alternate file gradle/docker.gradle

buildscript {
    repositories {
        jcenter()
        maven {
            url "https://plugins.gradle.org/m2/"
        }
    }

    dependencies {
        classpath 'com.bmuschko:gradle-docker-plugin:3.2.0'
        classpath 'gradle.plugin.com.garyclayburg:dockerPreparePlugin:1.3.2'
    }
}

apply plugin: com.bmuschko.gradle.docker.DockerRemoteApiPlugin

import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage

apply plugin: com.garyclayburg.docker.DockerPreparePlugin
dockerprepare {
    dockerSrcDirectory "${project.rootDir}/src/main/dockerlayer"
    commonService = ['org.springframework.boot:spring-boot-starter-web']
}

task buildDocker(type: DockerBuildImage, dependsOn: 'dockerLayerPrepare') {
    springBoot.executable = false
    description = "Package application as Docker image"
    group = "Docker"
    inputDir = project.file(dockerprepare.dockerBuildDirectory)
    tags = ["synconsole:latest".toString(), "synconsole:${project.version}".toString()]
}

The docker tags used here are particular to my project, but the rest is fairly generic. It should be usable in any gradle based jhipster project.

A few things about this solution:

• All of the above issues are fixed
• This uses a gradle plugin I created to do most of the work such as allowing the spring boot app to use the docker cache. I am not aware of a maven equivalent build plugin that does this.
• This configuration overrides the Dockerfile from src/main/docker/Dockerfile. The actual Dockerfile and start script used comes from the plugin itself. Check it out on github
• This version doesn't have any jhipster specific things in it (like $JHIPSTER_SLEEP)

So what do you guys think, would something like this be useful to the project?

[PierreBesson]

Thanks for the feedback ! Please let us some time to review your suggestion. Just a few remarks that I can do upfront :

I like that you are trying to solve the "big layer" problem inherent in the fact that we are using fat jars. I have thought of a few ways to fix this using for example the spring-boot-thin-wrapper (but I never implemented it yet). However I would much rather prefer if the solution would use an existing Dockerfile so that it can be customized by the user, even if this dockerfile is actually modified in the build process.

Also I would much rather prefer a solution that would work for both gradle and maven and keep it simple and stable.

Concerning running the process inside docker as a user other than root, yes we would welcome improvements on this part, remember that some of it was first implemented over 2 years ago !

As for being able to pass command line args, it's would be a nice thing to have but I personally prefer to use env variables.

Thanks again for the feedback, I wished more people would reach out to us with suggestions like this !

[pascalgrimaud]

Thanks for your feedback @gclayburg. Here some comments:

• about KILL signal: what do you suggest to improve it ? I think you can have the same problem with or without Docker, if on the machine, someone decided to stop the application with kill -9. The best would be to use Spring Boot Actuator endpoints.

• I don't see 'big layer', I can see 5 layers, and compressed size at 105MB (https://hub.docker.com/r/jhipster/jhipster-sample-app/tags/) -> it's fine for me

• as Pierre, a common Dockerfile for both Gradle and Maven is simplier for us, although we already depends on external plugin, we try to keep it simple

• root user: PR is welcome on this part, so yes, it can be improved

• base image: PR is welcome too. We should use a specific version

[PierreBesson]

@gclayburg Apparently creating layered images is currently under investigation by the spring boot team: spring-projects/spring-boot#12545
So maybe you could share your experience upstream to work on a global solution to the problem.

[gclayburg]

You are right about KILL, @pascalgrimaud. The way it is right now, if you run a docker stop command, docker ends up sending a kill -9 to the app and a clean shutdown is not possible. Of course, many apps may not notice this if they don't have any particular special @PreDestroy shutdown hooks they must execute. For those that do, it is a big deal.

One way to fix it to do something like this in the Docker file:

ENTRYPOINT ["./bootrunner.sh"]

and this at the end of bootrunner.sh:

  exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -jar "${HOME}/app.jar" "$@"

This does 2 things actually - it allows us to pass command like arguments to the docker image being run and it allows OS signals to reach the JVM. This is important if you are stopping the container with the docker tools or other higher level tools like kubernetes.

There are some cases where command line options to the app work better than environment variables. For my app,I ran into an issue back with spring boot 1.4.x where certain parameters were not parsed consistently by Spring. I could get it to work using the command line arguments, but not with environment variables. BTW, the Spring team has since fixed that issue with Spring Boot 1.5

If you don't do the exec at launch time, docker will run a shell to execute the app. This shell will not forward OS signals to the application. The way I understand it, docker sends signals to PID 1 within the container only. Most of the time PID 1 is the shell that forks the app. After we do the exec ourselves. the JVM becomes PID 1 and is then joins the docker party. Shutdowns are cleaner and faster.

The full bootrunner.sh script I use works with a regular jar or war created from maven or gradle as well as a layered jar/war created from my dockerprepare gradle plugin. It looks like this:

https://github.com/gclayburg/dockerPreparePlugin/blob/master/src/main/resources/defaultdocker/bootrunner.sh

#!/bin/sh
date_echo(){
    datestamp=$(date "+%F %T")
    echo "${datestamp} $*"
}
#exec the JVM so that it will get a SIGTERM signal and the app can shutdown gracefully

if [ -d "${HOME}/app/WEB-INF" ]; then
  #execute springboot expanded war, which may have been constructed from several image layers
  date_echo "exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -cp ${HOME}/app org.springframework.boot.loader.WarLauncher $*"
  # shellcheck disable=SC2086
  exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -cp "${HOME}/app" org.springframework.boot.loader.WarLauncher "$@"
elif [ -d "${HOME}/app" ]; then
  #execute springboot expanded jar, which may have been constructed from several image layers
  date_echo "exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -cp ${HOME}/app org.springframework.boot.loader.JarLauncher $*"
  # shellcheck disable=SC2086
  exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -cp "${HOME}/app" org.springframework.boot.loader.JarLauncher "$@"
elif [ -f "${HOME}/app.jar" ]; then
  # execute springboot jar
  date_echo "exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -jar ${HOME}/app.jar $*"
  # shellcheck disable=SC2086
  exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -jar "${HOME}/app.jar" "$@"
else
  date_echo "springboot application not found in ${HOME}/app or ${HOME}/app.jar"
  exit 1
fi 

And yes all but the layering issue can be addressed with a simple replacement default Dockerfile and bootrunner.sh. I could see about creating a PR for you if it would help.

[gclayburg]

About the layering issue, I understand you would prefer to share a single Dockerfile between the maven and gradle options. I think that could be setup with some modifications to the Dockerfile and bootrunner.sh so that it could execute the app from a single war file if that is present in the docker image or from the layers created from the gradle dockerprepare plugin. That way the app would be stored more efficiently in the docker image when built with gradle, but otherwise act the same as if it were built with maven. I'll look into that.

[PierreBesson]

I have just done some testing using https://github.com/dsyer/spring-boot-thin-launcher and I am able to create a layered build although with only 2 layers: /repository which is a local maven repository with all the dependencies and the thin jar with the current application class. Compared to your approach is not as fine grained but it could be a solution.

[gclayburg]

I wonder how well that kind of solution would work over time in a docker environment. As the app evolves, dependencies will get upgraded. Wouldn't that lead to a docker image that has lots of unused old dependencies? Once the docker layers are added, the files in there are always there.

[PierreBesson]

I don't think so. Anyway you will need to clean your build dir when you rebuild the Docker image otherwise it might increase in size every time you change dependencies.

[gclayburg]

Ok, I created a Dockerfile and bootrunner.sh to fix these issues that will also work with both maven and gradle. I put them in a vanilla spring boot starter project here

These files will also work in a jhipster generated project if they are placed in src/main/docker

I'm not sure if it makes sense or not, but I had to play some games with the Dockerfile and emptyfile.txt since it doesn't really support conditional ADD commands.

[PierreBesson]

I have started implementing things. For now I have not added the following:

• use a fixed docker image version: because it is convenient to have the version auto updated for us.
• layered build: let's think this through and add this later.

[FireWolf2007]

Guys, sorry about some off topic.
Recently I'm discovered some old solutions in build.gradle related to the docker.
Project used plugin io.spring.dependency-management with version 1.0.3.RELEASE
It's ok for io.spring.dependency-management 1.X, but it's obsolete for io.spring.dependency-management 2.X and it's template not worked in newer plugin version (Reference https://docs.spring.io/spring-boot/docs/2.0.2.RELEASE/gradle-plugin/reference/html/).

plugins {
    id "io.spring.dependency-management" version "1.0.3.RELEASE"
}
...
bootRepackage {
   mainClass = 'ru.inquarta.prozvon.server.ProzvonApp'
}

springBoot {
    mainClass = 'ru.inquarta.prozvon.server.ProzvonApp'
    executable = true
    buildInfo()
}

it's replaced with more simple construction without additional version definition in pugins section:

bootWar {
     mainClassName = 'ru.inquarta.prozvon.server.ProzvonApp'
     launchScript()
}

May be in upcoming 5 version update this section?
I'll ready to make PR, or may be topic starter make this modification?

[PierreBesson]

I have implemented many of the suggested improvements so I'm closing this issue. If additional improvements to our docker build are needed in the future, please open other feature requests. @gclayburg I'm sorry I don't think we can base our gradle build on your plugin for now even though I recognize that it's quite nice !