Multistage build docker #115

mattwr18 · 2019-01-03T22:06:33Z

fixes #93

roschaefer · 2019-01-04T00:29:12Z

.travis.yml

-  - docker build --build-arg BUILD_COMMIT=$TRAVIS_COMMIT -t humanconnection/nitro-web .
-  - docker-compose -f docker-compose.yml up -d
+  - docker build --build-arg BUILD_COMMIT=$TRAVIS_COMMIT --target production -t humanconnection/nitro-web .
+  - docker-compose -f docker-compose.travis.yml up -d


Suggested change

- docker-compose -f docker-compose.travis.yml up -d

- docker-compose -f docker-compose.yml up -f docker-compose.travis.yml up -d

@mattwr18 I suspect this is the reason for the failing build. Without the default docker-compose.yml frontend and backend are not joining the same network and cannot reach each other. Also we wait-on http://localhost:3000 later in .travis.yml and if we don't expose port 3000 we will wait forever.

Great, I'll give this a go! I am exposing 3000 on line 34... although I believe it is waiting cause it is not spinning up properly, although, it is failing silently

Are you sure? I believe with docker-compose you have to expose all the ports in the .yml file explicitly. (Although I never looked it up)

roschaefer · 2019-01-04T00:29:55Z

docker-compose.travis.yml

+
+services:
+  backend:
+    image: humanconnection/nitro-web:builder


Suggested change

image: humanconnection/nitro-web:builder

I think it's in the base docker-compose.yml so not needed here.

Do you know the reason for having yet another docker-compose.yml just for travis? Because of shared folders. Travis struggles to delete them after running the tests, thus silently failing our deployments. The reason why Travis cannot delete the files is that the docker user on Travis creates the files, not the user who is later trying to delete them.

roschaefer · 2019-01-04T17:09:03Z

.travis.yml

@@ -19,7 +19,7 @@ before_install:

 install:
  - docker build --build-arg BUILD_COMMIT=$TRAVIS_COMMIT --target production -t humanconnection/nitro-web .
-  - docker-compose -f docker-compose.yml -f docker-compose.travis.yml up -d
+  - docker-compose -f docker-compose.yml up -f docker-compose.travis.yml up -d


no no, the up is there already 😉 @mattwr18

ahh, right... I thought I had missed it yesterday from your review above... it's still failing the build though

roschaefer · 2019-01-04T17:09:59Z

@mattwr18 great work, can you tell me how large the docker image is on your machine after this PR?

mattwr18 · 2019-01-04T17:24:56Z

@mattwr18 great work, can you tell me how large the docker image is on your machine after this PR?

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
<none>                          <none>              c94475d09e1e        6 seconds ago       318MB

A little bigger than a third of the size it was!!
Do you understand why the Repository and Tag are ? when I first built the image from the Dockerfile in the master branch it looked like humanconnection/nitro-web latest 4a714bb66e4c 21 hours ago 902MB

roschaefer · 2019-01-04T17:46:13Z

@mattwr18 yes, you can add a repository and a tag. See our build server and the deploy script:

docker build [...] -t humanconnection/nitro-web:latest .

adds the repository and

docker push humanconnection/nitro-web:latest

does the same for docker push.

mattwr18 · 2019-01-04T17:50:36Z

Output from build, command docker-compose -f docker-compose.yml -f docker-compose.travis.yml up -d

...
Creating nitro-web_webapp_1
...

but unfortunately still getting this output

docker-compose exec webapp yarn run lint
No container found for webapp_1
The command "docker-compose exec webapp yarn run lint" exited with 1.

mattwr18 · 2019-01-04T18:05:34Z

I've run the exact same commands as from the build locally and they have passed...

@mattwr18

@mattwr18 ouch, that was a hard to spot error. It struck my eye when I was switching back and forth between `docker-compose.override.yml` and `docker-compose.travis.yml`.

mattwr18 · 2019-01-04T18:29:39Z

ahhh duh!! sorry 😛

This should fix our build server. Probably the container immediately exited on Travis which is using the `builder` stage for testing.

mattwr18 · 2019-01-04T21:50:29Z

Size of the production image now

humanconnection/nitro-web       latest              aa9218a28cfd        5 minutes ago       319MB

Needed to run tests

appinteractive · 2019-01-04T22:59:41Z

Why not just copy all into the build image and trusting the dockerignore file? @mattwr18

mattwr18 · 2019-01-04T23:20:14Z

that's a good idea @appinteractive, right now we were just concerned with the minimum to get it passing the build, but we can definitely explore that route.

Currently, it's unexpectedly failing the build because of many linting errors, which we are not able to reproduce locally.
https://travis-ci.com/Human-Connection/Nitro-Web/builds/96354499#L1006

appinteractive · 2019-01-04T23:23:12Z

Yeah that’s because of failing config files for the Linter. Please try to copy the whole root dir and trust the dockerignore. After that you just copy the needed files to the production image.

That way the build will be more solid to change imho.

Dockerfile

cypress/integration/common/steps.js

appinteractive · 2019-01-08T19:02:48Z

Dockerfile

+FROM base as production
+ENV NODE_ENV=production
+COPY --from=build-and-test ./nitro-web/node_modules ./node_modules
+COPY --from=build-and-test ./nitro-web/plugins ./plugins


Why do you copy the plugins and static files over when you already copied everything in the base image?

Good point, carry over from before your above suggestion.

…3-minimize-docker-image-size-with-multi-stage-build

mattwr18 · 2019-01-09T22:48:17Z

@appinteractive @roschaefer I have gotten this build to pass reliably again and cleaned it up even more in the process, taking into account your suggestions. It is not an ideal situation, however.

The reason it was failing the build before @roschaefer is because for some reason the commands docker-compose exec webapp yarn run lint and yarn run test are being run from the production image, which after having removed the copy node_modules from build and test, does not include the devDependencies jest or eslint necessary to run the commands.

This is also the reason I needed to add the -e NODE_ENV=test... I was having a look at how the docker-compose -f docker-compse.yml -f docker-compose.travis.yml up -d command works and it seems like the Docker docs encourage to have a separate yml for production. See https://docs.docker.com/compose/extends/#different-environments and https://docs.docker.com/compose/production/ @roschaefer

I give up now after numerous attempts to reduce the docker image size. The only thing left is that I've put the NODE_ENV=.. configuration in the docker-compose.X.yml files, I think that's slightly cleaner

@mattwr18

This reverts commit 6b4601e. @mattwr18 for reasons I don't understand, our build server hangs https://travis-ci.com/Human-Connection/Nitro-Web/builds/96979338 Honestly, I start to think this is a bug in `docker-compose`

…tage-build

roschaefer · 2019-01-11T00:31:41Z

For the record:

roschaefer · 2019-01-11T11:31:14Z

@mattwr18 I'm done, I believe this is a bug with docker-compose.yml. My latest reverted change is an equivalent refactoring. I don't understand why the build was failing. Nevertheless, I think this PR is an improvement. I'll merge

done

roschaefer · 2019-01-11T12:34:55Z

@mattwr18 🎉

mattwr18 · 2019-01-11T12:56:49Z

Wow!! what happened!? I thought it was around 315MB, still an big improvement, but nothing the 114MB!!!
This is great!! I want to get more experience with this by multi-staging all the Dockerfiles I helped create!! Thanks for your help @appinteractive @roschaefer @ksinkar :)

roschaefer · 2019-01-11T13:06:23Z

@mattwr18 I assume it comes from the node_modules and the built files in styleguide that are not included. I still believe it can be compressed even more by not including the development dependencies. But after spending hours of try/error and being confused by the behaviour of docker-compose I leave that for some point in the future.

mattwr18 · 2019-01-11T13:26:10Z

Did you have a look at the links I shared with you here about the use of docker-compose in production?

…eo4j Import remote mongodb in neo4j

Multistage build docker

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

5778ef3

pr-triage bot added the PR: unreviewed label Jan 3, 2019

mattwr18 added 2 commits January 3, 2019 20:22

Configure docker-compose, travis multistage build

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

e88b674

Copy plugins from builder

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

c575565

roschaefer reviewed Jan 4, 2019

View reviewed changes

mattwr18 added 2 commits January 3, 2019 22:37

Link docker-compose to docker-compose.travis

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

744362a

Add missing up in docker-compose up

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

014392a

roschaefer reviewed Jan 4, 2019

View reviewed changes

Revert docker-compose up command in travis.yml

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

b8fb1bd

roschaefer and others added 3 commits January 4, 2019 19:52

Add folder /server to docker stage production

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

9541b81

Refactor multistage build names, copy locales in base

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

3e3065c

mattwr18 added 4 commits January 4, 2019 20:12

Initialize eslint

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

40aedb1

Copy eslint config, test files

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

11a7f0a

Copy components and store, remove in production

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

beac045

Needed to run tests

Fix lint issues

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

9f8a292

roschaefer reviewed Jan 5, 2019

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

appinteractive previously requested changes Jan 5, 2019

View reviewed changes

Dockerfile Show resolved Hide resolved

Dockerfile Outdated Show resolved Hide resolved

Dockerfile Outdated Show resolved Hide resolved

cypress/integration/common/steps.js Outdated Show resolved Hide resolved

appinteractive reviewed Jan 8, 2019

View reviewed changes

mattwr18 added 2 commits January 8, 2019 20:39

Remove redundant copies, run yarn install in production image

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

39f581a

pr-triage bot added PR: unreviewed and removed PR: reviewed-changes-requested labels Jan 8, 2019

mattwr18 added 7 commits January 9, 2019 18:54

Copy over styleguide/dist

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

1a234e7

Fix copy path

Verified

This commit was signed with the committer’s verified signature.

epdenouden Ewout Pieter den Ouden

GPG key ID: BD2A2FF1CC41549E

Verified
Learn about vigilant mode

cca0941

Add copy everything in build and test

0acca8c

Add back static and plugins copy from build and test

19a298d

Revert to passing build

b171074

Remove un-needed copies

6948c14

Merge branch 'master' of github.com:Human-Connection/Nitro-Web into 9…

bae7b7b

…3-minimize-docker-image-size-with-multi-stage-build

roschaefer force-pushed the 93-minimize-docker-image-size-with-multi-stage-build branch from a96221f to 6b4601e Compare January 10, 2019 23:54

roschaefer and others added 3 commits January 11, 2019 00:54

Try to help @mattwr18

6b4601e

I give up now after numerous attempts to reduce the docker image size. The only thing left is that I've put the NODE_ENV=.. configuration in the docker-compose.X.yml files, I think that's slightly cleaner

Revert "Try to help @mattwr18"

521a868

This reverts commit 6b4601e. @mattwr18 for reasons I don't understand, our build server hangs https://travis-ci.com/Human-Connection/Nitro-Web/builds/96979338 Honestly, I start to think this is a bug in `docker-compose`

Merge branch 'master' into 93-minimize-docker-image-size-with-multi-s…

8ee8839

…tage-build

roschaefer approved these changes Jan 11, 2019

View reviewed changes

roschaefer merged commit 9825b41 into master Jan 11, 2019

roschaefer deleted the 93-minimize-docker-image-size-with-multi-stage-build branch January 11, 2019 11:31

pr-triage bot added PR: merged and removed PR: unreviewed labels Jan 11, 2019

roschaefer pushed a commit that referenced this pull request Mar 20, 2019

Merge pull request #115 from Human-Connection/100_import_mongodb_to_n…

c0dc7e3

…eo4j Import remote mongodb in neo4j

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multistage build docker #115

Multistage build docker #115

mattwr18 commented Jan 3, 2019

roschaefer Jan 4, 2019

mattwr18 Jan 4, 2019

roschaefer Jan 4, 2019

roschaefer Jan 4, 2019

roschaefer Jan 4, 2019

roschaefer Jan 4, 2019

mattwr18 Jan 4, 2019

roschaefer commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

roschaefer commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

appinteractive commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

appinteractive commented Jan 4, 2019 •

edited

Loading

appinteractive Jan 8, 2019

mattwr18 Jan 8, 2019

mattwr18 commented Jan 9, 2019

roschaefer commented Jan 11, 2019

roschaefer commented Jan 11, 2019 •

edited

Loading

roschaefer commented Jan 11, 2019

mattwr18 commented Jan 11, 2019

roschaefer commented Jan 11, 2019

mattwr18 commented Jan 11, 2019

	- docker-compose -f docker-compose.travis.yml up -d
	- docker-compose -f docker-compose.yml up -f docker-compose.travis.yml up -d

Multistage build docker #115

Multistage build docker #115

Conversation

mattwr18 commented Jan 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roschaefer commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

roschaefer commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

appinteractive commented Jan 4, 2019

mattwr18 commented Jan 4, 2019

appinteractive commented Jan 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattwr18 commented Jan 9, 2019

roschaefer commented Jan 11, 2019

roschaefer commented Jan 11, 2019 • edited Loading

roschaefer commented Jan 11, 2019

mattwr18 commented Jan 11, 2019

roschaefer commented Jan 11, 2019

mattwr18 commented Jan 11, 2019

appinteractive commented Jan 4, 2019 •

edited

Loading

roschaefer commented Jan 11, 2019 •

edited

Loading