Add docker support #2272

fazlerabbi37 · 2019-06-19T22:09:14Z

Docker is one of the most popular container platform that makes it easier to build and deploy application. This pull request will add docker support for development setup of openstreetmap-website.

This PR is build on top of an old abandoned PR (#1290) and follows the Installation guide provided with this project.

Please keep in mind that I am just starting out with Docker and feel free to suggest and discuss anything that comes to mind. I will try my best to answer them. 😄

Todo:

make docker file for the rails app
make docker file for the PostgreSQL database
make a docker-compose file to tie up both containers
fix deployment when docker files are on docker directory
fix rake test:db error
populating the database
documentation
configure an unprivileged user

added Dockerfile for the ruby on rails app inside a separate docker directory. this will build a image from the official ruby (specifically ruby:2.5-slim).

…rror postgresql-client throws a dpkg error when it can't create a softlink in for man page and aborts installation. found a solve at stack overflow and modified Dockerfile to add a comment to mention the problem with slove source.

pnorman · 2019-06-19T22:23:12Z

docker/Dockerfile

+#install packages
+#fixes dpkg man page softlink error while installing postgresql-client [source: https://github.com/debuerreotype/debuerreotype/issues/10#issuecomment-438342078]
+RUN mkdir -p /usr/share/man/man1 && mkdir -p /usr/share/man/man7
+RUN apt-get update && apt-get install curl -y


Each time apt-get is run you should update and then remove the APT cache files at the end of the command.

So that the cache files don't make it into the docker image

Isn't this just building a container for local use though? Not anything that's going to be published?

Sure, but what I've read indicates it as a best practice in general

pnorman · 2019-06-19T22:24:06Z

docker/Dockerfile

+RUN mkdir -p /usr/share/man/man1 && mkdir -p /usr/share/man/man7
+RUN apt-get update && apt-get install curl -y
+RUN curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh && bash nodesource_setup.sh && rm nodesource_setup.sh
+RUN apt-get install -y --no-install-recommends ruby-dev libarchive-dev libmagickwand-dev libxml2-dev libxslt1-dev build-essential libpq-dev libsasl2-dev imagemagick libffi-dev locales postgresql-client-9.6 nodejs && apt-get clean && rm -rf /var/lib/apt/lists/*


The rm -rf /var/lib/apt/lists/* doesn't reclaim the space since the space is still used by earlier layers of the dockerfile.

This all looks horribly wrong anyway - why is is creating directories by hand?

And downloading and installing node from upstream using a dodgy "run this shell script as root" install method isn't going to fly. Why not just use the ubuntu packaged one?

This all looks horribly wrong anyway - why is is creating directories by hand?

as I have mentioned on the comment above the command, while installing postgresql-client, because update-alternatives can't create symbolic link, dpkg throws an error and exits with code 100. A quick search pointed to this stack answer which solves the problem.

And downloading and installing node from upstream using a dodgy "run this shell script as root" install method isn't going to fly. Why not just use the ubuntu packaged one?

the image ruby:2.5-slim usages Debian 9 (stretch) as base image. Though I could install nodejs using the office Debian repo , I couldn't install npm using the apt-get command as it gives E: Unable to locate package npm error. I followed the official download instruction which leads to NodeSource's GitHub instruction and installed nodejs and npm.

If there is any better way to solve this, I would be happy to hear. 😊

If there is any better way to solve this, I would be happy to hear. blush

I'm also not a Docker expert, but it might be worth considering starting from a different base image, like Ubuntu 18.04, rather than starting from a ruby base image. This would more closely match the installations instructions, and also the Vagrant build.

I'm also not a Docker expert, but it might be worth considering starting from a different base image, like Ubuntu 18.04, rather than starting from a ruby base image.

Neither am I. As far as I understand the purpose of using a ruby image it to remove the legwork required to install ruby on an Ubuntu base image. Maybe someone with better docker understanding can explain it better? ping @pnorman

I am building on top of the previous PR (#1290) and at first I wanted to see if I can build the web and db images using docker with the new code because the code base and the installation procedure has changed since that PR. If we all agree on the pros, we can change the base image to Ubuntu 18.04.

pnorman · 2019-06-19T22:25:17Z

docker/Dockerfile

+RUN apt-get update && apt-get install curl -y
+RUN curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh && bash nodesource_setup.sh && rm nodesource_setup.sh
+RUN apt-get install -y --no-install-recommends ruby-dev libarchive-dev libmagickwand-dev libxml2-dev libxslt1-dev build-essential libpq-dev libsasl2-dev imagemagick libffi-dev locales postgresql-client-9.6 nodejs && apt-get clean && rm -rf /var/lib/apt/lists/*
+RUN npm install yarn -g


Docs state bundle exec rake yarn:install. If there's a reason to differ, can you add a comment?

Different thing - you need to install yarn first (unfortunately it's not packaged in Ubuntu like it is in Fedora) and then bundle exec rake yarn:install runs yarn to install the modules listed in package.json.

mmd-osm · 2019-06-20T04:25:50Z

It may be worthwhile taking a look at: https://github.com/developmentseed/osm-seed

gravitystorm · 2019-06-20T08:05:54Z

This pull request will add docker support for development setup of openstreetmap-website.

While I'm happy to add Docker support, I feel like we're ending up with a lot of duplication in installation instructions. We have the human-readable INSTALL.md instructions, the travis config, the Vagrant provisioning scripts and now a Dockerfile - all of these cover similar ground.

Is there any way that we can simplify things, or at least reduce some of the duplication? It's becoming increasingly hard to make sure these are all in line when we make changes.

…ed jobs from bundle install modified Dockerfile to separate apt-get cache cleaning command, added a comment explaining why and how npm is added and removed --jobs flag from bundle install command

added docker_postgres.sh for PostgreSQL extension and functions that installs PostgreSQL Btree-gist Extension, compiles libpgosm.so and installs maptile_for_point, tile_for_point and xid_to_int4 functions

added Dockerfile.postgres for the postgres database app. this will build a image from the official postgres (specifically postgres:9.4) image.

Firefishy · 2019-06-20T12:03:22Z

Please give me some time to review this over the weekend. I have a very good understanding of docker.

fazlerabbi37 · 2019-06-21T14:35:37Z

So it would seem I have hit a wall. 😞 I can build the application and run database migrations and nodejs package installation with yarn when the docker files are on the root directory but the database test command fails giving the following error:

The HashDiff constant used by this gem conflicts with another gem of a similar name.  As of version 1.0 the HashDiff constant will be completely removed and replaced by Hashdiff.  For more information see https://github.com/liufengyun/hashdiff/issues/45.
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

However, when the docker files are moved to docker directory the docker-compose build command fails with docker_web_1 exited with code 10 error and I can't find any reference of the error code. 😶

I have changed the context when I moved the docker file to docker directory. If anyone can shade some light on this, would appreciate it. 😄

gravitystorm · 2019-06-21T14:53:38Z

database test command fails giving the following error:

This is probably because the default settings for connecting to the database are to use a Unix domain socket, whereas I suspect the Docker config is set up to use networking between the containers. Unfortunately I can't provide any guidance here as to what is best from a Docker point of view!

gravitystorm · 2019-06-21T14:57:54Z

There's a few commandline things which are pretty important to having a useful development environment, such as:

Running the test suite
Running rake commands (e.g. for database migrations)
Running the console
Connecting to the database with psql

I'd like to make sure all of these things are possible in the Dockerized development setup, since at the moment I only see mention of running the webserver. With Vagrant, for example, you can just vagrant ssh into the virtual machine and do all of these things.

If they are possible already (great!) then it would be worthwhile documenting how that works, since it's not obvious to me at least!

fazlerabbi37 · 2019-06-22T10:18:03Z

ok I have some good news. I was able solve the deployment problem mentioned on my previous comment. I forgot to change the volume mount point when I moved the docker-compose.yml to docker directory from root directory and docker was unable to find the files necessary. Silly mistake ruined my weekend 😶. But unfortunately haven't made any progress with rake test:db error yet.

There's a few commandline things which are pretty important to having a useful development environment, such as:

Running the test suite
Running rake commands (e.g. for database migrations)
Running the console
Connecting to the database with psql

I'd like to make sure all of these things are possible in the Dockerized development setup, since at the moment I only see mention of running the webserver. With Vagrant, for example, you can just vagrant ssh into the virtual machine and do all of these things.

@gravitystorm they all are possible but for some reason I have setback regarding the rake test:db command. Might have something to do with the UNIX socket vs Docker tpc connection as you have mentioned in your comment. And yes you can also use bash on all of them.

If they are possible already (great!) then it would be worthwhile documenting how that works, since it's not obvious to me at least!

I just went through the old PR (#1290) and found that @paulsturgess made a nice doc for docker usage. Will try to re-use/re-purpose that and push it asap. Cheers!

mmd-osm · 2019-06-22T10:22:41Z

The developmentseed repo I mentioned earlier in this issue uses TCP connections instead of unix sockets to connect to he postgresql database.

Also, check out https://github.com/hadolint/hadolint to get some recommendations on your Docker file, based on best practices.

fazlerabbi37 · 2019-06-22T10:28:54Z

The developmentseed repo I mentioned earlier in this issue uses TCP connections instead of unix sockets to connect to he postgresql database.

come to think of it, wouldn't that UNIX socket vs Docker tpc connection issue be applicable for rake db:migrate command as well? As far as I understand it also connects to the db PostgreSQL image and creates tables and function and it works ok. If so why it fails only on rake test:db. While I investigate that may be @gravitystorm and @tomhughes can help me understand the process better!?

robbyoconnor · 2019-07-13T04:12:47Z

You should also configure an unprivileged user for the Docker images.

fazlerabbi37 · 2019-07-21T14:38:56Z

You should also configure an unprivileged user for the Docker images.

hi @robbyoconnor. I don't know a lot about unprivileged user in docker but saw bundle complaining about it being running as root so thanks for the unprivileged user creation pointer. I am still stuck with the basics ie. running the test properly and populating the database. So for now I will add this to do and try to finish it after the basic tasks are done.

note to self: This medium articles seems to have a good explanation. Might come handy.

robbyoconnor · 2019-07-21T18:22:28Z

@fazlerabbi37 here's the thing: Docker does have isolation HOWEVER as root -- you can escape it...in theory is it likely..probably not but I'm not willing to chance it. Always treat root in a docker container as equal to root on the host...you are root, you can escape namespaces.

…ntomjs via npm modified Dockerfile to changes postgresql-client package from postgresql-client-10 to postgresql-client, and added phantomjs installation via npm

mmd-osm · 2019-07-21T21:23:20Z

docker/Dockerfile.postgres

@@ -1,4 +1,4 @@
-FROM postgres:9.4
+FROM postgres:11



Production still uses 9.5 as of now.

tomhughes · 2019-07-21T21:24:42Z

Only because of the xid index issues though, which relate to how we do the diff feed. I use 11 for my development setup.

robbyoconnor · 2019-07-21T23:42:21Z

docker/docker-compose.yml

+  web:
+    image: openstreetmap-website:v1
+    build:
+      context: ..


Why not put the Dockerfiles in this directory?

Which directory? Right now all docker related files are in one directory named docker.

Oh wait, nevermind.

robbyoconnor · 2019-07-21T23:42:32Z

docker/docker-compose.yml

+  db:
+    image: openstreetmap-db:v1
+    build:
+      context: ..


Same here as above?

modified Dockerfile.postgres file to changes docker_postgres.sh location from db to docker

modified Dockerfile.postgres file to add app location setup and gem install via bundle

modified Dockerfile.postgres file to move database function directory from /db to /app/db

modified docker_postgres.sh to fix the location of libpgosm as it moved from /db/functions to /app/db/functions

modified Dockerfile to add osmosis package needed for populating database with osm extract

modified DOCKER.md to add osmosis instructions needed to use osmosis and populate database

modified Dockerfile.postgres to add ruby compilation from source as we need ruby=>2.4 and also removed bundle installation via gem install command because bundle is already compiled installed while ruby source is being compiled and installed.

fazlerabbi37 · 2019-09-10T09:31:54Z

I am really sorry for not being able to push any update since July but I have good news to share. Docker support is now ready for review. 🎉

/cc: @mmd-osm @robbyoconnor have a look. can't re-request review for you guys.

tiendat-trinh · 2019-09-24T09:19:31Z

I am really sorry for not being able to push any update since July but I have good news to share. Docker support is now ready for review.

/cc: @mmd-osm @robbyoconnor have a look. can't re-request review for you guys.

Hi @fazlerabbi37 , I'm a newbie in osm, I have tried you docker-compose file and I got the error

ActiveRecord::PendingMigrationError (

Migrations are pending. To resolve this issue, run:

    bin/rails db:migrate RAILS_ENV=development

):

Then, I have ran

docker exec -it openstreetmap-website /bin/bash -c "bin/rails db:migrate RAILS_ENV=development"

But, it's still error.

fazlerabbi37 · 2019-09-24T09:41:40Z

@goonux did you followed this instructions to build it? If not please give it a try. In the meantime I will try to do a fresh build with latest master branch code. 😊

tiendat-trinh · 2019-09-24T09:45:16Z

@goonux did you followed this instructions to build it? If not please give it a try. In the meantime I will try to do a fresh build with latest master branch code.

I'm sorry, I didn't read it yet, I will try now. Thanks.

…ap-website into docker

mmd-osm · 2019-10-17T05:47:48Z

TODO: create dedicated user & group to run the server, don't just use root.

fazlerabbi37 · 2019-10-19T11:04:27Z

TODO: create dedicated user & group to run the server, don't just use root.

@mmd-osm I am making very slow progress on this due to SotM Asia 2019. meanwhile, it would be great if you can point me towards a few resource on this topic.

jalessio · 2019-10-20T02:25:24Z

I took a run at improving on this PR by adding my own knowledge of Docker best practices along with some of the recommendations in this PR. I created PR #2406 which allows you to very quickly have the Rails app up and running using docker-compose.

Please have a look - thanks.

gravitystorm · 2019-10-23T15:17:53Z

Closing in favour of #2409 , feel free to reopen if necessary.

fazlerabbi37 added 2 commits June 20, 2019 03:51

Add Dockerfile for rails app

ad237cd

added Dockerfile for the ruby on rails app inside a separate docker directory. this will build a image from the official ruby (specifically ruby:2.5-slim).

pnorman reviewed Jun 19, 2019

View reviewed changes

fazlerabbi37 added 4 commits June 20, 2019 17:37

Modify Dockerfile to separate apt cache clean, npm comment, and remov…

4eeb36a

…ed jobs from bundle install modified Dockerfile to separate apt-get cache cleaning command, added a comment explaining why and how npm is added and removed --jobs flag from bundle install command

Add docker_postgres.sh for PostgreSQL extension and functions

caaa350

added docker_postgres.sh for PostgreSQL extension and functions that installs PostgreSQL Btree-gist Extension, compiles libpgosm.so and installs maptile_for_point, tile_for_point and xid_to_int4 functions

Add Dockerfile.postgres for postgres app

b6d373f

added Dockerfile.postgres for the postgres database app. this will build a image from the official postgres (specifically postgres:9.4) image.

Add docker-compose.yml to build the complete app

30ad01e

Firefishy self-requested a review June 20, 2019 12:01

fazlerabbi37 mentioned this pull request Jul 21, 2019

Invalid CSS when running tests #1931

Closed

fazlerabbi37 added 6 commits July 22, 2019 03:03

modify Dockerfile.postgres to fix script path

447c55e

modify Dockerfile.postgres to upgrade postgres image to 11

0cf0d0c

modify Dockerfile to change postgresql-client package and install pha…

18ef6b6

…ntomjs via npm modified Dockerfile to changes postgresql-client package from postgresql-client-10 to postgresql-client, and added phantomjs installation via npm

modify docker-compose.yml to fix context path

f75eea4

add DOCKER.md file with docker setup instructions

0d15f1c

add .dockerignore file to list ignorable file for docker

c4c450f

mmd-osm reviewed Jul 21, 2019

View reviewed changes

docker/Dockerfile.postgres

@@ -1,4 +1,4 @@

FROM postgres:9.4

FROM postgres:11

Copy link

Contributor

mmd-osm Jul 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Production still uses 9.5 as of now.

robbyoconnor reviewed Jul 21, 2019

View reviewed changes

fazlerabbi37 added 9 commits September 9, 2019 12:40

moved docker_postgres.sh to docker directory

ce38a10

modify Dockerfile.postgres file to change pgsql script location

b03bfd1

modified Dockerfile.postgres file to changes docker_postgres.sh location from db to docker

modify Dockerfile.postgres file to add app location and gem install

c6edb29

modified Dockerfile.postgres file to add app location setup and gem install via bundle

modify Dockerfile.postgres file to move db function directory

8383453

modified Dockerfile.postgres file to move database function directory from /db to /app/db

modify docker_postgres.sh to fix libpgosm location

cb96ddd

modified docker_postgres.sh to fix the location of libpgosm as it moved from /db/functions to /app/db/functions

modify Dockerfile to add osmosis

a1413af

modified Dockerfile to add osmosis package needed for populating database with osm extract

modify DOCKER.md to add osmosis instructions

02851d2

modified DOCKER.md to add osmosis instructions needed to use osmosis and populate database

modify Dockerfile.postgres to change owner to postgres

004227e

fazlerabbi37 changed the title ~~[WIP] Add docker support~~ Add docker support Sep 10, 2019

fazlerabbi37 requested review from tomhughes, pnorman and gravitystorm September 10, 2019 09:43

Merge branch 'master' into docker

96383be

Merge branch 'master' into docker

35ad5bd

mmd-osm mentioned this pull request Oct 4, 2019

Remove need for shared library PostgreSQL functions #2383

Merged

fazlerabbi37 added 2 commits October 6, 2019 20:06

Merge branch 'master' into docker

4900551

Merge branch 'master' of https://github.com/openstreetmap/openstreetm…

7dd9810

…ap-website into docker

mmd-osm mentioned this pull request Oct 17, 2019

Can't proceed after bundler install #2401

Closed

jalessio mentioned this pull request Oct 20, 2019

Add Docker Compose Support #2406

Closed

8 tasks

jalessio mentioned this pull request Oct 22, 2019

Add Docker Compose Support for Development Environment #2409

Merged

gravitystorm closed this Oct 23, 2019

Add docker support #2272

Add docker support #2272

Conversation

fazlerabbi37 commented Jun 19, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmd-osm commented Jun 20, 2019 • edited

gravitystorm commented Jun 20, 2019

Firefishy commented Jun 20, 2019

fazlerabbi37 commented Jun 21, 2019

gravitystorm commented Jun 21, 2019

gravitystorm commented Jun 21, 2019

fazlerabbi37 commented Jun 22, 2019

mmd-osm commented Jun 22, 2019 • edited

fazlerabbi37 commented Jun 22, 2019 • edited

robbyoconnor commented Jul 13, 2019

fazlerabbi37 commented Jul 21, 2019 • edited

robbyoconnor commented Jul 21, 2019

Choose a reason for hiding this comment

tomhughes commented Jul 21, 2019

Choose a reason for hiding this comment

fazlerabbi37 Jul 22, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fazlerabbi37 commented Sep 10, 2019 • edited

tiendat-trinh commented Sep 24, 2019

fazlerabbi37 commented Sep 24, 2019

tiendat-trinh commented Sep 24, 2019

mmd-osm commented Oct 17, 2019

fazlerabbi37 commented Oct 19, 2019

jalessio commented Oct 20, 2019

gravitystorm commented Oct 23, 2019

fazlerabbi37 commented Jun 19, 2019 •

edited

mmd-osm commented Jun 20, 2019 •

edited

mmd-osm commented Jun 22, 2019 •

edited

fazlerabbi37 commented Jun 22, 2019 •

edited

fazlerabbi37 commented Jul 21, 2019 •

edited

fazlerabbi37 Jul 22, 2019 •

edited

fazlerabbi37 commented Sep 10, 2019 •

edited