Add Docker Compose Support for Development Environment #2409

jalessio · 2019-10-22T19:58:04Z

This is a cleaned up version of PR #2406 which was built upon the work of @fazlerabbi37 in PR #2272.

Introduces the usage of docker-compose for creating a development environment for this project
Adds DOCKER.md for Docker-specific installation instructions
~~Adds a Makefile for providing shortcuts for running common docker-compose commands~~

There is certainly still room for improvement here, including pruning unneeded dependencies in the top-level Dockerfile, but I propose merging this working version and iterating from that point (particularly since many of the changes will apply regardless of the installation method). I think we will benefit from feedback of a few people actually using this before making too many more changes.

DOCKER.md

jalessio · 2019-10-28T01:06:15Z

@gravitystorm any thoughts on how to move this forward? I believe it's in a usable state and offers a quick way to get up and running with this project locally. I've been using it locally for over a week now without any issues.

I could definitely benefit from a few people trying it out from scratch and providing feedback, but perhaps that's more likely to occur once it's on master?

@zerebubuth / @Firefishy / @mmd-osm - would any of you be willing to give this branch a quick run-through locally since you were all active in the previous PR (#2406)? Thanks!

SomeoneElseOSM · 2019-10-28T09:09:06Z

@jalessio Maybe write an OSM diary entry (or similar) explaining what this is and how it would be useful to people - explain why would they benefit from having a local copy of this code, and what Docker is and how that helps them to do that? I appreciate that there's some of that at the top of #2272 , but a bit more info would also be useful.

mmd-osm · 2019-10-28T17:50:44Z

Test instructions are here: #2401 (comment)

mmd-osm · 2019-10-29T09:09:05Z

I tested the set up yesterday, and it looks good. It took forever to download phantomjs and I ended up removing it from the Dockerfile. phantomjs project is currently in some "suspended" status, not sure if we'd better replace it or not.

@jalessio : as most of the editing API calls run via CGImap in production these days, we might consider (optionally) including CGImap https://github.com/zerebubuth/openstreetmap-cgimap/blob/master/Dockerfile as well, Some people claim is this notoriously difficult to set up, so adding this to docker compose seems worthwhile to me.
Dispatching to Rails port / cgimap could be done using lighttpd (https://github.com/zerebubuth/openstreetmap-cgimap/blob/master/lighttpd.conf) as a reverse proxy.

gravitystorm · 2019-10-29T09:15:06Z

@gravitystorm any thoughts on how to move this forward?

I'll have another look later in the week - it looks like you've already taken care of a few other things that I was going to query.

as most of the editing API calls run via CGImap these days in production, we might consider (optionally) including CGImap

No, please don't. I'm happy to see stuff in this repository to help with setting up development environments, but we deliberately leave production choices out of consideration.

mmd-osm · 2019-10-29T09:18:07Z

we deliberately leave production choices out of consideration

We'll add it to the CGImap repo then, no worries. I don't really care which repo hosts this config, as long as we offer this option at all.

icemagno · 2019-10-29T10:45:51Z

As a suggestion i would like to say that you could build your own image and publish at Docker Hub. So we would just run it. Maybe the database image and the website image. The only flaw I see is regarding the configuration file. The OSM guys could accept database configuration via environment variables and not only by YAML. I think the OSM WEBSITE is not very flexible and was build in a very
monilithic old fashion.

gravitystorm

Overall looks great to me! Just a few changes required to bring this into line with INSTALL.md

Talking of which, there should be a paragraph added to the top of INSTALL.md to describe that you can use Docker to set up your development environment, just like you can with Vagrant.

gravitystorm · 2019-10-30T09:43:08Z

Dockerfile

+RUN mkdir -p /usr/share/man/man1 && \
+    mkdir -p /usr/share/man/man7
+
+# npm is not available in Debian repo so following official instruction [source: https://github.com/nodesource/distributions/blob/master/README.md#debinstall]


I have more knowledge of Ubuntu than Debian, but as far as I can tell:

ruby:2.5 is an alias for ruby:2.5-buster (https://hub.docker.com/_/ruby/)

buster contains npm (https://packages.debian.org/buster/npm)

but we don't need npm anyway, only nodejs and yarn

Best to use FROM: ruby:2.5 rather than FROM: ruby:2.5-buster... The images upstream will pull in changes.

gravitystorm · 2019-10-30T09:45:36Z

Dockerfile

+
+# install npm packages
+RUN npm install -g --unsafe-perm \
+      phantomjs-prebuilt \


Debian buster comes with a phantomjs package. Should we just install that instead?

@gravitystorm I moved to installing phantomjs via apt.

gravitystorm · 2019-10-30T09:47:57Z

Dockerfile

+# install npm packages
+RUN npm install -g --unsafe-perm \
+      phantomjs-prebuilt \
+      yarn


I find using npm to install yarn a bit of a roundabout way to go. Could we install the yarn package using the yarn repository instead?

@gravitystorm Agreed! I'm now installing yarn via apt.

gravitystorm · 2019-10-30T09:54:20Z

DOCKER.md

+
+See `CONFIGURE.md` for information on how to manage users and enable OAuth for iD, JOSM etc.
+
+### Tests


Running the tests section should come before the loading an extract or additional configuration sections. This is because the goal for the user is to ensure their development environment is set up correctly, as validated by the tests

gravitystorm · 2019-10-30T09:57:00Z

DOCKER.md

@@ -0,0 +1,100 @@
+# Using Docker and Docker Compose to run OpenStreetMap


"Using Docker and Docker Compose to set up The Rails Port for development and testing" - based on the wording in the current INSTALL.md and to make it clear that these docker images are designed for development and testing, not necessarily for production.

gravitystorm · 2019-10-30T10:01:19Z

DOCKER.md

+
+Run the Rails database migrations:
+
+    docker-compose run --rm web rake db:migrate


For consistency and reliability, we should have ... bundle exec rake db:migrate here

gravitystorm · 2019-10-30T10:01:32Z

DOCKER.md

+
+### Tests
+
+    docker-compose run --rm web rake test:db


bundle exec

gravitystorm · 2019-10-30T10:03:46Z

Dockerfile

+    dpkg-reconfigure --frontend=noninteractive locales && \
+    update-locale LANG=en_GB.UTF-8
+
+ENV LANG en_GB.UTF-8


There needs to be a bit about bundle exec rake yarn:install somewhere, either here in the docker config, or elsewhere in the manual installation notes.

I'm not calling the yarn installer during Docker image build.

icemagno · 2019-10-30T11:49:19Z

What about config the database connection using environment vars?

gravitystorm · 2019-10-31T10:10:31Z

What about config the database connection using environment vars?

@icemagno What's your usecase? Would this be for your development environment, or for production deployments?

icemagno · 2019-10-31T23:09:53Z

@gravitystorm I'm developing a distributed software with about 50 microservices in Spring boot and must follow some Standards in the continuous delivery and infrastructure deployment. I already have it running and using a old OSM server I've created. Now I have a Docker conteiner running a very nice and rich styled OSM and want to create a software to create and edit OSM and custom features ( from the Brazilian Gov. Metadata Standard ). So I'm trying some options mostly (actualy a must) in Docker. The installation way I seen here is good and working but I need some more flexible because I need to document the installation process to the future and it must follow the current standard wich is one script for each container and must be possible to replace any container without affect others. So I need one script to build and run the database and another to do the same with the website. No docker compose is allowed even if it is the easy way to do it. Each build and run process must be documented alone.

mmd-osm · 2019-11-01T07:17:15Z

This blog post https://blog.codeship.com/running-rails-development-environment-docker/ should give you some idea how to set up database.yml to fetch database user and password from environment variables. Besides the requirements you describe seem rather out of scope for this pull request.

icemagno · 2019-11-01T09:52:47Z

@mmd-osm thanks. I'll give a look but you may consider use it anyway because this is the best way to offer a Docker image. Under my viewpoint it is not a good idea to create hard coded configurations in a Docker deploy. It is extremely boring editing files before building and sometimes I think the Docker premisse of "fire-and-go" is broken. And the user must be free to choose database names, hosts and etc.

Almost forgot: congratulations. You guys are doing a very good work.

mmd-osm · 2019-12-23T20:09:29Z

as most of the editing API calls run via CGImap these days in production, we might consider (optionally) including CGImap
No, please don't. I'm happy to see stuff in this repository to help with setting up development environments, but we deliberately leave production choices out of consideration.

FYI. A complete stack consisting of Rails port, Postgres db, cgimap and lighttpd is now described in more detail in zerebubuth/openstreetmap-cgimap#213.

It leverages the Postgres and Rails port set up outlined in this issue, and extends docker-compose-yml by two additional services for cgimap and lighttpd.

jalessio · 2019-12-31T14:15:06Z

TODO - possibly switch to Debian Bionic base image to match other parts of this repo. See #2473

migurski · 2020-02-16T01:53:52Z

I just ran through the complete process of running this Docker environment, to test the user experience of a random passerby who’d like to contribute an edit to the site. @jalessio walked me through setting up Docker-Compose, initializing a clean new database, importing data using Osmosis (from a small PBF extract), and verifying that everything worked at localhost:3000.

I made the smallest change I could think of: updating the header text in a way that was obviously visible (history/schmistory, etc.). Rails hot-reloaded everything, and I was able to see my change immediately:

As a Rails novice who doesn't expect to become an expert, this has been a super useful way to understand the moving parts of the website and test potential changes. I can imagine helping a designer implement an idea here and test it out live, useful to getting new ideas into OSM.

I know that Docker configs can fall out of date over time, but as we’ve seen with openstreetmap/chef#266 tests for this can be added to our Travis configuration and images can be published to Docker Hub where a casual developer can try them out.

Firefishy · 2020-02-20T17:27:21Z

docker-compose.yml

+      POSTGRES_HOST_AUTH_METHOD: trust
+      POSTGRES_DB: openstreetmap
+    volumes:
+      - ./docker-db-data:/var/lib/postgresql/data


This should really be a data volume, rather than a file mapping.
See: https://docs.docker.com/compose/compose-file/compose-file-v2/#volume-configuration-reference

jalessio · 2020-03-02T03:27:09Z

DOCKER.md

+
+    docker-compose run --rm web bundle exec rake test:db
+
+### Loading an OSM extract


Since loading an extract via osmosis stopped working after PR #2432 I'm inclined to remove this section until I can offer a working solution.

There’s a work around available: #2543 (comment)

Just add it to the description. You would need to call psql in any case to correct the various Postgres sequences, so the additional pain to add and drop a column should be close to zero.

It’s a super-ugly workaround, almost to the point that I think it’d be harmful just to recommend that people monkey with the schema pre- and post-import. Let’s remove the section for now?

It’s as ugly as having to tweak sequences. My point was, it was already ugly before, we’re not adding much to that by this workaround.

Osmosis 0.47.1 now correctly writes to the DB so this section should work fine.

jalessio · 2020-03-02T03:32:37Z

Dockerfile

+
+# Install NodeJS packages
+ADD package.json yarn.lock /app/
+RUN yarn


@gravitystorm I realize you've been consistently asking for using the bundle exec form of commands for consistency (which I appreciate), but I'll try to convince you of this exception...

I toiled with getting this to work via bundle exec rake db:migrate, but I couldn't get it to run without adding a big chunk of the repository into the image during build. It ended up adding a bunch of extra files to the image just so that I could run bundle exec rake db:migrate instead yarn directly. I believe the result is the same in the end so I went with the simpler option.

That said, perhaps someone else could compile a small list of files necessary to successfully run rake commands during the image build?

You're already done bundle install which should install everything bundle exec might need - if you don't use bundle exec then you have no idea if things are using the expected modules, or even if they will work at all.

jalessio · 2020-03-02T03:39:05Z

I made a number of changes to this PR and I believe I have addressed all the comments / requests for changes (except one! see below). I have also rebased it against master.

TODO

Update docker-compose config to use a data volume per @Firefishy (Add Docker Compose Support for Development Environment #2409 (review))
Remove instructions for using osmosis? (since they don't work any more)
Get tests passing in Docker!

This reverts commit 20dbf89.

Co-authored-by: Tobias <t@tobiasjordans.de>

- Added db/functions/functions.sql to docker-compose DB image - Added `file`, `libgd-dev`, and `unzip` packages to help tests pass

…ose volumes

migurski · 2021-01-28T20:41:12Z

I’ve rebased this PR to keep it updated. @gravitystorm I’ve confirmed that the directory litter you pointed out in a previous review has been fixed since late last year. I believe we’ve addressed all feedback.

gravitystorm · 2021-02-03T18:10:19Z

Great, thanks @migurski and everyone else who contributed!

I did find a first-run issue with the tests, involving i18n-js. But during my investigations (checking the web interface, running the tests again) everything worked, so something triggered the file generation. And now I can't recreate it, so I just mention it in case it comes up again for other people at some point. But I didn't think it was a showstopper, so I'm happy to merge.

tomhughes

Apologies for submitting this after the PR has been merged but I hadn't realised the PR was ready for review.

tomhughes · 2021-02-03T18:17:19Z

.github/workflows/docker.yml

+    - name: Start Docker-Compose
+      run: |
+        docker-compose up -d
+        sleep 15 # let the DB warm up a little


That's a failure waiting to happen... Is there really not a way to actually detect when it's ready instead of guessing?

tomhughes · 2021-02-03T18:17:49Z

.github/workflows/docker.yml

+    runs-on: ubuntu-20.04
+    steps:
+    - name: Checkout source
+      uses: actions/checkout@v1


This is out of date - there is a v2 now.

tomhughes · 2021-02-03T18:18:21Z

.github/workflows/docker.yml

+        curl -siL http://127.0.0.1:3000/api/0.6/node/1 | grep 'Null Island'
+    - name: Test Complete Suite
+      run: |
+        docker-compose run --rm web bundle exec rails test:db


I always use rake test:db but I admit to not being able to keep up with the constant changes about what is run via rails and what is run via rake...

tomhughes · 2021-02-03T18:21:35Z

config/boot.rb

@@ -1,4 +1,4 @@
 ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)

 require "bundler/setup" # Set up gems listed in the Gemfile.
-require "bootsnap/setup" # Speed up boot time by caching expensive operations.
+require "bootsnap/setup" if ENV.fetch("ENABLE_BOOTSNAP", "true") == "true" # Speed up boot time by caching expensive operations.


I'm sorry but there has to be a better solution to the bootsnap problem that polluting core rails setup files with this nonsense.

tomhughes · 2021-02-03T18:23:18Z

docker-compose.yml

+    ports:
+      - "54321:5432"
+    environment:
+      POSTGRES_HOST_AUTH_METHOD: trust


If you're doing trust auth why did you put a password in the database configuration?

tomhughes · 2021-02-03T18:30:33Z

docker/postgres/openstreetmap-postgres-init.sh

+psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" <<-EOSQL
+    CREATE USER openstreetmap PASSWORD 'openstreetmap';
+    GRANT ALL PRIVILEGES ON DATABASE openstreetmap TO openstreetmap;
+EOSQL


As you appear to be using trust auth there shouldn't be any need to set a password and it would also be best not to set superuser unless have to - the only thing that is likely to be needed for installing the extension which os done as the postgres user anyway.

tomhughes · 2021-02-03T18:32:23Z

Dockerfile

+      unzip \
+      yarnpkg \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*


Why is this removing part of apt's internal state?

tomhughes · 2021-02-03T18:33:24Z

Dockerfile

+
+# Install current Osmosis
+RUN curl -OL https://github.com/openstreetmap/osmosis/releases/download/0.47.2/osmosis-0.47.2.tgz \
+ && tar -C /usr/local -xzf osmosis-0.47.2.tgz


Hard coding an omsosmis version number is a bad idea as it will need updating when osmosis changes. Why are we wasting time downloading osmosis anyway? How many people will use it? It's not like it even works very well for loading an API database which I assume is what you have it here for...

tomhughes · 2021-02-03T18:34:11Z

Dockerfile

+
+# Install NodeJS packages
+ADD package.json yarn.lock /app/
+RUN yarnpkg install


This should be done as bundle exec rake yarn:install and not by running yarn directly.

tomhughes · 2021-02-03T18:38:32Z

config/docker.database.yml

+  adapter: postgresql
+  database: openstreetmap
+  username: openstreetmap
+  password: openstreetmap


If you're using trust auth (which you appear to be) then you shouldn't need a password.

migurski · 2021-02-03T19:06:46Z

Thanks @gravitystorm, so excited to see this merged! @tomhughes, I’ll follow up on your points in #3081 and I’ll continue to keep a close eye on any Docker-related issues that emerge in this repo.

gravitystorm · 2021-02-04T10:36:46Z

Apologies for submitting this after the PR has been merged but I hadn't realised the PR was ready for review.

And I'm sorry for merging it without checking if you were happy with it. It's one of the downsides of PRs with 100+ comments, I guess - maybe we should get better at closing and reopening PRs after several rounds of changes.

And I'm kicking myself about the sbin bit, since I'd flagged that up in my own head, but then forgot all about it by the time I'd built and rebuilt the containers a few times ☹️

jalessio mentioned this pull request Oct 22, 2019

Add Docker Compose Support #2406

Closed

8 tasks

gravitystorm reviewed Oct 23, 2019

View reviewed changes

DOCKER.md Outdated Show resolved Hide resolved

DOCKER.md Outdated Show resolved Hide resolved

gravitystorm mentioned this pull request Oct 23, 2019

Add docker support #2272

Closed

8 tasks

jalessio mentioned this pull request Oct 27, 2019

Can't proceed after bundler install #2401

Closed

jalessio mentioned this pull request Oct 28, 2019

Vagrant and Docker installations are unmaintained #2344

Open

gravitystorm requested changes Oct 30, 2019

View reviewed changes

gravitystorm added the dx Developer Experience label Oct 31, 2019

gravitystorm added the changes requested Changes are needed before the PR will be merged or reviewed again label Nov 13, 2019

mmd-osm mentioned this pull request Dec 23, 2019

Updated Dockerfile and lighttpd.conf zerebubuth/openstreetmap-cgimap#213

Merged

migurski mentioned this pull request Feb 18, 2020

Remove the users.nearby column #2439

Merged

Firefishy reviewed Feb 20, 2020

View reviewed changes

jalessio force-pushed the docker-compose-take2 branch from 753b670 to 7dc7996 Compare March 2, 2020 03:22

jalessio commented Mar 2, 2020

View reviewed changes

jalessio and others added 19 commits January 28, 2021 12:03

Update INSTALL.md to reference Docker as an option

6bcd214

Added new parallel test job for docker-compose

f07314c

Added API read to docker-compose test

b635cf6

Replaced raw PGSQL testing file with Osmosis-driven XML file

c17cb84

Bumped Ruby version to 2.7 to match CI config

b71a6fd

Expanded yarn call to fix test failure

188fc8f

Noted yarn step in Docker instructions

1ab5e3b

Update INSTALL.md to restore original graf

c936063

This reverts commit 20dbf89.

Switched tests to "openstreetmap"

bc98021

Added minimal Docker workflow to test Github Actions

810a941

Extended Docker action to cover full setup and tests

1e5cc59

Migrated from ruby:2.7 Docker image to ubuntu:20.04

8baebab

Applied documentation suggestions from code review

60f8fb9

Co-authored-by: Tobias <t@tobiasjordans.de>

Made Bootsnap required only when asked-for

c66041c

Prevent node_modules from mounting to save rake yarn:install step

2da3ce6

Added complete Rails test suite

033c3b2

- Added db/functions/functions.sql to docker-compose DB image - Added `file`, `libgd-dev`, and `unzip` packages to help tests pass

Switched to Docker volumn for DB data

8c2db90

Create directories for traces and images

e19e58f

Added /home/osm/traces and /home/osm/images to persistent Docker-comp…

4fac47a

…ose volumes

migurski force-pushed the docker-compose-take2 branch from d37f606 to 4fac47a Compare January 28, 2021 20:38

gravitystorm approved these changes Feb 3, 2021

View reviewed changes

gravitystorm merged commit 86dbcb5 into openstreetmap:master Feb 3, 2021

gravitystorm mentioned this pull request Feb 3, 2021

Docker build failing #3080

Closed

tomhughes reviewed Feb 3, 2021

View reviewed changes

migurski mentioned this pull request Feb 3, 2021

Address post-merge Docker Compose cleanup issues #3081

Closed

6 tasks

migurski mentioned this pull request Feb 4, 2021

Post-merge Docker Compose cleanup issues #3083

Merged


		See `CONFIGURE.md` for information on how to manage users and enable OAuth for iD, JOSM etc.

		### Tests

		@@ -0,0 +1,100 @@
		# Using Docker and Docker Compose to run OpenStreetMap


		Run the Rails database migrations:

		docker-compose run --rm web rake db:migrate


		docker-compose run --rm web bundle exec rake test:db

		### Loading an OSM extract

Add Docker Compose Support for Development Environment #2409

Add Docker Compose Support for Development Environment #2409

Conversation

jalessio commented Oct 22, 2019 • edited

jalessio commented Oct 28, 2019

SomeoneElseOSM commented Oct 28, 2019

mmd-osm commented Oct 28, 2019

mmd-osm commented Oct 29, 2019 • edited

gravitystorm commented Oct 29, 2019

mmd-osm commented Oct 29, 2019

icemagno commented Oct 29, 2019

gravitystorm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Firefishy Feb 20, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

icemagno commented Oct 30, 2019

gravitystorm commented Oct 31, 2019

icemagno commented Oct 31, 2019

mmd-osm commented Nov 1, 2019

icemagno commented Nov 1, 2019 • edited

mmd-osm commented Dec 23, 2019

jalessio commented Dec 31, 2019

migurski commented Feb 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmd-osm Mar 2, 2020 • edited

Choose a reason for hiding this comment

migurski Mar 2, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jalessio commented Mar 2, 2020 • edited

TODO

migurski commented Jan 28, 2021

gravitystorm commented Feb 3, 2021

tomhughes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

migurski commented Feb 3, 2021 • edited

gravitystorm commented Feb 4, 2021

jalessio commented Oct 22, 2019 •

edited

mmd-osm commented Oct 29, 2019 •

edited

Firefishy Feb 20, 2020 •

edited

icemagno commented Nov 1, 2019 •

edited

mmd-osm Mar 2, 2020 •

edited

migurski Mar 2, 2020 •

edited

jalessio commented Mar 2, 2020 •

edited

migurski commented Feb 3, 2021 •

edited