Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Docker Compose Support #2406

Closed
wants to merge 51 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
5fdf823
Add Dockerfile for rails app
fazlerabbi37 Jun 19, 2019
7e5553c
Modify Dockerfile to fix comment for postgresql-client installation e…
fazlerabbi37 Jun 19, 2019
39269fd
Modify Dockerfile to separate apt cache clean, npm comment, and remov…
fazlerabbi37 Jun 20, 2019
1e5dfea
Add docker_postgres.sh for PostgreSQL extension and functions
fazlerabbi37 Jun 20, 2019
c03fcd0
Add Dockerfile.postgres for postgres app
fazlerabbi37 Jun 20, 2019
c66841a
Add docker-compose.yml to build the complete app
fazlerabbi37 Jun 20, 2019
b19e46b
modify Dockerfile.postgres to fix script path
fazlerabbi37 Jul 21, 2019
e8d364b
modify Dockerfile.postgres to upgrade postgres image to 11
fazlerabbi37 Jul 21, 2019
5d685e7
modify Dockerfile to change postgresql-client package and install pha…
fazlerabbi37 Jul 21, 2019
91b69b1
modify docker-compose.yml to fix context path
fazlerabbi37 Jul 21, 2019
a1ab596
add DOCKER.md file with docker setup instructions
fazlerabbi37 Jul 21, 2019
59b399e
add .dockerignore file to list ignorable file for docker
fazlerabbi37 Jul 21, 2019
cf63d94
modify DOCKER.md to add database population
fazlerabbi37 Jul 23, 2019
afdc9e9
modify DOCKER.md to add app config
fazlerabbi37 Sep 6, 2019
ed004c7
moved docker_postgres.sh to docker directory
fazlerabbi37 Sep 9, 2019
d382d4c
modify Dockerfile.postgres file to change pgsql script location
fazlerabbi37 Sep 9, 2019
da0b13c
modify Dockerfile.postgres file to add app location and gem install
fazlerabbi37 Sep 9, 2019
27ff12e
modify Dockerfile.postgres file to move db function directory
fazlerabbi37 Sep 9, 2019
7754f8f
modify docker_postgres.sh to fix libpgosm location
fazlerabbi37 Sep 9, 2019
d51662c
modify Dockerfile to add osmosis
fazlerabbi37 Sep 10, 2019
bc75e4e
modify DOCKER.md to add osmosis instructions
fazlerabbi37 Sep 10, 2019
f759f40
modify Dockerfile.postgres to add ruby compilation
fazlerabbi37 Sep 10, 2019
d64c706
modify Dockerfile.postgres to change owner to postgres
fazlerabbi37 Sep 10, 2019
bfc205f
Reorganize docker-compose files
jalessio Oct 19, 2019
2702431
Require newer version of docker-compose
jalessio Oct 19, 2019
5fcbabe
Re-org includes following directory re-org
jalessio Oct 19, 2019
50dce9a
Use currently 'supported' version of Postgres
jalessio Oct 19, 2019
8c4a16f
Move 'ADD' later in build process to preserve layer caches
jalessio Oct 19, 2019
2e887d0
Dockerfile cleanup to minimize image layers
jalessio Oct 19, 2019
a62ead6
Drop unneeded image names from docker-compose.yml
jalessio Oct 19, 2019
6b77b8d
Update build context for 'db'
jalessio Oct 19, 2019
c68ce09
More re-org of main Dockerfile for better layer caching
jalessio Oct 19, 2019
56019ec
Remove dockerignore file for less build surprises
jalessio Oct 19, 2019
2179344
Fix db Dockerfile path
jalessio Oct 19, 2019
2104509
Port forward Postgres to a less common port to avoid conflicts
jalessio Oct 19, 2019
2bd5abb
Rename Postgres init script
jalessio Oct 20, 2019
707da60
Build shared object in Dockerfile, not at run time
jalessio Oct 20, 2019
b8a2a9a
Use PG11. Don't install Ruby - just get headers for PG extension.
jalessio Oct 20, 2019
4105547
Add db name to database URL for docker-compose
jalessio Oct 20, 2019
609afda
Temporary(?) Makefile for docker-compose shortcuts
jalessio Oct 20, 2019
c9e8721
This project doesn't require PostGIS
jalessio Oct 20, 2019
7fcfbe9
Remove line from Makefile which was not working as expected
jalessio Oct 20, 2019
a3a989f
Remove need for shared library PostgreSQL functions
jalessio Oct 20, 2019
e28ab32
First pass modifications on Docker README
jalessio Oct 20, 2019
8be4a40
Use Docker volume to persist Postgres data on host filesystem
jalessio Oct 22, 2019
37142f1
Exlude PGDATA directory from Docker build context
jalessio Oct 22, 2019
b397c05
Use 'openstreetmap' user to connect to db
jalessio Oct 22, 2019
c0ec8bc
Add a helper command for populating the database
jalessio Oct 22, 2019
7748dd4
DOCKER readme tweaks
jalessio Oct 22, 2019
ba08eec
Move db config to yml file
jalessio Oct 22, 2019
cb2881c
Add helper command for running tests w/Docker
jalessio Oct 22, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
docker-db-data
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,6 @@ public/attachments
public/export
storage
tmp

# docker-compose database directory
docker-db-data
97 changes: 97 additions & 0 deletions DOCKER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Using Docker and Docker Compose to run OpenStreetMap

Using [Docker](https://www.docker.com/) will allow you to install the OpenStreetMap application and all its dependencies in Docker images and then run them in containers, almost with a single command. You will need to install Docker and Docker Compose on your development machine:

- [Install Docker](https://docs.docker.com/install/)
- [Install Docker Compose](https://docs.docker.com/compose/install/)

These instructions gloss over the precise details of the dependencies and their configuration but these can be found in full detail at [INSTALL.md](INSTALL.md).

The first step is to fork/clone the repo to your local machine. The repository is reasonably large (~150MB) and it's unlikely that you need the full history. If you are happy to wait for it all to download, run:

git clone https://github.com/openstreetmap/openstreetmap-website.git

To clone only the most recent version (~23MB), instead use a 'shallow clone':

git clone --depth=1 https://github.com/openstreetmap/openstreetmap-website.git

Now change working directory to the `openstreetmap-website`:

cd openstreetmap-website

### Storage setup

cp config/example.storage.yml config/storage.yml

### Database

cp config/docker.database.yml config/database.yml

### App configuration

cp config/settings.yml config/settings.local.yml

### Installation

In the root directory run:

docker-compose build

Now if this is your first time running or you have removed cache this will take some time to complete. So grab tea/coffee and sit tight. Once the Docker images have finished building you can launch the images as containers.

To launch the app run:

docker-compose up -d

This will launch one Docker container for each 'service' specified in `docker-compose.yml` and run them in the background. There are two options for inspecting the logs of these running containers:

- You can tail logs of a running container with a command like this: `docker-compose logs -f web` or `docker-compose logs -f db`.
- Instead of running the containers in the background with the `-d` flag, you can launch the containers in the foreground with `docker-compose up`. The downside of this is that the logs of all the 'services' defined in `docker-compose.yml` will be intermingled. If you don't want this you can mix and match - for example, you can run the database in background with `docker-compose up -d db` and then run the Rails app in the foreground via `docker-compose up web`.

### Migrations

While the `db' service is running, open another terminal windows and run:

docker-compose run --rm web rake db:migrate

### Node.js modules

We use Yarn to manage the Node.js modules required for the project:

docker-compose run --rm web rake yarn:install

Once these are complete you should be able to visit the app at http://localhost:3000

If localhost does not work, you can use the IP address of the docker-machine.

### Tests

docker-compose run --rm web rake test:db

### Bash

If you want to get into a web container and run specific commands you can fire up a throwaway container to run bash in via:

docker-compose run --rm web bash

Alternatively, if you want to use the already-running `web` container then you can `exec` into it via:

docker-compose exec web bash

Similarly, if you want to `exec` in the db container use:

docker-compose exec db bash

### Populating the database

This installation comes with no geographic data loaded. You can either create new data using one of the editors (Potlatch 2, iD, JOSM etc) or by loading an OSM extract.

After installing but before creating any users or data, import an extract with [Osmosis](https://wiki.openstreetmap.org/wiki/Osmosis) and the `--write-apidb` task. The `web` container comes with `osmosis` pre-installed. So to populate data with a `.osm.pbf` use the following command:

docker-compose run --rm web osmosis \
--read-pbf /path/to/file.osm.pbf \
--write-apidb \
host="db" \
database="openstreetmap" \
user="openstreetmap" \
validateSchemaVersion="no"
52 changes: 52 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
FROM ruby:2.5

# fixes dpkg man page softlink error while installing postgresql-client [source: https://stackoverflow.com/a/52655008/5350059]
RUN mkdir -p /usr/share/man/man1 && \
mkdir -p /usr/share/man/man7

# npm is not available in Debian repo so following official instruction [source: https://github.com/nodesource/distributions/blob/master/README.md#debinstall]
RUN curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh && \
bash nodesource_setup.sh && \
rm -f nodesource_setup.sh

# install packages
RUN apt-get update && \
apt-get install --no-install-recommends -y \
build-essential \
curl \
imagemagick \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder why imagemagick is still in the list. GPX preview image generation was moved to gd2-ffij earlier this year, and I'm not aware of other parts of the website depending on imagemagick.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The imagemagick requirements are still listed in the "Minimum Requirements" section of INSTALL.md so if they are dropped here they should probably be dropped there as well.

https://github.com/openstreetmap/openstreetmap-website/blob/master/INSTALL.md#minimum-requirements

Copy link
Contributor

@mmd-osm mmd-osm Oct 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it seems the INSTALL.md file also needs a bit of cleanup.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mmd-osm is there anything in the test suite which would verify whether this dependency is needed? If not, any suggestions on a suitable test I could add?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we got rid of imagemagick, but it seems that 6c20244 introduced mini_magick, which depends on imagemagick. This is only used to resize avatar images and might be a bit heavyweight for the task at hand. Maybe @tomhughes knows more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't believe there is any other supported approach. It only uses it via the command line though so there is no need for the development package.

libarchive-dev \
libffi-dev \
libmagickwand-dev \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one should also be reviewed, not sure it is still needed.

libpq-dev \
libsasl2-dev \
libxml2-dev \
libxslt1-dev \
locales \
nodejs \
osmosis \
postgresql-client \
ruby-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

# install npm packages
RUN npm install -g --unsafe-perm \
phantomjs-prebuilt \
yarn

# Setup app location
RUN mkdir -p /app
WORKDIR /app

# Install gems
ADD Gemfile* /app/
RUN bundle install

# Setup local
RUN sed -i -e 's/# en_GB.UTF-8 UTF-8/en_GB.UTF-8 UTF-8/' /etc/locale.gen && \
echo 'LANG="en_GB.UTF-8"'>/etc/default/locale && \
dpkg-reconfigure --frontend=noninteractive locales && \
update-locale LANG=en_GB.UTF-8

ENV LANG en_GB.UTF-8
22 changes: 22 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
docker-build:
docker-compose build

docker-up:
docker-compose up -d

docker-db-migrate:
docker-compose run --rm web rake db:migrate

docker-test:
docker-compose run --rm web rake test:db

docker-populate-db:
wget https://download.geofabrik.de/north-america/us/district-of-columbia-latest.osm.pbf
docker-compose run --rm web osmosis \
-verbose \
--read-pbf district-of-columbia-latest.osm.pbf \
--write-apidb \
host="db" \
database="openstreetmap" \
user="openstreetmap" \
validateSchemaVersion="no"
20 changes: 20 additions & 0 deletions config/docker.database.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# This configuration is tailored for use with docker-compose. See DOCKER.md for more information.

development:
adapter: postgresql
database: openstreetmap
username: openstreetmap
password: openstreetmap
host: db
encoding: utf8

# Warning: The database defined as 'test' will be erased and
# re-generated from your development database when you run 'rake'.
# Do not set this db to the same as development or production.
test:
adapter: postgresql
database: osm_test
username: postgres
password:
host: db
encoding: utf8
24 changes: 24 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
version: "3"

services:
web:
build:
context: .
volumes:
- .:/app
ports:
- "3000:3000"
command: bundle exec rails s -p 3000 -b '0.0.0.0'
depends_on:
- db

db:
build:
context: .
dockerfile: docker/postgres/Dockerfile
ports:
- "54321:5432"
environment:
POSTGRES_DB: openstreetmap
volumes:
- ./docker-db-data:/var/lib/postgresql/data
4 changes: 4 additions & 0 deletions docker/postgres/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
FROM postgres:11

# Add db init script to install OSM-specific Postgres functions/extensions.
ADD docker/postgres/openstreetmap-postgres-init.sh /docker-entrypoint-initdb.d/
11 changes: 11 additions & 0 deletions docker/postgres/openstreetmap-postgres-init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/bash
set -ex

# Create 'openstreetmap' user
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" <<-EOSQL
CREATE USER openstreetmap PASSWORD 'openstreetmap';
GRANT ALL PRIVILEGES ON DATABASE openstreetmap TO openstreetmap;
EOSQL

# Create btree_gist extensions
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE EXTENSION btree_gist" openstreetmap