Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docker support #2272

Closed
wants to merge 28 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
ad237cd
Add Dockerfile for rails app
fazlerabbi37 Jun 19, 2019
3fe73f0
Modify Dockerfile to fix comment for postgresql-client installation e…
fazlerabbi37 Jun 19, 2019
4eeb36a
Modify Dockerfile to separate apt cache clean, npm comment, and remov…
fazlerabbi37 Jun 20, 2019
caaa350
Add docker_postgres.sh for PostgreSQL extension and functions
fazlerabbi37 Jun 20, 2019
b6d373f
Add Dockerfile.postgres for postgres app
fazlerabbi37 Jun 20, 2019
30ad01e
Add docker-compose.yml to build the complete app
fazlerabbi37 Jun 20, 2019
447c55e
modify Dockerfile.postgres to fix script path
fazlerabbi37 Jul 21, 2019
0cf0d0c
modify Dockerfile.postgres to upgrade postgres image to 11
fazlerabbi37 Jul 21, 2019
18ef6b6
modify Dockerfile to change postgresql-client package and install pha…
fazlerabbi37 Jul 21, 2019
f75eea4
modify docker-compose.yml to fix context path
fazlerabbi37 Jul 21, 2019
0d15f1c
add DOCKER.md file with docker setup instructions
fazlerabbi37 Jul 21, 2019
c4c450f
add .dockerignore file to list ignorable file for docker
fazlerabbi37 Jul 21, 2019
076073b
modify DOCKER.md to add database population
fazlerabbi37 Jul 23, 2019
fc139b0
modify DOCKER.md to add app config
fazlerabbi37 Sep 6, 2019
29f6833
Merge branch 'master' into docker
fazlerabbi37 Sep 9, 2019
ce38a10
moved docker_postgres.sh to docker directory
fazlerabbi37 Sep 9, 2019
b03bfd1
modify Dockerfile.postgres file to change pgsql script location
fazlerabbi37 Sep 9, 2019
c6edb29
modify Dockerfile.postgres file to add app location and gem install
fazlerabbi37 Sep 9, 2019
8383453
modify Dockerfile.postgres file to move db function directory
fazlerabbi37 Sep 9, 2019
cb96ddd
modify docker_postgres.sh to fix libpgosm location
fazlerabbi37 Sep 9, 2019
a1413af
modify Dockerfile to add osmosis
fazlerabbi37 Sep 10, 2019
02851d2
modify DOCKER.md to add osmosis instructions
fazlerabbi37 Sep 10, 2019
6ce9e77
modify Dockerfile.postgres to add ruby compilation
fazlerabbi37 Sep 10, 2019
004227e
modify Dockerfile.postgres to change owner to postgres
fazlerabbi37 Sep 10, 2019
96383be
Merge branch 'master' into docker
fazlerabbi37 Sep 20, 2019
35ad5bd
Merge branch 'master' into docker
fazlerabbi37 Sep 24, 2019
4900551
Merge branch 'master' into docker
fazlerabbi37 Oct 6, 2019
7dd9810
Merge branch 'master' of https://github.com/openstreetmap/openstreetm…
fazlerabbi37 Oct 16, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions docker/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
*
!Gemfile
!Gemfile.lock
!db/functions
!db/docker_postgres.sh
99 changes: 99 additions & 0 deletions docker/DOCKER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Using Docker to run OpenStreetMap

Using [Docker](https://www.docker.com/) will allow you to install the OpenStreetMap application and all its' dependencies in a container, almost with a single command.

These instructions gloss over the precise details of the dependencies and their configuration but these can be found in full detail at [INSTALL.md](INSTALL.md).

The first step is to fork/clone the repo to your local machine. The repository is reasonably large (~150MB) and it's unlikely that you need the full history. If you are happy to wait for it all to download, run:
```
git clone https://github.com/openstreetmap/openstreetmap-website.git
```

To clone only the most recent version (~23MB), instead use a 'shallow clone':

```
git clone --depth=1 https://github.com/openstreetmap/openstreetmap-website.git
```

Now change working directory to the `openstreetmap-website`:

```
cd openstreetmap-website
```

### Storage setup

```
cp config/example.storage.yml config/storage.yml
```

### Database

```
cp config/example.database.yml config/database.yml
```

Set `username` to postgres and `host` to db leave the `password` blank


### App configuration

```
cp config/settings.yml config/settings.local.yml
```

### Installation

In the root directory run:

```
docker-compose -f docker/docker-compose.yml up
```
Now if this is your first time running or you have removed cache this will take some time to complete. So grab tea/coffee and seat tight. Upon successfull build it should show

### Migrations
While `docker-compose up` is running, open another terminal windows and run:

```
docker-compose -f docker/docker-compose.yml exec web bundle exec rake db:migrate
```

### Node.js modules
We use Yarn to manage the Node.js modules required for the project.:

```
docker-compose -f docker/docker-compose.yml exec web bundle exec rake yarn:install
```

Once these are complete you should be able to visit the app at http://localhost:3000

If localhost does not work, you can use the IP address of the docker-machine.

### Tests

```
docker-compose -f docker/docker-compose.yml exec web bundle exec rake test:db
```

### Bash

If you want to get onto the web container and run specific commands you can fire up bash via:

```
docker-compose -f docker/docker-compose.yml exec web /bin/bash
```

Similarly, if you want to get onto the db container use:

```
docker-compose -f docker/docker-compose.yml exec db /bin/bash
```

### Populating the database
This installation comes with no geographic data loaded. You can either create new data using one of the editors (Potlatch 2, iD, JOSM etc) or by loading an OSM extract.

After installing but before creating any users or data, import an extract with [Osmosis](https://wiki.openstreetmap.org/wiki/Osmosis) and the `--write-apidb` task. The `web` container comes with `osmosis` pre-installed. So to populate data with a `.osm.pbf` use the following command:

```
docker-compose -f docker/docker-compose.yml exec web osmosis --read-pbf /path/to/file.osm.pbf --write-apidb host="db" database="openstreetmap" user="postgres" validateSchemaVersion="no"
```
30 changes: 30 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
FROM ruby:2.5-slim

# install packages
# fixes dpkg man page softlink error while installing postgresql-client [source: https://stackoverflow.com/a/52655008/5350059]
RUN mkdir -p /usr/share/man/man1 && mkdir -p /usr/share/man/man7
RUN apt-get update && apt-get install curl -y
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each time apt-get is run you should update and then remove the APT cache files at the end of the command.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that the cache files don't make it into the docker image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this just building a container for local use though? Not anything that's going to be published?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but what I've read indicates it as a best practice in general

RUN apt-get clean && rm -rf /var/lib/apt/lists/*


#npm is not available in Debian repo so following official instruction [source: https://github.com/nodesource/distributions/blob/master/README.md#debinstall]
RUN curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh && bash nodesource_setup.sh && rm nodesource_setup.sh
RUN apt-get install -y --no-install-recommends ruby-dev libarchive-dev libmagickwand-dev libxml2-dev libxslt1-dev build-essential libpq-dev libsasl2-dev imagemagick libffi-dev locales postgresql-client nodejs osmosis
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
RUN npm install yarn phantomjs-prebuilt -g --unsafe-perm


# Setup app location
RUN mkdir -p /app
WORKDIR /app

# Install gems
ADD Gemfile* /app/
RUN bundle install
# Setup local
RUN sed -i -e 's/# en_GB.UTF-8 UTF-8/en_GB.UTF-8 UTF-8/' /etc/locale.gen && \
echo 'LANG="en_GB.UTF-8"'>/etc/default/locale && \
dpkg-reconfigure --frontend=noninteractive locales && \
update-locale LANG=en_GB.UTF-8

ENV LANG en_GB.UTF-8
26 changes: 26 additions & 0 deletions docker/Dockerfile.postgres
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
FROM postgres:11

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Production still uses 9.5 as of now.

ADD docker/docker_postgres.sh docker-entrypoint-initdb.d/docker_postgres.sh

# compiling ruby from source as official debian has ruby 2.3 and we need => 2.4
RUN apt-get update
RUN apt-get install -y curl build-essential libreadline-dev libssl-dev libcurl4-openssl-dev postgresql-server-dev-all zlib1g-dev libxml2-dev
RUN curl -L https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.4.tar.gz | tar zx
RUN cd ruby-2.6.4 && ./configure && make && make install
RUN cd .. && rm -r ruby-2.6.4
RUN apt-get clean && rm -rf /var/lib/apt/lists/*


# Setup app location
RUN mkdir -p /app
WORKDIR /app

# add database functions directory
ADD db/functions/ /app/db/functions/

# Install gems
ADD Gemfile* /app/
RUN bundle install

# change ownership to postgres as while running docker_postgres.sh postgres will need write access
RUN chown -R postgres /app/db
25 changes: 25 additions & 0 deletions docker/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
version: '2'
services:
web:
image: openstreetmap-website:v1
build:
context: ..

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not put the Dockerfiles in this directory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which directory? Right now all docker related files are in one directory named docker.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wait, nevermind.

dockerfile: docker/Dockerfile
volumes:
- ..:/app
ports:
- "3000:3000"
command: bundle exec rails s -p 3000 -b '0.0.0.0'
depends_on:
- db
environment:
DATABASE_URL: postgres://postgres@db:5432
db:
image: openstreetmap-db:v1
build:
context: ..

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here as above?

dockerfile: docker/Dockerfile.postgres
ports:
- "5432:5432"
environment:
POSTGRES_DB: openstreetmap
7 changes: 7 additions & 0 deletions docker/docker_postgres.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE EXTENSION btree_gist" openstreetmap
make -C db/functions libpgosm.so
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE FUNCTION maptile_for_point(int8, int8, int4) RETURNS int4 AS '/app/db/functions/libpgosm', 'maptile_for_point' LANGUAGE C STRICT" openstreetmap
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE FUNCTION tile_for_point(int4, int4) RETURNS int8 AS '/app/db/functions/libpgosm', 'tile_for_point' LANGUAGE C STRICT" openstreetmap
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE FUNCTION xid_to_int4(xid) RETURNS int4 AS '/app/db/functions/libpgosm', 'xid_to_int4' LANGUAGE C STRICT" openstreetmap