-
Notifications
You must be signed in to change notification settings - Fork 904
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Docker Compose Support #2406
Closed
Closed
Changes from all commits
Commits
Show all changes
51 commits
Select commit
Hold shift + click to select a range
5fdf823
Add Dockerfile for rails app
fazlerabbi37 7e5553c
Modify Dockerfile to fix comment for postgresql-client installation e…
fazlerabbi37 39269fd
Modify Dockerfile to separate apt cache clean, npm comment, and remov…
fazlerabbi37 1e5dfea
Add docker_postgres.sh for PostgreSQL extension and functions
fazlerabbi37 c03fcd0
Add Dockerfile.postgres for postgres app
fazlerabbi37 c66841a
Add docker-compose.yml to build the complete app
fazlerabbi37 b19e46b
modify Dockerfile.postgres to fix script path
fazlerabbi37 e8d364b
modify Dockerfile.postgres to upgrade postgres image to 11
fazlerabbi37 5d685e7
modify Dockerfile to change postgresql-client package and install pha…
fazlerabbi37 91b69b1
modify docker-compose.yml to fix context path
fazlerabbi37 a1ab596
add DOCKER.md file with docker setup instructions
fazlerabbi37 59b399e
add .dockerignore file to list ignorable file for docker
fazlerabbi37 cf63d94
modify DOCKER.md to add database population
fazlerabbi37 afdc9e9
modify DOCKER.md to add app config
fazlerabbi37 ed004c7
moved docker_postgres.sh to docker directory
fazlerabbi37 d382d4c
modify Dockerfile.postgres file to change pgsql script location
fazlerabbi37 da0b13c
modify Dockerfile.postgres file to add app location and gem install
fazlerabbi37 27ff12e
modify Dockerfile.postgres file to move db function directory
fazlerabbi37 7754f8f
modify docker_postgres.sh to fix libpgosm location
fazlerabbi37 d51662c
modify Dockerfile to add osmosis
fazlerabbi37 bc75e4e
modify DOCKER.md to add osmosis instructions
fazlerabbi37 f759f40
modify Dockerfile.postgres to add ruby compilation
fazlerabbi37 d64c706
modify Dockerfile.postgres to change owner to postgres
fazlerabbi37 bfc205f
Reorganize docker-compose files
jalessio 2702431
Require newer version of docker-compose
jalessio 5fcbabe
Re-org includes following directory re-org
jalessio 50dce9a
Use currently 'supported' version of Postgres
jalessio 8c4a16f
Move 'ADD' later in build process to preserve layer caches
jalessio 2e887d0
Dockerfile cleanup to minimize image layers
jalessio a62ead6
Drop unneeded image names from docker-compose.yml
jalessio 6b77b8d
Update build context for 'db'
jalessio c68ce09
More re-org of main Dockerfile for better layer caching
jalessio 56019ec
Remove dockerignore file for less build surprises
jalessio 2179344
Fix db Dockerfile path
jalessio 2104509
Port forward Postgres to a less common port to avoid conflicts
jalessio 2bd5abb
Rename Postgres init script
jalessio 707da60
Build shared object in Dockerfile, not at run time
jalessio b8a2a9a
Use PG11. Don't install Ruby - just get headers for PG extension.
jalessio 4105547
Add db name to database URL for docker-compose
jalessio 609afda
Temporary(?) Makefile for docker-compose shortcuts
jalessio c9e8721
This project doesn't require PostGIS
jalessio 7fcfbe9
Remove line from Makefile which was not working as expected
jalessio a3a989f
Remove need for shared library PostgreSQL functions
jalessio e28ab32
First pass modifications on Docker README
jalessio 8be4a40
Use Docker volume to persist Postgres data on host filesystem
jalessio 37142f1
Exlude PGDATA directory from Docker build context
jalessio b397c05
Use 'openstreetmap' user to connect to db
jalessio c0ec8bc
Add a helper command for populating the database
jalessio 7748dd4
DOCKER readme tweaks
jalessio ba08eec
Move db config to yml file
jalessio cb2881c
Add helper command for running tests w/Docker
jalessio File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
docker-db-data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,3 +18,6 @@ public/attachments | |
public/export | ||
storage | ||
tmp | ||
|
||
# docker-compose database directory | ||
docker-db-data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
# Using Docker and Docker Compose to run OpenStreetMap | ||
|
||
Using [Docker](https://www.docker.com/) will allow you to install the OpenStreetMap application and all its dependencies in Docker images and then run them in containers, almost with a single command. You will need to install Docker and Docker Compose on your development machine: | ||
|
||
- [Install Docker](https://docs.docker.com/install/) | ||
- [Install Docker Compose](https://docs.docker.com/compose/install/) | ||
|
||
These instructions gloss over the precise details of the dependencies and their configuration but these can be found in full detail at [INSTALL.md](INSTALL.md). | ||
|
||
The first step is to fork/clone the repo to your local machine. The repository is reasonably large (~150MB) and it's unlikely that you need the full history. If you are happy to wait for it all to download, run: | ||
|
||
git clone https://github.com/openstreetmap/openstreetmap-website.git | ||
|
||
To clone only the most recent version (~23MB), instead use a 'shallow clone': | ||
|
||
git clone --depth=1 https://github.com/openstreetmap/openstreetmap-website.git | ||
|
||
Now change working directory to the `openstreetmap-website`: | ||
|
||
cd openstreetmap-website | ||
|
||
### Storage setup | ||
|
||
cp config/example.storage.yml config/storage.yml | ||
|
||
### Database | ||
|
||
cp config/docker.database.yml config/database.yml | ||
|
||
### App configuration | ||
|
||
cp config/settings.yml config/settings.local.yml | ||
|
||
### Installation | ||
|
||
In the root directory run: | ||
|
||
docker-compose build | ||
|
||
Now if this is your first time running or you have removed cache this will take some time to complete. So grab tea/coffee and sit tight. Once the Docker images have finished building you can launch the images as containers. | ||
|
||
To launch the app run: | ||
|
||
docker-compose up -d | ||
|
||
This will launch one Docker container for each 'service' specified in `docker-compose.yml` and run them in the background. There are two options for inspecting the logs of these running containers: | ||
|
||
- You can tail logs of a running container with a command like this: `docker-compose logs -f web` or `docker-compose logs -f db`. | ||
- Instead of running the containers in the background with the `-d` flag, you can launch the containers in the foreground with `docker-compose up`. The downside of this is that the logs of all the 'services' defined in `docker-compose.yml` will be intermingled. If you don't want this you can mix and match - for example, you can run the database in background with `docker-compose up -d db` and then run the Rails app in the foreground via `docker-compose up web`. | ||
|
||
### Migrations | ||
|
||
While the `db' service is running, open another terminal windows and run: | ||
|
||
docker-compose run --rm web rake db:migrate | ||
|
||
### Node.js modules | ||
|
||
We use Yarn to manage the Node.js modules required for the project: | ||
|
||
docker-compose run --rm web rake yarn:install | ||
|
||
Once these are complete you should be able to visit the app at http://localhost:3000 | ||
|
||
If localhost does not work, you can use the IP address of the docker-machine. | ||
|
||
### Tests | ||
|
||
docker-compose run --rm web rake test:db | ||
|
||
### Bash | ||
|
||
If you want to get into a web container and run specific commands you can fire up a throwaway container to run bash in via: | ||
|
||
docker-compose run --rm web bash | ||
|
||
Alternatively, if you want to use the already-running `web` container then you can `exec` into it via: | ||
|
||
docker-compose exec web bash | ||
|
||
Similarly, if you want to `exec` in the db container use: | ||
|
||
docker-compose exec db bash | ||
|
||
### Populating the database | ||
|
||
This installation comes with no geographic data loaded. You can either create new data using one of the editors (Potlatch 2, iD, JOSM etc) or by loading an OSM extract. | ||
|
||
After installing but before creating any users or data, import an extract with [Osmosis](https://wiki.openstreetmap.org/wiki/Osmosis) and the `--write-apidb` task. The `web` container comes with `osmosis` pre-installed. So to populate data with a `.osm.pbf` use the following command: | ||
|
||
docker-compose run --rm web osmosis \ | ||
--read-pbf /path/to/file.osm.pbf \ | ||
--write-apidb \ | ||
host="db" \ | ||
database="openstreetmap" \ | ||
user="openstreetmap" \ | ||
validateSchemaVersion="no" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
FROM ruby:2.5 | ||
|
||
# fixes dpkg man page softlink error while installing postgresql-client [source: https://stackoverflow.com/a/52655008/5350059] | ||
RUN mkdir -p /usr/share/man/man1 && \ | ||
mkdir -p /usr/share/man/man7 | ||
|
||
# npm is not available in Debian repo so following official instruction [source: https://github.com/nodesource/distributions/blob/master/README.md#debinstall] | ||
RUN curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh && \ | ||
bash nodesource_setup.sh && \ | ||
rm -f nodesource_setup.sh | ||
|
||
# install packages | ||
RUN apt-get update && \ | ||
apt-get install --no-install-recommends -y \ | ||
build-essential \ | ||
curl \ | ||
imagemagick \ | ||
libarchive-dev \ | ||
libffi-dev \ | ||
libmagickwand-dev \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This one should also be reviewed, not sure it is still needed. |
||
libpq-dev \ | ||
libsasl2-dev \ | ||
libxml2-dev \ | ||
libxslt1-dev \ | ||
locales \ | ||
nodejs \ | ||
osmosis \ | ||
postgresql-client \ | ||
ruby-dev && \ | ||
apt-get clean && \ | ||
rm -rf /var/lib/apt/lists/* | ||
|
||
# install npm packages | ||
RUN npm install -g --unsafe-perm \ | ||
phantomjs-prebuilt \ | ||
yarn | ||
|
||
# Setup app location | ||
RUN mkdir -p /app | ||
WORKDIR /app | ||
|
||
# Install gems | ||
ADD Gemfile* /app/ | ||
RUN bundle install | ||
|
||
# Setup local | ||
RUN sed -i -e 's/# en_GB.UTF-8 UTF-8/en_GB.UTF-8 UTF-8/' /etc/locale.gen && \ | ||
echo 'LANG="en_GB.UTF-8"'>/etc/default/locale && \ | ||
dpkg-reconfigure --frontend=noninteractive locales && \ | ||
update-locale LANG=en_GB.UTF-8 | ||
|
||
ENV LANG en_GB.UTF-8 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
docker-build: | ||
docker-compose build | ||
|
||
docker-up: | ||
docker-compose up -d | ||
|
||
docker-db-migrate: | ||
docker-compose run --rm web rake db:migrate | ||
|
||
docker-test: | ||
docker-compose run --rm web rake test:db | ||
|
||
docker-populate-db: | ||
wget https://download.geofabrik.de/north-america/us/district-of-columbia-latest.osm.pbf | ||
docker-compose run --rm web osmosis \ | ||
-verbose \ | ||
--read-pbf district-of-columbia-latest.osm.pbf \ | ||
--write-apidb \ | ||
host="db" \ | ||
database="openstreetmap" \ | ||
user="openstreetmap" \ | ||
validateSchemaVersion="no" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# This configuration is tailored for use with docker-compose. See DOCKER.md for more information. | ||
|
||
development: | ||
adapter: postgresql | ||
database: openstreetmap | ||
username: openstreetmap | ||
password: openstreetmap | ||
host: db | ||
encoding: utf8 | ||
|
||
# Warning: The database defined as 'test' will be erased and | ||
# re-generated from your development database when you run 'rake'. | ||
# Do not set this db to the same as development or production. | ||
test: | ||
adapter: postgresql | ||
database: osm_test | ||
username: postgres | ||
password: | ||
host: db | ||
encoding: utf8 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
version: "3" | ||
|
||
services: | ||
web: | ||
build: | ||
context: . | ||
volumes: | ||
- .:/app | ||
ports: | ||
- "3000:3000" | ||
command: bundle exec rails s -p 3000 -b '0.0.0.0' | ||
depends_on: | ||
- db | ||
|
||
db: | ||
build: | ||
context: . | ||
dockerfile: docker/postgres/Dockerfile | ||
ports: | ||
- "54321:5432" | ||
environment: | ||
POSTGRES_DB: openstreetmap | ||
volumes: | ||
- ./docker-db-data:/var/lib/postgresql/data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
FROM postgres:11 | ||
|
||
# Add db init script to install OSM-specific Postgres functions/extensions. | ||
ADD docker/postgres/openstreetmap-postgres-init.sh /docker-entrypoint-initdb.d/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
#!/bin/bash | ||
set -ex | ||
|
||
# Create 'openstreetmap' user | ||
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" <<-EOSQL | ||
CREATE USER openstreetmap PASSWORD 'openstreetmap'; | ||
GRANT ALL PRIVILEGES ON DATABASE openstreetmap TO openstreetmap; | ||
EOSQL | ||
|
||
# Create btree_gist extensions | ||
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE EXTENSION btree_gist" openstreetmap |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder why imagemagick is still in the list. GPX preview image generation was moved to gd2-ffij earlier this year, and I'm not aware of other parts of the website depending on imagemagick.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The imagemagick requirements are still listed in the "Minimum Requirements" section of
INSTALL.md
so if they are dropped here they should probably be dropped there as well.https://github.com/openstreetmap/openstreetmap-website/blob/master/INSTALL.md#minimum-requirements
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, it seems the
INSTALL.md
file also needs a bit of cleanup.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mmd-osm is there anything in the test suite which would verify whether this dependency is needed? If not, any suggestions on a suitable test I could add?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought we got rid of imagemagick, but it seems that 6c20244 introduced mini_magick, which depends on imagemagick. This is only used to resize avatar images and might be a bit heavyweight for the task at hand. Maybe @tomhughes knows more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well I don't believe there is any other supported approach. It only uses it via the command line though so there is no need for the development package.