Skip to content

Latest commit

 

History

History
365 lines (241 loc) · 11.3 KB

SETUP.md

File metadata and controls

365 lines (241 loc) · 11.3 KB
Raw

Setting up your development environment

There are a few different options to get started:

1. Manual setup for experienced developers (i.e. using an existing development environment):

Install prerequisites

  • MySQL 8.0+
  • PHP 8.0+ (with curl, gd, intl, json, mbstring, mcrypt, mysql, xml and zip extensions)
  • nginx (or other webserver)
  • Node.js 16
  • elasticsearch 6+
  • redis

Clone the git repository

git clone https://github.com/ppy/osu-web.git

Configure .env file

# copy the example file and edit the settings, the important ones are APP_* and DB_*
cp .env.example .env
vi .env

URL rewriting

# for nginx, with root set to the `public` folder of the repo
location / {
    try_files $uri $uri/ /index.php?$query_string;
}

Consult the laravel documentation for non-nginx

Initialize database

php artisan db:create

Install packages and build assets

# will also install composer and yarn
./build.sh

At this point you should be able to access the site via whatever webserver you configured.

2. Using Docker

  • First, install Docker and Docker Compose (on Windows, it's already part of Docker install).
  • Install git.
  • Obtain a GitHub token as it's required for several parts of the website.
  • If using Windows, make sure it's running at least build 2004 and install Ubuntu (or another Linux distro of choice) from Windows Store. Additionally:
    • Make sure it's running WSL2 (convert it if it's still using WSL1).
    • Open Docker settings, go to Resources → WSL Integration → Enable integration with additional distros (enable for the installed distro).
  • Open terminal (or Linux console on Windows).
  • Clone this repository.
  • Set GITHUB_TOKEN environment variable (usually by export GITHUB_TOKEN=ghs_...).
    • It'll be recorded to composer and app config so it doesn't need to be set again next time.
  • Run bin/docker_dev.sh. Make sure the repository folder is owned by the user executing this command (must be non-root). The site will be hosted at http://localhost:8080/.
  • Due to the nature of Docker (a container is killed when the command running in it finishes), the Yarn container will be run in watch mode.
  • Do note that the supplied Elasticsearch container uses a high (1+ GB) amount of RAM. Ensure that your system (or virtual machine, if running on Windows/macOS) has a necessary amount of memory allocated (at least 2 GB). If you can't (or don't want to), you can comment out the relevant elasticsearch lines in docker-compose.yml.
  • To run any of the below commands, make sure you are using the docker container: docker compose run --rm php.
    • To run artisan commands, run using docker compose run --rm php artisan.

Notes

The elasticsearch and db containers store their data to volumes, the containers will use data on these volumes if they already exist.

Elasticsearch

Existing Elasticsearch indices will be upgraded to new versions on start. Indices from a newer version cannot be used by older versions and downgrades are not supported.

If you need to use a previous version of elasticsearch, e.g. to run osu-elastic-indexer, you can specify a previous version in a docker-compose.override.yml file (volumes and environment must be specified, as well, not just image):

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.23
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    environment:
      action.auto_create_index: "false"
      discovery.type: single-node
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"

Note that older versions of Elasticsearch do not work on ARM-based CPUs.

osu-elastic-indexer currently cannot update indices using Elasticsearch 7; existing records can still be queried normally.

Mysql

The Mysql images provided by Docker and Mysql have different uids for the mysql user, if you are getting permission errors when starting the db container like

mysqld: File './binlog.index' not found (OS errno 13 - Permission denied)

update the ownership of the mysql data files:

docker compose run --rm db sh -c 'chown -R mysql:mysql /var/lib/mysql'

Windows

On Windows, the files inside Linux system can be found in Explorer from \\wsl$ location.

Default user home directory can be found inside home → <username>.

Due to difference in file permission and line endings, adjustments on git may be needed. Run these in the repository directory:

git config core.eol lf
git config core.filemode false

Docker hints

Services

There are multiple services involved:

  • php: main service for php server. Also serves as entry point for doing other stuff like testing etc
  • assets: builds assets. It sometimes behaves weirdly in which case try restarting it
  • job: runs queued job
  • schedule: runs scheduled job every 5 minutes
  • migrator: prepare database and elasticsearch (service should exit with status 0 after finishing its task)
  • beatmap-difficulty-lookup-cache: for computing beatmap difficulty attributes
  • notification-server: main service for notification websocket server
  • notification-server-dusk: notification server to be used by browser test
  • db: database server. Can be skipped by commenting it out and setting a different database instance
  • redis: cache and session server. Can be skipped just like db service
  • elasticsearch: search database. Can be skipped just like db service
  • nginx: proxies php and notification-server(-dusk) so they can be accessed under same host
  • score-indexer: Solo\Score indexer.
  • score-indexer-test: Solo\Score indexer used by tests.

Modifying environment (.env, .env.dusk.local) files

Sometimes a restart of notification-server and notification-server-dusk will be needed when changing those files.

Example commands

See if anything has stopped:

docker compose ps

Start docker in background:

bin/docker_dev.sh -d

Start single docker service:

docker compose start <servicename>

Restart single docker service:

docker compose restart <servicename>

Direct database access

Using own mysql client, connect to port 3306 or MYSQL_EXTERNAL_PORT if set when starting up docker.

Alternatively, there's mysql client installed in php service:

docker compose run --rm php mysql

Updating image

Docker images need to be occasionally updated to make sure they're running latest version of the packages.

docker compose build --no-cache

Faster php commands

When frequently running commands, doing docker compose run may feel a little bit slow. An alternative is by running the command in existing instance instead. For example to run artisan tinker:

docker compose exec php /app/docker/development/entrypoint.sh artisan tinker

Add an alias for the docker compose command so it doesn't need to be specified every time:

alias p='docker compose exec php /app/docker/development/entrypoint.sh'
p artisan tinker

(add the alias line to shell startup file; usually ~/.profile, ~/.zshrc, etc)

Development

Creating your initial user

In the repository directory:

php artisan tinker
>>> (new App\Libraries\UserRegistration(["username" => "yourusername", "user_email" => "your@email.com", "password" => "yourpassword"]))->save();

Generating assets

# build assets (should be done automatically if using docker)
yarn run development

Note that if you use the bundled docker compose setup, yarn/webpack will be already run in watch mode.

Reset the database + seeding sample data

php artisan migrate:fresh --seed

Run the above command to rebuild the database and populate it with sample data. In order for the seeder to seed beatmaps, you must enter a valid osu! API key as the value of the OSU_API_KEY property in the .env configuration file, as the seeder obtains beatmap data from the osu! API. The key can be obtained from the "Legacy API" section of your account settings page.

Continuous asset generation while developing

To continuously generate assets as you make changes to files (less, coffeescript) you can run webpack in watch mode.

yarn run watch

Email

You can watch emails being sent from osu-web by watching the storage/logs/laravel.log file. Make sure the MAIL_DRIVER is set to log.

Use the API from osu!

To connect from osu!(lazer) via the API offered by osu-web, you need to create a special OAuth password client with:

php artisan passport:client --password

You can then change the constants in the osu! repository (./osu.Game/Online/API/APIAccess.cs).

Testing

To run test, first copy .env.testing.example to .env.testing and .env.dusk.local.example to .env.dusk.local. Make sure to set ES_INDEX_PREFIX and all the databases to something other than production.

Once the env files are set, database for testing will need to be setup:

Initializing the test database

Tests should be run against an empty database, to initialize an empty database:

APP_ENV=testing php artisan migrate:fresh --no-interaction

or if using docker:

docker compose run --rm -e APP_ENV=testing php artisan migrate:fresh --no-interaction

IMPORTANT

If there are existing matching databases, they will be dropped!

PHP tests

PHP tests use PHPUnit, to run:

bin/phpunit.sh

or if using Docker:

docker compose run --rm php test phpunit

Regular PHPUnit arguments are accepted, e.g.:

bin/phpunit.sh --filter=Route --stop-on-failure

Test groups

Some tests are marked with a @group they require a specific service to be available. These groups can be used to exclude tests:

bin/phpunit.sh --exclude=RequiresScoreIndexer,RequiresBeatmapDifficultyLookupCache

or run only those tests:

bin/phpunit.sh --group=RequiresScoreIndexer
  • RequiresBeatmapDifficultyLookupCache: Requires beatmap-difficulty-lookup-cache to be running
  • RequiresScoreIndexer: Requires a score indexing schema to be set and score-indexer-test service to be running

Most tests require elasticsearch and redis to be available, so these are not optional.

Browser tests

Browser tests are run using Laravel Dusk:

bin/run_dusk.sh

or if using Docker:

# `compose exec` doesn't work here due to port conflict with dev instance
docker compose run --rm php test browser

Known Issues

The Dusk tests currently do not clean up completely, leaving behind test data in the database; the database should be reintialized after running a Dusk test.

Javascript tests

Javascript tests are run with Karma.

Karma is currently configured to use Headless Chrome by default; this will require Chrome or a standalone Headless Chrome to be already installed. If you are using Docker, Headless Chrome will already be installed in the container.

yarn karma start --single-run

or if using Docker:

docker compose run --rm php test js

Documentation

php artisan scribe:generate

Documentation will be generated in the docs folder in both html and markdown formats.