Deployment for self-hosting

[ojeytonwilliams]

Chapter should have at least one well documented (and hopefully inexpensive) way to deploy the site.

To that end, I'm going to try to deploy to https://railway.app/ and document the process.

[ojeytonwilliams]

It's been a while, but I've created a template https://railway.app/template/it9kIn. Sorry for the delay, @allella, but it's ready for you test.

The process is still quite involved, unfortunately. What you need to do is the following:

1. Setup an Auth0 account https://auth0.com/
2. Create an Auth0 Application https://manage.auth0.com/ > Applications > Applications > Create Application
3. Navigate to https://railway.app/new/template/it9kIn
4. Fill in the environment variable with placeholder text (they're all required, but it's easier to configure them later)
5. Wait until the Activity panel appears on the right
6. Go to Chapter Client > Settings > Domains > Generate Domain (you can customise the domain it generates)
7. Go to Chapter Server > Settings > Domains > Generate Domain
8. Click on Settings (above the panel) > Shared Variables > production
9. Add AUTH0_AUDIENCE, AUTH0_DOMAIN (from dashboard > Applications > APIs > Auth0 Management API and dashboard -> Applications -> your-application -> Settings, respectively)
10. Add CLIENT_LOCATION with the domain you generated
11. Add USE_AUTH0=true
12. Go to Chapter Client > Variables > Raw Editor and add NEXT_PUBLIC_USE_AUTH0=${{shared.USE_AUTH0}} NEXT_PUBLIC_AUTH0_DOMAIN=${{shared.AUTH0_DOMAIN}} NEXT_PUBLIC_AUTH0_CLIENT_ID=Your client id from the dashboard NEXT_PUBLIC_AUTH0_AUDIENCE=${{shared.AUTH0_AUDIENCE}} NEXT_PUBLIC_SERVER_URL=Your full server url # must be a complete url e.g. https://www.example.com NEXT_PUBLIC_CLIENT_URL=${{shared.CLIENT_LOCATION}} NEXT_PUBLIC_DEPLOYMENT_ENVIRONMENT=production
13. Go to Chapter Server > Variables > Raw Editor and add AUTH0_AUDIENCE=${{shared.AUTH0_AUDIENCE}} AUTH0_DOMAIN=${{shared.AUTH0_DOMAIN}} CLIENT_LOCATION=${{shared.CLIENT_LOCATION}} USE_AUTH0=${{shared.USE_AUTH0}} COOKIE_DOMAIN=up.railway.app SESSION_SECRET=SetThisTo32orMoreRandomCharacters UNSUBSCRIBE_SECRET=SetThisTo32orMoreRandomCharacters CHAPTER_EMAIL=Whatever you like <chapter@email.address> SES_ACCESS_KEY_ID= SES_SECRET_ACCESS_KEY= DATABASE_URL=${{Postgres.DATABASE_URL}} PGDATABASE=${{Postgres.PGDATABASE}} PGHOST=${{Postgres.PGHOST}} PGPASSWORD=${{Postgres.PGPASSWORD}} PGPORT=${{Postgres.PGPORT}} PGUSER=${{Postgres.PGUSER}} EMAIL_SERVICE=one of (smtp, sendgrid and ses) EMAIL_HOST= EMAIL_PASSWORD= EMAIL_USERNAME= SENDGRID_KEY= SENDGRID_EMAIL=

Depending on which email service you'd like to use, you need (EMAIL_USERNAME, EMAIL_PASSWORD and EMAIL_HOST), (SENDGRID_KEY) or (SES_ACCESS_KEY_ID and SES_SECRET_ACCESS_KEY). That is for SMTP, Sendgrid and SES respectively.

Configure Auth0

Once this is done and you have your client URL, it should be added to your Auth0 application's Allowed Callback URLs, Allowed Logout URLs and Allowed Web Origins.

Prepare the database

# starting from the root of the repo
# install railway cli locally https://docs.railway.app/develop/cli
railway login
railway link
# connect to the server, not the client:
railway service
# install dependencies
npm i
# run the migration
railway run npm run prisma migrate deploy
# go to your site and login with the user you want to be an admin
railway run node server/prisma/seed/promoteToOwner.js

And, finally, that's it. All being well, your chapter instance should be up and running.

I know that's a lot, so please let me know if there's anything that needs to be clarified or doesn't work as expected. Hopefully it can be refined in time!

[allella]

Thanks. I'll work on running through these notes and hopefully in the next few days.

[allella]

@ojeytonwilliams I started deploying the Railway template and found it asking for more and more access to GitHub. This seems like too much.

Is all of the following permissions really needed just to deploy an app on Railway?

[allella]

More specifically, why would it need access to All repositories?

If I'm going to limit it, then would I select my fork of Chapter under "Only select repositories" ? If so, do I need to sync up with the upstream before deploying?

[ojeytonwilliams]

Yes, that's correct. Since Railway asks for quite a lot of permissions, I only grant it access to specific repositories - the fork of Chapter in this case. And, yes, it would be a good idea to bring your fork up to date.

[gikf]

Sorry, I didn't see added instructions earlier.

9. Add AUTH0_AUDIENCE, AUTH0_DOMAIN (from dashboard > applications > APIs > Auth0 Management API and dashboard > APIs >Auth0 Management API, respectively)

Second one should be dashboard -> Applications -> settings I think.

---

I've followed instructions, but I'm getting both client and server builds failing.

Before starting builds, I had to manually change Source Repo both for client and server (they were both gikf/chapter Server I believe).

In the build logs, errors start from postInstall.js

Log

#11 83.38 --------------------------
#11 83.38 🎉 WELCOME TO CHAPTER 🎉
#11 83.38 --------------------------
#11 83.38 fatal: not a git repository (or any of the parent directories): .git

#11 83.39 Error: Command failed: git remote -v
#11 83.39 fatal: not a git repository (or any of the parent directories): .git
#11 83.39
#11 83.39     at checkExecSyncError (node:child_process:871:11)
#11 83.39     at execSync (node:child_process:943:15)
#11 83.39     at setup (/app/scripts/postInstall.js:13:18)
#11 83.39     at Object.<anonymous> (/app/scripts/postInstall.js:55:1)

#11 83.39     at Module._compile (node:internal/modules/cjs/loader:1159:14)
#11 83.39     at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
#11 83.39     at Module.load (node:internal/modules/cjs/loader:1037:32)
#11 83.39     at Module._load (node:internal/modules/cjs/loader:878:12)
#11 83.39     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
#11 83.39   status: 128,
#11 83.39   signal: null,
#11 83.39   output: [
#11 83.39     null,
#11 83.39     '',
#11 83.39     'fatal: not a git repository (or any of the parent directories): .git\n'
#11 83.39   ],
#11 83.39   pid: 328,
#11 83.39   stdout: '',
#11 83.39   stderr: 'fatal: not a git repository (or any of the parent directories): .git\n'
#11 83.39 }
#11 83.39 You don't have a .env
#11 83.39 Copying .env.example to .env
#11 83.39 Copied!

Then docker build fails

Log

#13 [stage-0  9/10] RUN --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-client/next/cache,target=/app/client/.next/cache --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-node_modules/cache,target=/app/node_modules/.cache npm run build
 
#13 0.708 npm
WARN config production Use `--omit=dev` instead.
#13 0.724
#13 0.724 > chapter@0.0.1 build
#13 0.724 > cross-env NODE_ENV=production docker compose -f docker-compose.yml build
#13 0.724
#13 0.816 node:events:491
#13 0.816       throw er; // Unhandled 'error' event
#13 0.816       ^
#13 0.816
 
#13 0.816 Error: spawn docker ENOENT
#13 0.816     at ChildProcess._handle.onexit (node:internal/child_process:283:19)
#13 0.816     at onErrorNT (node:internal/child_process:476:16)
#13 0.816     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
#13 0.816 Emitted 'error' event on ChildProcess instance at:
#13 0.816     at ChildProcess._handle.onexit (node:internal/child_process:289:12)
#13 0.816     at onErrorNT (node:internal/child_process:476:16)
#13 0.816     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
#13 0.816   errno: -2,
#13 0.816   code: 'ENOENT',
#13 0.816   syscall: 'spawn docker',
#13 0.816   path: 'docker',
#13 0.816   spawnargs: [ 'compose', '-f', 'docker-compose.yml', 'build' ]
#13 0.816 }
#13 0.816
#13 0.816 Node.js v18.12.1
 
#13 ERROR: process "/bin/bash -ol pipefail -c npm run build" did not complete successfully: exit code: 1
-----
> [stage-0  9/10] RUN --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-client/next/cache,target=/app/client/.next/cache --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-node_modules/cache,target=/app/node_modules/.cache npm run build:
#13 0.816     at onErrorNT (node:internal/child_process:476:16)
#13 0.816     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
#13 0.816   errno: -2,
#13 0.816   code: 'ENOENT',
#13 0.816   syscall: 'spawn docker',
#13 0.816   path: 'docker',
 
#13 0.816   spawnargs: [ 'compose', '-f', 'docker-compose.yml', 'build' ]
#13 0.816 }
#13 0.816
#13 0.816 Node.js v18.12.1
-----
 
Dockerfile:25
-------------------
23 |     # build phase
24 |     COPY . /app/.
25 | >>> RUN --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-client/next/cache,target=/app/client/.next/cache --mount=type=cache,id=s/011ac8ac-6370-4bab-b006-a7ab10c98946-node_modules/cache,target=/app/node_modules/.cache npm run build
26 |
27 |
-------------------
 
ERROR: failed to solve: process "/bin/bash -ol pipefail -c npm run build" did not complete successfully: exit code: 1
 
Error: Docker build failed

[ojeytonwilliams]

Second one should be dashboard -> Applications -> settings I think.

Ooops. I'd describe it as dashboard -> Applications -> your-application -> Settings, but, yes, my instructions were wrong. Thanks, I'll update them!

[ojeytonwilliams]

Okay, so the postInstall error just needs to be suppressed. It's only telling us that it cannot set up an upstream remote and we don't need that for deployment.

As for the rest of the errors, it looks like the template doesn't include the custom build command. Instead it runs npm run build from the root. That's no good for us, because railway doesn't support docker compose (yet, hopefully they add it).

Since the client and server both rely on the /common directory, I can't simply set the root to be /client and /server respectively. Instead, you have to configure those commands separately.

For the client service go to

• Settings > Build > Build Command and set it to npm run -w=client build

Similarly for the server service

• Settings > Build > Build Command and set it to npm run -w=server build

Monorepos are hard to deploy, to say the least. If we can migrate the server to NextJS, this whole thing should get a lot easier, but I'm not sure that's realistic at the moment.

[gikf]

That helped with the builds 🙆‍♂️. Although that either went awry, or database need to be additionally initialized for the server. Adding && npm run db:migrate:reset to the build command helped with it.

Couple notes:

• I'm still a bit confused whether there really isn't way to start initial build & deployment on the railway page, without some shenanigans with removing linked repository and adding it again. After the first build & deployment was attempted, re-deployment is triggered automatically.
• Railway copies client and server addresses without the https:// this will cause errors during build (client) or after deploying (server). So it'd be good to put emphasis on using the full url.
• Auth0 application that I configured for local development worked fine after adding railway urls. Somehow when creating new one something was not working.

[ojeytonwilliams]

Hey @gikf, sorry for the delay getting back to you. I need to take a little time off, but I should be back to 100% next week.

[gikf]

@ojeytonwilliams No worries!

[ojeytonwilliams]

@gikf #2607 adds a quick and dirty script to help setting up the db. I've also updated my instructions with a quick guide to that.

I'm still a bit confused whether there really isn't way to start initial build & deployment on the railway page, without some shenanigans with removing linked repository and adding it again. After the first build & deployment was attempted, re-deployment is triggered automatically.

Well, it's always possible to railway up to trigger deployments manually. Other than that pushing commits or modifying environment variables should trigger a new one. Or are you saying those things did not trigger new deployments?

Railway copies client and server addresses without the https:// this will cause errors during build (client) or after deploying (server). So it'd be good to put emphasis on using the full url.

Do you remember which URLs you were copying?

Auth0 application that I configured for local development worked fine after adding railway urls. Somehow when creating new one something was not working.

I've added a little section about configuring Auth0, but, ultimately, people will need to check Auth0's docs (Chapter doesn't require anything non-standard, to my knowledge).

[gikf]

Seeding db via the railway cli does the job. Everything worked as intended with the following notes (to not forget about):

• Railway cli has to be used from the project directory, with installed dependencies.
• Repo linked to the template does not have the server/prisma/seed/promoteToOwner.js. I didn't want to touch that, so I've changed the role from db gui in railway.
• Using in railway custom build commands to use the client/server workspace.
• Adding {serverUrl}/login to auth0's allowed callback urls.

---

Other than that pushing commits or modifying environment variables should trigger a new one. Or are you saying those things did not trigger new deployments?

Not sure about push, but before the first deploy, modifying environment variables did not trigger it. After first deploy (triggered differently - ie. disconnect and connect repo) changing environment variables was triggering new deploy.

Do you remember which URLs you were copying?

chapter-client-production.up.railway.app and chapter-server-production.up.railway.app

[allella]

@ojeytonwilliams I got further along and am asked "Where should we clone this repo?" when clicking "Configure" on the "Deploy Chapter" pages.

I left the default value "Chapter Client" and filled out all of the variables with the word "placeholder" and when it tries to deploy I got "Failed to Deploy" and then another "Oh No" error on the second attempt.

When I restart the process on the template page it highlights the "repo name" as being a problem. Do I need to set this to match the name of my fork? allella/chapter

Default Clone Value

**Second Error **

[ojeytonwilliams]

@allella it looks like the problem is that https://github.com/allella/Chapter-Client already exists. When I tried re-doing the process it would not let me re-use a fork.

@gikf cheers, I've updated the instructions. I left the Auth0 instructions alone, though, because the live config uses the client url rather than {serverUrl}/login and works just fine 🤷‍♂️ Also, I've updated the source repo, so there shouldn't be any issue with missing scripts.

[allella]

@ojeytonwilliams yeah, it had an error on the first try, so I'm it may have created that repo but never made a deployable app.

In any case, is the idea to have it creating yet another copy of the repo, as in https://github.com/allella/Chapter-Client, or should I be telling it to use my main chapter repo https://github.com/allella/chapter ?

I purposely haven't read the Railway docs because I wanted to approach this as someone who's never used Railway and see if the instructions we're providing are sufficient to not have to understand their way of doing everything.

[ojeytonwilliams]

Yep, that sounds right.

And, yes, unfortunately it does mean creating a new fork. I think the idea with Templates is that they're for newcomers to get up and running quickly, so the assumption is that they won't already have the repo. It's a bit of an oversight.

I'll update the instructions, anyway.

One thing to note: you can always change the source repo after getting set up.

EDIT: actually, it looks like they've just changed things today and now it's possible to deploy using an existing fork!