RUNBOOK.md

Origami Bower Registry

Install Financial Times GitHub repositories as Bower components

Code

origami-bower-registry

Service Tier

Platinum

Lifecycle Stage

Decommissioned

Primary URL

https://origami-bower-registry.ft.com

Host Platform

Heroku

Contains Personal Data

No

Contains Sensitive Data

No

Can Download Personal Data

No

Can Contact Individuals

No

Failover Architecture Type

ActiveActive

Failover Process Type

FullyAutomated

Failback Process Type

FullyAutomated

Data Recovery Process Type

NotApplicable

Release Process Type

PartiallyAutomated

Rollback Process Type

PartiallyAutomated

Key Management Process Type

Manual

Architecture

This is mostly a Node.js application but with the following external components:

• An S3 bucket which contains a cache of JSON data
• An organisation-wide Webhook on the Financial-Times GitHub

Loading Data

1. When this service starts, the first thing it does is fetches a JSON file from an S3 bucket. If this bucket isn't available then the application will ignore it – the JSON data is a cache, and is much faster to access than crawling the GitHub API
2. Having loaded the cached data (or not), the application makes several requests to the GitHub API to fetch every repository name in the Financial-Times GitHub organisation. It then discards the original cached JSON data
3. The application replaces the JSON file in the S3 bucket with a new fresh copy of the data in GitHub, so that next time the application starts it has the latest data

Ingestion

1. When a repository is created, renamed, or deleted on the Financial Times GitHub, an organisation-wide webhook makes a request to this service
2. The service then performs step 2 and 3 in "Loading Data" outlined above, ensuring that all of the data is fresh

First Line Troubleshooting

There are a few things you can try before contacting the Origami team:

1. Verify that GitHub and S3 are up. Either of these being down could cause downtime for this application. See GitHub's status page and the Registry Data's __gtg endpoint.
2. Restart all of the dynos across the production EU and US Heroku apps (pipeline here)

Second Line Troubleshooting

If the application is failing entirely, you'll need to check a couple of things:

1. Did a deployment just happen? If so, roll it back to bring the service back up (hopefully)
2. Check the Heroku metrics page for both EU and US apps, to see what CPU and memory usage is like (pipeline here)
3. Check the Splunk logs (see the monitoring section of this runbook for the link)

If only a few things aren't working, the Splunk logs (see monitoring) are the best place to start debugging. Always roll back a deploy if one happened just before the thing stopped working – this gives you the chance to debug in the relative calm of QA.

Monitoring

• Grafana dashboard: graph memory, load, and number of requests
• Pingdom check (Production EU): checks that the EU production app is responding
• Pingdom check (Production US): checks that the US production app is responding
• Sentry dashboard (Production): records application errors in the production app
• Sentry dashboard (QA): records application errors in the QA app
• Splunk (Production): query application logs

Failover Details

Our Fastly config automatically routes requests between the production EU and US Heroku applications. If one of those regions is down, Fastly will route all requests to the other region.

Data Recovery Details

The data stored in S3 is only used as a cache, recovery is not necessary as it's rebuilt from GitHub whenever the application starts.

Release Details

The application is deployed to QA whenever a new commit is pushed to the main branch of this repo on GitHub. To release to production, the QA application must be manually promoted through the Heroku interface.

Key Management Details

This service uses two keys:

1. GitHub (with read permissions only)
2. AWS (read/write permissions for a single S3 bucket)

The process for rotating these keys is manual, via the GitHub and AWS interfaces.