Skip to content
This repository has been archived by the owner on Apr 3, 2024. It is now read-only.

5. Trouble Shooting and Support

Jump to bottom
Ben Wong edited this page Dec 4, 2018 · 7 revisions

Tools

Credentials

Credentials/master-key are stored in 1-Password or in the environment settings of heroku. The keys for google auth in development mode and the AWS bucket for the initial data import are held rails credential store, but are not used in the general running of the app.

What normal looks like

  • Background jobs are kicked off every 5 minutes
  • Web requests should < 10s to load
  • Some requests may take over 20s if very busy - this can happen when we are trying to request appointment data from a system known as DRS. Pages are:
    • Api::WorkOrdersController#notes_and_appointments
    • Api::WorkOrdersController#description

Things that can be done

  • Verify the API is running

    • via swagger
    • via the rails console: 
      > Hackney::WorkOrder.find '01902630'
=> #<Hackney::WorkOrder:0x000055cf9472fd08 @reference="01902630", @rq_ref="03529606",
   @prop_ref="00034545", @created=Thu, 02 Aug 2018 14:26:21 +0000, @date_due=Fri ...

  • Restart the dynos

    • can be done at any time, the workers should be able to handle a restart without any data loss

  • Restore Neo4j

    • shut down the workers (scale the feed-worker & worker dynos to zero)
    • restore to a last known good backup
    • destroy all the existing jobs via the delayed_job web console
    • restart the workers
    • the feed system will catch up

  • Restoring postgres

    • postgres only contains the delayed job queue - restoring an old backup might cause the feed to lose some citations. However, you could follow the restore neo4j process and restore postgres before destroying all the jobs.

  • Stopping background tasks from being created

    • Edit the heroku scheduler config, but echo in front of each of the commands

In Case of Catastrophic Failure

Clone this wiki locally