Skip to content

Bat Team Tips

Matt Roberts edited this page Feb 18, 2022 · 21 revisions

Production Incidents

When we get a production incident from PagerDuty or DataDog alert, we typically go through the following.

Check Logs and Monitors

Take action

Support Issues

When we get a support issue in the #appeals-batteam Slack channel, we typically go through the following.

Replicate in Caseflow

Locate the appeal in Rails console

  • ssm into a production machine:
ssm caseflow prod auto
sudo su -c "source /opt/caseflow-certification/caseflow-certification_env.sh; cd /opt/caseflow-certification/src; bin/rails c"
  • Get the appeal/case id from the URL if possible
    • URL format for queue is https://appeals.cf.ds.va.gov/queue/appeals/<vacols_id or appeal_uuid>
  • Retrieve an AMA appeal
    • Given appeal uuid: Appeal.find_by(uuid: "b6a684f8-37f0-4856-a0ec-426d138545f6")
    • Given a docket number for an AMA appeal, it is in the format #{receipt_date.strftime('%y%m%d')}-#{id} (see def docket_number in appeal.rb), so just use the id number after the hyphen to grab the appeal: Appeal.find(id)
  • Retrieve a Legacy appeal
    • Given VACOLS id: LegacyAppeal.find_by(vacols_id: 2717083)
    • Given docket number:
docket_number = "1853273A"
vacols_id = VACOLS::Folder.find_by(tinum: docket_number).ticknum
appeal = LegacyAppeal.find_by(vacols_id: vacols_id)
  • Print the task tree for an appeal
puts appeal.structure_render(:id, :status, :updated_at)

To find similar real task trees, check out Task Tree docs

Locate the Intake models in Rails console

  • Similarly to AMA Appeals, HLRs and SCs are identified by UUID:
sc = SupplementalClaim.find_by(uuid: "77cdc5af-cc7b-4662-acd6-0ddfc3ed0136")
  • Given a VBMS Claim ID, we can also retrieve Caseflow's corresponding EPE:
epe = EndProductEstablishment.find_by(reference_id: CLAIM_ID)
  • Once you have any relevant model (even just a Veteran object), print the intake structure:
puts IntakeRenderer.render(obj)

Read more about the Intake Structure Renderer on its wiki page.

Keep the Intake Data Model handy, as a guide to the complicated relationships between intake models.

Search for existing solutions

Search on:

Find expertise

  • Ask Tech Lead of relevant team/product:
    • Echo (Queue, Reader): JC
    • Foxtrot (Intake, Decision Reviews): Yang
    • Tango (Hearings): Tomas
  • Ask Engineering Lead: Lowell

Sentry Alerts

When we investigate a Sentry alert, we typically do the following.

Inspect the Sentry alert

  • Remember to enable socks proxy sockson <mfa_pin>. See utility scripts and
  • Click on the alert link in Slack which loads https://sentry.ds.va.gov/department-of-veterans-affairs/caseflow/issues/... in a browser
  • Click the Comments tab to check for any previous work or ticket reference
  • Click the Events tab to check for frequency and/or patterns

Search for existing solutions

  • Scroll up in Slack for comments and reactions to similar alerts
  • Use the search bars to look for existing GitHub issues
  • See the other "Search for existing solutions" section above

Creating a GitHub issue/ticket

  • See the relevant section in the Bat Team manual.
  • Post ticket in Sentry alert's Comment tab and in a Slack thread for the alert.

All the Caseflow Sentry alerts.

Clone this wiki locally