How to do a Deployment

How to Perform a BFD Deployment

Follow this runbook to successfully build and deploy the BFD Services code to target environment(s).

1. Create a new GitHub tag-based Release

• Visit the Build Release Job in GitHub Actions
• Click "Run Workflow" and enter the desired inputs:

  Input	Description
  Use workflow from	Use Branch: master for production deployments
  The branch on which a release is based	Use master for production deployments
  Version to be used as tag and release	Enter the version you intend to release today
  Post-release Development version	Enter a next version for development
  AWS Region to upload artifacts to	Use us-east-1. Only change in the event of disaster recovery scenarios

• Monitor GitHub Actions log output for your Build Release job
• Upon Completion, this should generate:
  • a git tag reference that matches the Version to be used as a tag and release from above
  • a release based on this git tag reference
  • reference artifacts attached to the desired GitHub Release
  • matching versioned deployable artifacts stored in AWS Code Artifact (for disaster recovery: AWS Code Artifact in us-west-2)

---

2. Access WEB-based CloudBees SDA Cloud Operations Center (Jenkins) https://jenkins-east.cloud.cms.gov/bfd and select BFD from the Dashboard.

   • Optional: The pipeline includes a couple of fail-safes that can be optionally enabled to lock environments before moving on to the next stage of the pipeline. To create an exclusive lock on one or more environments, select Lockable Resources from the left-side list of actions; the UI provides an interface where each environment can be locked by toggling its checkbox. Locking an environment precludes another build from deploying to the locked environment.

3. From the list of Jenkins jobs, select the BFD Multibranch and Multistage Pipeline.

4. Scan for the new tag you created by clicking Scan Repository Now on the left sidebar.

   • Beneath the Scan Repository Now link, select the Scan Repository Log if you want to view the scan log and ensure the tag was picked up

5. Ensure no other jobs are currently building or queued up to build; it's also a good practice to check if another developer is performing work in the Test environment. This can be done by checking the Slack channel #bfd-builds. If the Test environment is not clear, then one should either postpone the build, or contact the developer to see if they can relinquish their claim to the Test environment.

6. While not required, it is convenient to notify BFD developers of the intent to perform a deployment; this can be done by posting a message to the #bfd-builds Slack channel. NOTE: This message should also be posted for the other environments such as prod-sbx & prod.

   • post message: /dibs on test to stake claim to the test environment.
   • post message: /dibs off test to release claim to the test environment.

7. Once the environment is clear, we can deploy from the new tag. In Jenkins, go to the "Tags" tab at the top of BFD Multibranch and Multistage Pipeline.

8. Find the tag in the list that corresponds to your new tag. Click into it.

9. When build commences, the Stage View is displayed showing the now activated current pipeline processing stage(s). If you need to see more detail of the pipeline build, then click the current build number (under Build History). Some useful actions that provide more detailed information and control:

   • Console Output provides a detailed view of the console output log.
   • Pipeline Steps provides a dynamic hierarchical view of the pipeline stages.
   • Pause/resume allows you to stop (and restart) the pipeline processing.

10. When the build pipeline finishes the deployment to Test, the pipeline may pause requesting a Manual Approval to proceed on to prod-sbx, prod. Select Proceed to cause the pipeline to continue its processing to PROD-SBX and PROD if you determine no manual testing is needed on this release.

11. Ensure all environment steps were green (successful).

    • At the conclusion of a deployment to each of the environments (prod, prod-sbx, test), the pipeline performs a step that runs a regression suite vs. the BFD API Services; the intent is to try to proactively verify that no functional or performance regression results from the deployment. It does this by capturing results from the deployment vs. a set of baseline numbers; this has historically been a top contributor to a deployment failures, mainly failing on performance degradation. If an environment fails, more detailed information can be found in:
      • for deployments based on master, we visualize historical regression results in AWS QuickSight for test, prod-sbx, and prod.
      • for for all deployments, Jenkins logs point operator to the raw AWS CloudWatch stored regression suite results

12. When the pipeline has completed the deploy from master, there's a couple of minor tasks that will need to be done:

    • Release your hold on the Test environment in #bfd-builds as needed (/dibs off test).
    • Notify our partners that a deployment to Prod has been completed; this is done by posting a message to the #bfd-users Slack channel. While there is no prescribed content of the message, a best practice is to link to the released tag you created in github which will list out all the changes present in the release:

@here :wave: [BFD has deployed release 2.45.0](https://github.com/CMSgov/beneficiary-fhir-data/releases/tag/2.45.0)