Module Installation Broke Automated Production Deploy

[7hunderbird]

Log of Events

• We got an alert in #cms-notifications
  • https://dsva.slack.com/archives/CJT90C0UT/p1715197444181499
• Edmund saw the alert first, created a thread in #sitewide-cms-platform
  • https://dsva.slack.com/archives/CT4GZBM8F/p1715197875453479
• Edmund created a zoom link
• Grace, Robbie, Tyler, and Edmund joined the zoom
• We tasked Grace with getting OOB deploy approval
• Erika was out of pocket for travel
• We began to review the log for the prod deploy in the jenkins job
  • http://jenkins.vfs.va.gov/job/deploys/job/cms-vagov-prod/1063/consoleFull
• The jenkins job runs the post-deploy-tasks.yml
  • https://github.com/department-of-veterans-affairs/devops/blob/e4e4ebddd07384a76f2af7f214c8727f0530bbb9/ansible/deployment/config/cms/tasks/post-deploy-tasks.yml
• When prod is deployed a new prod VM is created
• We ssh'd into the new prod virtual machine
  • Running drush status looked good
  • Tried drush en entity_usage_addons was already enabled
• Because the module was enabled in the database, we decided to manually roll forward the deploy with the module installed
• We got approval for the out-of-band deploy from Erika
• Using the deploy-oob we determined that the rollback scenario we needed to use was the 1) option and finish the deployment manually
• What broke was a rolling out of a module install
  • We need to make the folder before we change the permissions on the folder
• Edmund ran the commands in step 1, then ran the post deploy step in Jenkins
  • http://jenkins.vfs.va.gov/job/deploys/job/cms-vagov-prod/
• So we ran 2A, steps 1-4 to get the asg back to a state where it can accept a new deployment
  1. Abandon/remove new instance with:

  aws autoscaling complete-lifecycle-action \
     --region us-gov-west-1 \
     --auto-scaling-group-name "dsva-vagov-prod-cms-asg" \
     --lifecycle-hook-name launch-hook \
     --lifecycle-action-result ABANDON \
     --instance-id i-0f256f4eae72d5c87
  

  2. Scale-in ASG from 2 to 1 (this may change later as we move to HA)
  3. Remove latest launch-template (first need to set latest-1 to default)
  4. Set deployment_version tag on existing instance from ‘previous’ to ‘latest’
• We went back to this job, kicked it off with no wait:
  • http://jenkins.vfs.va.gov/job/deploys/job/cms-auto-deploy/
  • it failed on the "Set Deployment Site Alert" step
• we connected the previous prod virtual machine to manually install a module on the running instance
  • su - cms
  • composer require drupal/entity_usage_addons
• Tried the deploys/cms-auto-deploy again
  • it failed on the "Compare last passing to last release commit" step
• Since it was stopping the "the release commit is the same" we moved forward to the deploy job
  • http://jenkins.vfs.va.gov/job/deploys/job/cms-vagov-prod/1064/

[7hunderbird]

To discuss what we learned next.

• Ensure that the Ansible creates the folder that the translations file config will go into
• Would it make sense to have the Apache user manage all the folders/files that the CMS uses to release new modules?