environments.md

Environments & the Content Build Process

To enable end to end quality analysis, from the VA.gov CMS through to the public facing website, new feature review, and stakeholder demonstrations, multiple joined CMS/WEB environments may be created by any user with access to Tugboat. (More info on Tugboat.)

The following table lists all environments and CMS/WEB sites used in the development process for VA.gov:

Environment	Drupal (CMS)	Frontend (FE / WEB / Static)	FE Data source	Management
PROD Live Site	prod.cms.va.gov va.gov-cms: main	www.va.gov content-build: main vets-website: main	prod.cms.va.gov	BRD: Jenkins
STAGING Pre-release testing.	staging.cms.va.gov va.gov-cms: main Staging CMS uses Tugboat Preview content-build-branch-build (CMS PROD Mirror) as its data source. This Preview is refreshed daily at 7am UTC (2am EST, 1am EDT), using the latest Database and Asset file snapshot from AWS S3 at that time. * Staging CMS database edits will be overwritten the next time code is merged. * Staging CMS is not the source of data for the Staging front-end, meaning: edits you make in the Staging CMS will not cause the Staging front-end to be re-built, and your changes will not appear in the Staging.va.gov front-end.	staging.va.gov content-build: main vets-website: main	Tugboat Preview content-build-branch-build (CMS PROD Mirror) refreshed daily at 7am UTC (2am EST, 1am EDT), using the latest Database and Asset file snapshot from AWS S3 at that time.	BRD: Jenkins
LOCAL Local development	va-gov-cms.ddev.site	va-gov-cms.ddev.site va-gov-cms.ddev.site/static va-gov-cms.ddev.site/$URL?_format=static_html
CI / PR Pull Requests & Automated Testing	pr###-{hash}.ci.cms.va.gov va.gov-cms: specified branch	web-{hash}.ci.cms.va.gov pr###-{hash}.ci.cms.va.gov/static pr###-{hash}.ci.cms.va.gov/$URL?_format=static_html content-build: main, unless another branch is specified in the CMS vets-website: main It is not possible to specify an alternate vets-website branch until #6434 is complete	Tugboat: CMS > CMS Pull Request Environments > Base Preview, including Database and Asset files, refreshed daily at 10am UTC (5am EST, 4am EDT). Tugboat PR Previews need to be rebuilt (after 5am EST), or refreshed, for the database to stay up-to-date with Prod.	CMS-CI: Tugboat
Demos Demos & Training	cms-{hash}.demo.cms.va.gov	web-{hash}.demo.ci.cms.va.gov cms-{hash}.demo.cms.va.gov/static cms-{hash}.demo.cms.va.gov/$URL?_format=static_html	Tugboat: CMS > CMS Demo Environments > Base Preview, including Database and Asset files, refreshed daily at 9am UTC (4am EST, 3am EDT). Demo Previews need to be rebuilt (after 4am EST), or refreshed, for the database to stay up-to-date with Prod.	CMS-CI: Tugboat

Review Instances

NOTE: Review Instances are another system that provides test environments, for CI PRs in vets-website and vets-api repos. RIs are managed by the Platform team. (Feb 2023: Platform has an active project underway to replace Review Instances; Ray Messina, POC.) That's not a CMS concern, other than to note: Tugboat does also indirectly act as the Content Data Source for RIs.

Environment	Drupal (CMS)	Frontend (FE / WEB / Static)	FE Data source	Management
CI / PR for vets-website & vets-api Pull Requests & Automated Testing	None provided	{hash}.review.vetsgov-internal * vets-website or vets-api: specified branch * content-build: main	Review instances can't pull data directly from Tugboat. To work around this, when content-build merges to main, a process captures the data + files from CI (effectively, Tugboat Preview content-build-branch-build (CMS PROD Mirror) 7am UTC or latest refreshed data/assets) and pushes them to S3 to a different location than the every 15mins data + asset backup. Review Instances then pull and use the new S3 snapshot archive file. Therefore: the freshness of the data on a Review Instance will line up with timing for the last time content-build had a merge to main.	NA

Other

• Access: how to access these environments

• To preview CMS non-prod data: Publish the content in Tugboat PR Preview's CMS, then trigger a content release in the CMS (via /admin/content/deploy). You will see the content built from that Tugboat's CMS data. This will not include any changes to vets-website that are not in main.

• To preview vets-website changes along with CMS / content-build changes:

  • Staging Dark launch (for new content only). Allows preview of combined changes. Uses Staging CMS data & Frontend built from content-build main and vets-website main, so vets-website code changes must be merged to main. If vets-website changes are referenced by any existing CMS content, they should be placed behind a feature flag that can then be enabled on Staging.
  • In Tugboat: you must merge to vets-website main with your changes behind a feature flag. Then, enable the feature flag within the Tugboat instance. (TODO: is this a real thing? Tugboat runs vets-api?)

What is an Environment?

Environments are copies of the production site that are running newer code or have different content that needs to be tested before going live.

Environments can also be used for demonstrations or training, without worrying about disrupting production content.

Each Environment has both a CMS and a WEB site. The CMS is a content management system built with Drupal, and the WEB site is a static HTML site, built with Metalsmith.

The WEB build process consumes the content from the CMS in the same environment.

Important Concepts

• Each WEB site is made up of generated "static" files. This means that the WEB site reflects the content from the CMS at the time the WEB Build process was run.
• CMS Editors will not see changes in the WEB site until a "WEB Build" is triggered and the process completes successfully.
• The WEB site will not be accessible until at least one "WEB Build" has run successfully. This happens automatically for CI environments, but not yet for Demo environments. If you get an error such as "Forbidden" when visiting the WEB site, try running the Rebuild WEB process again.

Manually build a demo env: Step-by-step Instructions

1. Visit https://tugboat.vfs.va.gov/ and click on the "GitHub" link to log in with GitHub.

   

2. Click the CMS link and the click the "CMS Demo Environments" to visit the Demos page.

   

3. Scroll down to the "Base Previews" heading and select "Clone" from the "Actions" menu.

   

4. Your demo environment has been created. It will be titled "master". Scroll to the top of the page, click on the "Settings" link for your new environment, and give it a title starting with your initials and a hyphen. Then, click the "Save Configuration" button.

   

5. If you wish to copy prod.cms.va.gov using the latest content, do not change any additional options.

6. Click the "Preview" button for your environment to visit it.

   

NOTE: The WEB site for this environment will not work until you trigger a WEB Build process on the "Release Content" page.

WEB Build Process

Within each environment, the static HTML for the WEB site is occasionally "rebuilt" so that the latest content from that environment's CMS is used.

The WEB build process is tested in the CI system to ensure compatibility with the CMS content schema.

The WEB build process is triggered automatically by certain actions in the CMS or manually via the command line.

Build Triggers

The WEB instance of an environment is rebuilt when any of the following actions take place in the CMS:

• Facility Alert or Individual Facility Operating Status is created or updated.
• The "Rebuild WEB" button is pressed.
• @TODO: Document all current build triggers.

Note to Developers: Keep this list up to date to help content editors understand the process.

Rebuilding Environments Manually

There is a special button and form for rebuilding VA.gov environments. Use this to manually trigger either a WEB or CMS rebuild (or both), and optionally check out different code.

Step-by-step Instructions

1. Find the environment you would like to build in the CMS-CI site, and go to the preview.

2. Hover over the "Content" admin menu, and then click "Release Content".

   

3. If you wish to just trigger a WEB rebuild with the existing content, do not change any other options.

   

   For advanced users and developers:

   1. If you wish to change the branch of the WEB project, click "Select a different frontend branch/pull request". Start typing to search by the name of the branch or pull request you wish to use.

   

4. Press the "Release content" button. You will see the status of your build under the "Wait for the release to complete" heading. You may view the build logs by clicking the "View logs" link.

   

5. Once the REBUILD process is complete, you can click the "Go to front end" button under the "Access the frontend environment" heading.

   

6. That's it! If the process completed, you should see a site that looks like VA.gov.

Creating new Environments

Within the CMS-CI platform, in "CMS Demo Environments", users can create new environments by cloning the "master" Base Preview.

Hosting Architecture

All environments are hosted on VA GovCloud in AWS GovCloud.

The primary environments, DEV, STAGING, and PROD, are hosted in the BRD system.

Pull Request Environments and Ad Hoc environments are hosted in the CMS-CI system.

SOCKS proxy or PIV+GFE hardware is required for accessing VA internal network.

BRD: Jenkins

Build, Release, Deploy

jenkins.vfs.va.gov

• Source Code: https://github.com/department-of-veterans-affairs/devops/tree/master/ansible/build/roles/cms

• Runs Continuous Integration for about a dozen different applications with different requirements, including vets-website, vets-api, cms, and soon cms-ci.

• BRD Process is standardized across apps using Ansible playbooks and roles. See the DevOps Repo Documentation for more information.

  • The "Build" process creates the entire server image and permantently tags and archives it as an AMI.

  • The "Deploy" process delivers those images to the 3 "environments", DEV, STAGING, and PROD and runs whatever hooks are needed.

  • The "Release" process continuously delivers code to each BRD Environment

    • Primary branch commits are automatically deployed to DEV and STAGING Environments.
    • Git Tags and GitHub Releases are created automatically if those commits pass testing.
    • "Environments" in the context of BRD are really different networks.
    • The "servers" that actually run the apps are activated AMI images, placed into the desired "environment".

CMS-CI: Tugboat

CI Platform

tugboat.vfs.va.gov/

• CMS-CI refers to the VA's implementation of Tugboat.
• Provides an environment per Pull Request, and allows creation of ad-hoc environments with any name, on any desired branch or Pull Request.
• Provides a Web UI for getting information and managing these environments
• Installed with open source Ansible roles, plus a custom playbook.
• Deployment of new releases of CMS-CI and Tugboat is handled by BRD in way very similar to CMS.
• Provides a complete SDLC pipeline for Drupal code:
  • Creates new environments when a PR is open.
  • Automatically tests the environment and passes status to GitHub to allow or block merging.
  • Destroys and rebuilds PR Environments and runs the full test suite again on every git push.
  • If the PR is merged or closed, environment is destroyed.
  • Notifies GitHub of deployment success or failure, with lnks to the environments.
• Includes the FE/WEB Build process in the Drupal CI pipeline.
  • Front-end WEB project can be built inside PR environments by using the "Release Content" page.
  • End-to-end testing of CMS+WEB with Cypress:
    1. Make CMS updates: Change content, publish state, etc.
    2. Run WEB build command to rebuild static assets.
    3. Confirm CMS change is visible in WEB static assets.
• Runs on a single EC2 instance.

SSH access to CI environments

To access a CI environment via ssh, go to that environment's page on Tugboat and click the "Terminal" link on the line for the "php" service.

Resources

• https://va-gov.atlassian.net/wiki/spaces/VAGOV/pages/103448589/VA.gov+CMS+DevOps+2.0+Architecture+Notes
• https://va-gov.atlassian.net/wiki/spaces/VAGOV/pages/28770332/CMS+Infrastructure+CI+CD+Architecture+Proposal+3
• More info on DEPO(OCTO) Environments and Deployments

Table of Contents