Design Concept: Continuous Integration & Testing

[arcticicestudio]

This epic documents the design concept for the continuous integration & testing of Nord Docs to ensure a great project quality. Therefore a setup of stable and proven continuous integration providers will be used which attaches great importance to the support of open source projects.

🚧 This is a living document which means it is work in progress, not completed yet and will be extended!

All implementation details and requirements are documented and tracked in the corresponding issues:

• Continuous Integration & Testing #44 “Continuous Integration & Testing”

Circle CI

Circle CI is one of the most popular and proven CI providers that allows to automate pipelines from commit to deploy with a quick, safe, and scaled process. They support many languages and provide great and unique features like orbs, job orchestration, powerful caching, SSH or local builds for easy debugging and first-class Docker and GitHub integration (apps, hooks, etc.) support.
The extensive and well-written documentations make it easy to set up and configure it for the usage in projects.
Circle CI has been used for almost all open source projects of Arctic Ice Studio and will be used as primary CI provider for Nord Docs too.

Before starting to run builds, Circle CI must be authorized on a per project basis to access the code on GitHub. After the process has been authorized (GitHub OAuth) the project will be added to the dashboard and required webhooks to listen for new Git events will be automatically set up. See the documentation about how to get started and add a project to the dashboard.
NOTE: The latest config 2.1 features are currently still in Beta Preview and must be enabled explicitly for each project via Settings → Advanced Settings > “Enable build processing (preview)”! See the “Enabling Build Processing” documentation for more details.

The Docker image of the CI container will be the currently latest stable Node.js LTS version 10.x.x and the previous LTS version 8. The -browsers-legacy tag suffix can be used later on for end-to-end tests (#38) with Cypress. Next to Node.js these images currently include Chrome, Firefox, Java 8, and PhantomJS.

Travis CI

Travis CI is also one of the most popular and proven CI providers and will also be used for Nord Docs as a secondary CI provider. They provide similiar features like Circle CI and allow parallel builds with different system environment (Linux and macOS).

Update: Travis CI will not be used for the time being since Circle CI already provides all features that made Travis CI an additional solution, but Circle CI version 2.0 and 2.1 include a lot of more unique features and it developed activly with a great API, CLI, and powerful & flexible build configuration.

Codecov

Codecov is a service to improve the code review workflow and quality. They provide highly integrated tools to group, merge, archive, and compare coverage reports with an great integration for GitHub and many CI providers like the ones used for this project: Circle CI and Travis CI. The support for open source is given through the free plan.
The fantastic and detailed docs include all necessary information to set up the automated CI tasks that Nord Docs will go through.

To gather and upload all created coverage reports Nord Docs will use the codecov/codecov Circle CI orb which provides the upload command for easy integration into jobs and workflows. The specified flags can be everything like e.g. “unit” or “e2e” which are also configurable in .codecov.yml for custom paths which flag should be used.

Workflow

This section documents Nord Docs's continuous integration & testing workflow steps.

NOTE:: There are small differences between the setup for Circle CI and Travis CI so all steps are marked which provider the step will be applied to!

Important: If the build configuration make use of custom environment variables defined and injected through contexts or per-project environment variables make sure that these have been created through the Circle CI UI or via the API before running the project!

1. Checkout of the source code - Clone the GitHub repository to the CI container as working directory.
2. Print build and environment information - Prints details like the branch name, commit hash and container data. See the provider specific documentations for Circle CI and Travis CI.
3. Restore the dependency cache (node_modules) - Restore the node_modules folder based on the checksum of the package-lock.json file and an version prefix which allows to simply invalidate the cache. See the provider specific documentations for Circle CI and Travis CI.
4. Install the dependencies - Run NPM ci command to install all dependencies.
5. Install the Codecov CLI globally - To upload the coverage reports (generated in later steps) Codecov provides a NPM package as CLI which can be installed globally and called in CI build steps.
6. Run all configured linters - Run all installed and configured linters using the lint NPM script.
7. Run all tests - Run all unit, integration and end-to-end tests and generate the coverage report using the test NPM script.
8. Build a production bundle - Generate a production build with Gatsby using the build NPM script.
9. Save the dependency cache (node_modules) - Save the node_modules folder again to the cache based on the checksum of the package-lock.json file and the version prefix which allows to simply invalidate the cache. See the provider specific documentations for Circle CI and Travis CI.
10. Upload generated coverage reports to Codecov - Codecov provides a NPM package which detects the CI it runs on and automatically uploads the reports.
11. Store all generated artifacts - This includes the whole production bundle stored in the created public folder, the coverage and webpack bundle reports in the build/reports folder and the node_modules_ dependency folder to allow to reproduce the build locally with the exact same resolved dependencies.

GitHub Integration Configuration

To integrate all used services some GitHub repository settings must be made:

Configure protected branch rules - To enforce rules for branches to pass the CI build and include updates for changed code coverage stats GitHub provides a way to create rules for branches. Nord Docs will enforce a minimum of one required green build per PR.

Jest JUnit Reports

Jest is able to generate JUnit formatted reports using the jest-junit package. It can be defined in the reporters array in the main configuration file or specified with the --reporters CLI option. It will be added as additional reporter next to the default report format.
See Circle CI's documentation about to how to collect test data for more details about setups with Jest.

ESLint JUnit Reports

Next to the generated reports by Jest, it is also possible to create lint reports in the JUnit format by using builtin formatters. This is useful to create an overview of linted files and found errors/warnings. These reports can be evaluated by services like Circle CI and Codecov to create detailed code style analyzes.

NPM Scripts

To simplify the workflow, new NPM scripts will be added tailored for CI environments. This includes the generation of the JUnit formatted reports by Jest and ESLint.