checks.yaml

# Copyright 2021 Security Scorecard Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This is the source of truth for all check descriptions and remediation steps.
# Run `cd checks/main && go run /main` to generate `checks.json` and `checks.md`.
checks:
  Maintained:
    risk: High
    tags: supply-chain, security
    short: Determines if the project is "actively maintained".
    description: >-
      This check tries to determine if the project is "actively maintained".

      A project which is not active might not be patched, might not have its
      dependencies patched, or might not be actively tested and used.
      A low score is therefore considered `High` risk.

      A lack of active maintenance is not necessarily a problem.
      Some software, especially smaller utility functions, does not
      normally need to be maintained. For example, a library that
      determines if an integer is even would not normally need maintenance
      (it might only need to be modified if its underlying implementation
      language definition changed, e.g., by adding a new type).  A lack
      of active maintenance simply suggests that potential users
      should investigate further.

      The check currently works by looking whether the repo is archived or not.
      If it is archived, it returns the minimum score. If it is not,
      the check looks for commits within the last 90 days, and
      outputs the highest score if there are at least 1 commit/week during this period.
    remediation:
      - >-
        There is *NO* remediation work necessarily needed here from projects;
        this just indicates the project activity and maintenance commitment.
        External users should determine if this is the
        kind of software that would not normally need active maintenance.
  Dependency-Update-Tool:
    risk: High
    tags: supply-chain, security, dependencies
    short: Determines if the project uses a dependency update tool.
    description: >-
      This check tries to determine if the project uses a dependency update tool.

      Not updating dependencies makes a project vulnerable to known flaws and prone to attacks.
      A low score is therefore considered `High` risk.

      The checks looks for [dependabot](https://dependabot.com/docs/config-file/) or
      [renovatebot](https://docs.renovatebot.com/configuration-options/). This check only looks if
      it is enabled and does not ensure that it is run and pull requests are merged.
      A project may meet this criterion yet have a failing scorecard report;
      there are many ways to implement this criterion, and it's challenging
      for an automated tool (like scorecard) to detect them all.

    remediation:
      - >-
        Signup for automatic dependency updates with dependabot or renovatebot and place the config
        file in the locations that are recommended by these tools.
  Binary-Artifacts:
    risk: High
    tags: supply-chain, security, dependencies
    short: Determines if the project has generated executable (binary) artifacts in the source repository.
    description: >-
      This check tries to determine if the project has generated executable (binary) artifacts in the source repository.

      Including generated executables in the source repository
      increases user risks.
      Many programming language systems can generate
      executables from source code (e.g., C/C++ generated machine code,
      Java `.class` files, Python `.pyc` files, and minified JavaScript).
      Users will often directly use executables if they are included in the
      source repository, leading to many dangerous behaviors.
      One problem is that reviews become ineffective and thus enable the
      use of obsolete or maliciously-subverted executables.
      Reviews generally review source code, not executables, since
      it's difficult to audit executables to ensure that they correspond.
      Since it's hard to audit this, over time
      the included executables might not correspond to the source code.
      Another problem is that it allows the executable generation process to
      atrophy, which can lead to an inability to create working
      executables.  These problems can be countered with verified
      reproducible builds, but it's easier to implement verified
      reproducible builds when executables are not included in the
      source repository (since the executable generation process is less
      likely to have atrophied).

      It's fine to include, in the source repository,
      files that are simultaneously reviewable source code and executables,
      since these are reviewable.
      (Some interpretive systems, such as many operating system shells,
      don't have a mechanism for storing generated executables that are
      different from the source file.)
      Similarly, we allow including in the source repository
      source code generated by other tools
      (e.g., by bison, yacc, flex, and lex).
      There are potential downsides to generated source code, but
      generated source code tends to be much easier to review and thus
      presents a lower risk. Generated source code is also often difficult
      for external tools to detect.

      We allow including generated documentation in source repositories.
      Generated documentation is intended for use by humans
      (not computers), and humans can take context into account.
      Thus, generated documentation doesn't pose the same level of risk.

      A low score is therefore considered `High` risk.
    remediation:
      - >-
        Remove the generated executable artifacts from the repository.
      - >-
        Build from source.
  Branch-Protection:
    risk: High
    tags: supply-chain, security, source-code, code-reviews
    short: Determines if the default and release branches are protected with GitHub's branch protection settings.
    description: >-
      [Branch protection](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches)
      can enable various rules to enforce certain workflows for
      branches, such as preventing rewriting of public history
      (e.g., a *force push*),
      requiring review before acceptance into a main branch,
      or passing certain status checks before acceptance into a main branch.

      Branch protection can reduce the risk of unintentional or malicious
      code from entering the "main" branch.
      Branch protection rules that prevent rewriting of public history
      (e.g., preventing *force push* of public branches) prevent history
      from changing without external notice.
      Branch protection rules that require status checks ensure that
      at least those checks are met before a change is accepted.
      Branch protection rules that require at least one other reviewer
      reviews greatly reduces the risk that a compromise of a contributor's
      account will lead to injection of malicious code. Review also increases
      the likelihood that an unintentional vulnerability in a contribution
      will be detected and fixed before the change is accepted.
      A low score is therefore considered `High` risk.

      Note, however, that requiring reviews for every proposed change
      is impractical for many projects, since many projects simply don't
      have that many active participants. For more discussion, see
      [Code Reviews](#code-reviews).

      In some cases these rules will need to be suspended.
      For example, if a past commit includes illegal content
      (such as child pornography), it may be impractical to hide the commit
      and in such cases the history may need to be rewritten.

      This check determines if the default and release branches are
      protected with GitHub's branch protection settings.
      The check only works when the token has [Admin
      access](https://github.community/t/enable-branch-protection-get-api-without-admin/14197)
      to the repository. This check determines if the default and release branches are
      protected.
    remediation:
      - >-
        Enable branch protection settings in your source hosting provider to
        avoid force pushes or deletion of your important branches.
      - >-
        For GitHub, check out the steps
        [here](https://docs.github.com/en/github/administering-a-repository/managing-a-branch-protection-rule).
  CI-Tests:
    risk: Low
    tags: supply-chain, testing
    short: Determines if the project runs tests before pull requests are merged.
    description: >-
      This check tries to determine if the project runs tests before pull
      requests are merged.

      Running tests helps developers catch mistakes early on.
      A low score is considered 'Low' risk.

      The check works by looking for a set of CI-system
      names in GitHub `CheckRuns` and `Statuses` among the recent commits (~30).
      A CI-system is considered well-known if its name contains any of the
      following: appveyor, buildkite, circleci, e2e, github-actions, jenkins,
      mergeable, test, travis-ci.
      A project may meet this criterion yet have a failing scorecard report;
      there are many ways to implement this criterion, and it's challenging
      for an automated tool (like scorecard) to detect them all.
      If a project's system was not detected and you think it should be, please
      [open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
    remediation:
      - Check-in scripts that run all the tests in your repository.
      - >-
        Integrate those scripts with a CI/CD platform that runs it on every pull
        request (e.g. [GitHub
        Actions](https://docs.github.com/en/actions/learn-github-actions/introduction-to-github-actions),
        [Prow](https://github.com/kubernetes/test-infra/tree/master/prow), etc).
  CII-Best-Practices:
    risk: Low
    tags: security-awareness, security-training, security
    short: Determines if the project has a CII Best Practices Badge.
    description: >-
      This check tries to determine if the project has earned a
      [CII Best Practices Badge](https://bestpractices.coreinfrastructure.org/).

      This badge tells us if the project is applying a particular
      set of security-focused best development practices
      for open source software.
      The CII Best Practices badge has 3 tiers (passing, silver, and gold);
      we give full credit if the
      [passing criteria are met](https://bestpractices.coreinfrastructure.org/criteria/0),
      as achieving a passing badge is a significant achievement
      for many projects.
      We give a little credit if the project is at least working to achieve
      a badge, and increasingly more as more criteria are met.

      For example, to earn the passing badge,
      the project MUST publish the process for
      reporting vulnerabilities on the project site,
      it MUST provide a working build system that can automatically rebuild
      the software from source code (where applicable),
      it MUST have a general policy that tests
      will be added to an automated test suite
      when major new functionality is added,
      it MUST meet various cryptography criteria where applicable,
      it MUST have at least one primary developer who knows how to
      design secure software,
      at least one of the project's primary developers MUST know of
      common kinds of errors that lead to vulnerabilities in this kind
      of software (as well as at least one method to counter or mitigate
      each of them), and at least
      one static code analysis tool (beyond compiler warnings and "safe"
      language modes) MUST be applied to any proposed major production
      release.

      A low score is considered 'Low' risk.
      The check uses the URL for the Git repo and the CII API.
    remediation:
      - >-
        Sign up for the [CII Best Practices
        program](https://bestpractices.coreinfrastructure.org/en).
  Code-Review:
    risk: High
    tags: supply-chain, security, source-code, code-reviews
    short: Determines if the project requires code review before pull requests (aka merge requests) are merged.
    description: >-
      This check tries to determine if the project requires code review before
      pull requests (aka merge requests) are merged.

      Reviewing code improves the quality of code in general,
      because reviews may detect various unintentional
      problems that can be fixed immediately before they are merged.
      Such problems include unintentional vulnerabilities, so unintentional
      vulnerabilities can be reduced through review.
      Reviews also make it
      more difficult for an attacker to insert malicious code
      (either as a malicious contributor or as an attacker who has
      subverted a contributor's account), because a reviewer might detect
      the subversion.
      This increased difficulty can even deter attackers.
      A low score is therefore considered `High` risk.

      However, requiring that all proposed changes be reviewed by someone
      else before merging them is impractical for some projects.
      A project with only one active participant cannot practically
      enforce multi-person review, and even a project with multiple
      active participants may not have enough active participation to be able
      to require review of all proposed changes.
      Such projects will not be able to practically enforce rules such as
      requiring a minimum number of reviewers, and administrators in
      practice will be exempt from reviews.
      Projects with a small number of active participants, but more than one,
      instead sometimes aim for a review of a percentage of proposals
      (e.g., "at least half of all proposed changes are reviewed").

      In addition, note that requiring review does not eliminate all risks.
      The other reviewer(s) might fail to notice unintentional vulnerabilities
      or malicious code. They might be colluding with a malicious developer.
      The "other" reviewers might even be the same person
      (aka a "sock puppet").

      In short, requiring others' review before accepting a change
      reduces risks to users,
      as those additional reviewers may detect problems early,
      but it is not a practical requirement for some projects.

      The check first tries to detect if Branch-Protection is enabled
      on the default branch ,and the number of reviewers is at least 1. If this
      fails, it checks if the recent (~30) commits have a Github-approved
      review or if the merger is different from the committer (implicit review).
      It also performs similar check for reviews using
      [Prow](https://github.com/kubernetes/test-infra/tree/master/prow#readme)
      (labels "lgtm" or "approved") and Gerrit ("Reviewed-on" and "Reviewed-by").
    remediation:
      - >-
        If the project has only one contributor, or does not have enough
        reviewers to practically require that all contributions be reviewed,
        try to recruit more maintainers to the project who will be
        willing to review others' work.
        Ideally at least some of these people will
        be from different organizations.
        If the project has very limited utility, consider expanding its
        intended utility so more people will be interested in improving
        it, and make that larger scope clear to potential contributors.
      - >-
        Follow security best practices by performing strict code reviews for
        every new pull request / merge request.
      - >-
        Make "code reviews" mandatory in your repository configuration. E.g.
        [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#require-pull-request-reviews-before-merging).
      - >-
        Enforce the rule for administrators / code owners as well. E.g.
        [GitHub](https://docs.github.com/en/github/administering-a-repository/about-protected-branches#include-administrators)
  Contributors:
    risk: Low
    tags: source-code
    short: Determines if the project has a set of contributors from multiple organizations (e.g., companies).
    description: >-
      This check tries to determine if the project has a set of contributors from
      multiple organizations (e.g., companies).

      Low score has 'Low' risk.

      Some projects cannot meet this requirement.
      Obviously projects with only one active participant
      cannot meet it.
      Many projects have a narrow scope and may not be able to attract
      the interest of multiple organizations.
      See [Code Reviews](#code-reviews) for added notes about
      projects with a small number of participants.

      The check works by looking at the authors of recent commits
      and checking the `Company` field on the GitHub user profile. A contributor
      must have at least 5 commits in the last 30 commits.
      The highest score is achieved when there are contributors from
      at least 3 different companies in the last 30 commits.
    remediation:
      - >-
        There is *NO* remediation work needed here. This is to provide some
        insights on which organization(s) have contributed to the project and
        making trust decisions based on that. But you can ask your contributors
        to join their respective organizations.
  Fuzzing:
    risk: Medium
    tags: supply-chain, security, testing
    short: Determines if the project uses fuzzing.
    description: >-
      This check tries to determine if the project uses fuzzing.

      Fuzzing is important to reduce the number of vulnerabilities in code.
      A low score is considered 'Medium' risk.

      The check currently works by checking if the repo name is in the
      [OSS-Fuzz](https://github.com/google/oss-fuzz) project list.
      A project may meet this criterion yet have a failing scorecard report;
      there are many ways to implement this criterion, and it's challenging
      for an automated tool (like scorecard) to detect them all.
    remediation:
      - >-
        Integrate the project with OSS-Fuzz by following the instructions
        [here](https://google.github.io/oss-fuzz/).
  Packaging:
    risk: Medium
    tags: supply-chain, security, releases
    short: Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.
    description:
      This check tries to determine if the project is published as a package that others can easily download, install, easily update, and uninstall.

      It's important that the project provide an easy way to
      download, install, update, and uninstall the software.
      It's particularly important to make it easy for users to
      receive security patches as updates.

      This is often done by creating a "package" that is
      easy to install and uninstall by a package manager.
      Many program language ecosystems have a generally-used packaging format
      supported by a language-level package manager
      tool and public package repository.
      Many operating system platforms also have at least one package format,
      tool, and public repository (in some cases the source repository
      generates system-independent source packages, which are then
      used by others to generate system executable packages).
      Container images are yet another way to package software.
      In some situations packaging is not sensible, but it's wise to package
      software in so many circumstances that it's worth checking for it.

      A low score is considered `Medium` risk.

      The check currently looks for [GitHub packaging workflows]( https://docs.github.com/en/packages/learn-github-packages/publishing-a-package)
      and language-specific GitHub Actions that upload the package to a corresponding hub, e.g., [Npm](https://www.npmjs.com/).
      There is a plan to add better support to query package manager hubs directly in the future, e.g., for [Npm](https://www.npmjs.com/), [PyPi](https://pypi.org/).
      A project may meet this criterion yet have a failing scorecard report;
      some widely-used tools are relatively easy to detect, but it's
      challenging for an automated tool (like scorecard) to detect
      them all.
      If scorecard fails to detect the way you publish a package and
      you think scorecard should support your use case, please
      [open an issue in the scorecard project](https://github.com/ossf/scorecard/issues/new/choose).
    remediation:
      - Publish your project as a [downloadable package](https://docs.github.com/en/packages/learn-github-packages/publishing-a-package).
      - Use a GitHub action to release your package to language-specific hubs.
  Pinned-Dependencies:
    risk: Medium
    tags: supply-chain, security, dependencies
    short: Determines if the project has declared and pinned its dependencies.
    description: >-
      This check tries to determine if the project is an application that
      has declared and pinned its dependencies.
      A "pinned dependency" is a dependency that is explicitly set to a
      specific version instead of allowing a range of versions.
      If this project is a library (not an application), this check should
      automatically pass (but see limitations below).

      It's important for applications to pin dependencies
      to ensure that checking and deployment are
      all done with the same software, reducing deployment risks, simplifying
      debugging, and enabling reproducibility.
      In some cases pinning dependencies can help
      mitigate compromised dependencies
      from undermining the security of the project (in the case where
      you've evaluated the pinned dependency, you are confident
      it's not compromised,
      and a later version is released that is compromised).
      In particular, pinning dependencies is one way to
      [counter dependency confusion (aka substitution) attacks](https://azure.microsoft.com/en-us/resources/3-ways-to-mitigate-risk-using-private-package-feeds/).
      In a dependency confusion attack, an application uses multiple feeds
      to acquire software packages (a "hybrid configuration"),
      and attackers can fool the user into using a malicious package
      via a feed that was not expected for that package.
      A risk of pinning dependencies is that it can inhibit software updates
      (e.g., because of a security vulnerability or because the pinned version
      is compromised);
      this can be mitigated by
      [having applications and *not* libraries pin to specific versions](https://jbeckwith.com/2019/12/18/package-lock/),
      using automated tools to notify applications when their dependencies are
      outdated, and by applications that *do* pin dependencies update quickly.
      Low score is therefore considered `Medium` risk.

      The checks works by (1) looking for the following files in the root
      directory: go.mod, go.sum (Golang), package-lock.json, npm-shrinkwrap.json
      (Javascript), requirements.txt, pipfile.lock (Python), gemfile.lock
      (Ruby), cargo.lock (Rust), yarn.lock (package manager), composer.lock
      (PHP), vendor/, third_party/, third-party/; (2) looks for
      unpinned dependencies in Dockerfiles, shell scripts and GitHub workflows.

      *Limitations:*
      This check should only apply to applications, as
      libraries shouldn't normally have enforced pinned dependencies.
      Unfortunately, Scorecard currently can't detect if a project
      is a library or application.
      Even when
      [application detection is added](https://github.com/ossf/scorecard/issues/689),
      it's always possible for an automated tool like Scorecard to
      incorrectly categorize software
      (especially in projects that include both libraries and applications).

      You can learn more about dependencies for projects on GitHub using
      [GitHub dependency graph](https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph).
    remediation:
      - >-
        First, determine if your project is producing a library or application.
        If it is a library, then generally you don't want to pin dependencies
        of library users. If your project is producing an application,
        consider the remediations below.
      - >-
        Declare all your dependencies with specific versions in your package
        format file (e.g. `package.json` for npm, `requirements.txt` for
        python). For C/C++, check in the code from a trusted source and add a
        `README` on the specific version used (and the archive SHA hashes).
      - >-
        If the package manager supports lock files (e.g. `package-lock.json` for
        npm), make sure to check these in the source code as well. These files
        maintain signatures for the entire dependency tree and saves from future
        exploitation in case the package is compromised.
      - >-
        For Dockerfiles and GitHub workflows, pin dependencies by hash. See example
        [main.yaml](https://github.com/ossf/scorecard/blob/f55b86d6627cc3717e3a0395e03305e81b9a09be/.github/workflows/main.yml#L27)
        and [Dockerfile](https://github.com/ossf/scorecard/blob/main/cron/worker/Dockerfile) examples.
      - >-
        To help update your dependencies after pinning them, use tools such as
        Github's [dependabot](https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/)
        or [renovate bot](https://github.com/renovatebot/renovate).
  SAST:
    risk: Medium
    tags: supply-chain, security, testing
    short: Determines if the project uses static code analysis.
    description: >-
      This check tries to determine if the project uses static code analysis.

      SAST tool may prevent known classes of bugs to be inadvertently
      introduced in the codebase. A low score is considered `Medium` risk.

      The checks currently looks for known Github apps
      such as [github-code-scanning](https://securitylab.github.com/tools/codeql)
      (codeql) and sonarcloud in the recent (~30) merged PRs. The
      check also looks for the use of "github/codeql-action" in a GitHub workflow.
      A project may meet this criterion yet have a failing scorecard report;
      there are many ways to implement this criterion and it's especially
      difficult for an automated tool (like scorecard) to detect them all.
    remediation:
      - >-
        Run CodeQL checks in your CI/CD by following the instructions
        [here](https://github.com/github/codeql-action#usage).
  Security-Policy:
    short: Determines if the project has published a security policy.
    tags: supply-chain, security, policy
    description: >-
      This check tries to determine if the project has published a security
      policy. It works by looking for a file named `SECURITY.md`
      (case-insensitive) in a few well-known directories.
    remediation:
      - >-
        Place a security policy file `SECURITY.md` in the root directory of your
        repository. This makes it easily discoverable by a vulnerability reporter.
      - >-
        The file should contain information on what constitutes a vulnerability
        and a way to report it securely (e.g. issue tracker with private issue
        support, encrypted email with a published public key).
  Signed-Releases:
    risk: High
    tags: supply-chain, security, releases
    short: Determines if the project cryptographically signs release artifacts.
    description: >-
      This check tries to determine if the project cryptographically signs release
      artifacts.

      Signed releases attest to the provenance of the artifact.
      A low score is considered 'High' risk.

      It works by looking for filenames: *.minisig
      (https://github.com/jedisct1/minisign), *.asc (pgp), *.sign. for the last
      5 GitHub releases. The check does not verify the signatures.
    remediation:
      - Publish the release.
      - Generate a signing key.
      - Download the release as an archive locally.
      - Sign the release archive with this key (should output a signature file).
      - Attach the signature file next to the release archive.
      - >-
        For GitHub, check out the steps
        [here](https://wiki.debian.org/Creating%20signed%20GitHub%20releases).
  Token-Permissions:
    risk: High
    tags: supply-chain, security, infrastructure
    short: Determines if the project's GitHub workflows follow the principle of least privilege.
    description: >-
      This check tries to determine if the project's GitHub workflows
      follow the principle of least privilege, i.e. if the GitHub tokens
      are set read-only by default.

      Attackers may use a compromised token with write access to push malicious code into the project.
      A low score is therefore considered `High` risk.

      For each workflow yaml file, the check looks
      for the permission definitions. To obtain the highest score, the permissions should be set as
      read-only at the [top level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#permissions)
      and the required write permissions should be declared at the [run-level](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idpermissions).
      The check cannot detect if the "read-only" GitHub permission settings is enabled, as there is no API available.
    remediation:
      - >-
        Set permissions as `read-all` or `contents: read` as described in
        GitHub's [documentation](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#permissions).
  Vulnerabilities:
    risk: High
    tags: supply-chain, security, vulnerabilities
    short: Determines if the project has open, known unfixed vulnerabilities.
    description: >-
      This check determines if the project has open, unfixed 
      vulnerabilities using the [OSV](https://osv.dev) service.

      An existing vulnerability is can readily be used by attackers,
      so a low score is considered `High` risk.
    remediation:
      - >-
        Fix the vulnerabilities. The details of each vulnerability can be found
        on <https://osv.dev>.