[Feature Request]: Introduce support for code coverage & enforcement (GSoC'24 4.1) #5343

BenHenning · 2024-02-17T00:18:53Z

Is your feature request related to a problem? Please describe.

The team cannot currently monitor and enforce minimum code coverage (as a proxy for functional behavioral coverage) in automated tests.

Describe the solution you'd like

Suggested milestones

Milestone 1: Introduce a new script to compute a per-unit code coverage percentage for a single file.
- Deliverable 1: Introduce a new scripting utility that can run code coverage for a specific Bazel test target, interpret the results, and return a proto for data processing.
- Deliverable 2: Introduce a new utility which can, given the proto from the deliverable 1 utility, generate a rich-text code coverage report in one of three formats:
  1. Markdown (for easy copying to GitHub)
  2. HTML (for easy local viewing)
- Deliverable 3: Update the test exemption check script & its exemption format such that each file now has two possible states:
  1. Exempt from having a test file (the current exemption behavior).
  2. An override of the code coverage to meet a specific minimum (which can be 0%).
Milestone 2: Integrate code coverage checking.
- Deliverable 1: Introduce a new script that uses the utility from deliverable 1 to perform code coverage analysis for a single target and the utility from deliverable 2 to optionally generate and output a rich-text report file for the test run. This script should always output the computed code coverage for a given file, and have a configuration option to fail if it's below a specified threshold.
- Deliverable 2: Introduce a new CI workflow which, similarly to the existing unit test workflow, runs a series of buckets for code coverage analysis using the new script from milestone 1. Report generation should be enabled & reports uploaded as artifacts. Code coverage should only run after tests are passing (on a per-bucket basis if possible).
- Deliverable 3: Fix/replace the cancellation workflow to ensure re-runs of CI correctly and quickly terminate all existing workflows that are running.
- Deliverable 4: Introduce a wiki page explaining how to use the code coverage tool, provide advice on how to write tests with good behavioral coverage, and explain the limitations of the code coverage tool (i.e. all the cases it does not correctly count coverage for a specific line).
- Deliverable 5: File issues for all cases where the code coverage tool misses or incorrectly counts code coverage for future work.

Technical hints / guidance

Top-level components needed for code-coverage support

coverage.proto: Describes the new scripting data structures needed for code coverage.
CoverageRunner.kt: A new scripting utility for running code coverage for a selected target.
CoverageReporter.kt: A new scripting utility for generating an HTML report of code coverage.
RunCoverage.kt: A new script for running code coverage locally.
scripts/src/java/org/oppia/android/scripts/testfile/TestFileCheck.kt: Existing utility that needs to be updated to include support for code coverage enforcement.
scripts/src/java/org/oppia/android/scripts/proto/script_exemptions.proto: Existing exemptions definition that needs to be updated per the TestFileCheck changes.
.github/workflows/unit_tests.yml: Existing CI workflow that needs to be updated in order to support code coverage.
.github/workflows/workflow_canceller.yml: Existing CI workflow that needs to be fixed.
(Other files for wiki changes).

Some key technical notes

This spec overview does not include details for the Yaml or wiki changes.
The test file exemptions textproto file will need to be rebuilt. The simplest way to do this would be to update TestFileCheck to include a text proto regenerate argument (as used in other scripts).
workflow_canceller.yml changes may required manually introducing support for workflow cancellation, or finding an available open source utility that works correctly for dynamic matrix jobs.
unit_tests.yml will probably be extended to introduce a new job that blocks on the test run (in the same way as check_tests) except it (note that this may require a new script not spec'ed out in this overview):
- 1. Intreprets the same matrix data as the main test run but omits the failing tests using the outputs from the test runs themselves.
- 1. Starts a new test matrix that runs the RunCoverage.kt script for each affected file corresponding to each affected test. For example, if StateFragmentTest.kt is one of the affected tests, then RunCoverage would be run for StateFragment.kt (which would then correspond to StateFragmentTest per its behavior).
- Note that the min enforced coverage for now should be something very modest, like 10%. This number will be increased in future changes outside the scope of this project (as the number may not be believable until we understand the limitations of what can be covered by JaCoCo).
BazelClient.kt may need to be updated to include a function for running coverage on a specific target & output the standard output lines from that (which can then be parsed to get the report to generate the CoverageReport proto object).
All new components require new corresponding test files. All updated components will need their tests updated based on the changes to those components.

Suggested files to add/change

coverage.proto:

message CoverageReport {
  string bazel_test_target = 1; // The Bazel test target that was run.
  repeated CoveredFile covered_file = 2; // Files with some coverage in this report.
}

message CoveredFile {
  string file_path = 1; // Relative to the project root.
  string file_sha1_hash = 2; // SHA-1 hash of the file content at the time of report (to guard against changes).
  repeated CoveredLine covered_line = 3; // Lines of code covered in the report.
}

message CoveredLine {
  int32 line_number = 1; // 0-starting line number of the covered line.
  Coverage coverage = 2; // Detected coverage.
  
  enum Coverage {
    FULL, // This line was fully covered by the test.
    PARTIAL, // This line was partially covered by the test, indicating some branches not being covered.
    NONE // This line was not executed during the test.
  }
}

CoverageReporter.kt:

class CoverageReporter {
  fun generateRichTextReport(report: CoverageReport, format: ReportFormat): String
  fun computeCoverageRatio(file: String, report: CoverageReport): Float // Returns in [0, 1].
  
  enum class ReportFormat {
    MARKDOWN,
    HTML
  }
}

CoverageRunner.kt:

class CoverageRunner {
  // Uses bazel coverage to run code coverage on the specified test target & interprets the results to generate and return a CoverageReport.
  fun runWithCoverageAsync(bazelTestTarget: String): Deferred<CoverageReport>
}

RunCoverage.kt:

// Usage:
//   bazel run //scripts:run_coverage -- <repo_root> <relative_test_file> <report_output_dir> [min_coverage=<int_perc>] [format=markdown,html]
//
// Examples:
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports min_coverage=20
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports format=markdown
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports format=html
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports min_coverage=20 format=markdown,html
//
// The script:
// 1. Finds the corresponding test file specific to the file to be checked.
//   a. If the file has no test & isn't exempted, the script should fail.
//   b. If the file has no test & is exempted, the script should output as such.
// 2. Uses CoverageRunner to run the corresponding test file to generate a report.
// 3. Outputs code coverage percentage based on the report.
//   a. If the file has a code coverage percentage exemption, it's also outputted at this point.
// 4. If formats are specified, CoverageReporter should be used to generate reports for each specified format in the destination report_output_dir directory.
// 5. If min_coverage is specified, compares the computed code coverage with the minimum specified. The script should fail iff the file has less than the minimum specified coverage.
fun main(vararg args: String)

class RunCoverage {
  fun runCoverage(targetFile: String, outputFormats: List<CoverageReporter.ReportFormat>)
}

script_exemptions.proto:

// The old TestFileExemptions proto (for reference).
message TestFileExemptionsOld {
  repeated string exempted_file_path = 1;
}

message TestFileExemptions {
  repeated TestFileExemption test_file_exemption = 1;
  
  message TestFileExemption {
    string exempted_file_path = 1;
    oneof exemption_type {
      bool test_file_not_required = 2;
      int32 override_min_coverage_percent_required = 3;
    }
  }
}

Describe alternatives you've considered

No response

Additional context

This is the high-level tracking issue corresponding to https://github.com/oppia/oppia/wiki/Google-Summer-of-Code-2024#41-code-coverage-support-and-enforcement.

Note that this project includes work that relate to the following issues: #1497, #1726, #1727, and #1728.

The text was updated successfully, but these errors were encountered:

BenHenning · 2024-02-17T00:20:01Z

NB: This issue is locked to ensure that it only contains updates for requirements. If you have questions regarding this project, please use its corresponding discussion category: https://github.com/oppia/oppia-android/discussions/categories/gsoc-q-a-4-1-code-coverage-support.

BenHenning · 2024-03-01T01:27:11Z

NB: The main issue comment has been updated to include the technical details directly rather than a link to a Gist (so that we can easily change this if needed). None of the technical requirements have actually changed.

Rd4dev · 2024-06-04T03:02:32Z

Milestone 1

PR 1.1

Introduce RunCoverage and CoverageRunner utilities to run code coverage for a specific bazel test target.

Starting Date	Creation Date	Merge Date
27-05-2024	01-05-2024	03-06-2024

Tasks

Task	Due Date
⬜ Setup the workspace with bazel 6.2.0.	27-05-2024
⬜ Introduce RunCoverage script.	27-05-2024
⬜ Introduce CoverageRunner utility to call Bazel command.	28-05-2024
⬜ Add implementation in BazelClient to execute Bazel coverage command.	29-05-2024
⬜ Add tests for execution of Bazel coverage command through RunCoverage script.	01-05-2024

PR 1.2

Updating the test exemption script.

Starting Date	Creation Date	Merge Date
01-06-2024	03-06-2024	05-06-2024

Tasks

Task	Due Date
⬜ Update the test exemption check script proto.	01-06-2024
⬜ Update the exemption_type data.	02-06-2024
⬜ Add / Update tests for TestFileCheckTest scripts as required.	03-06-2024

PR 1.3

Update / Implement script to run code coverage for a specific file replacing test target argument.

Starting Date	Creation Date	Merge Date
03-06-2024	08-06-2024	10-06-2024

Tasks

Task	Due Date
⬜ Update the script to take in a file as argument instead of target.	03-05-2024
⬜ Check with the exempted test file list to log the coverage ratio to 0%.	03-06-2024
⬜ Add functionality to map the file name to related test targets.	04-06-2024
⬜ Include execution for related Test and LocalTest Targets.	06-06-2024
⬜ Add tests for execution of Bazel coverage command with filename.	08-06-2024

PR 1.4

Interpreting the results and building proto for data processing.

Starting Date	Creation Date	Merge Date
08-06-2024	13-06-2024	15-06-2024

Tasks

Task	Due Date
⬜ Extract the coverage data path from the results.	08-06-2024
⬜ Introduce coverage.proto to store the produced coverage data.	08-06-2024
⬜ Parse the coverage data and store them to the proto.	10-06-2024
⬜ Isolate the coverage data to only contain the target coverage data.	11-06-2024
⬜ Include coverage data to proto for any related tests - 'LocalTest'.	12-06-2024
⬜ Add tests to validate acquired coverage data stored in the proto.	13-06-2024

PR 1.5

Introduce CoverageReporter utility and generate the code coverage report.

Starting Date	Creation Date	Merge Date
13-06-2024	20-06-2024	22-06-2024

Tasks

Task	Due Date
⬜ Introduce CoverageReporter utility to generate the code coverage report.	13-06-2024
⬜ Compute the coverage Ratio.	13-06-2024
⬜ Generate Markdown report with proto.	15-06-2024
⬜ Generate HTML report with proto.	18-06-2024
⬜ Add tests to validate the Markdown and HTML report generation and coverage ratios.	20-06-2024

Buffer Time

To work on reviews and code changes.

Starting Date	Creation Date	Merge Date
22-06-2024	26-06-2024	28-07-2024

Midpoint Evaluation

Code Reviews and Evaluations

Starting Date	Creation Date	Merge Date
28-06-2024	08-07-2024	12-07-2024

Milestone 2

PR 2.1

Introduce new CI workflow for code coverage

Starting Date	Creation Date	Merge Date
12-07-2024	22-07-2024	24-07-2024

Tasks

Task	Due Date
⬜ Introduce a new CI workflow that utilizes the coverage script.	13-07-2024
⬜ Configure it to run it with and without cache.	15-07-2024
⬜ Configure it to run as a series of buckets.	18-07-2024
⬜ Configure it to fail if it's below the threshold.	20-07-2024
⬜ Configure it to generate markdown reports.	22-07-2024
⬜ Analyze the new runs.	22-07-2024

PR 2.2

Upload generated markdown report as comment

Starting Date	Creation Date	Merge Date
22-07-2024	26-07-2024	28-07-2024

Tasks

Task	Due Date
⬜ Modify the workflow to upload markdown report as a comment.	24-07-2024
⬜ Test markdown report upload feature.	25-07-2024
⬜ Review and refine markdown upload implementation.	26-07-2024

PR 2.3

Fix/replace cancellation workflow

Starting Date	Creation Date	Merge Date
26-07-2024	01-08-2024	03-08-2024

Tasks

Task	Due Date
⬜ Identify issues with the current cancellation workflow.	26-07-2024
⬜ Execute and analyze cancellation with styfle - PR #2890.	28-07-2024
⬜ Develop replacement or fix for the cancellation workflow.	30-08-2024
⬜ Validate the new cancellation workflow.	01-08-2024

PR 2.4

Create wiki page explaining code coverage usage, limitations, file issues for coverage gaps, and test writing tips

Starting Date	Creation Date	Merge Date
01-08-2024	04-08-2024	06-08-2024

Tasks

Task	Due Date
⬜ Introduce new wiki page for "how to use the code coverage tool".	01-08-2024
⬜ Introduce new wiki page for "how to write tests with good behavioral coverage".	02-08-2024
⬜ Introduce new wiki page for "Limitations of the code coverage tool".	03-08-2024
⬜ File issues for missing / incorrect code coverages.	04-08-2024

Buffer Time

To work on reviews and code changes.

Starting Date	Creation Date	Merge Date
06-08-2024	10-08-2024	12-08-2024

Final Evaluation

Code Reviews and Evaluations

Starting Date	Creation Date	Merge Date
12-08-2024	26-08-2024	03-09-2024

Spare Intervals

Spare intervals to fix any further issues.

Starting Date	Creation Date	Merge Date
03-09-2024	N/A	End of GSoC period

BenHenning added enhancement End user-perceivable enhancements. triage needed labels Feb 17, 2024

BenHenning self-assigned this Feb 17, 2024

oppia locked and limited conversation to collaborators Feb 17, 2024

seanlip removed the triage needed label Feb 17, 2024

oppia unlocked this conversation May 8, 2024

seanlip assigned Rd4dev May 8, 2024

Rd4dev mentioned this issue Jun 4, 2024

Fix part of #5343: Introduce RunCoverageForTestTarget and CoverageRunner utilities to run code coverage for a specific bazel test target [Blocked: #4886]. #5420

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature Request]: Introduce support for code coverage & enforcement (GSoC'24 4.1) #5343

[Feature Request]: Introduce support for code coverage & enforcement (GSoC'24 4.1) #5343

BenHenning commented Feb 17, 2024 •

edited

Top-level components needed for code-coverage support

Some key technical notes

Suggested files to add/change

BenHenning commented Feb 17, 2024

BenHenning commented Mar 1, 2024

Rd4dev commented Jun 4, 2024

[Feature Request]: Introduce support for code coverage & enforcement (GSoC'24 4.1) #5343

[Feature Request]: Introduce support for code coverage & enforcement (GSoC'24 4.1) #5343

Comments

BenHenning commented Feb 17, 2024 • edited

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Suggested milestones

Top-level components needed for code-coverage support

Some key technical notes

Suggested files to add/change

Describe alternatives you've considered

Additional context

BenHenning commented Feb 17, 2024

BenHenning commented Mar 1, 2024

Rd4dev commented Jun 4, 2024

Milestone 1

PR 1.1

Tasks

PR 1.2

Tasks

PR 1.3

Tasks

PR 1.4

Tasks

PR 1.5

Tasks

Buffer Time

Midpoint Evaluation

Milestone 2

PR 2.1

Tasks

PR 2.2

Tasks

PR 2.3

Tasks

PR 2.4

Tasks

Buffer Time

Final Evaluation

Spare Intervals

BenHenning commented Feb 17, 2024 •

edited