[ISSUE 7] Automated diagram generation

[widal001]

Overview

Description

This PR changes the approach to populating both the milestone_short_descriptions.md and the milestone_dependencies_diagram.mmd files by having the summary-level details about each milestone defined in a centralized YAML file then providing a CLI tool that automates the creation of these two files based on the data in that YAML file.

Fixes: #7

Motivation

There are three primary goals associated for this proposed shift:

• Single source of truth: Having milestone details (at least the summary level details) defined in a single place prevents the potential divergence of information about milestones between the places where that information is represented. When building out the consolidated yaml file, I noticed there were a few dependencies represented in the diagram that were not reflected in the markdown file.
• Machine-readable format: By defining the milestone details in a machine-readable format we also provide the option to expose those details in an API as well, which could be useful for the measurement dashboard milestone.
• Data validation: The other benefit of storing these details in a machine-readable format is that it becomes easier to apply data validation to those details and catch potential errors or inconsistencies. One issue caught by the basic validation that has already been implemented was a typo in some of the names of the Hackathon milestones.

Changes Proposed

• Copies milestone details and dependencies into documenation/milestones/mileston-summaries.md
• Creates a milestone-cli/milestone_cli python package which contains the following sections:
  • templates/ Jinja templates for rendering the diagram and the markdown files
  • schemas.py Defines pydantic models used to deserialize and validate the milestones from in the yaml file
  • utils.py Defines a set of utility functions used to populate the output files from the templates
  • cli.py Defines the CLI entry points that developers can use to validate the yaml file and populate the templates

PR Details

Instructions to Review

1. Checkout the PR locally: git checkout widal001:ISSUE-7-automated-diagram-generation
2. Pull the most recent changes from the branch: git pull
3. Change directory into the CLI tool: cd milestone-cli
4. Check that you have the pre-requisites installed:
   • Python version 3.10 or greater: python --version
   • Poetry: poetry --version
5. Install the package: poetry install
6. Activate a shell: poetry shell
7. Run the tests: pytest
8. Validate the yaml file: milestones validate
9. Comment out one of the description: lines and run milestones validate again, it should raise an error.
10. Uncomment that line and then populate the summary and diagram files using the following commands:
    • Diagram: milestones populate diagram ./diagram.mmd
    • Summary: milestones populate summary ./summary.md
11. Exit the shell: exit

TODO

• Finish migrating milestone details to milestone-summaries.yaml
  • descriptions
  • dependencies (i.e. check for parity with diagram)
• Improve documentation and docstrings
• Add a Makefile to streamline setup
• Add tox.ini and GitHub actions to enforce checks
• Expand validations

[acouch]

This is great! I got caught up on an issue with type annotations in python 11, which begs the question which versions of Python we should support 🤔 .

When I run in a env with 3.11, I get the following error:

poetry run milestones validate hello
Traceback (most recent call last):

  File "<string>", line 1, in <module>

RuntimeError: Type not yet supported: str | None

Will continue reviewing ASAP.

[widal001]

@acouch @lucasmbrown-usds I might close this PR so it's not pulling us away from some of the higher priority WIP we have in other PRs. Additionally I think we wanted to avoid moving milestone details into a format that is harder for non-technical users to access (e.g. yaml).

As I mentioned to you both, there might be another way to repurpose some of this code and achieve the underlying goals of:

1. Having a (machine-readable) single source of truth for Milestone status, dependencies, and other summary level details
2. Automatically creating/updating the milestone short descriptions doc and the milestone dependency diagram from this single source of truth

The TL;DR for this strategy is a potential shift to using GitHub issues as the canonical source source for Milestone summary level details and then using the GitHub API to generate and update the diagram and summary doc.

If we do move in that direction though I'll open a separate PR with the updated code.

[lucasmbrown-usds]

Thanks @widal001 , sounds like a reasonable call. This was a very cool and exciting prototype!