Edk2-Pytools v1.0.0

[Javagedes]

This is a discussion on the current proposal of Edk2-Pytool modifications necessary for a v1.0.0 release. Please find the attached document for the proposal.

Please provide feedback on current proposed changes, any additional changes you would like to see, and any User stories missed that need to be accounted for.

Edk2-Pytools Proposal.docx

Proposals

1. Convert stuart_<cmd> to stuart <cmd>.
2. Merge stuart_setup and stuart_ci_setup into stuart_init.
3. Remove Settings Managers, using Toml based configuration instead.
4. replace stuart_ci_build with --checks flag for stuart_build.
5. Add advanced configurability to python logging
6. Move to pythonic way of error and exception handling
7. move to pythonic style guide (snake_case functions)

Updates

Introduce command aliasing

It was requested by multiple people to a multiple build paradigm, or the ability to call invocables from other invocables. The current idea is to add command alias's to the configuration files as it allows the flexibility of developers to create combinations of commands as they see fit. This would probably be in a 1.1.0 release as a new feature rather than a part of the 1.0.0 release, but could be very beneficial.

Below would be an example of adding an alias to run stuart update and stuart build through an alias

[Alias.<os>?]
# stuart switch_build PlatformPkg1 -> stuart update -p PlatformPkg1; stuart build -p PlatformPkg1
switch_build = "stuart update -p {0}; stuart build -p {0}"

# stuart switch_build -p PlatformPkg1 -> stuart update -p PlatformPkg1; stuart build -p PlatformPkg1
switch_build = "stuart update -p {-p}; stuart build -p {-p}"

This can already be easily done on a user-by-user basis through scripts, but by enabling it in the configuration file, that functionality can easily be shared across a dev team where applicable. More thought will need to be put into this to ensure complex aliasing can be handled

Configuration Chaining

The original idea was to provide a singular override for configurations (repository level and package level override). It was brought up that additional override levels would be beneficial. The new idea is to follow the folder hierarchy up to the workspace root and overriding any configs along the way.

[jorticus-msft]

1. Feature Request:
   Add an option to stuart build that also invokes update if necessary, combining the two steps.
   My workflow consists of switching between branches and needing to update & rebuild, and this would save me time.

2. I'm not a fan of config.toml as a name to define a package, as it's not obvious this has anything to do with EDK2/UEFI. Perhaps something like project.toml, package.toml, MyPackageNamePkg.toml (though I recognize this last one is harder to grep for, but it matches our convention for naming DSC/FDFs).

3. I agree with the simplification of the stuart commandline, python logging changes, error handling, etc.

4. The chaining of configurations could benefit from one level between repository and package, to allow for multiple "families" of projects where each project shares the common family scope. Eg, imagine a structure like this:

- Repo (scope: common)
  - Family1 (scope: family1)
     - Package1
     - Package2
  - Family2 (scope: family2)
     - Package 1
     - Package 2

Though I guess this could also be implemented via Python functions or UefiBuilder.

5. Please make sure there's full backwards compatibility / a migration path available for the current UefiBuilder approach, at least in the near term. Otherwise switching between branches where one contains UefiBuilder and the other contains TOML will become a huge pain.

[Javagedes]

@jorticus-msft Thank you for your comments. I will need to look into your first point. It may be better to offer users the ability to set alias's - possibly in the repository config file (similar to a git alias). This way users could set their own combination of commands as they wish. I'd have to look into the feasibility of it.

For your second point, PackageName.toml would not be an issue. As it stands right now, the packages will need to be defined in the top level toml file to prevent conflicts with packages in submodules, so it should not be an issue to search for PkgName/PkgName.toml. Not having a standardized name has me concerned with handling your 4th point, however.

For your 4th point, I think it would be best to allow for any number of configuration chaining's by following the folder directory up searching for a toml file. This is where not having standardized names for the toml files might be an issue. Especially if, for some reason, toml files for things other than Edk2-Pytools exist. There are absolutely workarounds so I'll need to think about the best approach for that.

For your last point, I'm thinking that since edk2-pytools is well versioned, backwards compatibility should not be an issue. You will need to pip install the requirements file for the repo when you change though, so if that is a concern, or if you have any suggestions, I definitely want to make life easier during any transition period (or for repos that won't be moving to v1.0.0) The edk2-pytool-version specified in a repository pip-requirements.txt should always be able to build that repository. I do plan on making the migration path as simple as possible. One benefit is that no changes will be made to the UefiBuilder and how a platform is actually built. Only the infrastructure around that is being changed.