Review guidelines
Jirka Borovec edited this page Nov 29, 2020
·
12 revisions
“Reviews allow for discussion of proposed changes and help ensure that the changes meet the repository’s contributing guidelines and other quality standards.“ (from GitHub documentation)
- We value everyone putting time and effort into PyTorch Lightning, but not every change should be included.
- We encourage everyone to open issues describing what they want to do before going off and doing it.
- Every feature we add takes attention and maintenance, but most importantly- might break something else.
- Discuss the tradeoffs in the PR or associated issues.
Each of them has its own label and accordingly, they shall be put into a milestone - semantic versioning (1.F.0) for features and (1.F.B) for bug-fixes.
-
Features:
- all API changes (add or remove public argument, class method, etc.)
- behaviour changes - function does something else, or parameters accept another value ranges/types
-
Bugs:
- update behaviour of a function to work as expected/desired
- fix crashes with expected input arguments
- Decide if my PR is (a) ready to review or (b) I just need to complex testing of my contribution or (c) I want to share my work-in-progress and ask someone for help:
- (a) -> make it as regular PR
- (b,c) -> make it as Draft PR
- Make PR title reasonably short and sufficiently self-explainable so just after reading it anyone can get a basic idea of what it is about.
- Write short summary of your work, link issue which it fixes or closes, eventually what is missing to be done as a checkpoint-list.
- Add a label to classify the PR if it is fix or enhancement, logger to another part of code so reviewers can simpler take it according to their domain of expertise...
- Add milestone where it shall fit so we can put it to timeline, mainly distinguish if it goes to closes bugfix release 1.0.x or feature release 1.x.0
We have observed that we have some long hanging or abundant PRs. Let's get them done all - they shall land on Pl or get rejected for a reason (the feature/bugfix was implemented already in another PR or they are very outdated and it would be much easier to create a new one). To reach such flow, all PR which are not authored by core-contributor, shall be assigned to one core-contributor who will take care of it, in particular:
- decide if this PR shall land or be rejected
- taking close look and mentor PR author
- help if partial fixes requested by reviewers if needed
- keep the PR actively in progress till it lands :]
- Follow the steps from self submitting PR above ^^.
- Make sure that the PR title summarizes the change (not "fixes bug #2168").
- Make sure there's an issue associated with this PR. No issue associated? Look for the relevant issue or ask the author to create one.
- Make sure PR has the right labels (bugfix, docs, tests, enhancements, etc.).
- Make sure PR description includes:
- A detailed summary of what changed
- The motivation for the change
- For new features or enhancements- ask yourself if we want this change in the product?
- Refactors should always be in separate PRs than changes.
- Every PR should change/fix exactly one thing. Otherwise, ask the author to split into smaller PRs.
- Any code changes (may be skipped form typos in docs) both features and bug-fixes shall be cover by tests existing or newly added ones, If there is a bugfix and all tests were passing on master before and the PR does not add a new test to reveal the issue, ask the author to add it... and add label "waiting for author".
- Code is read way more often than it is written or modified, so readability is always important. You should aim to understand every changed line. If you don't understand it, it's likely someone else won't either. Ask questions and comments for things you don't understand.
- If you don’t think a variable name accurately represents what’s in it, suggest a better one.
- Preferably use GH suggestion tooling which makes clear what and a=how you recommend to change.
- When something has to change in the code, explain what should be changed and more importantly, why it should be changed (it is always useful to include links to external documentation/articles that provide more detailed information on the subject).
- Are there sufficient tests to ensure this change doesn’t break in the future?
- Read the tests and make sure they will break if the code does. The code in the tests is just as important as the code in the product.
- No tests? Ask the author to add them or add them yourself.
- Make sure error scenarios and edge cases are well handled.
- Would this work for a single node? multi-node?
- CPU? GPU? TPU?
- Anything with spawn needs to be serializable. Spawn is basically like: a car is driving down a single lane- this is the main process. When the spawn is called, that car gets copied into 8 other cars (8 GPUs). That car then goes into its own lane. The car is trained on that lane, but the original car isn’t... so consider what happens in those scenarios.
- DDP is like the same analogy but instead of copying the car 8 times, we copy it 7 times and the main car then joins those 7 to train like all.
- Make sure this works with 16-bit precision.
- Any bugfix, feature, substantial change in code behavior shall be included
- No need for typos, docs, and CI edits
- [this is a testing suggestion] make 2 line spacing between all items in the "active" section (unreleased changes), this shall prevent collisions in git, later before release we format the active section at once
- If you as a reviewer find the PR fine, and all tests are passing, you can place the label "Ready-to-go".
- These "Ready-to-go" labels shall not be used by the PR author 8-]
If you checked all the above and everything is looking good, you may approve the PR in your review or request changes. As a rule of thumb:
- Approve: if you are confident that the issue or feature is fully addressed and the content can be merged as it is now (no extra conditions in comments).
- Comment: if there are any doubts (in your understanding, in the general direction, etc.) but you want to express your opinion about a specific point.
- Request changes: if you are sure that the actual change will break something or has obvious issues.
Remember that if you approve a PR and it reaches the minimum number of required approvals and given that all tests pass, the branch will be automatically merged into master.
- Be nice and constructive, remember your first PR and what helped you / made you motivated to continue contributing.
- Use suggestions for simple edits when you know what it should be instead of leaving a long description (you can still add a short justification of your proposal) so the author can simply accept it.