Skip to content

Commit

Permalink
Rework peer requirement and warning system (#6205)
Browse files Browse the repository at this point in the history
**What's the problem this PR addresses?**

There are a number of issues with the current peer requirement and peer
warning system:

----

Firstly, since v4, we lost the ability to list all peer requirements in
the project and to explain missing peer dependency warnings, which is a
huge degradation to DX.

----

Secondly, also since v4, transitive peer warnings are no longer
displayed. The rationale given was "those are not actionable". But they
*are* actionable because we have `packageExtensions`.

More importantly, there are no indication nor discoverability when this
happens. Yarn exits without even a warning when transitive peer warnings
can put the project in an invalid state. Even when the user somehow
knows that there are transitive peer warnings, they are impossible to
investigate and fix without the proper tools.

----

Thirdly, as I just reported, the peer warning aggregation is aggregating
unrelated peer requests, causing `yarn explain peer-requirements` to
give incorrect advice. #6203

----

Finally, the peer resolution pipeline has been refactored and added to
many times over the years which make the code quite difficult to
understand. This is compounded by misplaced comments and overloading
terms like "peer descriptor" in the code.

----

Closes #5841
Closes #5977
Closes #6016
Closes #6118
Closes #6203


**How did you fix it?**

On the last problem, it is quite impossible to completely solve in one
go, so I only took small bites and at least rewrote some of the comments
to better capture the current peer pipeline. I have also unified the
terminology and change variable names to use those terminology to reduce
ambiguity. Maybe we should add these to the lexicon?

Let's say
- Package `A` (regular-)depends on `B`, which peer-depends on `C`
- Package `A` also depends on `C`

```
A@1.0.0 --> B@^1.0.0 (resolves to B@1.1.0) ==> C@^1.0.0
        --> C@^1.2.0 (resolves to C@1.3.0)
```

In the scope of this single level
- The package `A@1.0.0` is known as the **subject** or **requestee**
- The package `B@1.1.0` is known as the **requester**. Also called the
**virtualized package** in the code because of what the pipeline is
doing
- The ident `C` is known as the **peer ident** and the descriptor
`C@^1.0.0` is known as the **peer descriptor**
- The descriptor `C@^1.2.0` is the **provided descriptor**. Similarly,
`C@1.3.0` is the **provided locator/package**. **Provision** can refer
to any of those depending on context.
- Note that the provision is not necessarily `A`'s dependency. `A`
itself could be used if its ident *were* matching, or a package can be
not provided at all.

Based on those we can create some composite data structures:
- A **peer request** encapsulates a requester + peer ident.
- If a subject does not provide to the request itself, but instead
requests a peer dependency under the same ident, that subject and the
new peer request it issues are both said to **forward** the original
peer request.
- A **peer requirement** encapsulates a subject + peer ident + one or
more peer requests. This captures the fact that a single subject that
depends on multiple peer requesters can satisfy multiple peer requests
at the same time
- Note that peer requests and peer requirements is a many-to-many
relationship. A peer request is included in multiple peer requirements
if the requester is depended upon multiple times (which can happen due
to deduplication).
- A peer requirement is said to be a **root peer requirement** if the
subject does not forward the peer requests.

----

With those concepts established, the core of this PR changes the peer
requirement and peer warning system to use a tree-based structure. The
nodes of the tree are either peer requests or peer requirements. A peer
request's children are the requests it forwards, and a peer
requirement's children are the requests it includes. Note that a tree
can only ever include peer requirements and requests regarding a single
peer ident.

![Untitled
Diagram](https://github.com/yarnpkg/berry/assets/41266433/166595b0-b7be-4e8c-a589-64a7d16ac462)

By storing all peer requirement nodes in `Project`, other places can
easily display information on peer requirements/requests/warnings by
retrieving them form the tree (using the new tree-view UI, even). This
allows us to reimplement `yarn explain peer-requirements` and `yarn
explain peer-requirements <hash>` to list all requirements and explain
any peer requirement (even without warnings) respectively. This also
fixes #6203 because the peer requirement nodes correctly group the peer
requests.


![image](https://github.com/yarnpkg/berry/assets/41266433/7f40f8f3-53da-48d4-bb10-38a8c23ccec4)

----

To solve the discoverability problem without bringing back the clutter,
I've added an additional CTA for transitive peer warnings, so all
transitive peer warnings are reduced to a single line.

```
➤ YN0000: ┌ Post-resolution validation
➤ YN0002: │ workspace-b@workspace:workspace-b doesn't provide p (p427bb), requested by y.
➤ YN0086: │ Some peer dependencies are incorrectly met by your project; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0086: │ Some peer dependencies are incorrectly met by dependencies; run yarn explain peer-requirements for details.
➤ YN0000: └ Completed
```

A user wishing to view the transitive warnings can do so with the
reimplemented `yarn explain peer-requirements` command.

----

Lastly, I don't know how much of the original peer requirement and
warning system matter for BC. For now, I have recreated the data
structures created by the original system, except the aggregated
warnings which is the wrong abstraction as noted. I have added TODO
comments to remove those code for the next major. Please advice if the
original system can be safely removed in a minor.

**Checklist**
<!--- Don't worry if you miss something, chores are automatically
tested. -->
<!--- This checklist exists to help you remember doing the chores when
you submit a PR. -->
<!--- Put an `x` in all the boxes that apply. -->
- [x] I have read the [Contributing
Guide](https://yarnpkg.com/advanced/contributing).

<!-- See
https://yarnpkg.com/advanced/contributing#preparing-your-pr-to-be-released
for more details. -->
<!-- Check with `yarn version check` and fix with `yarn version check
-i` -->
- [x] I have set the packages that need to be released for my changes to
be effective.

<!-- The "Testing chores" workflow validates that your PR follows our
guidelines. -->
<!-- If it doesn't pass, click on it to see details as to what your PR
might be missing. -->
- [x] I will check that all automated PR checks pass before the PR gets
reviewed.

---------

Co-authored-by: Maël Nison <nison.mael@gmail.com>
  • Loading branch information
clemyan and arcanis committed May 17, 2024
1 parent 8504172 commit bfa6489
Show file tree
Hide file tree
Showing 5 changed files with 539 additions and 331 deletions.
34 changes: 34 additions & 0 deletions .yarn/versions/594fa39a.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
releases:
"@yarnpkg/cli": minor
"@yarnpkg/core": minor
"@yarnpkg/plugin-essentials": minor

declined:
- "@yarnpkg/plugin-compat"
- "@yarnpkg/plugin-constraints"
- "@yarnpkg/plugin-dlx"
- "@yarnpkg/plugin-exec"
- "@yarnpkg/plugin-file"
- "@yarnpkg/plugin-git"
- "@yarnpkg/plugin-github"
- "@yarnpkg/plugin-http"
- "@yarnpkg/plugin-init"
- "@yarnpkg/plugin-interactive-tools"
- "@yarnpkg/plugin-link"
- "@yarnpkg/plugin-nm"
- "@yarnpkg/plugin-npm"
- "@yarnpkg/plugin-npm-cli"
- "@yarnpkg/plugin-pack"
- "@yarnpkg/plugin-patch"
- "@yarnpkg/plugin-pnp"
- "@yarnpkg/plugin-pnpm"
- "@yarnpkg/plugin-stage"
- "@yarnpkg/plugin-typescript"
- "@yarnpkg/plugin-version"
- "@yarnpkg/plugin-workspace-tools"
- "@yarnpkg/builder"
- "@yarnpkg/doctor"
- "@yarnpkg/extensions"
- "@yarnpkg/nm"
- "@yarnpkg/pnpify"
- "@yarnpkg/sdks"
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ describe(`Features`, () => {
async ({path, run, source}) => {
const {stdout} = await run(`install`);

expect(stdout).toMatch(/no-deps is listed by your project with version 1\.1\.0, which doesn't satisfy what mismatched-peer-deps-lvl0 \(p[a-f0-9]{5}\) and other dependencies request \(1\.0\.0\)/);
expect(stdout).toMatch(/no-deps is listed by your project with version 1\.1\.0 \(p[a-f0-9]{5}\), which doesn't satisfy what mismatched-peer-deps-lvl0 and other dependencies request \(1\.0\.0\)/);
},
),
);
Expand Down
275 changes: 187 additions & 88 deletions packages/plugin-essentials/sources/commands/explain/peerRequirements.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import {BaseCommand} from '@yarnpkg/cli';
import {Configuration, MessageName, Project, StreamReport, structUtils, semverUtils, formatUtils, PeerWarningType} from '@yarnpkg/core';
import {Command, Option} from 'clipanion';
import {Writable} from 'stream';
import * as t from 'typanion';
import {BaseCommand} from '@yarnpkg/cli';
import type {PeerRequestNode} from '@yarnpkg/core/sources/Project';
import {Configuration, MessageName, Project, StreamReport, structUtils, formatUtils, treeUtils, PeerWarningType, miscUtils, type LocatorHash} from '@yarnpkg/core';
import {Command, Option} from 'clipanion';
import {Writable} from 'stream';
import * as t from 'typanion';

// eslint-disable-next-line arca/no-default-export
export default class ExplainPeerRequirementsCommand extends BaseCommand {
Expand All @@ -13,24 +14,25 @@ export default class ExplainPeerRequirementsCommand extends BaseCommand {
static usage = Command.Usage({
description: `explain a set of peer requirements`,
details: `
A set of peer requirements represents all peer requirements that a dependent must satisfy when providing a given peer request to a requester and its descendants.
A peer requirement represents all peer requests that a subject must satisfy when providing a requested package to requesters.
When the hash argument is specified, this command prints a detailed explanation of all requirements of the set corresponding to the hash and whether they're satisfied or not.
When the hash argument is specified, this command prints a detailed explanation of the peer requirement corresponding to the hash and whether it is satisfied or not.
When used without arguments, this command lists all sets of peer requirements and the corresponding hash that can be used to get detailed information about a given set.
When used without arguments, this command lists all peer requirements and the corresponding hash that can be used to get detailed information about a given requirement.
**Note:** A hash is a six-letter p-prefixed code that can be obtained from peer dependency warnings or from the list of all peer requirements (\`yarn explain peer-requirements\`).
`,
examples: [[
`Explain the corresponding set of peer requirements for a hash`,
`Explain the corresponding peer requirement for a hash`,
`$0 explain peer-requirements p1a4ed`,
], [
`List all sets of peer requirements`,
`List all peer requirements`,
`$0 explain peer-requirements`,
]],
});

hash = Option.String({
required: false,
validator: t.cascade(t.isString(), [
t.matchesRegExp(/^p[0-9a-f]{5}$/),
]),
Expand All @@ -46,108 +48,205 @@ export default class ExplainPeerRequirementsCommand extends BaseCommand {

await project.applyLightResolution();

return await explainPeerRequirements(this.hash, project, {
stdout: this.context.stdout,
});
if (typeof this.hash !== `undefined`) {
return await explainPeerRequirement(this.hash, project, {
stdout: this.context.stdout,
});
} else {
return await explainPeerRequirements(project, {
stdout: this.context.stdout,
});
}
}
}

export async function explainPeerRequirements(peerRequirementsHash: string, project: Project, opts: {stdout: Writable}) {
export async function explainPeerRequirement(peerRequirementsHash: string, project: Project, opts: {stdout: Writable}) {
const root = project.peerRequirementNodes.get(peerRequirementsHash);
if (typeof root === `undefined`)
throw new Error(`No peerDependency requirements found for hash: "${peerRequirementsHash}"`);

const seen = new Set<LocatorHash>();
const makeTreeNode = (request: PeerRequestNode): treeUtils.TreeNode => {
if (seen.has(request.requester.locatorHash)) {
return {
value: formatUtils.tuple(formatUtils.Type.DEPENDENT, {locator: request.requester, descriptor: request.descriptor}),
children: request.children.size > 0
? [{value: formatUtils.tuple(formatUtils.Type.NO_HINT, `...`)}]
: [],
};
}

seen.add(request.requester.locatorHash);
return {
value: formatUtils.tuple(formatUtils.Type.DEPENDENT, {locator: request.requester, descriptor: request.descriptor}),
children: Object.fromEntries(
Array.from(request.children.values(), child => {
return [
structUtils.stringifyLocator(child.requester),
makeTreeNode(child),
];
}),
),
};
};

const warning = project.peerWarnings.find(warning => {
return warning.hash === peerRequirementsHash;
});

if (typeof warning === `undefined`)
throw new Error(`No peerDependency requirements found for hash: "${peerRequirementsHash}"`);

const report = await StreamReport.start({
configuration: project.configuration,
stdout: opts.stdout,
includeFooter: false,
includePrefix: false,
}, async report => {
const Marks = formatUtils.mark(project.configuration);
const mark = warning ? Marks.Cross : Marks.Check;

report.reportInfo(MessageName.UNNAMED, `Package ${
formatUtils.pretty(project.configuration, root.subject, formatUtils.Type.LOCATOR)
} is requested to provide ${
formatUtils.pretty(project.configuration, root.ident, formatUtils.Type.IDENT)
} by its descendants`);

report.reportSeparator();

report.reportInfo(MessageName.UNNAMED, formatUtils.pretty(project.configuration, root.subject, formatUtils.Type.LOCATOR));
treeUtils.emitTree({
children: Object.fromEntries(
Array.from(root.requests.values(), request => {
return [
structUtils.stringifyLocator(request.requester),
makeTreeNode(request),
];
}),
),
}, {
configuration: project.configuration,
stdout: opts.stdout,
json: false,
});

switch (warning.type) {
case PeerWarningType.NotCompatibleAggregate: {
report.reportInfo(MessageName.UNNAMED, `We have a problem with ${formatUtils.pretty(project.configuration, warning.requested, formatUtils.Type.IDENT)}, which is provided with version ${structUtils.prettyReference(project.configuration, warning.version)}.`);
report.reportInfo(MessageName.UNNAMED, `It is needed by the following direct dependencies of workspaces in your project:`);

report.reportSeparator();

for (const dependent of warning.requesters.values()) {
const dependentPkg = project.storedPackages.get(dependent.locatorHash);
if (!dependentPkg)
throw new Error(`Assertion failed: Expected the package to be registered`);

const descriptor = dependentPkg?.peerDependencies.get(warning.requested.identHash);
if (!descriptor)
throw new Error(`Assertion failed: Expected the package to list the peer dependency`);

const mark = semverUtils.satisfiesWithPrereleases(warning.version, descriptor.range)
? Marks.Check
: Marks.Cross;

report.reportInfo(null, ` ${mark} ${structUtils.prettyLocator(project.configuration, dependent)} (via ${structUtils.prettyRange(project.configuration, descriptor.range)})`);
report.reportSeparator();

if (root.provided.range === `missing:`) {
const problem = warning ? `` : ` , but all peer requests are optional`;

report.reportInfo(MessageName.UNNAMED, `${mark} Package ${
formatUtils.pretty(project.configuration, root.subject, formatUtils.Type.LOCATOR)
} does not provide ${
formatUtils.pretty(project.configuration, root.ident, formatUtils.Type.IDENT)
}${problem}.`);
} else {
const providedLocatorHash = project.storedResolutions.get(root.provided.descriptorHash);
if (!providedLocatorHash)
throw new Error(`Assertion failed: Expected the descriptor to be registered`);

const providedPackage = project.storedPackages.get(providedLocatorHash);
if (!providedPackage)
throw new Error(`Assertion failed: Expected the package to be registered`);

report.reportInfo(MessageName.UNNAMED, `${mark} Package ${
formatUtils.pretty(project.configuration, root.subject, formatUtils.Type.LOCATOR)
} provides ${
formatUtils.pretty(project.configuration, root.ident, formatUtils.Type.IDENT)
} with version ${
structUtils.prettyReference(project.configuration, providedPackage.version ?? `0.0.0`)
}, ${warning ? `which does not satisfy all requests.` : `which satisfies all requests`}`);

if (warning?.type === PeerWarningType.NodeNotCompatible) {
if (warning.range) {
report.reportInfo(MessageName.UNNAMED, ` The combined requested range is ${formatUtils.pretty(project.configuration, warning.range, formatUtils.Type.RANGE)}`);
} else {
report.reportInfo(MessageName.UNNAMED, ` Unfortunately, the requested ranges have no overlap`);
}
}
}
});

const transitiveLinks = [...warning.links.values()].filter(link => {
return !warning.requesters.has(link.locatorHash);
});

if (transitiveLinks.length > 0) {
report.reportSeparator();
report.reportInfo(MessageName.UNNAMED, `However, those packages themselves have more dependencies listing ${structUtils.prettyIdent(project.configuration, warning.requested)} as peer dependency:`);
report.reportSeparator();

for (const link of transitiveLinks) {
const linkPkg = project.storedPackages.get(link.locatorHash);
if (!linkPkg)
throw new Error(`Assertion failed: Expected the package to be registered`);

const descriptor = linkPkg?.peerDependencies.get(warning.requested.identHash);
if (!descriptor)
throw new Error(`Assertion failed: Expected the package to list the peer dependency`);
return report.exitCode();
}

const mark = semverUtils.satisfiesWithPrereleases(warning.version, descriptor.range)
? Marks.Check
: Marks.Cross;
export async function explainPeerRequirements(project: Project, opts: {stdout: Writable}) {
const report = await StreamReport.start({
configuration: project.configuration,
stdout: opts.stdout,
includeFooter: false,
includePrefix: false,
}, async report => {
const Marks = formatUtils.mark(project.configuration);

report.reportInfo(null, ` ${mark} ${structUtils.prettyLocator(project.configuration, link)} (via ${structUtils.prettyRange(project.configuration, descriptor.range)})`);
}
const sorted = miscUtils.sortMap(project.peerRequirementNodes, [
([, requirement]) => structUtils.stringifyLocator(requirement.subject),
([, requirement]) => structUtils.stringifyIdent(requirement.ident),
]);

for (const [,peerRequirement] of sorted.values()) {
if (!peerRequirement.root)
continue;

const warning = project.peerWarnings.find(warning => {
return warning.hash === peerRequirement.hash;
});

const allRequests = [...structUtils.allPeerRequests(peerRequirement)];
let andOthers;
if (allRequests.length > 2)
andOthers = ` and ${allRequests.length - 1} other dependencies`;
else if (allRequests.length === 2)
andOthers = ` and 1 other dependency`;
else
andOthers = ``;

if (peerRequirement.provided.range !== `missing:`) {
const providedResolution = project.storedResolutions.get(peerRequirement.provided.descriptorHash);
if (!providedResolution)
throw new Error(`Assertion failed: Expected the resolution to have been registered`);

const providedPkg = project.storedPackages.get(providedResolution);
if (!providedPkg)
throw new Error(`Assertion failed: Expected the provided package to have been registered`);

const message = `${
formatUtils.pretty(project.configuration, peerRequirement.hash, formatUtils.Type.CODE)
}${
warning ? Marks.Cross : Marks.Check
} ${
structUtils.prettyLocator(project.configuration, peerRequirement.subject)
} provides ${
structUtils.prettyLocator(project.configuration, providedPkg)
} to ${
structUtils.prettyLocator(project.configuration, allRequests[0]!.requester)
}${andOthers}`;

if (warning) {
report.reportWarning(MessageName.UNNAMED, message);
} else {
report.reportInfo(MessageName.UNNAMED, message);
}

const allRanges = Array.from(warning.links.values(), locator => {
const pkg = project.storedPackages.get(locator.locatorHash);
if (typeof pkg === `undefined`)
throw new Error(`Assertion failed: Expected the package to be registered`);

const peerDependency = pkg.peerDependencies.get(warning.requested.identHash);
if (typeof peerDependency === `undefined`)
throw new Error(`Assertion failed: Expected the ident to be registered`);

return peerDependency.range;
});

if (allRanges.length > 1) {
const resolvedRange = semverUtils.simplifyRanges(allRanges);

report.reportSeparator();

if (resolvedRange === null) {
report.reportInfo(MessageName.UNNAMED, `Unfortunately, put together, we found no single range that can satisfy all those peer requirements.`);
report.reportInfo(MessageName.UNNAMED, `Your best option may be to try to upgrade some dependencies with ${formatUtils.pretty(project.configuration, `yarn up`, formatUtils.Type.CODE)}, or silence the warning via ${formatUtils.pretty(project.configuration, `logFilters`, formatUtils.Type.CODE)}.`);
} else {
report.reportInfo(MessageName.UNNAMED, `Put together, the final range we computed is ${formatUtils.pretty(project.configuration, resolvedRange, formatUtils.Type.RANGE)}`);
}
} else {
const message = `${
formatUtils.pretty(project.configuration, peerRequirement.hash, formatUtils.Type.CODE)
}${
warning ? Marks.Cross : Marks.Check
} ${
structUtils.prettyLocator(project.configuration, peerRequirement.subject)
} doesn't provide ${
structUtils.prettyIdent(project.configuration, peerRequirement.ident)
} to ${
structUtils.prettyLocator(project.configuration, allRequests[0]!.requester)
}${andOthers}`;

if (warning) {
report.reportWarning(MessageName.UNNAMED, message);
} else {
report.reportInfo(MessageName.UNNAMED, message);
}
} break;

default: {
report.reportInfo(MessageName.UNNAMED, `The ${formatUtils.pretty(project.configuration, `yarn explain peer-requirements`, formatUtils.Type.CODE)} command doesn't support this warning type yet.`);
} break;
}
}
});


return report.exitCode();
}

0 comments on commit bfa6489

Please sign in to comment.