ci(integrity): ensure that no new duplicates are introduced

[belgattitude]

What does it do?

Introduce a check for duplicates in the CI. this is suggestion after #18745 is merged.

Why is it needed?

Will ensure no new duplicates are introduced over time (often by tools like dependabot...)

How to test it?

N/A

Related issue(s)/PR(s)

• chore(lock): deduplicate dependencies #18745

Future

Use renovatebot instead of dependabot

While yarn 3+ is very good at keeping duplicates low (pnpm 8+ too), the yarn.lock is often changed by bots (renovate, dependabot...). In my experience renovate is probably the best to use as it's able to dedupe the update PR's. An example of renovate.json5 could be

{
  "extends": ["config:base"],
  "enabled": true,
  "enabledManagers": ["npm", "docker-compose", "dockerfile", "github-actions"],
  "postUpdateOptions": [
    // https://docs.renovatebot.com/configuration-options/#postupdateoptions
    // Will run yarn dedupe --strategy highest
    'yarnDedupeHighest'
  ],
  // ... Kept short for brevity

[belgattitude]

While yarn dedupe does not "require" a prior install (it will do it for you), good tip is to pre-install to keep the benefit of yarn-nm-install (more than 2x faster thx to caching)

[belgattitude]

Will display the list and fail if there is duplicates

[joshuaellis]

what do you foresee as the process for individuals to solve this check? run the command?

[belgattitude]

In my experience yarn 3+ already dedupes very well.

Contributors that would add/remove/update a dependency in their pull requests won’t never trigger a ci check error here.

But if it does fail: ie the contributor edited the lock manually or because of a yarn issue, he will have to run the command: yarn dedupe to generate a new lock file then push it.

There’s sense to add an howto in the contributor guideline. But it will be rare.

in the other hand tools that directly modifies the lock might trigger a failure: Dependabot..,

if it does, someone will need to add a dedupe commit.

hope it clarifies

[innerdvations]

I have a question you can probably answer :)

If dependabot tends to cause dupes, what do you feel would be the best strategy for using it? Is there a way to trick it into behaving better? or will someone see the error and have to know they need to manually pull the PR and dedupe it before merging?

I like this idea but we'll need to set some sort of guide/policy for "when you see this failed test on dependabot, you need to take these specific steps" so that people don't get lost and confused.

[belgattitude]

Indeed. My personal journey into that was to move away from dependabot (and in general whatever directly change the lock file). Not sure about dependabot recent evolutions, but still seems

dependabot/dependabot-core#5830

There’s ways to circumvent that directly in the ci, cause we can detect the bot user and dedupe but it’s weird at the end.

I’d recommend to try and use renovate instead. You’ll find a note in the issue description. Both tools are great but renovate will dedupe

otherwise yes documentation or add a comment in the ci step to direct the people to an how-to

Another option is to not check for duplicate at all in the ci. Running a scheduled action every week that just dedupe and create a pr if something was deduplicated is very good too. Also possible just run it before releases… the idea is just to prevent them to accumulate without noticing