[Question] licenses and generate-disclaimer

[matzeeable]

Hi yarn team,

Awesome work on berry! I am working on an open source project (https://github.com/devowlio/wp-react-starter) and want to migrate yarn to berry. I want to benefit from faster install times also on my CI pipeline. Generally, I had a look at "issues" which can be caused by upgrading. The first thing I note:

yarn licenses
yarn generate-disclaimer

What happened to this commands? They seem not to be available / ported. Are they on the roadmap? Are there better alternatives?

Best regards,
Matthew 🙂

[arcanis]

Hello 👋

We felt like those commands had a too fringe use case to be part of the core offering. Additionally there are various ways such tools could be expanded and configured, and trying to support them (and keep up with feature requests) would put an unwanted additional burden on our team.

The good news is, those commands are exactly the kind of thing we had in mind when designing the plugin API, so it's entirely possible to implement them in userspace (and share your implementation with the community if you so wish).

The plugin documentation is admittedly a bit sparse at the moment as we're working on improving that. In the meantime, you can check the plugin tutorial on the website and the plugin sources in this repo to get an idea of how it works.

[matzeeable]

Hi again!

Thanks for your fast response. I had a look at the codebase unfortunately I do not really understand how to achieve the following thing:

Get all dependencies for a given package.json including deep dependencies including the metadata (license, version, ...). Can you give me a tip? Consider that yarn classic had an issue with workspaces: yarnpkg/yarn#5174 - is this resolved in berry? I dived into this a bit deeper and the following code snippet gives me all dependencies if I run it inside a package (also respects workspaces):

import { resolve } from "path";

class ProductionDependencies {
    private all: string[] = [];

    constructor(packageJsonPath: string) {
        this.fetchAll(Object.keys(require(packageJsonPath).dependencies));
    }

    public fetchAll(deps: string[]) {
        for (const dep of deps) {
            if (this.all.indexOf(dep) === -1) {
                // eslint-disable-next-line @typescript-eslint/no-var-requires
                const pkg = require(require.resolve(`${dep}/package.json`));
                this.fetchAll(Object.keys(pkg.dependencies || [])); // devDependencies is not needed for license check and disclaimer
                console.log(pkg.name, pkg.license || pkg.licenses || "UNKNOWN");
            }
        }
        this.all.push(...deps);
    }
}

new ProductionDependencies(resolve("./package.json"));

I am thinking of creating a public package - I also do not promise if I create the package because perhaps someone maintaining a similar package is open to create a yarn plugin. I do not know yet if it will be a yarn plugin or a standalone package so it can be used with npm, too. The most important part for me is the workspace-compatibility.

So, I will note further things in this issue (just for myself, perhaps it can be interesting for others):

1. Obtain a list of all dependencies with their metadata of a package.json (see above code snippet)
2. Configure a whitelist for packages and skip their license check ["create-react-context@0.2.3"]: string[]
3. Configure a whitelist for allowed licenses (string[]) for the license check
4. Check the license meta and fallback to UNKNOWN
5. If the found license is not UNKNOWN:
   1. Check if license starts with ( and pass it through spdx-expression-parse. Afterwards do the license check based on the parsed expression
   2. Otherwise check if license is a string[] and if not, parse it to a string[]
   3. Pass the license array to spdx-correct and do the license check based on the result
6. Otherwise obtain the license from other files (mark "guess" with e. g. MIT*, and print a warning)
   1. scripts/generate-licenses-js.js generates a list of unique parts of LICENSE texts so they can be infered when package.json#license cannot be resolved correctly. The generated RegEx parts are found here: scripts/licenses
   2. src/util/normalize-manifest/infer-license.js is the programmatically way to infer a license from a string (e. g. LICENSE, NOTICE, COPYING or README file)
7. Throw an error of not found licenses. The developer has three options to correct the issue
   • Add the package to the whitelist for packages (the developer opts-in to skip that license)
   • Contact the developer of the package and ask to create a valid license in their package.json file
   • Contribute to this package' compat table
8. Throw an error if license check failed for a package. The developer has three options:
   • Add the license of the package to the allowed licenses
   • Add the package to the whitelist for packages (the developer opts-in to skip that license)
   • Search an alternative package

The generate-disclaimer I think is more easier, but I will dive into this later.

This are just thoughts how it can be implemented - I got inspired by the already implemented yarn licenses list and license-checker.

[mhassan1]

I've just pushed up a plugin that adds yarn licenses: https://github.com/mhassan1/yarn-plugin-licenses

It only does yarn licenses list right now, inspired by Yarn v1. We can add more commands in the future. Also, it only works with the latest Berry sources from master (it uses treeUtils), so it's not quite production ready.

Check it out and let me know if you have any feedback.

[mhassan1]

@arcanis I've written a plugin that provides yarn licenses list: https://github.com/mhassan1/yarn-plugin-licenses. Assuming there's interest in the community, what would be the best way to share it?

[arcanis]

I think we should add a "community plugins" page on the website to list those 🤔

[mhassan1]

Did you have something in mind? I'd be happy to contribute that to the website, but I'm not sure where you would want it, or how you would want to manage it.

[noahnu]

I also created an open source plugin for auditing licenses: https://github.com/tophat/yarn-plugin-licenses. It's still early stage, and needs some cleanup to the project (mainly tests), but as it's written right now, it can complement @mhassan1's plugin, as the commands don't conflict (yarn licenses audit).

[matzeeable]

@noahnu thanks for your reply. Does the plugin you provide support per-package audit? We need to manage multiple licenses per project. :-)

[noahnu]

@noahnu thanks for your reply. Does the plugin you provide support per-package audit? We need to manage multiple licenses per project. :-)

@matzeeable I plan on cleaning it up a bit. At the moment it lets you specify a license validator (RegEx/predicate function) in a config file and then runs your license validator against all licenses for all installed dependencies (including transitive dependencies) in your project. It runs against the entire project at the moment, but wouldn't be much work to constrain the scope to a specific workspace, and allow finer filtering on dev dependencies vs. prod depepndencies vs. peer, etc.

It doesn't generate disclaimers. You can use @mhassan1's plugin for that.

[Eli-Black-Work]

Adding to the list:

We use license-checker-webpack-plugin. It's a license checker that's deeply integrated with Webpack, so it shows you only the licenses of packages that are still left after tree shaking 🙂

[jeroenhabets]

For people, stumbling upon this ticket via Getting Started > CLI Commands > Removed from core, please note this plugin is listed on Features > Plugins > Contrib plugins

P.S. Especially for people like me who overlook Kurt-von-Laven's mention on Apr 29, 2021 🤭

[ranisalt]

Second link is broken