[Feature Request] Support NX Monorepos

[brianespinosa]

Affected Packages

get-workspaces

Problem A

As of today, in an NX workspace, it is not possible for Changesets to discover any related packages that are configured for the workspace.

There are only two ways of configuring where packages might be in an NX repo:

1. project.json and nx.json
2. package.json and nx.json (also requires a workspace.json file)

With the first option, NX relies on a project.json file existing at the root of any packages in the monorepo, which for publishable packages, will also need to have a package.json alongside them.

With the second option, there is a workspace.json file at the root that references all packages in the monorepo.

Note: If there is a workspaces.json file, NX will not glob for project.json files in the directories. It appears that readWorkspaceConfiguration is available as part of @nrwl/devkit.

Proposed solution

1. If @manypkg/get-packages does not return any packages, check for nx.json at our monorepo root. If present, use readWorkspaceConfiguration() from @nrwl/devkit to get the workspace tree instead of trying to implement ourselves.
2. Define @nrwl/devkit as an optional dependency, which will already be present in any NX monorepo.

Problem B

By default, NX does not compile publishable code adjacent to publishable packages, and instead usually has a root /dist directory where all build outputs go which are namespaced to the lib/app. This compiled output will also contain a copy of the package.json with all relevant dependencies added to it, and is what we should be publishing.

The build output is configurable in NX per package and there is likely a function as part of @nrwl/devkit that also returns configured output for each package

Not every package is necessarily publishable, again configurable, and thus will not get built to the /dist directory

Proposed solution

1. Use @nrwl/devkit to find which package(s) are publishable and publish their assets directly from the configured output directory

---

It is also worth noting that there are a couple other issues already opened that are asking for support for this, but are asking for ways to configure all of this by hand, versus using the tool set in @nrwl/devkit:

• Allow passing a custom glob to find packages #654
• Changeset doesn't detect monorepo context in simple nx monorepo without workspaces #761
• when using pnpm and publishConfig.directory publish errors out #773

[brianespinosa]

I suspect there will be other changes needed to some of the release plan functions in order to make this work, but I have not dug into the code far enough to comment on those, but I will come back and update the top of this ticket with affected packages once I know more.

I plan on poking around a bit with these proposed solutions using @nrwl/devkit sometime this weekend. I know that there was some POC work last year in #654 but it seems to be using some bespoke solutions to determine what is in the workspace, which could break if NX ever updates how they allow configuration. IMO it would be much safer (and less work) for us to use actual NX packages to find what we need.

Additional thought... once we have this working, we could create an NX plugin within the org that could be added to an NX repository and would have all of this stuff configured and added by default. I know we already have changesets/bot and changesets/action.

[brianespinosa]

It is also worth noting that manypkg has some (lovely) checks for single version rule in a monorepo. NX monorepos by default already following this rule as packages that get published will get their dependencies automatically assembled into the package.json file at build time. Additionally, manypkg has some great utilities for executing scrips on one or many packages within a monorepo. These features are already part of NX. Based on this... I don't see anyone using manypkg within an NX monorepo at all, and therefore, adding support for NX to @manypkg/get-packages would likely require them to bail out of a lot of the edge cases that NX would add, all for us to be able to use their @manypkg/get-packages internally here.

I think the best approach for us is to remove the error throwing in our current get-packages and switch back to using that everywhere, and do our check for NX repo there. Our own get-packages would still use @manypkg/get-packages under the hood, but would have the additional logic to check for nx.json if no packages are returned from @manypkg/get-packages.

[brianespinosa]

Will likely need #802

[Andarist]

Use @nrwl/devkit to find which package(s) are publishable and publish their assets directly from the configured output directory

Isn't "publishable" easily detectable with our usual logic? So the only missing piece here would be to change the directory from which we call into the publish command?

@nrwl/devkit

I'm not yet sold on using this dependency - if the logic for finding this stuff is easy then I think it would make sense to reimplement it. I've asked this in the linked PR but perhaps this is a better place for the discussion so gonna ask it here too... is this package always directly installed by Nx users?

Do you know if optional peer dependencies are automatically installed by npm? this could have been a blocker for this solution

Additional thought... once we have this working, we could create an NX plugin within the org that could be added to an NX repository and would have all of this stuff configured and added by default. I know we already have changesets/bot and changesets/action.

If we can make things easier for certain use cases - I'm all up for it.

Additionally, manypkg has some great utilities for executing scrips on one or many packages within a monorepo. These features are already part of NX. Based on this... I don't see anyone using manypkg within an NX monorepo at all, and therefore, adding support for NX to @manypkg/get-packages would likely require them to bail out of a lot of the edge cases that NX would add, all for us to be able to use their @manypkg/get-packages internally here.

Yeah, maybe... although I'm still somewhat concerned about potential problems with optional peer deps etc. I'm happy to give this more thought later on though. Note that the script runner from manypkg isn't ever part of what you install when using Changesets so that's not a big deal. In here, we only care about finding the package locations etc. @manypkg/get-packages is just a generic tool that has this goal - so I still think that it's worth adding support for this there. It could still be useful for some other CLIs trying to do the same, "simple", thing - locating packages.

[airtonix]

I personally think that changesets has too much tight coupling of strategies within itself.

It should really look to refactor into a plugin based tool where:

• formatting changelogs is a plugin
• discovering the changed packages graph, again another plugin
• logic that generates a pull request, yes... another plugin

[JakeGinnivan]

@airtonix feel free to weigh in on the RFC at #849 which covers point 2.

[esakal]

Anyone figure out how to use changeset with applications managed in NX monorepo with the existing changeset support?

[Ambroos]

We recently got started with our Nx monorepo, and didn't want to invest too much time into switching release methods right away (and assumed changesets would just work with Nx). When it didn't I spent some time coming up with a workaround that might be helpful for others to make changesets work.

The strategy: creating a dummy lerna.json on the fly to make changesets detect package directories. There are two variants of this: one based on the source directories for the version command, and another one with dist directories (more hacky) for publish.

It's definitely a hack, but it does work and doesn't require package patching. Here's the full setup, which involves package.json, project.json for any projects you want to support, a few scripts and customisations for the GH action.

With this you just have to run the wrapped changeset script via npm/pnpm/yarn rather than running it directly / via npx. Args still work.

{
  "name": "@org/your-monrepo",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "changeset": "node ./scripts/changeset-wrap.mjs",
    "publish-combo-changeset": "echo \"{}\" > lerna.json && pnpm exec nx run-many -t changeset-prepare-publish && pnpm exec changeset publish"
  }

{
  "name": "my-lib",
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "sourceRoot": "libs/my-lib/src",
  "projectType": "library",
  "targets": {
    "build": {
      ...
    },
    "changeset-prepare-publish": {
      "cache": false,
      "executor": "nx:run-commands",
      "dependsOn": ["build"]
      "options": {
        "command": "node ./utils/changeset-publish-prepare.mjs dist/my-lib"
      }
    },
    ...
  }
}

# In your changeset version / release GH action
      - name: Hack Nx support into Changeset
        run: node ./scripts/changeset-ci-prepare.mjs

      - name: Changeset - create pull request / publish packages
        id: changesets
        uses: changesets/action@v1.4.5
        with:
          publish: pnpm publish-combo-changeset
          # the default version command is OK thanks to the action script executed before this

# .gitignore
lerna.json

/**
 * create-src-lerna-json.mjs
 */
import nx from '@nx/devkit'

export default async function createSrcLernaJson() {
  const projectRoots = await nx
    .createProjectGraphAsync({ exitOnError: true, resetDaemonClient: true })
    .then((projectGraph) =>
      Object.entries(nx.readProjectsConfigurationFromProjectGraph(projectGraph).projects)
        .filter(([, p]) => {
          // Filter projects, we filter on a specific target existing. Easy to customise
          return 'changeset-prepare-publish' in (p.targets ?? {})
        })
        .map(([, p]) => `./${p.root}`),
    )

  const content = {
    ['not a real lerna.json']: 'This is a dummy lerna.json to support changeset',
    packages: projectRoots,
  }

  fs.writeFileSync('./lerna.json', JSON.stringify(content, null, 2))
  return content
}

/**
 * changeset-ci-prepare.mjs
 */
import createSrcLernaJson from './create-src-lerna-json.mjs'
createSrcLernaJson().then((content) => {
  console.log('Created dummy lerna.json:', content)
})

/**
 * changeset-wrap.mjs
 */
import fs from 'fs'
import { execSync } from 'child_process'
import createSrcLernaJson from './create-src-lerna-json.mjs'

// Verify we are in root by checking nx.json exists
if (!fs.existsSync('./nx.json')) {
  console.log('This script can only run from repo root.')
  process.exit(1)
}

createSrcLernaJson()
  .then(() => {
    // Run changeset, passing all args.
    const args = process.argv.slice(2).join(' ')
    execSync(`pnpm exec changeset ${args}`, { stdio: 'inherit' })
  })
  .then(() => {
    cleanup()
  })
  .catch((e) => {
    cleanup()
    // Exit with error
    console.error(e)
    process.exit(1)
  })

function cleanup() {
  // remove lerna.json
  fs.unlinkSync('./lerna.json')
}

process.on('SIGINT', function () {
  cleanup()
  process.exit()
})

/**
 * add-to-lerna-json.mjs
 */

import fs from 'fs'

// Verify we are in root by checking nx.json exists
if (!fs.existsSync('./nx.json')) {
  console.log('This script can only run from repo root.')
  process.exit(1)
}

// get dist path from argv
const distPath = process.argv[2]

const lernaJson = JSON.parse(fs.readFileSync('./lerna.json', 'utf8'))
let existingProjects = lernaJson.packages ?? []

fs.writeFileSync(
  './lerna.json',
  JSON.stringify(
    {
      packages: [...existingProjects, distPath],
    },
    null,
    2,
  ),
)