Composer 2.2 breaks updates in monorepos with certain dependency structures

[anomiex]

How are you running Renovate?

WhiteSource Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

Please select which platform you are using if self-hosting.

No response

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Describe the bug

Consider a monorepo where package C depends on B which depends on A, and C has composer.lock committed. If renovate is attempting an update in both C and at least one of A or B, composer 2.2 will choke because it was told to update only the dependency and not the monorepo packages.

This worked with composer 2.1 and earlier as those versions would update path deps in this situation. git bisect indicates that it was changed by composer/composer#10157.

Minimal reproduction repository: https://github.com/anomiex/renovate-test/

Renovate reports failure at anomiex/renovate-test#7 (comment)

If you happen to decide this is at root a bug in composer rather than renovate, I ask you to please file it with them as I expect you'll make a better case for them changing behavior here than I would. I wouldn't know how to respond if they say "just do composer update composer/semver monorepo/a monorepo/b --with-dependencies instead", as that does indeed work.

Relevant debug logs

Logs

DEBUG: manager.getUpdatedPackageFiles() reuseExistinbranch=false(branch="renovate/composer-semver-3.x")
DEBUG: Starting search at index 103(packageFile="c/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Found match at index 103(packageFile="c/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Found composer lock file(packageFile="c/composer.json", branch="renovate/composer-semver-3.x")
DEBUG: Contents updated(packageFile="c/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Starting search at index 103(packageFile="b/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Found match at index 103(packageFile="b/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Contents updated(packageFile="b/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Starting search at index 73(packageFile="a/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Found match at index 73(packageFile="a/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: Contents updated(packageFile="a/composer.json", branch="renovate/composer-semver-3.x")
{
  "depName": "composer/semver"
}
DEBUG: composer.updateArtifacts(c/composer.json)(branch="renovate/composer-semver-3.x")
DEBUG: composer command(branch="renovate/composer-semver-3.x")
{
  "cmd": "composer",
  "args": "update composer/semver --with-dependencies --ignore-platform-reqs --no-ansi --no-interaction --no-scripts --no-autoloader --no-plugins"
}
DEBUG: Using docker to execute(branch="renovate/composer-semver-3.x")
{
  "image": "php"
}
DEBUG: Resolved version(branch="renovate/composer-semver-3.x")
{
  "toolName": "composer",
  "constraint": "^2.1",
  "resolvedVersion": "2.2.3"
}
DEBUG: No tag or tagConstraint specified(branch="renovate/composer-semver-3.x")
{
  "image": "docker.io/renovate/php"
}
DEBUG: Fetching Docker image: docker.io/renovate/php(branch="renovate/composer-semver-3.x")
DEBUG: Finished fetching Docker image(branch="renovate/composer-semver-3.x")
DEBUG: Executing command(branch="renovate/composer-semver-3.x")
{
  "command": "docker run --rm --name=renovate_php --label=renovate_child -v \"/mnt/renovate/gh/anomiex/renovate-test\":\"/mnt/renovate/gh/anomiex/renovate-test\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e COMPOSER_CACHE_DIR -e COMPOSER_AUTH -w \"/mnt/renovate/gh/anomiex/renovate-test/c\" docker.io/renovate/php bash -l -c \"install-tool composer 2.2.3 && composer update composer/semver --with-dependencies --ignore-platform-reqs --no-ansi --no-interaction --no-scripts --no-autoloader --no-plugins\""
}
DEBUG: rawExec err(branch="renovate/composer-semver-3.x")
{
  "err": {
    "killed": false,
    "code": 2,
    "signal": null,
    "cmd": "docker run --rm --name=renovate_php --label=renovate_child -v \"/mnt/renovate/gh/anomiex/renovate-test\":\"/mnt/renovate/gh/anomiex/renovate-test\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e COMPOSER_CACHE_DIR -e COMPOSER_AUTH -w \"/mnt/renovate/gh/anomiex/renovate-test/c\" docker.io/renovate/php bash -l -c \"install-tool composer 2.2.3 && composer update composer/semver --with-dependencies --ignore-platform-reqs --no-ansi --no-interaction --no-scripts --no-autoloader --no-plugins\"",
    "stdout": "Installing tool composer v2.2.3\nComposer version 2.2.3 2021-12-31 12:18:53\n",
    "stderr": "Loading composer repositories with package information\nUpdating dependencies\nYour requirements could not be resolved to an installable set of packages.\n\n  Problem 1\n    - Root composer.json requires monorepo/b @dev -> satisfiable by monorepo/b[dev-master].\n    - monorepo/b dev-master requires monorepo/a @dev -> found monorepo/a[dev-master] but these were not loaded, likely because it conflicts with another require.\n\nUse the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.\n",
    "message": "Command failed: docker run --rm --name=renovate_php --label=renovate_child -v \"/mnt/renovate/gh/anomiex/renovate-test\":\"/mnt/renovate/gh/anomiex/renovate-test\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e COMPOSER_CACHE_DIR -e COMPOSER_AUTH -w \"/mnt/renovate/gh/anomiex/renovate-test/c\" docker.io/renovate/php bash -l -c \"install-tool composer 2.2.3 && composer update composer/semver --with-dependencies --ignore-platform-reqs --no-ansi --no-interaction --no-scripts --no-autoloader --no-plugins\"\nLoading composer repositories with package information\nUpdating dependencies\nYour requirements could not be resolved to an installable set of packages.\n\n  Problem 1\n    - Root composer.json requires monorepo/b @dev -> satisfiable by monorepo/b[dev-master].\n    - monorepo/b dev-master requires monorepo/a @dev -> found monorepo/a[dev-master] but these were not loaded, likely because it conflicts with another require.\n\nUse the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.\n",
    "stack": "Error: Command failed: docker run --rm --name=renovate_php --label=renovate_child -v \"/mnt/renovate/gh/anomiex/renovate-test\":\"/mnt/renovate/gh/anomiex/renovate-test\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e COMPOSER_CACHE_DIR -e COMPOSER_AUTH -w \"/mnt/renovate/gh/anomiex/renovate-test/c\" docker.io/renovate/php bash -l -c \"install-tool composer 2.2.3 && composer update composer/semver --with-dependencies --ignore-platform-reqs --no-ansi --no-interaction --no-scripts --no-autoloader --no-plugins\"\nLoading composer repositories with package information\nUpdating dependencies\nYour requirements could not be resolved to an installable set of packages.\n\n  Problem 1\n    - Root composer.json requires monorepo/b @dev -> satisfiable by monorepo/b[dev-master].\n    - monorepo/b dev-master requires monorepo/a @dev -> found monorepo/a[dev-master] but these were not loaded, likely because it conflicts with another require.\n\nUse the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.\n\n    at ChildProcess.exithandler (child_process.js:383:12)\n    at ChildProcess.emit (events.js:400:28)\n    at ChildProcess.emit (domain.js:475:12)\n    at maybeClose (internal/child_process.js:1058:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:293:5)"
  }
}
INFO: Composer requirements cannot be resolved(branch="renovate/composer-semver-3.x")
DEBUG: composer.updateArtifacts(b/composer.json)(branch="renovate/composer-semver-3.x")
DEBUG: No composer.lock found(branch="renovate/composer-semver-3.x")
DEBUG: composer.updateArtifacts(a/composer.json)(branch="renovate/composer-semver-3.x")
DEBUG: No composer.lock found(branch="renovate/composer-semver-3.x")
DEBUG: Updated 3 package files(branch="renovate/composer-semver-3.x")

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction repository in the bug description

[rarkins]

What logic do you think Renovate could implement or do differently? And is this caused by grouping rules you added or because the same dependency is updated in two subfolders?

[anomiex]

Hmm. Poking at this again, I've convinced myself there really is a composer bug here too as it happens even if no updates are being done. I filed that as composer/composer#10431.

But I'm not entirely sure that when they fix it that it won't still leave a problem here.

And is this caused by grouping rules you added or because the same dependency is updated in two subfolders?

The problem (theoretically if they fix composer/composer#10431 in the most minimal manner) is that the changes to a/composer.json and/or b/composer.json would need an update to hashes in c/composer.lock, but composer is not allowing that because it's restricted to only updating the composer/semver package.

On the other hand, they might decide to go back to fully "unlocking" all path repo sourced packages, in which case this problem would go away.

What logic do you think Renovate could implement or do differently?

You'd probably have to look at the path repos declared in composer.json to detect other packages that are being sourced from within the monorepo, and include any that are direct dependencies in the composer update <package> --with-dependencies command.

Or maybe there's a composer command to list out dependencies that are sourced from the monorepo?

[anomiex]

So good news, it looks like the fix in composer/composer#10431 will indeed probably make this issue go away without changes to renovate! I suppose we'll have to wait for Composer to release a version 2.2.4 to fully test that though.

[anomiex]

I see they released 2.2.4. Test repo no longer has a problem!