Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created VSIX via vsce package --yarn does not include not-hoisted deep dependency #432

Open
yhatt opened this issue Mar 23, 2020 · 8 comments
Assignees
Labels
bug Issue identified by VS Code Team member as probable bug help wanted Issues identified as good community contribution opportunities

Comments

@yhatt
Copy link

yhatt commented Mar 23, 2020

Think about a simple package like this, developed with yarn 1.22.4:

{
  "name": "vsce-yarn-lacked-dependency",
  "description": "example",
  "version": "0.0.1",
  "publisher": "yhatt",
  "repository": "https://github.com/yhatt/vsce-yarn-lacked-dependency",
  "engines": {
    "vscode": "^1.40.0"
  },
  "activationEvents": [
    "*"
  ],
  "main": "./extension.js",
  "devDependencies": {
    "vsce": "^1.74.0"
  },
  "dependencies": {
    "markdown-it": "^10.0.0"
  }
}
// extension.js
const md = require("markdown-it")();

module.exports = {
  activate: () => console.log(md.render("Hello, world!"))
};

The creation of extension through yarn vsce package --yarn will success, but will fail to activate in VS Code.

abstractExtensionService.ts:396 Activating extension 'yhatt.vsce-yarn-lacked-dependency' failed: Cannot find module 'entities/lib/maps/entities.json'
Require stack:
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/lib/common/entities.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/lib/common/utils.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/lib/index.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/index.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/extension.js
- /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js
- /Applications/Visual Studio Code.app/Contents/Resources/app/out/bootstrap-amd.js
- /Applications/Visual Studio Code.app/Contents/Resources/app/out/bootstrap-fork.js.

It would work correctly if created through npm with yarn vsce package.

I've compared the structure in both of VSIX archives, and noticed the VSIX from yarn has included only a dependency entities@1.1.2 hoisted from devDependenciess (vsce -> markdown-it@8.4.2 -> entities@1.1.2).

VSIX from yarn VSIX from npm
.
└── node_modules
    ├── argparse
    │   └── lib
    │       ├── action
    │       │   ├── append
    │       │   └── store
    │       ├── argument
    │       └── help
    ├── entities (1.1.2)
    │   ├── lib
    │   ├── maps
    │   └── test
    ├── linkify-it
    │   └── lib
    ├── markdown-it
    │   ├── bin
    │   ├── dist
    │   └── lib
    │       ├── common
    │       ├── helpers
    │       ├── presets
    │       ├── rules_block
    │       ├── rules_core
    │       └── rules_inline
    ├── mdurl
    ├── sprintf-js
    │   ├── demo
    │   ├── dist
    │   ├── src
    │   └── test
    └── uc.micro
        ├── categories
        │   ├── Cc
        │   ├── Cf
        │   ├── P
        │   └── Z
        └── properties
            └── Any
.
└── node_modules
    ├── argparse
    │   └── lib
    │       ├── action
    │       │   ├── append
    │       │   └── store
    │       ├── argument
    │       └── help
    ├── linkify-it
    │   └── lib
    ├── markdown-it
    │   ├── bin
    │   ├── dist
    │   ├── lib
    │   │   ├── common
    │   │   ├── helpers
    │   │   ├── presets
    │   │   ├── rules_block
    │   │   ├── rules_core
    │   │   └── rules_inline
    │   └── node_modules
    │       └── entities (2.0.0)
    │           └── lib
    ├── mdurl
    ├── sprintf-js
    │   ├── demo
    │   ├── dist
    │   ├── src
    │   └── test
    └── uc.micro
        ├── categories
        │   ├── Cc
        │   ├── Cf
        │   ├── P
        │   └── Z
        └── properties
            └── Any

Look at the difference of the structure about entities package.

The extension expects to use a not-hoisted entities@2.0.0 in dependencies (markdown-it@10.0.0 -> entities@2.0.0), not entities@1.1.2 in vsce's devDependency.

In fact, there is not-hoisted entities@2.0.0 in /node_modules/markdown-it/node_modules while development even if installed packages by yarn. I suppose the process of packaging via yarn has some wrong.

Workaround

An available workaround is just using npm by omit --yarn.

However, it has not a worth in a few case. The author of extension has to use --yarn if using resolutions field (only supported in yarn), for resolving some vulnerabilities in deep dependency. vsce package via npm would throw an error due to the different structure of node_modules between npm and yarn.

For example, the following is to fix a vulnerability in deep dependency of Puppeteer v2.1.0 by yarn.

{
  "dependencies": {
    "puppeteer": "^2.1.0"
  },
  "resolutions": {
    "**/extract-zip/mkdirp": "^0.5.3",
  }
}

vsce package using npm will fail by the different structure.

$ vsce package
 ERROR  Command failed: npm list --production --parseable --depth=99999 --loglevel=error
npm ERR! missing: mkdirp@0.5.1, required by extract-zip@1.6.7

Sometimes I've met this trouble in the extension developed by me: marp-team/marp-vscode#35, marp-team/marp-vscode#130 (comment)

UPDATE: npm v7 has shipped with support for yarn.lock and can generate fully deterministic dependency tree. Just using npm v7 might not need to worry about incorrect packaging.

@joaomoreno joaomoreno added bug Issue identified by VS Code Team member as probable bug help wanted Issues identified as good community contribution opportunities labels Mar 31, 2020
@joaomoreno joaomoreno self-assigned this Mar 31, 2020
@Kapelianovych
Copy link

The same for me too. Extension cannot be package neither by yarn, nor by npm.

npm ERR! missing: @babel/parser@7.10.2, required by hegel-language-server@0.0.43
npm ERR! missing: @babel/plugin-proposal-class-properties@7.10.1, required by hegel-language-server@0.0.43
npm ERR! missing: @babel/plugin-proposal-nullish-coalescing-operator@7.10.1, required by hegel-language-server@0.0.43

And so on.

@viktomas
Copy link
Contributor

viktomas commented Jun 26, 2020

We faced the same issue with gitlab-vscode-extension, this is my writeup:

Problem

The correct npm.ts implementation discards module version and packages incorrect version of NPM module.

For our extension, this happens because we have two different versions of ajv in our dependencies. Production code uses v6 and test code v5. The vsce ls (and so vsce package) picks the dev dependency.

Example project

The steps that follow can reproduce the issue reliably on gitlab-vscode-exteions.

  1. clone the extension git@gitlab.com:gitlab-org/gitlab-vscode-extension.git
  2. checkout git checkout v3.0.0
  3. install dependencies yarn
  4. observe that the production code depends on ajv@6 by running yarn list --prod
    ├─ har-validator@5.1.3
     │  ├─ ajv@^6.5.5
    
  5. vsce ls --yarn will say that we will package the module straight in node_modules
    vsce ls --yarn | grep ajv
    ...
    node_modules/ajv/package.json
    
  6. that's incorrect version:
    cat node_modules/ajv/package.json | grep version
    "version": "5.5.2",
    cat node_modules/har-validator/node_modules/ajv/package.json | grep version
    "version": "6.12.2",

Possible fixes

The issue is in the flattening logic. When we remove the version and then "ignore" all other versions we are potentially using incorrect dependencies. The only way to prevent this is IMO honouring the dependency structure given by yarn and not attempting to flatten it.

This can result in very subtle and hard to debug issues. Or production incidents

@johnsoncodehk
Copy link

In case someone made the same mistake as me, I just fixed it like this:

yarn && vsce package --yarn

@yhatt
Copy link
Author

yhatt commented Oct 16, 2020

How about always using npm instead of yarn for dependency resolution if vsce was using npm >= v7? npm v7 can generate fully deterministic dependency tree based on yarn.lock so can expect more reliable packaging.
https://blog.npmjs.org/post/621733939456933888/npm-v7-series-why-keep-package-lockjson

Someone may feel like a bit strange to use npm even if running vsce with --yarn, but it would be much better than encountering production incident as like as brought in Marp, GitLab, and Microsoft Azure Logic Apps.

@yhatt
Copy link
Author

yhatt commented Oct 19, 2020

Let me write down I met this trouble in our extension marp-vscode again. How to reproduce:

git clone https://github.com/marp-team/marp-vscode.git
cd ./marp-vscode
git checkout 5f285dc
yarn && yarn package

The following error would output to developer console if tried to activate created VSIX.

abstractExtensionService.ts:717 Activating extension 'marp-team.marp-vscode' failed: Cannot find module 'cssesc'
Require stack:
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/selectors/className.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/parser.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/processor.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/index.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-minify-selectors/dist/index.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/@marp-team/marp-core/lib/marp.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/lib/extension.js
- /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/loader.js
- /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/bootstrap-amd.js
- /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/bootstrap-fork.js.
_logMessageInConsole @ abstractExtensionService.ts:663

It can fix by adding --no-yarn option to vsce command in package npm-script: marp-team/marp-vscode#178

@yoyo930021
Copy link

yoyo930021 commented Nov 8, 2020

We have the same problem when publish Vetur.
We downgrade to old version for solve this problem at the end.

@roland-reed
Copy link

Any updates on this issue? I have the same problem here, downgrade to 1.76 still seems not to work.

kyliau added a commit to kyliau/vscode-ng-language-service that referenced this issue Dec 4, 2020
`semver` is a prod dependency of `vscode-languageclient`.
In the old build pipeline, `vscode-languageclient` is declared in
`client/package.json`.
When yarn installs dependencies in the `client` directory, it will put `semver`
next to `vscode-languageclient` in `client/node_modules`.

In the new build pipeline, `vscode-languageclient` is declared in root
`package.json`.
When yarn installs dependencies in the root directory, it chooses not to hoist
`semver`, instead puts it in a nested node_modules under
`node_modules/vscode-languageclient/node_modules`.

There is a bug in vsce that causes it to ignore nested `node_modules`,
resulting in a `vsix` build that is broken. The issue is trackeed
[here](microsoft/vscode-vsce#432), but it does not
look like a fix will come any time soon.

For now we should install our dependencies using NPM.
kyliau added a commit to kyliau/vscode-ng-language-service that referenced this issue Dec 4, 2020
`semver` is a prod dependency of `vscode-languageclient`.
In the old build pipeline, `vscode-languageclient` is declared in
`client/package.json`.
When yarn installs dependencies in the `client` directory, it will put `semver`
next to `vscode-languageclient` in `client/node_modules`.

In the new build pipeline, `vscode-languageclient` is declared in root
`package.json`.
When yarn installs dependencies in the root directory, it chooses not to hoist
`semver`, instead puts it in a nested node_modules under
`node_modules/vscode-languageclient/node_modules`.

There is a bug in vsce that causes it to ignore nested `node_modules`,
resulting in a `vsix` build that is broken. The issue is trackeed
[here](microsoft/vscode-vsce#432), but it does not
look like a fix will come any time soon.

For now we should install our dependencies using NPM.
kyliau added a commit to kyliau/vscode-ng-language-service that referenced this issue Dec 7, 2020
`semver` is a prod dependency of `vscode-languageclient`.
In the old build pipeline, `vscode-languageclient` is declared in
`client/package.json`.
When yarn installs dependencies in the `client` directory, it will put `semver`
next to `vscode-languageclient` in `client/node_modules`.

In the new build pipeline, `vscode-languageclient` is declared in root
`package.json`.
When yarn installs dependencies in the root directory, it chooses not to hoist
`semver`, instead puts it in a nested node_modules under
`node_modules/vscode-languageclient/node_modules`.

There is a bug in vsce that causes it to ignore nested `node_modules`,
resulting in a `vsix` build that is broken. The issue is trackeed
[here](microsoft/vscode-vsce#432), but it does not
look like a fix will come any time soon.

For now we should install our dependencies using NPM.
kyliau added a commit to angular/vscode-ng-language-service that referenced this issue Dec 7, 2020
`semver` is a prod dependency of `vscode-languageclient`.
In the old build pipeline, `vscode-languageclient` is declared in
`client/package.json`.
When yarn installs dependencies in the `client` directory, it will put `semver`
next to `vscode-languageclient` in `client/node_modules`.

In the new build pipeline, `vscode-languageclient` is declared in root
`package.json`.
When yarn installs dependencies in the root directory, it chooses not to hoist
`semver`, instead puts it in a nested node_modules under
`node_modules/vscode-languageclient/node_modules`.

There is a bug in vsce that causes it to ignore nested `node_modules`,
resulting in a `vsix` build that is broken. The issue is trackeed
[here](microsoft/vscode-vsce#432), but it does not
look like a fix will come any time soon.

For now we should install our dependencies using NPM.
lirc572 added a commit to lirc572/TD-VSC that referenced this issue Jan 11, 2021
hoonoh added a commit to hoonoh/slonik-live-server that referenced this issue Nov 17, 2021
- package using npm
  (microsoft/vscode-vsce#432)
- add .vscodeignore
- remove invalid import
@pkit
Copy link

pkit commented Feb 11, 2022

Err, open 2 for years, still not fixed?

Have two dependencies with different chalk versions:

yarn why v1.22.17
[1/4] Why do we have the module "chalk"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "chalk@2.4.2"
info Has been hoisted to "chalk"
info Reasons this module exists
   - Hoisted from "@babel#core#@babel#code-frame#@babel#highlight#chalk"
   - Hoisted from "@sqltools#base-driver#@sqltools#log#pino-pretty#args#chalk"
info Disk size without dependencies: "44KB"
info Disk size with unique dependencies: "96KB"
info Disk size with transitive dependencies: "188KB"
info Number of shared dependencies: 6
=> Found "pino-pretty#chalk@4.1.2"
info This module exists because "@sqltools#base-driver#@sqltools#log#pino-pretty" depends on it.
info Disk size without dependencies: "56KB"
info Disk size with unique dependencies: "92KB"
info Disk size with transitive dependencies: "184KB"
info Number of shared dependencies: 5
Done in 0.16s.

But only one dependency is present in vsix file:

unzip -l my-extension-0.0.1.vsix | grep chalk
     6439  2022-01-18 22:14   extension/node_modules/chalk/index.js
     1921  2022-01-18 22:14   extension/node_modules/chalk/index.js.flow
     1109  2022-01-18 22:14   extension/node_modules/chalk/license
     1195  2022-01-18 22:14   extension/node_modules/chalk/package.json
    10774  2022-01-18 22:14   extension/node_modules/chalk/readme.md
     3133  2022-01-18 22:14   extension/node_modules/chalk/templates.js

I'm in awe actually, either say that you don't really support yarn or fix it.
As right now you de facto do not support yarn, because of this bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue identified by VS Code Team member as probable bug help wanted Issues identified as good community contribution opportunities
Projects
None yet
Development

No branches or pull requests

8 participants