HookWebpackError: ENOENT: no such file or directory, scandir

[danielemoroni]

Description

I did a fresh download and install, and I get the error in the title.

Reproduce

I literally followed the contributing guide. After downloading mambaforge:

1. git clone https://github.com/jupyterlite/jupyterlite
2. mamba env update --file .binder/environment.yml
3. mamba activate jupyterlite-dev
4. python -m pip install -r requirements-docs.txt -r requirements-lint.txt
5. doit

and I get the error

. build:js:app
yarn run v1.22.17
$ C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules.bin\lerna run build:prod --scope @jupyterlite/app
lerna notice cli v4.0.0
lerna info versioning independent
lerna notice filter including "@jupyterlite/app"
lerna info filter [ '@jupyterlite/app' ]
lerna info Executing command in 1 package: "yarn run build:prod"
lerna ERR! yarn run build:prod exited 2 in '@jupyterlite/app'
lerna ERR! yarn run build:prod stdout:
$ yarn clean && yarn build --mode=production
$ rimraf build **/build
$ webpack --mode=production
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! yarn run build:prod stderr:
[webpack-cli] ←[31mHookWebpackError: ENOENT: no such file or directory, scandir
at makeWebpackError (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\webpack\lib\HookWebpackError.js:48:9)
at C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\webpack\lib\Compilation.js:3059:12
at eval (eval at create (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\tapable\lib\HookCodeFactory.js:33:10), :23:1)
at fn (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\webpack\lib\Compilation.js:480:17)
at _next9 (eval at create (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\tapable\lib\HookCodeFactory.js:33:10), :21:1)
at eval (eval at create (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\tapable\lib\HookCodeFactory.js:33:10), :37:1)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:95:5)
-- inner error --
Error: ENOENT: no such file or directory, scandir
at Object.readdirSync (fs.js:1047:3)
at CacheBackend.provideSync (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\enhanced-resolve\lib\CachedInputFileSystem.js:233:32)
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
at WebpackFileSystem.listPaths (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackFileSystem.js:78:24)
at LicenseTextReader.readLicense (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\LicenseTextReader.js:37:54)
at PluginChunkReadHandler.processModule (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:46:62)
at C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:22:23
at WebpackModuleFileIterator.internalCallback (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:25:13)
at WebpackModuleFileIterator.iterateFiles (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:10:9)
at C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:20:32
at WebpackChunkModuleIterator.iterateModules (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackChunkModuleIterator.js:42:21)
caused by plugins in Compilation.hooks.processAssets
Error: ENOENT: no such file or directory, scandir
at Object.readdirSync (fs.js:1047:3)
at CacheBackend.provideSync (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\enhanced-resolve\lib\CachedInputFileSystem.js:233:32)
at WebpackFileSystem.listPaths (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackFileSystem.js:78:24)
at LicenseTextReader.readLicense (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\LicenseTextReader.js:37:54)
at PluginChunkReadHandler.processModule (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:46:62)
at C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:22:23
at WebpackModuleFileIterator.internalCallback (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:25:13)
at WebpackModuleFileIterator.iterateFiles (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:10:9)
at C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:20:32
at WebpackChunkModuleIterator.iterateModules (C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackChunkModuleIterator.js:42:21)←[39m
error Command failed with exit code 2.
error Command failed with exit code 2.
lerna ERR! yarn run build:prod exited 2 in '@jupyterlite/app'
error Command failed with exit code 2.
TaskFailed - taskid:build:js:app
Command failed: '['C:\Users\DanieleMoroni\mambaforge\envs\jupyterlite-dev\Library\bin\yarn.cmd', 'lerna', 'run', 'build:prod', '--scope', '@jupyterlite/app']' returned 2

Expected behavior

No build error

Context

• JupyterLite version: v0.1.0b2, commit 65a47e6 from 14 Mar 2022
• Operating System and version: Windows 10 21H2
• Browser and version: n/a

Browser Output

Paste the output from your browser Javascript console here.

[bollwyvl]

Hm, debugging webpack on windows is among my least favorite things. #76 has been open for a long time, but we just haven't gotten much interest from folk in fixing it. The LicenseWebpackPlugin is in use upstream on JupyterLab, which does have the dev experience tested on windows, so while the problem is manifesting there, it might be something deeper.

. build:js:app

So your log starts at the build of app: presumably it did install and build everything before then? What happens if you run it again? A couple times? Just trying to determine if there's something wrong with the setup.

C:\Users\DanieleMoroni\Documents\dev\jupyterlite\node_modules\license-webpack-plugin\dist\WebpackFileSystem.js

This looks like it might be generating paths that are "too long" for windows when it is trying to scandir a path that is above the arbitrary threshold.

There are a bunch of registry hacks to work around the issue, enable symlinks, and all other kinds of useful things, but in 2022, I still recommend moving the repo to something short like c:\dev\jl. You may want to do the same with your mambaforge: reinstall to e.g. c:\mf. Sometimes those 40 characters make all difference.

It's also important to ensure that things like OneNote, DropBox, etc. are not involved, as these can play havoc with the filesystem.

[danielemoroni]

Thank you bollwyvl

The doit log is fine up until building app. I reinstalled and moved everything to C:\ so to get shorter filenames, and there is no DropBox etc involved. But I still get the same error, even after running multiple times

(jupyterlite-dev) C:\jl>doit
-- lint:version:js:lite
-- lint:version:js:pyodide
-- lint:black
-- lint:pyflakes
-- lint:schema:self
-- lint:schema:piplite
-- lint:schema:validate:app\jupyter-lite.json
-- lint:schema:validate:app\lab\jupyter-lite.json
-- lint:schema:validate:app\repl\jupyter-lite.json
-- lint:schema:validate:app\retro\jupyter-lite.json
-- lint:schema:validate:app\retro\consoles\jupyter-lite.json
-- lint:schema:validate:app\retro\edit\jupyter-lite.json
-- lint:schema:validate:app\retro\notebooks\jupyter-lite.json
-- lint:schema:validate:app\retro\tree\jupyter-lite.json
-- lint:schema:validate:app\jupyter-lite.ipynb
-- lint:schema:validate:app\lab\jupyter-lite.ipynb
-- lint:schema:validate:app\repl\jupyter-lite.ipynb
-- lint:schema:validate:app\retro\jupyter-lite.ipynb
-- lint:schema:validate:app\retro\consoles\jupyter-lite.ipynb
-- lint:schema:validate:app\retro\edit\jupyter-lite.ipynb
-- lint:schema:validate:app\retro\notebooks\jupyter-lite.ipynb
-- lint:schema:validate:app\retro\tree\jupyter-lite.ipynb
-- setup:js
-- docs:typedoc:ensure
-- env:binder
-- lint:prettier
-- lint:eslint
-- build:docs:favicon
-- build:js:ui-components
-- build:js:py:pyolite
-- build:js:py:piplite
-- build:js:py:ipykernel
-- build:js:py:widgetsnbextension
-- build:js:piplite:wheels
-- build:py:jupyterlite:pre:readme
-- build:js:lib
. build:js:app
yarn run v1.22.17
$ C:\jl\node_modules.bin\lerna run build:prod --scope @jupyterlite/app
lerna notice cli v4.0.0
lerna info versioning independent
lerna notice filter including "@jupyterlite/app"
lerna info filter [ '@jupyterlite/app' ]
lerna info Executing command in 1 package: "yarn run build:prod"
lerna ERR! yarn run build:prod exited 2 in '@jupyterlite/app'
lerna ERR! yarn run build:prod stdout:
$ yarn clean && yarn build --mode=production
$ rimraf build **/build
$ webpack --mode=production
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! yarn run build:prod stderr:
[webpack-cli] ←[31mHookWebpackError: ENOENT: no such file or directory, scandir
at makeWebpackError (C:\jl\node_modules\webpack\lib\HookWebpackError.js:48:9)
at C:\jl\node_modules\webpack\lib\Compilation.js:3059:12
at eval (eval at create (C:\jl\node_modules\tapable\lib\HookCodeFactory.js:33:10), :23:1)
at fn (C:\jl\node_modules\webpack\lib\Compilation.js:480:17)
at _next9 (eval at create (C:\jl\node_modules\tapable\lib\HookCodeFactory.js:33:10), :21:1)
at eval (eval at create (C:\jl\node_modules\tapable\lib\HookCodeFactory.js:33:10), :37:1)
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
at processTicksAndRejections (internal/process/task_queues.js:95:5)
-- inner error --
Error: ENOENT: no such file or directory, scandir
at Object.readdirSync (fs.js:1047:3)
at CacheBackend.provideSync (C:\jl\node_modules\enhanced-resolve\lib\CachedInputFileSystem.js:233:32)
at WebpackFileSystem.listPaths (C:\jl\node_modules\license-webpack-plugin\dist\WebpackFileSystem.js:78:24)
at LicenseTextReader.readLicense (C:\jl\node_modules\license-webpack-plugin\dist\LicenseTextReader.js:37:54)
at PluginChunkReadHandler.processModule (C:\jl\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:46:62)
at C:\jl\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:22:23
at WebpackModuleFileIterator.internalCallback (C:\jl\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:25:13)
at WebpackModuleFileIterator.iterateFiles (C:\jl\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:10:9)
at C:\jl\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:20:32
at WebpackChunkModuleIterator.iterateModules (C:\jl\node_modules\license-webpack-plugin\dist\WebpackChunkModuleIterator.js:42:21)
caused by plugins in Compilation.hooks.processAssets
Error: ENOENT: no such file or directory, scandir
at Object.readdirSync (fs.js:1047:3)
at CacheBackend.provideSync (C:\jl\node_modules\enhanced-resolve\lib\CachedInputFileSystem.js:233:32)
at WebpackFileSystem.listPaths (C:\jl\node_modules\license-webpack-plugin\dist\WebpackFileSystem.js:78:24)
at LicenseTextReader.readLicense (C:\jl\node_modules\license-webpack-plugin\dist\LicenseTextReader.js:37:54)
at PluginChunkReadHandler.processModule (C:\jl\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:46:62)
at C:\jl\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:22:23
at WebpackModuleFileIterator.internalCallback (C:\jl\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:25:13)
at WebpackModuleFileIterator.iterateFiles (C:\jl\node_modules\license-webpack-plugin\dist\WebpackModuleFileIterator.js:10:9)
at C:\jl\node_modules\license-webpack-plugin\dist\PluginChunkReadHandler.js:20:32
at WebpackChunkModuleIterator.iterateModules (C:\jl\node_modules\license-webpack-plugin\dist\WebpackChunkModuleIterator.js:42:21)←[39m
error Command failed with exit code 2.
error Command failed with exit code 2.
lerna ERR! yarn run build:prod exited 2 in '@jupyterlite/app'
error Command failed with exit code 2.
TaskFailed - taskid:build:js:app
Command failed: '['C:\mf\envs\jupyterlite-dev\Library\bin\yarn.cmd', 'lerna', 'run', 'build:prod', '--scope', '@jupyterlite/app']' returned 2

[bollwyvl]

It looks like this might in fact be a problem upstream: xz64/license-webpack-plugin#111

In the short term: we could skip adding that plugin on windows, knowing that it would not create the build/third-party-licenses.json file. This is fine, as our release-oriented python builds, which are tested on windows, would continue to be built, once, on linux... and extensions would still report some licenses, likely.

Longer term: a PR for a CI job, running parallel to the "for real" build job, that just tries to do some basic dev stuff on windows: there's a good deal of the full build that hasn't been tested on windows at all, so we'd likely want to try the basics of e.g. doit lint build.

[bollwyvl]

We now have it working under CI on windows, skipping the license plugin: please pull again and give this a shot, and open a new issue for whatever else you might encounter: this is preferable to changing the issue name, which has strong SEO energy.

[danielemoroni]

Thank you, that worked!

I now get some errors at the env:lite stage, with a blocking error at env:lite:extensions, but that stage is after the build ones, so I can now spawn a working jupyterlite server. I'll open a separate non-urgent ticket for those errors.