Have Mu use its own virtual environment

[tjguk]

This is a retroactive issue for the work being done on the use-venv branch

[tjguk]

I've merged PRs #1058 and #1056 to get the windows installer running ok

[tjguk]

Running through a virtualenv on Windows works fine on a development box. When an installed is built and installed, the packaged Python can't find the built-in venv module and so can't create the venv.

[tjguk]

Assuming that the approach of python -mvenv will play poorly with an installed setup, I've switched to importing the module and using a module-level convenience function to create the virtual environment

[tjguk]

Sigh .. and now we can't import venv. (After adding enough logging to get us an error during the initial imports). I'll have to start digging into pynsist to understand how it gathers its modules

[tjguk]

Hmmm... and I can see that there's much happening in the Mu.launch.pyw part of the installer which results in a log file in -- uniquely -- %APPDATA% (ie not %LOCALAPPDATA%) so I missed it.

[tjguk]

... which comes from https://github.com/takluyver/pynsist/blob/master/nsist/__init__.py#L222-L257

[tjguk]

Looks like venv isn't part of the Python embeddable zip. We'll have to pick it up elsewhere

[tjguk]

So I added virtualenv as an explicit install dependency then attempted to use its cli_run entry point to create the venv. That now fails on on the installed version with "ModuleNotFoundError: No module named importlib_metadata".

I've got an idea that importlib_metadata is new in 3.8 (or 3.7) and we're shipping with an embedded 3.6

[tjguk]

(from virtualenv.base)

if sys.version_info >= (3, 8):
    from importlib.metadata import entry_points
else:
    from importlib_metadata import entry_points

So we'll have to include importlib_metadata as a dependency and/or consider upgrading to an embeddable 3.8. The problem there is that black has only published a wheel for 3.6 with a source dist for everything else.

[tjguk]

By the way I apologise somewhat for spamming everyone watching this, but I'm trying to keep track of what's happening and I assume you're all capable of muteing this particular issue. Thank you for your patience... :)

[tjguk]

We've now got to the point where it's trying to create the venv, But the virtualenv code is complaining about not finding the DLLs directory

Mu.launch.pyw.log

[ntoll]

This is epic work Tim. Thank you so much for your efforts. 🥇

[tjguk]

Just a bit of a pick-up as I've been fighting the infamous win32api ModuleNotFoundError when attempting to open the Jupyter REPL. NB my work is all very similar to the far more extensive analysis which @tmontes has done over on beeware/briefcase#381

I've just come to a particular realisation, though, and I'm noting it here so I don't lose track of it later: we're launching Mu via the Mu.launch.pyw script which is generated by pynsist. When we pip install pynsist into the temporary virtualenv which we use for packaging, we're pinning it to version 2.3. (I don't know if this was to avoid a specific issue or just following a general approach to pinning...). But the SCRIPT_TEMPLATE which generates that launcher has changed between 2.3 and the current 2.5. Specifically, it now adds this pair of lines:

# Ensure .pth files in pkgdir are handled properly
site.addsitedir(pkgdir)

which is what the pywin32 tools need in order to invoke their .pth file and add their subdirectories to sys.path. At least, that's my hypothesis which I'm about to test...

[tjguk]

Well that "works" in that it gets us over the win32api issue... but now the REPL is giving the classic:

Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

and restarting infinitely.

And the REPL isn't working in dev / venv mode.

Time for Round 2...

[tjguk]

To avoid confusion I'm closing this in favour of #1070