xontrib unable to load modules installed with xpip on fedora 30

[skewballfox]

xonfig

Issues with modules being loaded by xontrib via xpip

I suspect this is because fedora separates packages installed via sudo pip from system packages, and also has commands use system python
changing pythonpath to either the $HOME/.local/lib/python3.7 or the /usr/local/lib/python3.7 doesn't seem to aleviate the issue (noteworthy was before assignment $PYTHONPATH didn't exist)

I've managed to symlink the smaller files but modules like schedule or tab completers don't seem to cooperate with this

$ xonfig

Expected Behavior

modules install via xpip work without symlinking or have some way to specify the install path of xontribs

Current Behavior

third party modules not included in the install will not work out of the box

Steps to Reproduce

on fedora 30

xpip install module
xontrib load module

I believe this is isolated to fedora given it's likely related to how they seperate packages installed via sudo pip from those installed by sudo dnf install python3-... and provide a separate python command for system packages, perhaps wrapping all the extra modules into either one or a series of copr's would work, or just allowing some way to specify the path that xontrib searches allowing for xontribs to be installed via pip install --user.

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[gforsyth]

Thanks for reporting @skewballfox -- I don't have a Fedora install handy, but xpip is just an alias to the version of pip associated with your xonsh install, so xpip install --user is a valid command and should work.

I'm not sure if we can mutate the paths that xontribs are located in since it's an implicit namespace package, but it's probably worth looking in to.

[AstraLuma]

Not just an alias, it also adds sudo if it thinks its needed.

Considering that it's trying to cover conda, venv/pipx, system installs, whatever homebrew does, pyenv, and who knows what else, I'm not surprised there's some oddness going on.

If there was a reliable way of detecting xonsh is running under the "system" python environment, adding --user under that circumstance would make sense.

[AstraLuma]

I think the "system" environment would be:

• Shipped as part of the OS (Apple's Python)
• Installed by package manager (Linux package, homebrew general python packages)
• Windows global install

What doesn't count:

• virtual environment (including pipx, i believe)
• conda
• pyenv (maybe? They're meant to be general global pythons)

Special considerations:

• zipapp (only one known is my special packaging, which does special things to enable a user-writable site)

[evelineraine]

I have the same issue on Fedora 31.

The reason is that sys.path is not initialized with site-specific paths, in particular /usr/local/lib/python3.7/site-packages, which is where pip install install packages by default now.

And it's not initialized due to sys.flags.no_user_site being set to 1 in Xonsh. It's default value is 0 in a normal python shell, which is why all paths are set automagically.
It's read by site python module, which is implicitly imported on every Python interpreter start. This module does or doesn't populate sys.path with additional directories (/usr/local, ~/.local/, etc.) depending on the value of sys.flags.no_user_site flag.

This flag is read-only in Python, so it seems these are the only 2 ways to change it's value:

1. Run Python interpreter as python -s
2. Set PYTHONNOUSERSITE environmental variable

Could someone suggest why it's set to 1 in Xonsh?

[evelineraine]

(Update)

The -s flag is set downstream - in Fedora RPM Spec.

In xonsh.spec you can see the line:

pathfix.py -i "%{__python3} %{py3_shbang_opts}u" -p -n %{buildroot}%{_bindir}/xonsh

And %{py3_shbang_opts} evaluates to -s, causing this issue.

The same issue has been dealt with by Ansible ansible/ansible#62565.

The details of the package and PoC can be found in packages/xonsh.

Would you like me to propose the same fix to the package maintainer as Ansible did for it's Spec?

[scopatz]

Yes please do!

[swedneck]

this seems to still be an issue on fedora 32

[swedneck]

for those who see this and just want a solution, add this to your config file before any xontribs are loaded:

import sys
sys.path = sys.path + ['/usr/local/lib/python3.8/site-packages']

[rickysarraf]

I think I Have the same problem on Debian.

First, I don't like the idea of embedding sudo in calls to xpip, when the user did not ask. I'm not sure why that's the default.

rrs@priyasi /tmp [16]$ xpip                                                                                                                       
[sudo] password for rrs: 

Usage:   
  /usr/bin/python3 -m pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
.....snipped......

Then, I install it with the --user option.

rrs@priyasi /usr/share/doc/xonsh [2]$ xpip install --user xontrib-prompt-bar xontrib-prompt-vi-mode                                               
Collecting xontrib-prompt-bar
  Downloading xontrib_prompt_bar-0.1.0-py3-none-any.whl (4.4 kB)
Collecting xontrib-prompt-vi-mode
  Downloading xontrib-prompt-vi-mode-0.1.3.tar.gz (1.5 kB)
Building wheels for collected packages: xontrib-prompt-vi-mode
  Building wheel for xontrib-prompt-vi-mode (setup.py) ... done
  Created wheel for xontrib-prompt-vi-mode: filename=xontrib_prompt_vi_mode-0.1.3-py3-none-any.whl size=1787 sha256=b61b09b6a344c828acfa485393839c85fd824fb6d953d7d114ee01178d970d45
  Stored in directory: /root/.cache/pip/wheels/f1/95/f2/49d66e3e72455db63472de3e902df341a0ab1371722d4c9f80
Successfully built xontrib-prompt-vi-mode
Installing collected packages: xontrib-prompt-bar, xontrib-prompt-vi-mode
Successfully installed xontrib-prompt-bar-0.1.0 xontrib-prompt-vi-mode-0.1.3

I have no clue why that went under /root when it should actually be under my user home directory.

And after that, I still run into the same problem

rrs@priyasi /usr/share/doc/xonsh $ xontrib prompt_bar prompt_ret_code prompt_vi_mode                                                              
The following xontribs are enabled but not installed: 
   prompt_bar prompt_vi_mode
To install them run 
    xpip install xontrib-prompt-bar xontrib-prompt-vi-mode

I want to install these modules but only as a user.

rrs@priyasi /usr/share/doc/xonsh $ xpip install xontrib-prompt-bar xontrib-prompt-vi-mode                                                         
Requirement already satisfied: xontrib-prompt-bar in /root/.local/lib/python3.8/site-packages (0.1.0)
Requirement already satisfied: xontrib-prompt-vi-mode in /root/.local/lib/python3.8/site-packages (0.1.3)
rrs@priyasi /usr/share/doc/xonsh $ xontrib prompt_bar prompt_ret_code prompt_vi_mode                                                              
The following xontribs are enabled but not installed: 
   prompt_bar prompt_vi_mode
To install them run 
    xpip install xontrib-prompt-bar xontrib-prompt-vi-mode

[scopatz]

I think this might be related to #3636

[anki-code]

I just put this here for the future tests:

docker run --rm -it fedora /bin/bash \
    -c "yum install -y pip && pip install xonsh && xonsh -c 'xpip install xontrib-hist-format && xontrib load hist_format && hist-format --help'"

[anki-code]

I can't repeat this issue now.