GhostInstall issue with pyenv managed Python providers?

[shawnohare]

For some context, I use pyenv to manage all Python versions and virtual environments, since I have to develop in a variety of Python contexts. Specifically, I follow the instructions at https://neovim.io/doc/user/provider.html for installing the neovim python provider in a pyenv managed virtualenv.

Running :GhostInstall yields the following error:

ghost: installing dependencies from /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt
ghost: Collecting git+https://github.com/dpallot/simple-websocket-server.git (from -r /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt (line 1))
ghost:   Cloning https://github.com/dpallot/simple-websocket-server.git to /private/var/folders/rv/ffd1sx152t99m4mhmgtdqjgr0000gp/T/pip-req-build-ic1__kmx
ghost:   Running command git clone -q https://github.com/dpallot/simple-websocket-server.git /private/var/folders/rv/ffd1sx152t99m4mhmgtdqjgr0000gp/T/pip-req-build-ic1__kmx
ghost: Collecting python-slugify (from -r /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt (line 2))
ghost:   Using cached https://files.pythonhosted.org/packages/f5/ef/c868a9ac657405f051a8a501ac5633e769c54228716b8db7f8d717977e57/python-slugify-3.0.4.tar.gz
ghost: Collecting text-unidecode>=1.3 (from python-slugify->-r /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt (line 2))
ghost:   Using cached https://files.pythonhosted.org/packages/a6/a5/c0b6468d3824fe3fde30dbb5e1f687b291608f9473681bbf7dabbf5a87d7/text_unidecode-1.3-py2.py3-none-any.whl
ghost: Installing collected packages: text-unidecode, python-slugify, SimpleWebSocketServer
ghost:   Running setup.py install for python-slugify: started
ghost:     Running setup.py install for python-slugify: finished with status 'done'
ghost:   Running setup.py install for SimpleWebSocketServer: started
ghost:     Running setup.py install for SimpleWebSocketServer: finished with status 'done'
ghost: Successfully installed SimpleWebSocketServer-0.1.1 python-slugify-3.0.4 text-unidecode-1.3
ghost:
ghost: dependencies installed successfully
Encountered ModuleNotFoundError loading plugin at /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/rplugin/python3/ghost.py: No module named 'SimpleWebSocketServer'
Traceback (most recent call last):
  File "/Users/shawn.ohare/.local/opt/pyenv/versions/neovim3/lib/python3.7/site-packages/pynvim/plugin/host.py", line 145, in _load
    module = imp.load_module(name, file, pathname, descr)
  File "/Users/shawn.ohare/.local/opt/pyenv/versions/3.7.3/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/Users/shawn.ohare/.local/opt/pyenv/versions/3.7.3/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
ModuleNotFoundError: No module named 'SimpleWebSocketServer'
remote/host: python3 host registered plugins ['far']
remote/host: generated rplugin manifest: /Users/shawn.ohare/.local/share/nvim/rplugin.vim
:UpdateRemotePlugins executed. Please restart nvim
Press ENTER or type command to continue

It looks like when :GhostInstall is run, it installed its dependencies in my global python version (installed via pyenv) since the --user flag is used. For example:

> ~/opt/pyenv/versions/3.7.3/bin/pip list
Package        Version
-------------- -------
pip            19.0.3
python-slugify 3.0.4
setuptools     40.8.0
text-unidecode 1.3

If instead I install the vim-ghost python dependencies manually inside the neovim python 3 provider virtualenv via,

> ~/opt/pyenv/versions/neovim3/bin/pip install python-slugify SimpleWebSocketServer

then :GhostInstall in neovim appears to complete successfully:

ghost: installing dependencies from /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt
ghost: Collecting git+https://github.com/dpallot/simple-websocket-server.git (from -r /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt (line 1))
ghost:   Cloning https://github.com/dpallot/simple-websocket-server.git to /private/var/folders/rv/ffd1sx152t99m4mhmgtdqjgr0000gp/T/pip-req-build-sg_4aeqh
ghost:   Running command git clone -q https://github.com/dpallot/simple-websocket-server.git /private/var/folders/rv/ffd1sx152t99m4mhmgtdqjgr0000gp/T/pip-req-build-sg_4aeqh
ghost: Requirement already satisfied (use --upgrade to upgrade): SimpleWebSocketServer==0.1.1 from git+https://github.com/dpallot/simple-websocket-server.git in /Users/shawn.ohare/.local/opt/pyenv/versions/3.7.3/envs/neovim3/lib/python3.7/site-packages (from -r /Users/shawn.ohare/.loca
l/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt (line 1))
ghost: Requirement already satisfied: python-slugify in /Users/shawn.ohare/.local/opt/pyenv/versions/3.7.3/envs/neovim3/lib/python3.7/site-packages (from -r /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt (line 2)) (3.0.4)
ghost: Requirement already satisfied: text-unidecode>=1.3 in /Users/shawn.ohare/.local/opt/pyenv/versions/3.7.3/envs/neovim3/lib/python3.7/site-packages (from python-slugify->-r /Users/shawn.ohare/.local/share/nvim/site/pack/minpac/start/vim-ghost/requirements_posix.txt (line 2)) (1.3)

ghost:
ghost: dependencies installed successfully
remote/host: python3 host registered plugins ['far', 'ghost.py']
remote/host: generated rplugin manifest: /Users/shawn.ohare/.local/share/nvim/rplugin.vim
:UpdateRemotePlugins executed. Please restart nvim

Is there a way for the vim-ghost python dependency installer to detect whether a virtualenv is being used for the neovim python provider and install its dependencies in that? I'm not familiar enough with plugins that use python to know where they generally should be installing their dependencies. (Like with pip installed system executables though, it seems the only sane thing is to use a separate virtualenv for each plugin to avoid dependency issues.)

I'm not entirely sure if this is a misconfiguration on my part, but I run a fairly mundane pyenv setup with the exception of setting PYENV_ROOT.

[raghur]

I'm not very savvy with pyenv (or even virtualenv).. right now, the installer script picks up the python used by vim/nvim and does a python -m pip to install requirements. See

vim-ghost/python3/ghost_install.py

Line 17 in 156f393

py3 = sys.executable

One possibility I can think of is that if the path to the python interpreter isn't a standard path (only on linux/mac), then assume it's a venv and don't use the --user arg... However, that just seems flaky and asking for trouble in general :)

[shawnohare]

Looking at this a bit more, I think the underlying issue is that user site packages (python -m site --user-site) is not included in sys.path for virtual environments (at least those created by the builtin venv module).

Given the above, it might make sense to check whether the user site packages are in the python path:

if site.USER_SITE not in sys.path:
    # Do not use --user flag with pip
else:
   # Install deps to user site

However, the predicate above is also satisfied by some system pythons, so perhaps just a note in the documentation indicating that if one is using a virtual env for the vim/nvim Python provider, they should install the vim-ghost deps manually?