Repository navigation
Unexpected --user behavior since version 23.1 #11982
Description
Description
Hey everyone,
I'm using pip typically in virtual environments (managed by PyCharm). Since version 23.1 I get an error message when installing packages.
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
A little research told me, that a --user flag within a venv does not really make sense. However, it seems that some kind of default --user is set since 23.1.
A workaround I found is to add the --no-user flag.
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy --no-user
Collecting numpy
Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2
For completeness, downgrading to 23.0.1 with the --no-user flag gives the me the behavior I'm used to.
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>python.exe -m pip install --upgrade pip==23.0.1 --no-user
Collecting pip==23.0.1
Using cached pip-23.0.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 23.1
Uninstalling pip-23.1:
Successfully uninstalled pip-23.1
Successfully installed pip-23.0.1
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
Collecting numpy
Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2
[notice] A new release of pip is available: 23.0.1 -> 23.1
[notice] To update, run: python.exe -m pip install --upgrade pip
Not sure if this is a bug or a feature. Maybe I'm also missing something very obvious. But at least it is documented now for anyone experiencing the same issue :)
Expected behavior
No response
pip version
23.1
Python version
3.9
OS
Windows 10
How to Reproduce
create a venv with pip 23.1
try to install a package with pip install <any-package-name>
Output
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
Code of Conduct
- I agree to follow the PSF Code of Conduct.
Activity
pfmoore commentedon Apr 19, 2023
Pip 23.1 does not have a default to
--user. However it's possible you are picking up a user config of some sort. Please can you provide the output ofpip config debugin both the failing and the successful cases?flowmeadow commentedon Apr 20, 2023
Sure, thanks in advance.
Pip 23.0.1 (successful case):
Pip 23.1 (Failing case):
pfmoore commentedon Apr 20, 2023
So that's your issue. In the failing case, you have a
baseconfig that setsuserto true. This looks like the Windows Store installation of Python. I'm not sure why they set a pip config (I think it's a workaround for a permission issue with Windows Store apps) but IMO settinguserglobally like this is a bad idea. But it's what we have to deal with, unfortunately.This was caused by
That issue was extensively discussed, but it appears that the case of the Windows Store distribution wasn't known (or wasn't considered). IMO this is a sufficiently serious problem that it requires a bugfix. I'm not sure it's possible to simply revert the change (reverting a merge commit is a bit beyond my git skills!) so we'll need someone to work on a fix ASAP. @pelson as the author of the original change, are you able to look at this? Alternatively, if anyone can tell me the right git incantation to revert #11487, I can do that and we can revisit the feature for a future pip version.
Thanks for reporting this @flowmeadow!
pfmoore commentedon Apr 20, 2023
As a quick proof of concept, I've created #11983 to patch out the BASE config. This might form the basis of a minimal fix if we can't find a better solution.
pfmoore commentedon Apr 20, 2023
I also opened python/cpython#103646 to discuss with CPython whether there's a better way than a pre-installed
pip.inito achieve what they are trying to do with the Store installation.flowmeadow commentedon Apr 20, 2023
Thanks a lot for the analysis. I can confirm that its the Windows Store installation. I remember having a very bad feeling using it. Well, at least this was the first issue that arose in almost two years.
pfmoore commentedon Apr 20, 2023
Generally, I think the Store install should be fine. There's some quirks like this, but they mostly seem to work. And having people use the distribution and report issues like you have is the only way we'll know what needs improving, so your help is appreciated.
abhinavsingh commentedon Apr 20, 2023
jFYI, this is also impacting
proxy.pyworkflows e.g. https://github.com/abhinavsingh/proxy.py/actions/runs/4717368971/jobs/8450593742pfmoore commentedon Apr 20, 2023
@abhinavsingh that's nothing to do with this issue. That's the deprecation of
pkg_resources, which is #11975 (and which is now fixed in main, and awaiting a bugfix release which will come once we've established what fixes we need to include).pfmoore commentedon Apr 20, 2023
By the way @flowmeadow, as a temporary workaround, you should be able to set the
useroption tofalsein a config file that's higher priority than the "base" one (or just set$PIP_USER=0in your environment).notatallshaw commentedon Apr 20, 2023
My understanding is that this is the model of Windows Store installs, the base install is in a global space that any specific user may not have permissions to modify, therefore the application must store user specific data in a user space.
But unlike Linux distros which manage the whole Python pipeline for the system install such as installing Python packages like Pandas using their own distribution method outside Pip there is no such infrastructure on the Windows Store and users are expected to use Pip install.
Maybe a Windows specific exception will need to be made here as it's model of users is not a 1-1 mapping to Unix-like model of users and root anyway.
I'm going to ping @zooba just in case he wants to comment here (and maybe correct any misunderstandings).
I do wishfully also wonder if the default behavior of installing Python on Windows could in the future switch from the the Windows Store to
wingetand behave more like Linux distros? But I'm throwing that idea out with no particular understanding of the problems, so it may be a faulty idea.zooba commentedon Apr 20, 2023
Show me a Linux distro that downloads our installer and runs it with unreliable command line options and I'll consider it (spoiler: no I won't 😉
wingetis not a good way to manage Python installs, I'm afraid).I've put plenty of explanation on the CPython bug already. My fundamental disagreement is with configurations located in
sys.base_prefixbeing used whensys.base_prefix != sys.prefix, as it makes it impossible to use asys.prefixconfiguration in something that may be used as a base environment.Perhaps
sys.base_prefixconfiguration files should use a different filename, or different section names or something. Or some extra setting to say "please consider the settings in this file for venvs created from this environment".I can imagine useful scenarios for files in both
sys.base_prefixand alsosys.prefix when ==sys.base_prefix, but they all assume the distributor knows what you should use by default (e.g. platform name, index-url, etc.). If pip wants to say that distributors don't know the right install locations, that's fine, we distributors will just patch source files instead of adding overridable config settings.pfmoore commentedon Apr 20, 2023
To be clear, there's no such thing as a "
sys.prefixconfiguration" as you describe it. If you look in the documentation, the location{sys.prefix}/pip.iniisn't mentioned as a source of config data, although in the code, it's used as the location of the "site" data when you're not in a virtual environment. I think that's probably just a problem with the docs, though, and what you mean here is the "site" location.I do agree that it's wrong to have the "base" config come from the same place as the "site" config of the system Python, precisely because as you say it means that the system site config "leaks" into a virtual environment. I think that's a flaw in the design of #9752 which we should fix1.
As a general point, I personally don't consider pip's configuration files as intended for distributors - they are for users and site admins to manage. Distributors shouldn't be using them, and very definitely shouldn't be relying on what's in them. If distributors need configuration options, they should discuss that with the pip maintainers and we can see how best to address such a need. But that's just my personal view - I'd be interested if the other @pypa/pip-committers agree with me on that.
I think that's an interesting use case that we should discuss further. It may be a reasonable use for certain types of config. One key difference if a distributor patches source files is that the support burden is squarely on the distributor in that case. One problem with diagnosing this issue was that it took a while to realise that the offending user config wasn't actually set by the user, and the user didn't know about it.
Footnotes
As a minor point, I'm not keen on having a different basename for the file - knowing that pip's config file is always called
pip.iniis a useful consistency for people trying to work out what's going on. ↩omarirfa commentedon Apr 20, 2023
Same issue here, for now I just reverted back to pip 21.3 and everything worked again. Although, in my case both pip 23.0.1 and pip 23.1 are failing with the same error
omarirfa commentedon Apr 20, 2023
For solution 1 how do u find a file that is higher priority than the base file? I was wondering if its one of these files I will have to change?
If it is one of the user or global files, I have tried adding this to the pip.ini, but it did not work:
For solution 2, I tried using
$PIP_USER=0while in the venv it gives an error saying it is not a recognized internal or external command. Is there a specific way to set or run this?pradyunsg commentedon Apr 21, 2023
@pfmoore How do you feel about reverting and reopening #9752, and we can come back around to re-adding that in a follow up change for 23.2?
pfmoore commentedon Apr 21, 2023
@pradyunsg As I said above I'd be happy to do that but I don't know how (in git) to revert a merge commit. If you can confirm how I do that I'd be far happier with that approach.
On the longer term resolution of #9752, I'd point out that this problem has reinforced my dislike of having the "base" environment leak into virtual environments like this, so I'd actually oppose #9752 in its stated form. I'd like a much better justification of why the global setting isn't sufficient for the original use case, and assuming the change can be justified, I'd want the "base" config to be very explicitly not the same as the "site" config of the base environment. So I think there's more than just "re-adding" the feature involved here, more like "re-evaluating" and "re-designing" it in a way that accounts for the way the Windows Store distribution uses the existing config mechanism.
pradyunsg commentedon Apr 21, 2023
Whoop, I missed that while skimming the issue! Sorry about that!
git revert -m 1 <hash>(at PyCon US, so my replies might be slow!)
pfmoore commentedon Apr 21, 2023
Ah! I tried that, but was confused that there was no reversion of the news entry. But I guess that makes sense, as it was merged into the changelog when I did the release. Thanks for confirming.
#11987
layday commentedon Apr 22, 2023
@pfmoore FYI, you reopened #9752, but it's still locked for comments - not sure if that was intentional.
pfmoore commentedon Apr 22, 2023
Thanks for the heads up - I'll fix that!