Mariatta , a Canadian developer, asked on python -m pip on Twitter, asking her to talk about this idiom and explain how it works.
I recently learned that you need to write python -m pip instead of the usual pip install, but now I can’t remember who I heard from. Probably from @brettsky or @zooba . Do any of you have a blog post so I can share it with readers?
- Mariatta (@mariatta
) October 29, 2019 ( https://twitter.com/mariatta/status/1189243515739561985?ref_src=twsrc%5Etfw )
I'm not sure what I told Mariatta about python -m pip , but there is every chance that it was me, since I asked that this instruction for installing packages using PyPI be written this way since 2016. So, this article should explain what python -m pip is and why you should use it when starting pip .
What is python -m pip?
To get started, python -m pip executes pip using the version of Python you specified for the python statement. Thus,
/usr/bin/python3.7 -m pip
means that you execute pip for the interpreter located in
/usr/bin/python3.7
. You can read the documentation about the
-m
flag if you don’t know how it works (by the way, it is extremely useful).
Why use python -m pip instead of pip / pip3?
You can say: “Okay, but why can't I just use pip by running the pip command?” The answer is: “Yes, but you will have less control over it.” I will explain what “control less” means by example.
Suppose I have two versions of Python installed, for example, Python 3.7 and 3.8 (this is very common among people who work on Mac OS or Linux, not to mention the fact that you probably wanted to play with Python 3.8, and you already had Python 3.7). So, if you type pip in the terminal, for which Python interpreter will you install the package?
Without more detailed information you will not know the answer. First you will need to understand what lies in PATH, that is,
/usr/bin
goes first or
/usr/local/bin
(which are the most common places to install Python, by the way usually
/usr/local/
goes first). So, you remember where you installed Python 3.7 and 3.8 and that they were different directories, and you will know what came in PATH first. Suppose you installed both manually, perhaps Python 3.7.3 was already preinstalled on your system, and you installed Python 3.7.5. In this case, both versions of Python are installed in
/usr/local/bin
. Can you tell me now what pip is now attached to?
You do not know the answer. If you do not know when each version was installed, and you understand that the latest version of pip was written to
/usr/local/bin/pip
, but you do not know which interpreter will be used for the pip command. Now you can say: “I always install the latest versions, so that means that Python 3.8.0 will be installed last, because it is newer than, say, 3.7.5.” Well, but what happens when Python 3.7 comes out .6? Your pip would no longer be used from Python 3.8, but from Python 3.7.
When you use python -m pip with the specific python interpreter you need, all the uncertainty disappears. If I write python3.8 -m pip , I know exactly which pip will be used and that the package will be installed for Python 3.8 (the same thing would happen if I pointed out python3.7).
If you use Windows, then you have an additional incentive to use python -m pip , since it allows pip to update itself. Mostly because pip.exe is considered running when you write pip install --upgrade pip . At this point, Windows will not allow you to reinstall pip.exe . However, if you do python-m pip install --upgrade pip , you work around this problem as python.exe is launched, not pip.exe .
And what happens when I am in an activated environment?
Usually, when I explain the essence of this article to people, there is always someone who will say: "I always use the virtual environment, and this does not apply to me." Well, for starters, it would ALWAYS be good to use a virtual environment! (I will explain why I think so in one of my next articles!) But to be honest, I would still insist on using python -m pip , even if, strictly speaking, this is not necessary.
Firstly, if you use Windows, you will still want to use python-m pip so that you can update pip in your environment.
Secondly, even if you use a different operating system, I would say that you still need to use python-m pip , since it will work regardless of the situation. He will warn you about a mistake if you forget to activate the environment, and any person who watches you will adopt the best practices. And personally, I do not think that saving 10 keystrokes is a significant price for not using good practice. And this command will also help you prevent errors when writing automation scripts that will perform obviously incorrect operations if you forget to activate the environment.
Personally, when I use any tool whose work depends on which interpreter it starts up, I always use
-m
, regardless of whether the virtual environment is activated or not. It's always important for me to understand which Python interpreter I'm using.
ALWAYS use the environment! Do not put everything in a row in the global interpreter!
When we talk about how to avoid confusion when installing in Python, I want to emphasize that we don’t have to install anything in the global Python interpreter when we work locally (containers are another matter entirely)! If this is the pre-installed Python of your system, then if you install some incompatible version of the library that your OS relies on, then you will actually break the system.
But even if you install a copy of python separately for you, I still strongly recommend not to put it directly in local development. Ultimately, in your projects, you will use various packages that may conflict with each other, and you will not have a clear idea of the dependencies within your projects. It is much better to use environments to isolate individual projects and tools for them from each other. The Python community uses two types of environments: virtual environments and conda environments. There is even a special way to install Python tools in isolation.
If you need to install the tool
For a stand- alone installation of the tool, I can recommend using pipx . Each tool will receive its own virtual environment, so as not to conflict with others. Thus, if you want to have only one installation, for example, Black , you can work without accidentally breaking your only installation of mypy .
If you need an environment for the project (and you do not use conda)
When you need to create an environment for a project, I personally always turn to venv and virtual environments. It is included in stdlib Python, so it is always available using python-m venv (unless, of course, you are using Debian or Ubuntu, in which case you may need to install the python3-venv apt package). A bit of history: I actually deleted the old pyvenv command that Python installed to create virtual environments using venv , for the same reasons why you need to use python -m pip instead of pip . That is, it is not clear for which interpreter you created the virtual environment using the old pyvenv command . And remember that you do not need to activate the environment in order to use the interpreter contained in it, because
.venv/bin/python
works as well as activating the environment and entering the python command.
Today, some developers still prefer virtualenv , as it is available in Python 2 and has some additional features. Personally, I have little interest in additional features, and having integrated venv means that I don't need to use pipx to install virtualenv on every machine. But if venv doesn't suit your needs and you want a virtual environment, then see if virtualenv offers what you need.
If you use conda
If you use conda , you can use conda environments to get the same effect that virtual environments provided by venv can offer. I'm not going to go into whether you need to use conda or venv in your specific situation, but if you use conda , then you know that you can (and should) create conda environments for your work, instead of installing everything in your own system installation. So you can get a clear understanding of what dependencies your project has (and this is a good reason to use miniconda instead of a full-fledged anaconda , since the former is less than a tenth of the volume of the latter).
There are always containers
Working in a container is a way not to deal with the environment at all, since your entire “machine” will become a separate environment. Until you have installed Python in the container system, you should be able to safely make a global installation so that your container remains simple and straightforward.
I repeat that you really understand the essence ...
Do not install anything in your global Python interpreter! Always try to use the environment for local development!
I can no longer say how many times I had to help someone who thought that pip installed in one Python interpreter, but actually installed in another. And this immeasurable amount also applies to those times when people broke the entire system or wondered why they could not install something that contradicted some other thing that they had previously set for another project, etc. due to the fact that they did not bother to configure the environment on their local machine.
Therefore, so that you and I can sleep peacefully, use python-m pip and try to always use the environment.