Install

The best tool that can be used to create virtual environments is the venv module, which is part of the standard library since Python 3.3.

venv is built into Python, and most users don’t need to install anything. However, Debian/Ubuntu users will need to run sudo apt-get install python3-venv to make it work (due to Debian not installing some components that venv needs by default). [1]

The alternative (and original, and previously standard) virtual environment tool is virtualenv. It works with Python 2.7, and has a couple extra fetures (that you generally won’t need). virtualenv can be installed with your system package manager, or pip install --user virtualenv.

Which one to use? Probably venv. Both tools achieve the same goal in similar ways. And if one of them does not work, you can try the other and it might just work better.

(Terminology note: most of the time, the names of both tools are used interchargeably, “venv” was often used as an abbreviation for “virtualenv” before the stdlib tool was created)

Create

To create a virtual environment named env, you need to run the venv tool with the Python you want to use in that environment.

Linux:   $ python3 -m venv env
Windows: > py -m venv env

or, if you’re using virtualenv:

$ python3 -m virtualenv env
> py -m virtualenv env

Afterwards, you will end up with a folder named env that contains folders named bin (Scripts on Windows — contains executables and scripts installed by packages, including python), lib (contains code), and include (contains C headers).

Both tools install pip and setuptools, but venv does not ship with wheel. In addition, the default versions tend to be more-or-less outdated. Let’s upgrade them real quick: [2]

$ env/bin/python -m pip install --upgrade pip setuptools wheel
> env\Scripts\python -m pip install --upgrade pip setuptools wheel

Use

There are three ways of working with virtual environments interactively (in a shell):

• activation (run source env/bin/activate on *nix; env\Scripts\activate on Windows) — it simplifies work and requires less typing, although it can sometimes fail to work properly. (After installing scripts, hash -r may be necessary on *nix to use them.)

• executing env/bin/python (env\Scripts\python) and other scripts directly, as activation only changes $PATH and some helper variables — those variables are not mandatory for operation, running the correct python is, and that method is failsafe.

• in subshells (IMO, it’s bad UX)

Whichever method you use, you must remember that without doing any of these things, you will still be working with the system Python.

For non-interactive work (eg. crontab entries, system services, etc.), activation and subshells are not viable solutions. In these cases, you must always use the full path to Python.

Here are some usage examples (paths can be relative, of course):

## *nix, activation ##
$ source /path/to/env/bin/activate
(env)$ pip install Django
(env)$ deactivate

## *nix, manual execution ##
$ /path/to/env/bin/pip install Django

## Windows, activation ##
> C:\path\to\env\Scripts\activate
(venv)> pip install Django
(venv)> deactivate

## Windows, manual execution ##
> C:\path\to\env\Scripts\pip install Django

## Windows, updating pip/setuptools/wheel ##
> C:\path\to\env\Scripts\python -m pip install -U pip setuptools wheel

The same principle applies to running Python itself, or any other script installed by a package. (With Django’s manage.py, calling it as ./manage.py requires activation, or you can run venv/bin/python manage.py.)

$ oldenv/bin/pip freeze > requirements.txt
$ python3 -m venv newenv
$ newenv/bin/pip install -r requirements.txt
(You may rm -rf oldenv now if you desire)

Frequently Asked Questions

Footnotes

Use the right tool for the job

By calling another process, you introduce a third-party dependency. That dependency isn’t controlled by your code, and your code becomes more fragile. The problems include:

• the program is not installed, or even available, for the user’s OS of choice

• the program is not in the $PATH your process gets

• the hard-coded path is not correct on the end user’s system

• the program is in a different version (eg. GNU vs. BSD, updates/patches), which means different option names or other behaviors

• the program’s output is not what you expected due to user config (including locale)

• error reporting is based on numeric exit codes, and the meaning of those differs between programs (if they have meaning besides 0/1 in the first place)

On the other hand, if your code uses a lot of subprocesses, perhaps you should stay with Bash. You can do the harder parts with Python, Ruby, or some other language by calling them from within your Bash script.

Don’t spawn subprocesses if there’s an alternative

Spawning a subprocess always incurs a (minor) [1] performance hit minor compared to the alternatives. With that in mind, and the resiliency issues listed above, you should always try to find an alternative for the external command.

The simplest ones are the basic Unix utilities. Replace grep, sed and awk with string operations and regular expressions. Filesystem utilities will have equivalents — for Python, in os or shutil. Your language of choice can also handle things like networking (don’t call curl), file compression, working with date/time…

Similarly, you should check if there are packages available that already do what you want — library bindings or re-implementations. And if there isn’t, perhaps you could help the world by writing one of those and sharing it?

One more important thing: if the program uses the same language as your code, then you should try to import the code and run it from the same process instead of spawning a process, if this is feasible.

Security considerations: shells, spaces, and command injection

We come to the most important part of this article: how to spawn subprocesses without compromising your system. When you spawn a subprocess on a typical Unix system, fork() is called, and your process is copied. Many modern Unix systems have a copy-on-write implementation of that syscall, meaning that the operation does not result in copying all the memory of the host process over. Forking is (almost) immediately followed by calling execve() (or a helper function from the exec family) [2] in the child process — that function transforms the calling process into a new process [3]. This technique is called fork-exec and is the typical way to spawn a new process on Unix. [4]

There are two ways to access this API, from the C perspective:

• directly, by calling fork() and exec*() (or posix_spawn()), and providing an array of arguments passed to the process, or

• through the shell (sh), usually by calling system(). As Linux’s manpage for system(3) puts it,

  The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:

  execl("/bin/sh", "sh", "-c", command, (char *) 0);
  

If you go through the shell, you pass one string argument, whereas exec*() demands you to specify arguments separately. Let’s write a sample program to print all the arguments it receives. I’ll do it in Python to get a more readable output.

#!/usr/bin/env python3
import sys
print(sys.argv)

Let’s see what appears:

$ ./argv.py foo bar
['./argv.py', 'foo', 'bar']
$ ./argv.py 'foo bar'
['./argv.py', 'foo bar']
$ ./argv.py foo\ bar baz
['./argv.py', 'foo bar', 'baz']

$ ./argv.py $(date)
['./argv.py', 'Sat', 'Sep', '2', '16:54:52', 'CEST', '2017']
$ ./argv.py "$(date)"
['./argv.py', 'Sat Sep  2 16:54:52 CEST 2017']

$ ./argv.py /usr/*
['./argv.py', '/usr/X11', '/usr/X11R6', '/usr/bin', '/usr/include', '/usr/lib', '/usr/libexec', '/usr/local', '/usr/sbin', '/usr/share', '/usr/standalone']
$ ./argv.py "/usr/*"
['./argv.py', '/usr/*']

$ ./argv.py $EDITOR
['./argv.py', 'nvim']

$ $PWD/argv.py foo bar
['/Users/kwpolska/Desktop/blog/subprocess/argv.py', 'foo', 'bar']
$ ./argv.py a{b,c}d
['./argv.py', 'abd', 'acd']

$ python argv.py foo bar | cat
['argv.py', 'foo', 'bar']
$ python argv.py foo bar > foo.txt
$ cat foo.txt
['argv.py', 'foo', 'bar']

$ ./argv.py foo; ls /usr
['./argv.py', 'foo']
X11@        X11R6@      bin/        include/    lib/        libexec/    local/      sbin/       share/      standalone/

As you can see, the following things are handled by the shell (the process is unaware of this occurring):

• quotes and escapes

• expanding expressions in braces

• expanding variables

• wildcards (glob, *)

• redirections and pipes (> >> |)

• command substitution (backticks or $(…))

• running multiple commands on the same line (; && || &)

The list is full of potential vulnerabilities. If end users are in control of the arguments passed, and you go through the shell, they can execute arbitrary commands or even get full shell access. Even in other cases, you’ll have to depend on the shell’s parsing, which introduces an unnecessary indirection.

TL;DR: How to do this properly in your language of choice

To ensure spawning subprocess is done securely, do not use the shell in between. If you need any of the operations I listed above as part of your command — wildcards, pipes, etc. — you will need to take care of them in your code; most languages have those features built-in.

In C (Unix)

Perform fork-exec by yourself, or use posix_spawn(). This also lets you communicate with the process if you open a pipe and make it stdout of the child process. Never use system().

In Python

Use the subprocess module. Always pass shell=False and give it a list of arguments. With asyncio, use asyncio.create_subprocess_exec (and not _shell), but note it takes *args and not a list. Never use os.system and os.popen.

In Ruby

Pass arrays to IO.popen. Pass multiple arguments to system() (system(["ls", "ls"]) or system("ls", "-l")). Never use %x{command} or backticks.

In Java

Pass arrays to Runtime.exec. Pass multiple arguments or list to ProcessBuilder.

In PHP

All the standard methods go through the shell. Try escapeshellcmd(), escapeshellarg() — or better, switch to Python. Or anything, really.

In Go

os/exec and os.StartProcess are safe.

In Node.js

Use child_process.execFile or child_process.spawn with shell set to false.

Elsewhere

You should be able to specify multiple strings (using variadic arguments, arrays, or otherwise standard data structures of your language of choice) as the command line. Otherwise, you might be running into something shell-related.

The part where I pretend I know something about Windows

On Windows, argument lists are always passed to processes as strings (Python joins them semi-intelligently if it gets a list). Redirections and variables work in shell mode, but globs (asterisks) are always left for the called process to handle.

Some useful functions are implemented as shell built-ins — in that case, you need to call it via the shell.

Internals: There is no fork() on Windows. Instead, CreateProcess(), ShellExecute(), or lower-level spawn*() functions are used. cmd.exe /c is called in shell calls.

Glossary and questions

Installing Python

This guide will focus on installing CPython 2.7 and 3.x (latest), using the standard distribution. This choice is satisfactory for most people. Third-party distributions, while handy in some cases, are not needed for most. (See Distributions for arguments)

Throughout this guide, I’ll refer to the Python interpreter executable as python. The exact name depends on your system and desired version. On most OSes, python is Python 2 and python3 is 3; python2 should also exist. On Arch Linux, python is Python 3. On Windows, use the py launcher.

Installing packages

To install third-party packages, you should use pip, the Python package manager. If you’re using Windows or macOS (from Homebrew), pip is included with your copy of Python. If you’re on Linux and installed Python from a system repository, install the correct system package (python-pip, python3-pip). If you compiled your own Python, pip is also included.

To run pip, use py -m pip (Windows), python -m pip (other platforms), or the short pip/pip3 commands.

NEVER use sudo pip. This can cause numerous problems:

• conflicts between packages installed by pip and your system package manager

• pip modifying system packages, leading to issues when updating them, or breaking dependencies

• no isolation between package versions, which is sometimes needed to satisfy dependencies

Note that a package install is specific to the Python interpreter used to run pip. Packages installed to a virtualenv are separate from system packages; packages installed for “global” Python 2.7 are separate from 3.6 packages. Virtual environments generally don’t use the system packages, unless specifically enabled during creation.

Some distros have popular packages in their repositories. Sometimes they’re good; in other cases they’re terribly outdated or they lack important components, making package managers angry and sick of supporting a 2-year-old version. (Especially since most bugs are closed with “we’ve fixed that long ago”)

Editors and IDEs

Another important thing a developer should take care of is the choice of an editor. This is an important decision, and is the reason for many holy wars in the programmer community.

A good editor should have syntax highlighting for all languages you need to work with. It should also have features like visual block/multiple selections, sophisticated find-and-replace, file finding, code completion, and many more minor but helpful features.

Then there’s the difference between IDEs and text editors. Text editors are simpler, whereas IDEs try to include many extra things not necessarily related to writing code. IDEs often use more resources, but you won’t notice it with a modern computer (especially with a SSD).

The best IDE out there is PyCharm from JetBrains. It has both a free Community and paid Professional edition. The JetBrains folks are experts at IDEs — they have fully-fledged tools for many languages. Their Python solution offers a plethora of options that aid programmers in their work. Also, if you work with Java, or otherwise more than one IDEA-supported language, then install IntelliJ IDEA and the Python plugin (which has the same features as PyCharm). Students can get free Professional/Ultimate licenses for JetBrains products.

I also spend a lot of time in Vim (neovim/VimR to be precise). Vim is the most powerful text editor out there, and with the right set of plugins it can beat IDEs at speed and productivity. Vim has a steep learning curve, but it’s worth it — you can do large changes with just a few keystrokes. Vim is considered so good that many IDEs (Visual Studio, IntelliJ IDEA/PyCharm) have Vim emulation plugins.

Another option is Visual Studio Code — it’s a text editor, but can offer many IDE-like features with the right set of plugins. It’s Electron-based architecture, or effectively being based on top of Google’s Chromium, is unfortunate and can lead to terrible performance on lower-end machines, and on higher-end ones in some cases. (In my experience, it’s better than Atom.) You can also try Sublime Text ($80).

But really, almost any editor will do. But please avoid IDLE, the editor included with Python. It lacks some of the most basic things — it doesn’t even have an option to show line numbers. Not to mention its ugliness. Also, don’t use Notepad and TextEdit. Those are too simple, and Notepad has encoding issues.

Update history

2018-09-21

Link to python-virtual-environments post.

2017-07-19

Better description of problems caused by using sudo pip.

2017-07-10

Added notes about not removing built-in Pythons.

2017-07-07

Spelling fixes and updates to the virtualenv usage section.

Requirements and Desired Results

You will need:

• a Python project

• a setup.py file using setuptools

• the following directory structure:

  • entry_points_project/
  • • my_project/
    • • __init__.py
      • __main__.py
    • setup.py

(entry_points_project is also where the README and other auxiliary files go, while my_project contains all the Python code.)

When you’re done, you will have a project that can be executed by:

• python -m my_project

• my_project

Provided that you have your Python directory and its Scripts\ subdirectory on the %PATH%, this will also work in Windows.

Step 1: create a __main__.py file

In order to implement the first desired result, you need to create a __main__.py file in your package. This file needs to contain a main() function that takes no arguments, and also a special passage to determine code to run:

entry_points_project/my_project/__main__.py (Source)

1. The if __name__ == "__main__": idiom, as documented here, is used to check whether this is executed as the top-level file, or if it has been imported by someone else (in this case, executing the main() function is not always intended).

2. The main() function must not take any arguments, because that’s how entry_points executes things.

Step 2: adjust setup.py accordingly

This is the real deal: create the entry points in your setup.py file.

entry_points_project/setup.py (Source)

1. You must use setuptools, otherwise this won’t work.

2. The most important piece of code is the entry_points declaration (unsurprisingly).

3. The declaration reads

"name_of_executable = module.with:function_to_execute"

4. If you are developing a GUI application (in Tkinter, PyQt/PySide, wxPython, PyGTK, PyGame…), you should change the declaration to gui_scripts. On *nix, this makes no difference, but on Windows, it means that running your script by opening the created .exe files does not show a console window. Note that stdout/stderr do not work in that mode under Windows, which can lead to spurious application crashes. (GUI-only processes cannot use stdout/stderr because they don’t have a console attached)

5. You can create multiple scripts this way. You can also have multiple console_scripts and gui_scripts in one setup file.

All code samples are freely reusable, but if you mention where you got them from, it’d be really nice.