Posts about guidehttps://chriswarrick.com/A rarely updated blog, mostly about programming.Sun, 20 Feb 2022 22:00:00 GMThttps://github.com/Kwpolska/YetAnotherBlogGeneratorPython Virtual Environments in Five MinutesChris Warrickhttps://chriswarrick.com/blog/2018/09/04/python-virtual-environments/Tue, 04 Sep 2018 18:15:00 GMThttps://chriswarrick.com/blog/2018/09/04/python-virtual-environments/ In Python, virtual environments are used to isolate projects from each other (if they require different versions of the same library, for example). They let you install and manage packages without administrative privileges, and without conflicting with the system package manager. They also allow to quickly create an environment somewhere else with the same dependencies. Virtual environments are a crucial tool for any Python developer. And at that, a very simple tool to work with. In Python, virtual environments are used to isolate projects from each other (if they require different versions of the same library, for example). They let you install and manage packages without administrative privileges, and without conflicting with the system package manager. They also allow to quickly create an environment somewhere else with the same dependencies.

Virtual environments are a crucial tool for any Python developer. And at that, a very simple tool to work with.

Let’s get started!

Install

The best tool that can be used to create virtual environments is the venv module, which is part of the standard library since Python 3.3.

venv is built into Python, and most users don’t need to install anything. However, Debian/Ubuntu users will need to run sudo apt-get install python3-venv to make it work (due to Debian not installing some components that venv needs by default). [1]

The alternative (and original, and previously standard) virtual environment tool is virtualenv. It works with Python 2.7, and has a couple extra fetures (that you generally won’t need). virtualenv can be installed with your system package manager, or pip install --user virtualenv.

Which one to use? Probably venv. Both tools achieve the same goal in similar ways. And if one of them does not work, you can try the other and it might just work better.

(Terminology note: most of the time, the names of both tools are used interchargeably, “venv” was often used as an abbreviation for “virtualenv” before the stdlib tool was created)

Create

To create a virtual environment named env, you need to run the venv tool with the Python you want to use in that environment.

Linux:   $ python3 -m venv env
Windows: > py -m venv env

or, if you’re using virtualenv:

$ python3 -m virtualenv env
> py -m virtualenv env

Afterwards, you will end up with a folder named env that contains folders named bin (Scripts on Windows — contains executables and scripts installed by packages, including python), lib (contains code), and include (contains C headers).

Both tools install pip and setuptools, but venv does not ship with wheel. In addition, the default versions tend to be more-or-less outdated. Let’s upgrade them real quick: [2]

$ env/bin/python -m pip install --upgrade pip setuptools wheel
> env\Scripts\python -m pip install --upgrade pip setuptools wheel

Where to store virtual environments?

While the tools allow you to put your virtual environments anywhere in the system, it is not a desirable thing to do. There are two options:

  1. Have one global place for them, like ~/virtualenvs.

  2. Store them in each project’s directory, like ~/git/foobar/.venv.

The first option can be easier to manage, there are tools that can help manage those (eg. virtualenvwrapper, shell auto-activation scripts, or the workon functions described below). The second option is equally easy to work with, but comes with one caveat — you must add the venv directory to your .gitignore file (or .git/info/exclude if you don’t want to commit changes to .gitignore), since you don’t want it in your repository (it’s binary bloat, and works only on your machine).

If you pick the global virtual environment store option, you can use the following short function (put it in .bashrc / .zshrc / your shell configuration file) to get a simple way to activate an environment (by running workon foo). virtualenvwrapper also has a workon feature, although I don’t think virtualenvwrapper is really necessary and too helpful — the workon feature is handy though, and so here’s a way to do it without virtualenvwrapper:

export WORKON_HOME=~/virtualenvs
function workon {
     source "$WORKON_HOME/$1/bin/activate"
}

And for PowerShell fans, here’s a workon.ps1 script:

$WORKON_HOME = "$home\virtualenvs"
& "$WORKON_HOME\$($args[0])\Scripts\activate.ps1"

And for cmd.exe fans… you should switch to PowerShell, it’s a very nice and friendly shell (though perhaps requiring some effort to learn how to be productive with it).

Use

There are three ways of working with virtual environments interactively (in a shell):

  • activation (run source env/bin/activate on *nix; env\Scripts\activate on Windows) — it simplifies work and requires less typing, although it can sometimes fail to work properly. (After installing scripts, hash -r may be necessary on *nix to use them.)

  • executing env/bin/python (env\Scripts\python) and other scripts directly, as activation only changes $PATH and some helper variables — those variables are not mandatory for operation, running the correct python is, and that method is failsafe.

  • in subshells (IMO, it’s bad UX)

Whichever method you use, you must remember that without doing any of these things, you will still be working with the system Python.

For non-interactive work (eg. crontab entries, system services, etc.), activation and subshells are not viable solutions. In these cases, you must always use the full path to Python.

Here are some usage examples (paths can be relative, of course):

## *nix, activation ##
$ source /path/to/env/bin/activate
(env)$ pip install Django
(env)$ deactivate

## *nix, manual execution ##
$ /path/to/env/bin/pip install Django

## Windows, activation ##
> C:\path\to\env\Scripts\activate
(venv)> pip install Django
(venv)> deactivate

## Windows, manual execution ##
> C:\path\to\env\Scripts\pip install Django

## Windows, updating pip/setuptools/wheel ##
> C:\path\to\env\Scripts\python -m pip install -U pip setuptools wheel

The same principle applies to running Python itself, or any other script installed by a package. (With Django’s manage.py, calling it as ./manage.py requires activation, or you can run venv/bin/python manage.py.)

Moving/renaming/copying environments?

If you try to copy or rename a virtual environment, you will discover that the copied environment does not work. This is because a virtual environment is closely tied to both the Python it was created with, and the location it was created in. (The “relocatable” option of virtualenv does not work and is deprecated.) [3]

However, this is very easy to fix. Instead of moving/copying, just create a new environment in the new location. Then, run pip freeze > requirements.txt in the old environment to create a list of packages installed in it. With that, you can just run pip install -r requirements.txt in the new environment to install packages from the saved list. (Of course, you can copy requirements.txt between machines. In many cases, it will just work; sometimes, you might need a few modifications to requirements.txt to remove OS-specific stuff.)

$ oldenv/bin/pip freeze > requirements.txt
$ python3 -m venv newenv
$ newenv/bin/pip install -r requirements.txt
(You may rm -rf oldenv now if you desire)

Note that it might also be necessary to re-create your virtual environment after a Python upgrade, [4] so it might be handy to keep an up-to-date requirements.txt for your virtual environments (for many projects, it makes sense to put that in the repository).

To manage those requirements.txt files in a more orgnized yet still simple way, you might be interested in pip-tools.

Frequently Asked Questions

I’m using virtualenv. Do I need to install it for each Python I want to use it with?

In most cases, you can use virtualenv -p pythonX env to specify a different Python version, but with some Python version combinations, that approach might be unsuccessful. (The venv module is tied to the Python version it’s installed in.)

I’m the only user on my system. Do I still need virtual environments?

Yes, you do. First, you will still need separation between projects, sooner or later. Moreover, if you were to install packages system-wide with pip, you might end up causing conflicts between packages installed by the system package manager and by pip. Running sudo pip is never a good idea because of this.

I’m using Docker. Do I still need virtual environments?

They are still a good idea in that case. They protect you against any bad system-wide Python packages your OS image might have (and one popular base OS is famous for those). They don’t introduce any extra overhead, while allowing to have a clean environment and the ability to re-create it outside of Docker (eg. for local development without Docker)

What about Pipenv?

Pipenv is a dependency management tool. It isn’t compatible with most workflows, and comes with many issues. In my opinion, it’s not worth using (Also, that thing about it being an officially recommended tool? Turns out it’s not true.)

I also wrote a blog post detailing concerns with that tool, titled Pipenv: promises a lot, delivers very little.

Consider using pip-tools instead.

Footnotes

]]>Pythonbest practicesdevelguidePythonvenvvirtual environmentsvirtualenvSpawning subprocesses smartly and securelyChris Warrickhttps://chriswarrick.com/blog/2017/09/02/spawning-subprocesses-smartly-and-securely/Sat, 02 Sep 2017 18:40:00 GMThttps://chriswarrick.com/blog/2017/09/02/spawning-subprocesses-smartly-and-securely/ As part of your code, you may be inclined to call a command to do something. But is it always a good idea? How to do it safely? What happens behind the scenes? As part of your code, you may be inclined to call a command to do something. But is it always a good idea? How to do it safely? What happens behind the scenes?

This article is written from a general perspective, with a Unix/C bias and a very slight Python bias. The problems mentioned apply to all languages in most environments, including Windows.

Use the right tool for the job

By calling another process, you introduce a third-party dependency. That dependency isn’t controlled by your code, and your code becomes more fragile. The problems include:

  • the program is not installed, or even available, for the user’s OS of choice

  • the program is not in the $PATH your process gets

  • the hard-coded path is not correct on the end user’s system

  • the program is in a different version (eg. GNU vs. BSD, updates/patches), which means different option names or other behaviors

  • the program’s output is not what you expected due to user config (including locale)

  • error reporting is based on numeric exit codes, and the meaning of those differs between programs (if they have meaning besides 0/1 in the first place)

On the other hand, if your code uses a lot of subprocesses, perhaps you should stay with Bash. You can do the harder parts with Python, Ruby, or some other language by calling them from within your Bash script.

Don’t spawn subprocesses if there’s an alternative

Spawning a subprocess always incurs a (minor) [1] performance hit minor compared to the alternatives. With that in mind, and the resiliency issues listed above, you should always try to find an alternative for the external command.

The simplest ones are the basic Unix utilities. Replace grep, sed and awk with string operations and regular expressions. Filesystem utilities will have equivalents — for Python, in os or shutil. Your language of choice can also handle things like networking (don’t call curl), file compression, working with date/time…

Similarly, you should check if there are packages available that already do what you want — library bindings or re-implementations. And if there isn’t, perhaps you could help the world by writing one of those and sharing it?

One more important thing: if the program uses the same language as your code, then you should try to import the code and run it from the same process instead of spawning a process, if this is feasible.

Security considerations: shells, spaces, and command injection

We come to the most important part of this article: how to spawn subprocesses without compromising your system. When you spawn a subprocess on a typical Unix system, fork() is called, and your process is copied. Many modern Unix systems have a copy-on-write implementation of that syscall, meaning that the operation does not result in copying all the memory of the host process over. Forking is (almost) immediately followed by calling execve() (or a helper function from the exec family) [2] in the child process — that function transforms the calling process into a new process [3]. This technique is called fork-exec and is the typical way to spawn a new process on Unix. [4]

There are two ways to access this API, from the C perspective:

  • directly, by calling fork() and exec*() (or posix_spawn()), and providing an array of arguments passed to the process, or

  • through the shell (sh), usually by calling system(). As Linux’s manpage for system(3) puts it,

    The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:

    execl("/bin/sh", "sh", "-c", command, (char *) 0);

If you go through the shell, you pass one string argument, whereas exec*() demands you to specify arguments separately. Let’s write a sample program to print all the arguments it receives. I’ll do it in Python to get a more readable output.

#!/usr/bin/env python3
import sys
print(sys.argv)

Let’s see what appears:

$ ./argv.py foo bar
['./argv.py', 'foo', 'bar']
$ ./argv.py 'foo bar'
['./argv.py', 'foo bar']
$ ./argv.py foo\ bar baz
['./argv.py', 'foo bar', 'baz']

$ ./argv.py $(date)
['./argv.py', 'Sat', 'Sep', '2', '16:54:52', 'CEST', '2017']
$ ./argv.py "$(date)"
['./argv.py', 'Sat Sep  2 16:54:52 CEST 2017']

$ ./argv.py /usr/*
['./argv.py', '/usr/X11', '/usr/X11R6', '/usr/bin', '/usr/include', '/usr/lib', '/usr/libexec', '/usr/local', '/usr/sbin', '/usr/share', '/usr/standalone']
$ ./argv.py "/usr/*"
['./argv.py', '/usr/*']

$ ./argv.py $EDITOR
['./argv.py', 'nvim']

$ $PWD/argv.py foo bar
['/Users/kwpolska/Desktop/blog/subprocess/argv.py', 'foo', 'bar']
$ ./argv.py a{b,c}d
['./argv.py', 'abd', 'acd']

$ python argv.py foo bar | cat
['argv.py', 'foo', 'bar']
$ python argv.py foo bar > foo.txt
$ cat foo.txt
['argv.py', 'foo', 'bar']

$ ./argv.py foo; ls /usr
['./argv.py', 'foo']
X11@        X11R6@      bin/        include/    lib/        libexec/    local/      sbin/       share/      standalone/

As you can see, the following things are handled by the shell (the process is unaware of this occurring):

  • quotes and escapes

  • expanding expressions in braces

  • expanding variables

  • wildcards (glob, *)

  • redirections and pipes (> >> |)

  • command substitution (backticks or $(…))

  • running multiple commands on the same line (; && || &)

The list is full of potential vulnerabilities. If end users are in control of the arguments passed, and you go through the shell, they can execute arbitrary commands or even get full shell access. Even in other cases, you’ll have to depend on the shell’s parsing, which introduces an unnecessary indirection.

TL;DR: How to do this properly in your language of choice

To ensure spawning subprocess is done securely, do not use the shell in between. If you need any of the operations I listed above as part of your command — wildcards, pipes, etc. — you will need to take care of them in your code; most languages have those features built-in.

In C (Unix)

Perform fork-exec by yourself, or use posix_spawn(). This also lets you communicate with the process if you open a pipe and make it stdout of the child process. Never use system().

In Python

Use the subprocess module. Always pass shell=False and give it a list of arguments. With asyncio, use asyncio.create_subprocess_exec (and not _shell), but note it takes *args and not a list. Never use os.system and os.popen.

In Ruby

Pass arrays to IO.popen. Pass multiple arguments to system() (system(["ls", "ls"]) or system("ls", "-l")). Never use %x{command} or backticks.

In Java

Pass arrays to Runtime.exec. Pass multiple arguments or list to ProcessBuilder.

In PHP

All the standard methods go through the shell. Try escapeshellcmd(), escapeshellarg() — or better, switch to Python. Or anything, really.

In Go

os/exec and os.StartProcess are safe.

In Node.js

Use child_process.execFile or child_process.spawn with shell set to false.

Elsewhere

You should be able to specify multiple strings (using variadic arguments, arrays, or otherwise standard data structures of your language of choice) as the command line. Otherwise, you might be running into something shell-related.

The part where I pretend I know something about Windows

On Windows, argument lists are always passed to processes as strings (Python joins them semi-intelligently if it gets a list). Redirections and variables work in shell mode, but globs (asterisks) are always left for the called process to handle.

Some useful functions are implemented as shell built-ins — in that case, you need to call it via the shell.

Internals: There is no fork() on Windows. Instead, CreateProcess(), ShellExecute(), or lower-level spawn*() functions are used. cmd.exe /c is called in shell calls.

]]>Programmingbest practicesCdevelguideLinuxPythonsecuritysubprocessUnixSetting up a Python development environmentChris Warrickhttps://chriswarrick.com/blog/2017/07/03/setting-up-a-python-development-environment/Mon, 03 Jul 2017 10:40:00 GMThttps://chriswarrick.com/blog/2017/07/03/setting-up-a-python-development-environment/ Setting up Python is usually simple, but there are some places where newcomers (and experienced users) need to be careful. What versions are there? What’s the difference between Python, CPython, Anaconda, PyPy? Those and many other questions may stump new developers, or people wanting to use Python. Setting up Python is usually simple, but there are some places where newcomers (and experienced users) need to be careful. What versions are there? What’s the difference between Python, CPython, Anaconda, PyPy? Those and many other questions may stump new developers, or people wanting to use Python.

Note: this guide is opinionated.

Glossary and questions

Python versions: 2 vs 3

The Python community has undergone sort of a schism in recent years. Python 3, released in 2008, broke backwards compatibility: deprecated some bad constructs and libraries (eg. raw_input() became input() and the original Python 2 function that ran code input by users is gone; print() became a function; many things that returned lists now are iterators — zip, range), and completely remodelled strings (which are now Unicode by default, and the interpreter behavior is stricter when the wrong type is used)

For new code, you should use Python 3. Most popular packages support Python 3, and many of them support both Pythons at the same time. The early bugs were ironed out in the first few point releases, some features that made porting easier were added (back).

But what if you end up needing Python 2 later? No problem: you can learn the differences in a short time, and with the help of a few libraries (eg. six) you can easily write code that is compatible with Python 2 and 3 at the same time, using the same codebase (most libraries out there do that).

Python 2 will go EOL and lose official support and updates in 2020.

Read more: Python 2 or Python 3 on Python Wiki

Can I run multiple Pythons on the same machine?

Yes. Note that multiple Python interpreters are completely separate: they have their own pip and packages, and you can’t run Python 2 code in a Python 3 interpreter. You need to specify which interpreter to use when installing packages and running some scripts (eg. pip2, pip3 or python3 -m pip).

It’s best to limit yourself to the latest Python 2 and 3 versions. Python is backwards-compatible within the major release, so Python 2.7 runs code written with older 2.x versions in mind.

Implementations

A programming language is an abstract construct. To run code written in that language, an interpreter or compiler needs to be written. In Python’s case, there’s a plethora of implementations. Some of them are:

  • CPython is the reference implementation. This is the implementation distributed on https://python.org/ and as part of many operating systems. Most Python features are first implemented in CPython, and then they are ported to other implementations. If you don’t know what to choose, use CPython.

  • PyPy is a fast implementation, written in a subset of Python. It’s compatible with Python 2.7 and 3.5 (beta support). It can run all pure Python code, and many extension libraries that use CFFI.

  • IronPython is a .NET CLR implementation. It can integrate with .NET code.

  • Jython is a Java JVM implementation. It can integrate with Java code, as well as other JVM languages.

Read more: Python Implementations on Python Wiki

Distributions

There are also Python (CPython) distributions. They ship the CPython interpreter and add some extra packages/features. They are maintained by other communities or corporate entities.

The most popular third-party distribution is Anaconda from Continuum Analytics. It’s popular for data scientists, and includes over 100 packages, with extra pre-built binaries available from the conda package manager.

I personally recommend to avoid Anaconda:

  • Most packages have binary wheels for Windows, macOS and Linux (yes, Linux!) making the installation as simple as pip install numpy.

  • You waste disk space for packages Anaconda installs that you won’t ever need.

  • It’s provided by some random for-profit company.

  • I’ve seen bugs that were not reproducible outside of Anaconda.

  • You can still do data science using the official distribution. There’s nothing special about Anaconda.

Read more: Python distributions on Python Wiki

Can I make .exe files from Python programs?

Yes, you can. There are tools for this — PyInstaller is the best one. Note that you usually need to run it on the destination operating system. And remember that “compiling” to exe files like that is not a security measure — your source code is still easily recoverable. (It’s not a security measure in other languages either, even if getting source code back might be more expensive/tricky in those.)

Where to learn Python? Where to get help?

The choice of learning material is important. If you get a bad book, it might discourage you from learning (because it’s boring), or may teach you bad/outdated practices.

If you can already program in another language, I recommend the official Python tutorial. For newcomers to programming, I recommend Think Python or Automate the Boring Stuff with Python. They teach Python 3, and (mostly) best practices.

If you need help, try #python on freenode IRC, the Tutor or Python-list mailing lists, or a bunch of other communities. (I’m a regular on #python)

Installing Python

This guide will focus on installing CPython 2.7 and 3.x (latest), using the standard distribution. This choice is satisfactory for most people. Third-party distributions, while handy in some cases, are not needed for most. (See Distributions for arguments)

Throughout this guide, I’ll refer to the Python interpreter executable as python. The exact name depends on your system and desired version. On most OSes, python is Python 2 and python3 is 3; python2 should also exist. On Arch Linux, python is Python 3. On Windows, use the py launcher.

Windows

Download the installer(s): https://www.python.org/downloads/

Those installers come with pip, and modern Python 3.x versions come with the py launcher. You can use that launcher to pick a specific Python version, eg.:

  • py -3 -m pip install <package>

  • py -2 somefile.py

  • py -2.7

  • py (default system version)

It’s recommended for most use, and mandatory for upgrading pip.

The 32-bit versions are more versatile. Most packages support both (the only exception I’m aware of is Tensorflow, which only allows 64-bit Python 3.5 as of now).

macOS

macOS ships with Python 2.7.10 (as of macOS Sierra). It’s not the latest version; it’s good enough for most people, but I still recommend installing your own (the system Python doesn’t include pip, for example). You can install the latest 2.7 version, as well as Python 3, using a package manager. I recommend Homebrew — it’s the most popular solution, and lets you install many other packages.

DO NOT use the python.org installers: they do not have uninstallers, so you will have outdated versions lying around after some time. There is no auto-update as well. DO NOT attempt to remove the system-installed Python, this will only damage your system and you’ll need to reinstall.

If you already have a package manager installed (MacPorts, Fink), don’t install a new one and just use the existing one.

  1. Install Homebrew.

  2. Run brew install python python3.

  3. You should now have python, python3, pip and pip3.

To update Homebrew and Python, run brew update.

Linux (and other Unix-like OSes)

On Linux, there usually are good enough packages in your OS repositories. You should be able to install the appropriate package for Python (2 and/or 3). Most (if not all) distributions require Python — do not remove the pre-installed packages, and be careful not to overwrite them with something newer.

If the version that ships with your distribution is too old, there are some options. There might be some repositories with better versions, eg. the deadsnakes PPA for Ubuntu. Then there’s the other option of compiling Python. There are some tools to help with this, like pyenv or pythonz (they can also manage multiple Python versions), or you can do it manually. The instructions depend on your exact requirements, but here’s a summary:

  1. Download the source distribution from Python.org and unpack it. Go into the unpacked source directory.

  2. Ensure you’ve got a functional C compiler and Python’s dependencies. You can usually use your system’s package manager to install the build dependencies of your system Python. Some dependencies are optional (eg. sqlite3 requires SQLite headers).

  3. Run ./configure --prefix=/opt/python3.6 and then make. (You may add other options to both. It will take a while.)

  4. Run make altinstall as root. Avoid make install, as it can override python executables.

Remember: compiling Python should be considered a last resort, unless you have very specific Python version requirements.

Installing packages

To install third-party packages, you should use pip, the Python package manager. If you’re using Windows or macOS (from Homebrew), pip is included with your copy of Python. If you’re on Linux and installed Python from a system repository, install the correct system package (python-pip, python3-pip). If you compiled your own Python, pip is also included.

To run pip, use py -m pip (Windows), python -m pip (other platforms), or the short pip/pip3 commands.

NEVER use sudo pip. This can cause numerous problems:

  • conflicts between packages installed by pip and your system package manager

  • pip modifying system packages, leading to issues when updating them, or breaking dependencies

  • no isolation between package versions, which is sometimes needed to satisfy dependencies

Note that a package install is specific to the Python interpreter used to run pip. Packages installed to a virtualenv are separate from system packages; packages installed for “global” Python 2.7 are separate from 3.6 packages. Virtual environments generally don’t use the system packages, unless specifically enabled during creation.

Some distros have popular packages in their repositories. Sometimes they’re good; in other cases they’re terribly outdated or they lack important components, making package managers angry and sick of supporting a 2-year-old version. (Especially since most bugs are closed with “we’ve fixed that long ago”)

User installs

At a small scale, you can install packages with pip for a single user. Use pip install --user PACKAGE to do this. If your package installs scripts, they will be installed to ~/.local/bin on Linux, and ~/Library/Python/X.Y/bin on macOS (X.Y is Python version), or you can use python -m if the package supports it.

For most people and projects, virtual environments are better. There are, however, use cases for putting some packages user-wide — if you don’t work on projects, but instead are doing one-off research projects, those are better suited by user-wide installs.

Virtual environments

I wrote a newer, more detailed post about virtualenvs: Python Virtual Environments in Five Minutes

Virtual environments are the best way to install and manage Python packages. Advantages include:

  • Isolation of projects and their requirements: if one app/package requires library version X, but another requires version Y, they can live in separate virtual environments

  • Independent from system-wide packages

  • Lightweight (an empty virtualenv is about 10 MB)

  • Simple to re-create in any place (pip freeze > requirements.txt → pip install -r requirements.txt)

Tools and management

There are two tools to facilitate creation of virtual environments: the older virtualenv project, and the newer venv module. The venv module is shipped with Python 3.x; some distributions may put it in a separate package or remove it altogether. Use whichever works for you. Virtualenv is compatible with more Python versions and cannot be broken by incompetent OS package maintainers (venv requires an extra package on Debian).

There are multiple schools of thought regarding virtualenv placement and content. Myself, I use virtualenvwrapper to manage virtualenvs and put them in ~/virtualenvs. Other people put virtualenvs inside their git repositories (but they must be in .gitignore) Virtualenvs should only contain packages installed with pip so they can be recreated quickly.

I also use the virtualenvwrapper plugin for Oh My Zsh, which also activates virtualenvs with the same name as a git repo, or the environment named by a .venv file.

Installation and usage

To install virtualenv user-wide, use pip install --user virtualenv. You can then use it with python -m virtualenv DIRECTORY. You may pass extra options, eg. interpreter to use (-p python3). Sometimes you need to install virtualenv for every Python version; usually, one copy is enough.

How to use them? This is a subject of heated debate in the Python community.

  • Some people believe that activating (source bin/activate on *nix; Scripts\activate on Windows) is the right thing to do and simplifies work.

  • Others think that you should use bin/python (or other scripts in that directory) directly, as activation only changes $PATH and some helper variables — those variables are not mandatory for operation, running the correct python is.

  • Others still think virtualenvs should be used in subshells.

In my opinion, if activating virtualenvs works in your environment, you should do it — it’s the most convenient option. There are, however, cases when activation fails, or is otherwise impossible — calling bin/python directly is your best bet in that case. If you are working inside shell scripts, do not activate virtualenvs. I’m not a fan of the subshell option, because it complicates stuff if you work on multiple projects, and requires tracking usage manually.

Upgrading and moving

Upgrading the system Python may make your virtualenvs unusable. For patch version upgrades, you can just update symlinks (see fix-venvs.sh). However, if the minor version changes, it’s best to re-create the virtualenv (you need to create requirements.txt ahead of time).

You cannot move a virtualenv between directories/machines or rename virtualenvs. You need to use pip freeze > requirements.txt, create a new virtualenv, and run pip install -r requirements.txt (you can then delete the old environment with a simple rm -rf)

Packages with C extensions (binary)

The situation improved drastically in the past year or so. Nowadays, almost all packages have a pre-compiled package available in PyPI. Those packages work for Windows, macOS, and Linux. There are packages for some of the most common offenders, including Pillow, lxml, PyQt5, numpy… However, there might still be packages without wheels on PyPI.

If there is no wheel for a package and you are on Windows, check out Christoph Gohlke’s unofficial binaries. If you can’t find any wheels online, you would have to resort to compiling it manually — this requires installing Visual Studio (Visual C++) in a version that matches your Python, and it’s kind of a pain to do.

If you are not on Windows, you must install a C compiler and toolchain. If you get a warning about missing Python.h, install the appropriate development package — for example, python-dev or python3-dev) on Debian/Ubuntu, python-devel or python3-devel on RHEL/Fedora. The package you’re trying to install might have other dependencies that you need to install (the -dev(el) part is important, too)

Other stuff

If you’re working on a project, use pip install -e . inside the project directory to install the package in your environment in development (editable) mode. This loads code directly from your repository — you don’t need to re-install on every change; you might need to re-install when your version number changes.

Editors and IDEs

Another important thing a developer should take care of is the choice of an editor. This is an important decision, and is the reason for many holy wars in the programmer community.

A good editor should have syntax highlighting for all languages you need to work with. It should also have features like visual block/multiple selections, sophisticated find-and-replace, file finding, code completion, and many more minor but helpful features.

Then there’s the difference between IDEs and text editors. Text editors are simpler, whereas IDEs try to include many extra things not necessarily related to writing code. IDEs often use more resources, but you won’t notice it with a modern computer (especially with a SSD).

The best IDE out there is PyCharm from JetBrains. It has both a free Community and paid Professional edition. The JetBrains folks are experts at IDEs — they have fully-fledged tools for many languages. Their Python solution offers a plethora of options that aid programmers in their work. Also, if you work with Java, or otherwise more than one IDEA-supported language, then install IntelliJ IDEA and the Python plugin (which has the same features as PyCharm). Students can get free Professional/Ultimate licenses for JetBrains products.

I also spend a lot of time in Vim (neovim/VimR to be precise). Vim is the most powerful text editor out there, and with the right set of plugins it can beat IDEs at speed and productivity. Vim has a steep learning curve, but it’s worth it — you can do large changes with just a few keystrokes. Vim is considered so good that many IDEs (Visual Studio, IntelliJ IDEA/PyCharm) have Vim emulation plugins.

Another option is Visual Studio Code — it’s a text editor, but can offer many IDE-like features with the right set of plugins. It’s Electron-based architecture, or effectively being based on top of Google’s Chromium, is unfortunate and can lead to terrible performance on lower-end machines, and on higher-end ones in some cases. (In my experience, it’s better than Atom.) You can also try Sublime Text ($80).

But really, almost any editor will do. But please avoid IDLE, the editor included with Python. It lacks some of the most basic things — it doesn’t even have an option to show line numbers. Not to mention its ugliness. Also, don’t use Notepad and TextEdit. Those are too simple, and Notepad has encoding issues.

Update history

2018-09-21

Link to python-virtual-environments post.

2017-07-19

Better description of problems caused by using sudo pip.

2017-07-10

Added notes about not removing built-in Pythons.

2017-07-07

Spelling fixes and updates to the virtualenv usage section.

]]>Pythonbest practicesdevelguideguidePythonSetting up a Python development environmentChris Warrickhttps://chriswarrick.com/blog/2017/07/03/setting-up-a-python-development-environment/Mon, 03 Jul 2017 10:40:00 GMThttps://chriswarrick.com/blog/2017/07/03/setting-up-a-python-development-environment/ Setting up Python is usually simple, but there are some places where newcomers (and experienced users) need to be careful. What versions are there? What’s the difference between Python, CPython, Anaconda, PyPy? Those and many other questions may stump new developers, or people wanting to use Python. Setting up Python is usually simple, but there are some places where newcomers (and experienced users) need to be careful. What versions are there? What’s the difference between Python, CPython, Anaconda, PyPy? Those and many other questions may stump new developers, or people wanting to use Python.

Note: this guide is opinionated.

Glossary and questions

Python versions: 2 vs 3

The Python community has undergone sort of a schism in recent years. Python 3, released in 2008, broke backwards compatibility: deprecated some bad constructs and libraries (eg. raw_input() became input() and the original Python 2 function that ran code input by users is gone; print() became a function; many things that returned lists now are iterators — zip, range), and completely remodelled strings (which are now Unicode by default, and the interpreter behavior is stricter when the wrong type is used)

For new code, you should use Python 3. Most popular packages support Python 3, and many of them support both Pythons at the same time. The early bugs were ironed out in the first few point releases, some features that made porting easier were added (back).

But what if you end up needing Python 2 later? No problem: you can learn the differences in a short time, and with the help of a few libraries (eg. six) you can easily write code that is compatible with Python 2 and 3 at the same time, using the same codebase (most libraries out there do that).

Python 2 will go EOL and lose official support and updates in 2020.

Read more: Python 2 or Python 3 on Python Wiki

Can I run multiple Pythons on the same machine?

Yes. Note that multiple Python interpreters are completely separate: they have their own pip and packages, and you can’t run Python 2 code in a Python 3 interpreter. You need to specify which interpreter to use when installing packages and running some scripts (eg. pip2, pip3 or python3 -m pip).

It’s best to limit yourself to the latest Python 2 and 3 versions. Python is backwards-compatible within the major release, so Python 2.7 runs code written with older 2.x versions in mind.

Implementations

A programming language is an abstract construct. To run code written in that language, an interpreter or compiler needs to be written. In Python’s case, there’s a plethora of implementations. Some of them are:

  • CPython is the reference implementation. This is the implementation distributed on https://python.org/ and as part of many operating systems. Most Python features are first implemented in CPython, and then they are ported to other implementations. If you don’t know what to choose, use CPython.

  • PyPy is a fast implementation, written in a subset of Python. It’s compatible with Python 2.7 and 3.5 (beta support). It can run all pure Python code, and many extension libraries that use CFFI.

  • IronPython is a .NET CLR implementation. It can integrate with .NET code.

  • Jython is a Java JVM implementation. It can integrate with Java code, as well as other JVM languages.

Read more: Python Implementations on Python Wiki

Distributions

There are also Python (CPython) distributions. They ship the CPython interpreter and add some extra packages/features. They are maintained by other communities or corporate entities.

The most popular third-party distribution is Anaconda from Continuum Analytics. It’s popular for data scientists, and includes over 100 packages, with extra pre-built binaries available from the conda package manager.

I personally recommend to avoid Anaconda:

  • Most packages have binary wheels for Windows, macOS and Linux (yes, Linux!) making the installation as simple as pip install numpy.

  • You waste disk space for packages Anaconda installs that you won’t ever need.

  • It’s provided by some random for-profit company.

  • I’ve seen bugs that were not reproducible outside of Anaconda.

  • You can still do data science using the official distribution. There’s nothing special about Anaconda.

Read more: Python distributions on Python Wiki

Can I make .exe files from Python programs?

Yes, you can. There are tools for this — PyInstaller is the best one. Note that you usually need to run it on the destination operating system. And remember that “compiling” to exe files like that is not a security measure — your source code is still easily recoverable. (It’s not a security measure in other languages either, even if getting source code back might be more expensive/tricky in those.)

Where to learn Python? Where to get help?

The choice of learning material is important. If you get a bad book, it might discourage you from learning (because it’s boring), or may teach you bad/outdated practices.

If you can already program in another language, I recommend the official Python tutorial. For newcomers to programming, I recommend Think Python or Automate the Boring Stuff with Python. They teach Python 3, and (mostly) best practices.

If you need help, try #python on freenode IRC, the Tutor or Python-list mailing lists, or a bunch of other communities. (I’m a regular on #python)

Installing Python

This guide will focus on installing CPython 2.7 and 3.x (latest), using the standard distribution. This choice is satisfactory for most people. Third-party distributions, while handy in some cases, are not needed for most. (See Distributions for arguments)

Throughout this guide, I’ll refer to the Python interpreter executable as python. The exact name depends on your system and desired version. On most OSes, python is Python 2 and python3 is 3; python2 should also exist. On Arch Linux, python is Python 3. On Windows, use the py launcher.

Windows

Download the installer(s): https://www.python.org/downloads/

Those installers come with pip, and modern Python 3.x versions come with the py launcher. You can use that launcher to pick a specific Python version, eg.:

  • py -3 -m pip install <package>

  • py -2 somefile.py

  • py -2.7

  • py (default system version)

It’s recommended for most use, and mandatory for upgrading pip.

The 32-bit versions are more versatile. Most packages support both (the only exception I’m aware of is Tensorflow, which only allows 64-bit Python 3.5 as of now).

macOS

macOS ships with Python 2.7.10 (as of macOS Sierra). It’s not the latest version; it’s good enough for most people, but I still recommend installing your own (the system Python doesn’t include pip, for example). You can install the latest 2.7 version, as well as Python 3, using a package manager. I recommend Homebrew — it’s the most popular solution, and lets you install many other packages.

DO NOT use the python.org installers: they do not have uninstallers, so you will have outdated versions lying around after some time. There is no auto-update as well. DO NOT attempt to remove the system-installed Python, this will only damage your system and you’ll need to reinstall.

If you already have a package manager installed (MacPorts, Fink), don’t install a new one and just use the existing one.

  1. Install Homebrew.

  2. Run brew install python python3.

  3. You should now have python, python3, pip and pip3.

To update Homebrew and Python, run brew update.

Linux (and other Unix-like OSes)

On Linux, there usually are good enough packages in your OS repositories. You should be able to install the appropriate package for Python (2 and/or 3). Most (if not all) distributions require Python — do not remove the pre-installed packages, and be careful not to overwrite them with something newer.

If the version that ships with your distribution is too old, there are some options. There might be some repositories with better versions, eg. the deadsnakes PPA for Ubuntu. Then there’s the other option of compiling Python. There are some tools to help with this, like pyenv or pythonz (they can also manage multiple Python versions), or you can do it manually. The instructions depend on your exact requirements, but here’s a summary:

  1. Download the source distribution from Python.org and unpack it. Go into the unpacked source directory.

  2. Ensure you’ve got a functional C compiler and Python’s dependencies. You can usually use your system’s package manager to install the build dependencies of your system Python. Some dependencies are optional (eg. sqlite3 requires SQLite headers).

  3. Run ./configure --prefix=/opt/python3.6 and then make. (You may add other options to both. It will take a while.)

  4. Run make altinstall as root. Avoid make install, as it can override python executables.

Remember: compiling Python should be considered a last resort, unless you have very specific Python version requirements.

Installing packages

To install third-party packages, you should use pip, the Python package manager. If you’re using Windows or macOS (from Homebrew), pip is included with your copy of Python. If you’re on Linux and installed Python from a system repository, install the correct system package (python-pip, python3-pip). If you compiled your own Python, pip is also included.

To run pip, use py -m pip (Windows), python -m pip (other platforms), or the short pip/pip3 commands.

NEVER use sudo pip. This can cause numerous problems:

  • conflicts between packages installed by pip and your system package manager

  • pip modifying system packages, leading to issues when updating them, or breaking dependencies

  • no isolation between package versions, which is sometimes needed to satisfy dependencies

Note that a package install is specific to the Python interpreter used to run pip. Packages installed to a virtualenv are separate from system packages; packages installed for “global” Python 2.7 are separate from 3.6 packages. Virtual environments generally don’t use the system packages, unless specifically enabled during creation.

Some distros have popular packages in their repositories. Sometimes they’re good; in other cases they’re terribly outdated or they lack important components, making package managers angry and sick of supporting a 2-year-old version. (Especially since most bugs are closed with “we’ve fixed that long ago”)

User installs

At a small scale, you can install packages with pip for a single user. Use pip install --user PACKAGE to do this. If your package installs scripts, they will be installed to ~/.local/bin on Linux, and ~/Library/Python/X.Y/bin on macOS (X.Y is Python version), or you can use python -m if the package supports it.

For most people and projects, virtual environments are better. There are, however, use cases for putting some packages user-wide — if you don’t work on projects, but instead are doing one-off research projects, those are better suited by user-wide installs.

Virtual environments

I wrote a newer, more detailed post about virtualenvs: Python Virtual Environments in Five Minutes

Virtual environments are the best way to install and manage Python packages. Advantages include:

  • Isolation of projects and their requirements: if one app/package requires library version X, but another requires version Y, they can live in separate virtual environments

  • Independent from system-wide packages

  • Lightweight (an empty virtualenv is about 10 MB)

  • Simple to re-create in any place (pip freeze > requirements.txt → pip install -r requirements.txt)

Tools and management

There are two tools to facilitate creation of virtual environments: the older virtualenv project, and the newer venv module. The venv module is shipped with Python 3.x; some distributions may put it in a separate package or remove it altogether. Use whichever works for you. Virtualenv is compatible with more Python versions and cannot be broken by incompetent OS package maintainers (venv requires an extra package on Debian).

There are multiple schools of thought regarding virtualenv placement and content. Myself, I use virtualenvwrapper to manage virtualenvs and put them in ~/virtualenvs. Other people put virtualenvs inside their git repositories (but they must be in .gitignore) Virtualenvs should only contain packages installed with pip so they can be recreated quickly.

I also use the virtualenvwrapper plugin for Oh My Zsh, which also activates virtualenvs with the same name as a git repo, or the environment named by a .venv file.

Installation and usage

To install virtualenv user-wide, use pip install --user virtualenv. You can then use it with python -m virtualenv DIRECTORY. You may pass extra options, eg. interpreter to use (-p python3). Sometimes you need to install virtualenv for every Python version; usually, one copy is enough.

How to use them? This is a subject of heated debate in the Python community.

  • Some people believe that activating (source bin/activate on *nix; Scripts\activate on Windows) is the right thing to do and simplifies work.

  • Others think that you should use bin/python (or other scripts in that directory) directly, as activation only changes $PATH and some helper variables — those variables are not mandatory for operation, running the correct python is.

  • Others still think virtualenvs should be used in subshells.

In my opinion, if activating virtualenvs works in your environment, you should do it — it’s the most convenient option. There are, however, cases when activation fails, or is otherwise impossible — calling bin/python directly is your best bet in that case. If you are working inside shell scripts, do not activate virtualenvs. I’m not a fan of the subshell option, because it complicates stuff if you work on multiple projects, and requires tracking usage manually.

Upgrading and moving

Upgrading the system Python may make your virtualenvs unusable. For patch version upgrades, you can just update symlinks (see fix-venvs.sh). However, if the minor version changes, it’s best to re-create the virtualenv (you need to create requirements.txt ahead of time).

You cannot move a virtualenv between directories/machines or rename virtualenvs. You need to use pip freeze > requirements.txt, create a new virtualenv, and run pip install -r requirements.txt (you can then delete the old environment with a simple rm -rf)

Packages with C extensions (binary)

The situation improved drastically in the past year or so. Nowadays, almost all packages have a pre-compiled package available in PyPI. Those packages work for Windows, macOS, and Linux. There are packages for some of the most common offenders, including Pillow, lxml, PyQt5, numpy… However, there might still be packages without wheels on PyPI.

If there is no wheel for a package and you are on Windows, check out Christoph Gohlke’s unofficial binaries. If you can’t find any wheels online, you would have to resort to compiling it manually — this requires installing Visual Studio (Visual C++) in a version that matches your Python, and it’s kind of a pain to do.

If you are not on Windows, you must install a C compiler and toolchain. If you get a warning about missing Python.h, install the appropriate development package — for example, python-dev or python3-dev) on Debian/Ubuntu, python-devel or python3-devel on RHEL/Fedora. The package you’re trying to install might have other dependencies that you need to install (the -dev(el) part is important, too)

Other stuff

If you’re working on a project, use pip install -e . inside the project directory to install the package in your environment in development (editable) mode. This loads code directly from your repository — you don’t need to re-install on every change; you might need to re-install when your version number changes.

Editors and IDEs

Another important thing a developer should take care of is the choice of an editor. This is an important decision, and is the reason for many holy wars in the programmer community.

A good editor should have syntax highlighting for all languages you need to work with. It should also have features like visual block/multiple selections, sophisticated find-and-replace, file finding, code completion, and many more minor but helpful features.

Then there’s the difference between IDEs and text editors. Text editors are simpler, whereas IDEs try to include many extra things not necessarily related to writing code. IDEs often use more resources, but you won’t notice it with a modern computer (especially with a SSD).

The best IDE out there is PyCharm from JetBrains. It has both a free Community and paid Professional edition. The JetBrains folks are experts at IDEs — they have fully-fledged tools for many languages. Their Python solution offers a plethora of options that aid programmers in their work. Also, if you work with Java, or otherwise more than one IDEA-supported language, then install IntelliJ IDEA and the Python plugin (which has the same features as PyCharm). Students can get free Professional/Ultimate licenses for JetBrains products.

I also spend a lot of time in Vim (neovim/VimR to be precise). Vim is the most powerful text editor out there, and with the right set of plugins it can beat IDEs at speed and productivity. Vim has a steep learning curve, but it’s worth it — you can do large changes with just a few keystrokes. Vim is considered so good that many IDEs (Visual Studio, IntelliJ IDEA/PyCharm) have Vim emulation plugins.

Another option is Visual Studio Code — it’s a text editor, but can offer many IDE-like features with the right set of plugins. It’s Electron-based architecture, or effectively being based on top of Google’s Chromium, is unfortunate and can lead to terrible performance on lower-end machines, and on higher-end ones in some cases. (In my experience, it’s better than Atom.) You can also try Sublime Text ($80).

But really, almost any editor will do. But please avoid IDLE, the editor included with Python. It lacks some of the most basic things — it doesn’t even have an option to show line numbers. Not to mention its ugliness. Also, don’t use Notepad and TextEdit. Those are too simple, and Notepad has encoding issues.

Update history

2018-09-21

Link to python-virtual-environments post.

2017-07-19

Better description of problems caused by using sudo pip.

2017-07-10

Added notes about not removing built-in Pythons.

2017-07-07

Spelling fixes and updates to the virtualenv usage section.

]]>Pythonbest practicesdevelguideguidePythonUnix locales vs Unicode (‘ascii’ codec can’t encode character…)Chris Warrickhttps://chriswarrick.com/blog/2017/06/18/unix-locales-vs-unicode/Sun, 18 Jun 2017 18:40:00 GMThttps://chriswarrick.com/blog/2017/06/18/unix-locales-vs-unicode/ You might get unusual errors about Unicode and inability to convert to ASCII. Programs might just crash at random. Those are often simple to fix — all you need is correct locale configuration. You might get unusual errors about Unicode and inability to convert to ASCII. Programs might just crash at random. Those are often simple to fix — all you need is correct locale configuration.

Has this ever happened to you?

Traceback (most recent call last):
  File "aogonek.py", line 1, in <module>
    print(u'\u0105')
UnicodeEncodeError: 'ascii' codec can't encode character '\u0105' in position 0: ordinal not in range(128)

Nikola: Could not guess locale for language en, using locale C

Input: ą
Desired ascii(): '\u0105'
Real ascii(): '\udcc4\udc85'

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    [...]
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

All those errors have the same root cause: incorrect locale configuration. To fix them all, you need to generate the missing locales and set them.

Check currently used locale

The locale command (without arguments) should tell you which locales you’re currently using. (The list might be shorter on your end)

$ locale
LANG="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

If any of those is set to C or POSIX, has a different encoding than UTF-8 (sometimes spelled utf8) is empty (with the exception of LC_ALL), or if you see any errors, you need to reconfigure your locale.

Check locale availability and install missing locales

The first thing you need to do is check locale availability. To do this, run locale -a. This will produce a list of all installed locales. You can use grep to get a more reasonable list.

$ locale -a | grep -i utf
<lists all UTF-8 locales>
$ locale -a | grep -i utf | grep -i en_US
en_US.UTF-8

The best locale to use is the one for your language, with the UTF-8 encoding. The locale will be used by some console apps for output. I’m going to use en_US.UTF-8 in this guide.

If you can’t see any UTF-8 locales, or no appropriate locale setting for your language of choice, you might need to generate those. The required actions depend on your distro/OS.

  • Debian, Ubuntu, and derivatives: install language-pack-en-base, run sudo dpkg-reconfigure locales

  • RHEL, CentOS, Fedora: install glibc-langpack-en

  • Arch Linux: uncomment relevant entries in /etc/locale.gen and run sudo locale-gen (wiki)

  • For other OSes, refer to the documentation.

You need a UTF-8 locale to ensure compatibility with software. Avoid the C and POSIX locales (it’s ASCII) and locales with other encodings (those aren’t used by ~anyone these days)

Configure system-wide

On some systems, you may be able to configure locale system-wide. Check your system documentation for details. If your system has systemd, run

sudo localectl set-locale LANG=en_US.UTF-8

Configure for a single user

If your environment does not allow system-wide locale configuration (macOS, shared server with generated but unconfigured locales), or if you want to ensure it’s always configured independently of system settings.

To do this, you need to edit the configuration file for your shell. If you’re using bash, it’s .bashrc (or .bash_profile on macOS). For zsh users, .zshrc. Add this line (or equivalent in your shell):

export LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

That should be enough. Note that those settings don’t apply to programs not launched through a shell.

Python/Windows corner: Python 3.7 will fix this on Unix by assuming UTF-8 if it encounters the C locale. On Windows, Python 3.6 is using UTF-8 interactively, but not when using shell redirections to files or pipes.

This post was brought to you by ą — U+0105 LATIN SMALL LETTER A WITH OGONEK.

]]>ProgrammingguidelocalePythonUnicodeUnixDeploying Python Web Applications with nginx and uWSGI EmperorChris Warrickhttps://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/Wed, 10 Feb 2016 14:00:00 GMThttps://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/ You’ve just written a great Python web application. Now, you want to share it with the world. In order to do that, you need a server, and some software to do that for you. The following is a comprehensive guide on how to accomplish that, on multiple Linux-based operating systems, using nginx and uWSGI Emperor. It doesn’t force you to use any specific web framework — Flask, Django, Pyramid, Bottle will all work. Written for Ubuntu, Debian, Fedora, CentOS 7, Alma Linux, Rocky Linux and Arch Linux (should be helpful for other systems, too). Now with an Ansible Playbook. Revision 8 (2022-02-20): works with Fedora 35, AlmaLinux 8, RockyLinux 8 While this guide is still valid, in 2026 I switched to Docker and Gunicorn for my Python web app deployments. Check out Deploying Python Web Applications with Docker for more details. You’ve just written a great Python web application. Now, you want to share it with the world. In order to do that, you need a server, and some software to do that for you.

The following is a comprehensive guide on how to accomplish that, on multiple Linux-based operating systems, using nginx and uWSGI Emperor. It doesn’t force you to use any specific web framework — Flask, Django, Pyramid, Bottle will all work. Written for Ubuntu, Debian, Fedora, CentOS 7, Alma Linux, Rocky Linux and Arch Linux (should be helpful for other systems, too). Now with an Ansible Playbook.

Revision 8 (2022-02-20): works with Fedora 35, AlmaLinux 8, RockyLinux 8

While this guide is still valid, in 2026 I switched to Docker and Gunicorn for my Python web app deployments. Check out Deploying Python Web Applications with Docker for more details.

CI status for the associated Ansible Playbook: ci-status

For easy linking, I set up some aliases: https://go.chriswarrick.com/pyweb and https://go.chriswarrick.com/uwsgi-tut.

Prerequisites

In order to deploy your web application, you need a server that gives you root and ssh access — in other words, a VPS (or a dedicated server, or a datacenter lease…). If you’re looking for a great VPS service for a low price, I recommend Hetzner Cloud (reflink [1]), which offers a pretty good entry-level VPS for €3.49 + VAT / month (with higher plans available for equally good prices). If you want to play along at home, without buying a VPS, you can create a virtual machine on your own, or use Vagrant with a Vagrant box for Fedora 35 (fedora/35-cloud-base).

Your server should also run a modern Linux-based operating system. This guide was written and tested on:

  • Ubuntu 18.04 LTS, 20.04 LTS or newer

  • Debian 10 (buster), 11 (bullseye) or newer

  • Fedora 33 or newer (with SELinux enabled and disabled)

  • CentOS 7 (with SELinux enabled and disabled) — manual guide should also work on RHEL 7.

  • AlmaLinux 8 (with SELinux enabled and disabled) — manual guide should also work on RHEL 8. Referred to as “EL8” collectively with Rocky Linux.

  • Rocky Linux 8 (with SELinux enabled and disabled) — manual guide should also work on RHEL 8. Referred to as “EL8” collectively with AlmaLinux.

  • Arch Linux

Debian 8 (jessie) 9 (stretch), Ubuntu 16.04 LTS, and Fedora 24 through 32 are not officially supported, even though they still probably work.

What if you’re using Docker? This guide does not apply, you may want to read Deploying Python Web Applications with Docker instead.

Users of other Linux distributions (and perhaps other Unix flavors) can also follow this tutorial. This guide assumes systemd as your init system; if you are not using systemd, you will have to get your own daemon files somewhere else. In places where the instructions are split three-way, try coming up with your own, reading documentation and config files; the Arch Linux instructions are probably the closest to upstream (but not always). Unfortunately, all Linux distributions have their own ideas when it comes to running and managing nginx and uWSGI.

nginx and uWSGI are considered best practices by most people. nginx is a fast, modern web server, with uWSGI support built in (without resorting to reverse proxying). uWSGI is similarly aimed at speed. The Emperor mode of uWSGI is recommended for init system integration by the uWSGI team, and it’s especially useful for multi-app deployments. (This guide is opinionated.)

Automate everything: Ansible Playbook

A Playbook that automates everything in this tutorial is available. ci-status

How to use

  1. Install Ansible on your control computer (not necessarily the destination server).

  2. Clone the Playbook from GitHub.

  3. Read README.md. You should also understand how Ansible works.

  4. Configure (change three files: hosts, group_vars/all, and group_vars/os_<destination OS>

  5. Make sure all the dependencies are installed on your destination server

  6. Run ansible-playbook -v nginx-uwsgi.yml -i hosts and watch magic happen.

  7. Skip over to End result and test your site.

The manual guide

Even though I personally recommend the Playbook as a much less error-prone way to set up your app, it might not be compatible with everyone’s system, or otherwise be the wrong solution. The original manual configuration guide is still maintained.

Even if you are using the Playbook, you should still read this to find out what happens under the hood, and to find out about other caveats/required configuration changes.

Getting started

Start by installing Python 3 (with venv), nginx and uWSGI. I recommend using your operating system’s packages. Make sure you are downloading the latest versions available for your OS (update the package cache first). For uWSGI, we need the logfile and python3 plugins. (Arch Linux names the python3 plugin python; the logfile plugin may be built-in — check with your system repositories!). I’ll also install Git to clone the tutorial app, but it’s optional if your workflow does not involve Git.

Ubuntu, Debian:

apt install python3 python3-venv uwsgi uwsgi-emperor uwsgi-plugin-python3 nginx-full git

Fedora:

dnf install python3 uwsgi uwsgi-plugin-python3 uwsgi-logger-file nginx git

CentOS 7:

yum install epel-release
yum install python36 uwsgi uwsgi-plugin-python36 uwsgi-logger-file nginx git wget

EL8 (AlmaLinux 8, Rocky Linux 8):

dnf install epel-release
dnf install python36 uwsgi uwsgi-plugin-python3 uwsgi-logger-file nginx git wget

Arch Linux:

pacman -S python uwsgi uwsgi-plugin-python nginx git

Preparing your application

This tutorial will work for any web framework. I will use a really basic Flask app that has just one route (/), a static hello.png file and a favicon.ico for demonstration purposes. The app is pretty basic, but all the usual advanced features (templates, user logins, database access, etc.) would work without any other web server-related config. Note that the app does not use app.run(). While you could add it, it would be used for local development and debugging only, and would have to be prepended by if __name__ == '__main__': (if it wasn’t, that server would run instead of uWSGI, which is bad)

The app will be installed somewhere under the /srv directory, which is a great place to store things like this. I’ll choose /srv/myapp for this tutorial, but for real deployments, you should use something more distinguishable — the domain name is a great idea.

If you don’t use Flask, this tutorial also has instructions for other web frameworks (Django, Pyramid, Bottle) in the configuration files; it should be adjustable to any other WSGI-compliant framework/script nevertheless.

We’ll start by creating a virtual environment, which is very easy with Python 3:

mkdir /srv/myapp
python3 -m venv --prompt myapp /srv/myapp/venv

(The --prompt option is not supported on some old versions of Python, but you can just skip it if that’s the case, it’s just to make the prompt after source bin/activate more informative.)

Now, we need to put our app there and install requirements. An example for the tutorial demo app:

cd /srv/myapp
git clone https://github.com/Kwpolska/flask-demo-app appdata
venv/bin/pip install -r appdata/requirements.txt

I’m storing my application data in the appdata subdirectory so that it doesn’t clutter the virtual environment (or vice versa). You may also install the uwsgi package in the virtual environment, but it’s optional.

What this directory should be depends on your web framework. For example, for a Django app, you should have an appdata/manage.py file (in other words, appdata is where your app structure starts). I also assumed that the appdata folder should have a static subdirectory with all static files, including favicon.ico if you have one (we will add support for both in nginx).

At this point, you should chown this directory to the user and group your server is going to run as. This is especially important if uwsgi and nginx run as different users (as they do on Fedora). Run one of the following commands:

Ubuntu, Debian:

chown -R www-data:www-data /srv/myapp

Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux):

chown -R uwsgi:nginx /srv/myapp

Arch Linux:

chown -R http:http /srv/myapp

Configuring uWSGI and nginx

We need to write a configuration file for uWSGI and nginx.

uWSGI configuration

Start with this, but read the notes below and change the values accordingly:

[uwsgi]
socket = /srv/myapp/uwsgi.sock
chmod-socket = 775
chdir = /srv/myapp/appdata
master = true
binary-path = /srv/myapp/venv/bin/uwsgi
virtualenv = /srv/myapp/venv
module = flaskapp:app
uid = www-data
gid = www-data
processes = 1
threads = 1
plugins = python3,logfile
logger = file:/srv/myapp/uwsgi.log

Save this file as:

  • Ubuntu, Debian: /etc/uwsgi-emperor/vassals/myapp.ini

  • Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux): /etc/uwsgi.d/myapp.ini

  • Arch Linux: /etc/uwsgi/vassals/myapp.ini (create the directory first and chown it to http: mkdir -p /etc/uwsgi/vassals; chown -R http:http /etc/uwsgi/vassals)

The options are:

  • socket — the socket file that will be used by your application. It’s usually a file path (Unix domain socket). You could use a local TCP socket, but it’s not recommended.

  • chdir — the app directory.

  • binary-path — the uWSGI executable to use. Remove if you didn’t install the (optional) uwsgi package in your virtual environment.

  • virtualenv — the virtual environment for your application.

  • module — the name of the module that houses your application, and the object that speaks the WSGI interface, separated by colons. This depends on your web framework:

    Framework Flask, Bottle Django Pyramid
    Package module where app is defined project.wsgi
    (project is the package with settings.py)    		 module where app is defined
    Callable Flask: app instance
    Bottle: app = bottle.default_app()    		 application app = config.make_wsgi_app()
    Module package:app project.wsgi:application package:app
    Caveats Make sure app is not in an if __name__ == '__main__': block Add environment variable for settings:
    env = DJANGO_SETTINGS_MODULE=project.settings    		 Make sure app is not in an if __name__ == '__main__': block (the demo quickstart does that!)

  • uid and gid — the names of the user account to use for your server. Use the same values as in the chown command above.

  • processes and threads — control the resources devoted to this application. Because this is a simple hello app, I used one process with one thread, but for a real app, you will probably need more (you need to see what works the best; there is no algorithm to decide). Also, remember that if you use multiple processes, they don’t share memory (you need a database to share data between them).

  • plugins — the list of uWSGI plugins to use. For Arch Linux, use plugins = python (the logfile plugin is always active). For CentOS 7 only (i.e. not for EL8), use plugins = python36.

  • logger — the path to your app-specific logfile. (Other logging facilities are available, but this one is the easiest, especially for multiple applications on the same server)

  • env — environment variables to pass to your app. Useful for configuration, may be specified multiple times. Example for Django: env = DJANGO_SETTINGS_MODULE=project.settings

You can test your configuration by running uwsgi --ini /path/to/myapp.ini (disable the logger for stderr output or run tail -f /srv/myapp/uwsgi.log in another window).

If you’re using Fedora, CentOS, or EL8, there are two configuration changes you need to make globally: in /etc/uwsgi.ini, disable the emperor-tyrant option (which we don’t need, as it sets uid/gid for every process based on the owner of the related .ini config file — we use one global setup) and set gid = nginx. We’ll need this so that nginx can talk to your socket.

nginx configuration

We need to configure our web server. Here’s a basic configuration that will get us started:

Save this file as:

  • Ubuntu, Debian: /etc/nginx/sites-enabled/myapp.conf

  • Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux): /etc/nginx/conf.d/myapp.conf

  • Arch Linux: add include /etc/nginx/conf.d/*.conf; to your http directive in /etc/nginx/nginx.conf and use /etc/nginx/conf.d/myapp.conf

server {
    # for a public HTTP server:
    listen 80;
    # for a public HTTPS server:
    # listen 443 ssl;
    server_name localhost myapp.local;
    location / {
        include uwsgi_params;
        uwsgi_pass unix:/srv/myapp/uwsgi.sock;
    }
    location /static {
        alias /srv/myapp/appdata/static;
    }
    location /favicon.ico {
        alias /srv/myapp/appdata/static/favicon.ico;
    }
}

Note that this file is a very basic and rudimentary configuration. This configuration is fine for local testing, but for a real deployment, you will need to adjust it:

  • set listen to 443 ssl and create a http→https redirect on port 80 (you can get a free SSL certificate from Let’s Encrypt; make sure to configure SSL properly).

  • set server_name to your real domain name

  • you might also want to add custom error pages, log files, or change anything else that relates to your web server — consult other nginx guides for details

  • nginx usually has some server already enabled by default — edit /etc/nginx/nginx.conf or remove their configuration files from your sites directory to disable it

Service setup

After you’ve configured uWSGI and nginx, you need to enable and start the system services.

For Arch Linux

All you need is:

systemctl enable nginx emperor.uwsgi
systemctl start nginx emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi

For Fedora, CentOS, EL8

Make sure you followed the extra note about editing /etc/uwsgi.ini earlier and run:

systemctl enable nginx uwsgi
systemctl start nginx uwsgi

Verify the service is running with systemctl status uwsgi

If you disabled SELinux, this is enough to get an app working and you can skip over to the next section.

If you want to use SELinux, you need to do the following to allow nginx to read static files:

setenforce 0
chcon -R system_u:system_r:httpd_t:s0 /srv/myapp/appdata/static
setenforce 1

We now need to install a SELinux policy (that I created for this project; updated 2020-05-02) to allow nginx and uWSGI to communicate. Download nginx-uwsgi.pp and run:

semodule -i nginx-uwsgi.pp

Hopefully, this is enough (you can delete the file). In case it isn’t, please read SELinux documentation, check audit logs, and look into audit2allow.

For Ubuntu and Debian

Ubuntu and Debian (still!) use LSB services for uWSGI. Because LSB services are awful, we’re going to set up our own systemd-based (native) service.

Start by disabling the LSB service that comes with Ubuntu and Debian:

systemctl stop uwsgi-emperor
systemctl disable uwsgi-emperor

Copy the .service file from the uWSGI systemd documentation to /etc/systemd/system/emperor.uwsgi.service. Change the ExecStart line to:

ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi-emperor/emperor.ini

You can now reload systemd daemons and enable the services:

systemctl daemon-reload
systemctl enable nginx emperor.uwsgi
systemctl reload nginx
systemctl start emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi. (Ignore the warning about no request plugin)

End result

Your web service should now be running at http://localhost/ (or wherever you set up server to listen).

If you used the demo application, you should see something like this (complete with the favicon and image greeting):

/images/nginx-uwsgi-demo.png

If you want to test with cURL:

curl -v http://localhost/
curl -I http://localhost/favicon.ico
curl -I http://localhost/static/hello.png

Troubleshooting

Hopefully, everything works. If it doesn’t:

  • Check your nginx, system (journalctl, systemctl status SERVICE) and uwsgi (/srv/myapp/uwsgi.log) logs.

  • Make sure you followed all instructions.

  • If you get a default site, disable that site in nginx config (/etc/nginx/nginx.conf or your sites directory).

  • If you have a firewall installed, make sure to open the ports your web server runs on (typically 80/443). For firewalld (Fedora, CentOS, EL8):

firewall-cmd --add-service http
firewall-cmd --add-service https

  • If it still does not work, feel free to ask in the comments, mentioning your distribution, installation method, and what doesn’t work.

Can I use Docker?

This blog post is written for systems running standalone. But Docker is a bit special, in that it offers a limited subset of OS features this workflow expects. The main issue is with user accounts, which generally work weird in Docker, and I had issues with setuid/setgid as used by uWSGI. Another issue is the lack of systemd, which means that another part of the tutorial fails to apply.

This tutorial uses uWSGI Emperor, which can run multiple sites at once, and offers other management features (such as seamless code restarts with touch /etc/uwsgi/vassals/myapp.ini) that may not be useful or easy to use in a Docker environment. You’d probably also run uWSGI and nginx in separate containers in a typical Docker deployment.

Regardless, many parts of this tutorial can be used with Docker, although with the aforementioned adjustments. I have done some work on this topic. This tutorial has an Ansible Playbook attached, and the tutorial/playbook are compatible with five Linux distros in multiple versions. How do I know that there were no unexpected bugs in an older version? I could grab a Vagrant image or set up a VM. I do that when I need specific testing, but doing it for each of the distros on each update would take at least half an hour, probably even more. Yeah, that needs automating. I decided to use GitHub Actions for the CI, which can run anything, as long as you provide a Dockerfile.

The Docker images were designed to support running the Playbook and testing it. But the changes, setups and patches could be a good starting point if you wanted to make your own Docker containers that could run in production. You can take a look at the Docker files for CI The images support all 5 distros using their base images, but you could probably use Alpine images, or the python docker images; be careful not to mix Python versions in the latter case.

In 2026, I gave up on setting up Python applications with uWSGI using the system Python and virtual environments. I’ve switched all my deployments to Docker containers with Gunicorn as the application server. Nowadays, Docker is a much better choice, as not having to deal with system Python and system virtual environments makes deployments much easier. I wrote up my experiences and shared the configuration files in Deploying Python Web Applications with Docker.

]]>PythonAnsibleArch LinuxDjangoFlaskguideInternetLinuxnginxPythonsystemduWSGIPython Apps the Right Way: entry points and scriptsChris Warrickhttps://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/Mon, 15 Sep 2014 16:00:00 GMThttps://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/ There are multiple ways to write an app in Python. However, not all of them provide your users with the best experience. One of the problems some people encounter is writing launch scripts. The best way to handle this is the Entry Points mechanism of Setuptools, and a __main__.py file. It’s quite easy to implement. If you’re interested, read on to learn more! There are multiple ways to write an app in Python. However, not all of them provide your users with the best experience.

One of the problems some people encounter is writing launch scripts. The best way to handle this is the Entry Points mechanism of Setuptools, and a __main__.py file. It’s quite easy to implement. If you’re interested, read on to learn more!

Requirements and Desired Results

You will need:

(entry_points_project is also where the README and other auxiliary files go, while my_project contains all the Python code.)

When you’re done, you will have a project that can be executed by:

  • python -m my_project

  • my_project

Provided that you have your Python directory and its Scripts\ subdirectory on the %PATH%, this will also work in Windows.

Looking for a project template?

If you want to create a well-structured project with release automation and some other goodies, check out my Python Project Template.

Step 1: create a __main__.py file

In order to implement the first desired result, you need to create a __main__.py file in your package. This file needs to contain a main() function that takes no arguments, and also a special passage to determine code to run:

entry_points_project/my_project/__main__.py (Source)

import sys
def main(args=None):
    """The main routine."""
    if args is None:
        args = sys.argv[1:]
    print("This is the main routine.")
    print("It should do something interesting.")
    # Do argument parsing here (eg. with argparse) and anything else
    # you want your project to do. Return values are exit codes.
if __name__ == "__main__":
    sys.exit(main())

  1. The if __name__ == "__main__": idiom, as documented here, is used to check whether this is executed as the top-level file, or if it has been imported by someone else (in this case, executing the main() function is not always intended).

  2. The main() function must not take any arguments, because that’s how entry_points executes things.

Step 2: adjust setup.py accordingly

This is the real deal: create the entry points in your setup.py file.

entry_points_project/setup.py (Source)

from setuptools import setup
setup(
    name="my_project",
    version="0.1.0",
    packages=["my_project"],
    entry_points={
        "console_scripts": [
            "my_project = my_project.__main__:main"
        ]
    },
)

  1. You must use setuptools, otherwise this won’t work.

  2. The most important piece of code is the entry_points declaration (unsurprisingly).

  3. The declaration reads

"name_of_executable = module.with:function_to_execute"

  1. If you are developing a GUI application (in Tkinter, PyQt/PySide, wxPython, PyGTK, PyGame…), you should change the declaration to gui_scripts. On *nix, this makes no difference, but on Windows, it means that running your script by opening the created .exe files does not show a console window. Note that stdout/stderr do not work in that mode under Windows, which can lead to spurious application crashes. (GUI-only processes cannot use stdout/stderr because they don’t have a console attached)

  2. You can create multiple scripts this way. You can also have multiple console_scripts and gui_scripts in one setup file.

All code samples are freely reusable, but if you mention where you got them from, it’d be really nice.

]]>Pythonbest practicesdevelguidePythonAdventures in Windows: Music Player DaemonChris Warrickhttps://chriswarrick.com/blog/2013/09/01/mpd-on-windows/Sun, 01 Sep 2013 17:15:00 GMThttps://chriswarrick.com/blog/2013/09/01/mpd-on-windows/ Recently, I had to reinstall Windows. One of the things I had to set up was MPD, the Music Player Daemon. This is a short guide on how to do this. Recently, I had to reinstall Windows. One of the things I had to set up was MPD, the Music Player Daemon.

This is a short guide on how to do this.

Step 1: get MPD and MPC

Download MPD (mpd-x.y.z-win32.zip, replacing x.y.z with whatever is the current available version of this file) and MPC (mpc-x.y-win32-zip). I suggest using C:\mpd as your base path. This post assumes you actually used this value.

Put everything from the /mpd-x.y.z-win32/bin/ directory of the mpd-x.y.z-win32.zip archive into your base path (C:\mpd). From the mpc-x.y-win32.zip archive, take mpc.exe and libmpdclient-2.dll and put them there.

Step 2: create a config file

You need to create a configuration file. A sample one is included in /mpd-x.y.z-win32/doc/mpdconf.sample. Use the winmm output type.

Warning: Windows Notepad will not work, because it does not understand Unix-style line endings (\n instead of \r\n).

For db_file and log_file, I recommend a file in C:\mpd\data\. I also recommend saving your config as C:\mpd\data\mpd.conf (in fact, this is yet another assumption we make in this guide)

Multi Boot Corner: keep parts of the config the same

Because I have a dual-boot system, there are some directions for multi-boot users.

Your music_directory, playlist_directory and state_file must be the same on all the systems in your multi-boot environment. It is recommended to use the same db_file.

Your audio outputs must have the same names.

Step 3: test your configuration

Open up two Command Prompt windows. In the first one, run:

cd C:\mpd
mpd data\mpd.conf

If you see this message:

Failed to load database: Failed to open database file "<name>": No such file or directory

then you can ignore it. If you see another message, fix whatever it states. You may also use --stdout or -v to get more output messages.

Multi Boot path

If you followed the Multi Boot instructions correctly, you might be hearing music already. If you do not, use your other Command Prompt window and run:

cd C:\mpd
mpc play

If it outputs the name of a song, a [playing] line, all is well (if you cannot hear music, unmute your audio).

If you do not have a mpd database yet, use the Single path.

If MPD works, press Ctrl+C in the window where you ran ``mpd datampd.conf.``

Single path

In the other Command Prompt window, run:

cd C:\mpd
mpc update
mpc add "FILENAME"
mpc play

(FILENAME should be a name of a file in your music library. Make sure to keep the quotes.)

And now you should hear music. If you do not, try to unmute, then read the output of mpc and try to fix it.

If MPD works, press Ctrl+C in the window where you ran ``mpd datampd.conf.``

Step 3.5: test GMPC

Start the Gnome Music Player Client. Click Forward, Connect, Forward, Close and all should be well.

Step 4: create a Windows service

In order to run the daemon in the daemon mode, i.e. hidden, you need to perform some extra steps. You cannot use Startup in the Start Menu, you also cannot use the Run registry keys (this results in an ugly Command Prompt window running all the time that you cannot close or hide in the tray).

How to solve this problem? Just run it as a Windows service. That is very easy to do.

Open a Command Prompt window as an administrator (right-click on Command Prompt in the Start Menu/on the Start Screen and choose Run as administrator).

In that window, run one very easy command (make sure to copy-paste this exactly!):

sc create mpd binPath= "c:\mpd\mpd.exe c:\mpd\data\mpd.conf"

If it said:

  • [SC] CreateService SUCCESS, congratulations — you are almost done! Skip down to Service configuration.

  • [SC] OpenSCManager FAILED 5: Access is denied., you need an Administrator Command Prompt.

  • something else, check your spelling or Google the error.

Service configuration

That is the easiest part of this guide.

Start the Services console. You can get to it by typing services.msc into the Start Menu/Screen search (Run for XP or older).

In the tool, find the mpd service. Go to the Log On tab, choose This account: and enter your credentials there. Hit Apply and go to the General tab, on which you should choose the Startup type to be Automatic (Delayed Start). Finish by pressing Start. MPD should be running and configured properly. You can now hit OK and close the Services window, along with the command prompts you have open.

Warning: if you change your Windows password, you need to change your password in the Services console as well!

Just in case: uninstalling the service

First off, stop the service (in the Services console or through sc stop mpd in an administrator command prompt).

Then, run sc delete mpd.

]]>WindowsguidempdWindows