Learning curves and the deception of “simple”

Microsoft Word is simple, and a great beginner’s writing tool. You can make text bold with a single click. You can also make it blue in two clicks. But it’s easy to make an inconsistent mess. To make section headers, many users may just make the text bold and a bit bigger, without any consistency or semantics [1]. Making a consistent document with semantic formatting is hard in Word. Adding section numbering requires you to select a heading and turn it into a list. There’s also supposedly some magic involved, that magic doesn’t work for me, and I have to tell Word to update the heading style. Even if you try doing things nicely, Word will randomly break, mess up the styles, mix up styles and inline ad-hoc formatting, and your document may look differently on different computers.

LaTeX is very confusing to a beginner, and has a massive learning curve. And you can certainly write \textbf{hello} everywhere. But with some learning, you’ll be producing beautiful documents. You’ll define a \code{} command that makes code monospace and adds a border today, but it might change the background and typeset in Comic Sans tomorrow if you so desire. You’ll use packages that can render code from external files with syntax highlighting. Heading numbering is on by default, but it can easily be disabled for a section. LaTeX can also automatically put new sections on new pages, for example. LaTeX was built for scientific publishing, so it has stellar support for maths and bibliographies, among other things.

Let’s now talk about programming. Python is simple, and a great beginner’s programming language. You can write hello world in a single line of code. The syntax is simpler, there are no confusing leftovers from C (like the index-based for loop) or machine-level code (like break in switch), no pointers in sight. You also don’t need to write classes at all; you don’t need to write a class only to put a public static void main(String[] args) method there [2]. You don’t need an IDE, you can just write code using any editor (even notepad.exe will do for the first day or so), you can save it as a .py file and run it using python whatever.py.

Your code got more complicated? No worry, you can split it into multiple .py files, use import name_of_other_file_without_py and it will just work. Do you need more structure, grouping into folders perhaps? Well, forget about python whatever.py, you must use python -m whatever, and you must cd to where your code is, or mess with PYTHONPATH, or install your thing with pip. This simple yet common action (grouping things into folders) has massively increased complexity.

The standard library is not enough [3] and you need a third-party dependency? You find some tutorial that tells you to pip install, but pip will now tell you to use apt. And apt may work, but it may give you an ancient version that does not match the tutorial you’re reading. Or it may not have the package. Or the Internet will tell you not to use Python packages from apt. So now you need to learn about venvs (which add more complexity, more things to remember; most tutorials teach activation, venvs are easy to mess up via basic operations like renaming a folder, and you may end up with a venv in git or your code in a venv). Or you need to pick one of the many one-stop-shop tools to manage things.

In other ecosystems, an IDE is often a necessity, even for beginners. The IDE will force you into a project system (maybe not the best or most common one by default, but it will still be a coherent project system). Java will force you to make more than one file with the “1 public class = 1 file” rule, and it will be easy to do so, you won’t even need an import.

Do you want folders? In Java or C#, you just create a folder in the IDE, and create a class there. The new file may have a different package/namespace, but the IDE will help you to add the correct import/using to the codebase, and there is no risk of you using too many directories (including something like src) or using too few (not making a top-level package for all your code) that will require correcting all imports. The disruption from adding a folder in Java or C# is minimal.

The project system will also handle third-party packages without you needing to think about where they’re downloaded or what a virtual environment is and how to activate it from different contexts. A few clicks and you’re done. And if you don’t like IDEs? Living in the CLI is certainly possible in many ecosystems, they have reasonable CLI tools for common management tasks, as well as building and running your project.

PEP 723 solves a very niche problem: dependency management for single-file programs. Improving life for one-off things and messy code was apparently more important to the packaging community than any other improvements for big projects.

By the way, you could adapt this lesson to static and dynamic typing. Dynamic typing is easier to get started with and requires less typing, but compile-type checking can prevent many bugs — bugs that require higher test coverage to catch with dynamic typing. That’s why the JS world has TypeScript, that’s why mypy/pyright/typing has gained a lot of mindshare in the Python world.

…is looking pretty bleak

On the other hand, if you look at the 2023 contribution graphs for most packaging tools, you might be worried about the state of the packaging ecosystem.

• Pip has had a healthy mix of contributors and a lot of commits going into it.

• Pipenv and setuptools have two lead committers, but still a healthy amount of commits.

• Hatch, however, is a one-man-show: Ofek Lev (the project founder) made 184 commits, the second place belongs to Dependabot with 6 commits, and the third-place contributor (who is a human) has five commits. The bus factor of Hatch and Hatchling is 1.

The non-PyPA tools aren’t doing much better:

• Poetry has two top contributors, but at least there are four human contributors with a double-digit number of commits.

• PDM is a one-man-show, like Hatch.

• Rye has one main contributor, and three with a double-digit number of commits; note it’s pretty new (started in late April 2023) and it’s not as popular as the others.

The classic stack

The classic Python packaging stack consists of many semi-related tools. Setuptools, probably the oldest tool of the group, and itself based on distutils, which is part of the standard library (although it will be removed in Python 3.12), is responsible for installing a single package. It previously used setup.py files to do its job, which required arbitrary code execution. It then added support for non-executable metadata specification formats: setup.cfg, and also pyproject.toml (partially still in beta). However, you aren’t supposed to use setup.py files directly these days, you’re supposed to be using pip. Pip installs packages, usually from the PyPI, but it can also support other sources (such as git repositories or the local filesystem). But where does pip install things? The default used to be to install globally and system-wide, which meant you could introduce conflicts between packages installed by pip and apt (or whatever the system package manager is). Even with a user-wide install (which pip is likely to attempt these days), you can still end up with conflicts, and you can also have conflicts in which package A requests X version 1.0.0, but package B expects X version 2.0.0—but A and B are not at all related and could live separately with their preferred version of X. Enter venv, a standard library descendant of virtualenv, which can create a lightweight virtual environment for packages to live in. This virtual environment gives you the separation from system packages and from different environments, but it is still tied to the system Python in some ways (and if the system Python disappears, the virtual environment stops working).

A few extra tools would be used in a typical packaging workflow. The wheel package enhances Setuptools with the ability to generate wheels, which are ready-to-install (without running setup.py). Wheels can either be pure-Python and be installed anywhere, or they can contain pre-compiled extension modules (things written in C) for a given OS and Python (and there’s even a standard that allows building and distributing one wheel for all typical Linux distros). The wheel package should be an implementation detail, something existing inside Setuptools and/or pip, but users need to be aware of it if they want to make wheels on their system, because virtual environments produced by venv do not have wheel installed. Regular users who do not maintain their own packages may sometimes be told that pip is using something legacy because wheel is not installed, which is not a good user experience. Package authors also need twine, whose sole task is uploading source distributions or wheels, created with other tools, to PyPI (and there’s not much more to say about that tool).

…and a few extensions

Over the years, there have been a few tools that are based on things from the classic stack. For example, pip-tools can simplify dependency management. While pip freeze lets you produce a file with everything installed in your environment, there is no way to specify the dependencies you need, and get a lock file with specific versions and transitive dependencies (without installing and freezing everything), there is no easy way to skip development dependencies (e.g. IPython) when you pip freeze, and there is no workflow to update all your dependencies with just pip. pip-tools adds two tools, pip-compile which takes in requirements.in files with the packages you care about, and produces a requrirements.txt with pinned versions of them and all transitive dependencies; and also pip-sync, which can install requirements.txt and removes things not listed in it.

Another tool that might come in useful is virtualenvwrapper, which can help you manage (create and activate) virtual environments in a central location. It has a few bells and whistles (such as custom hooks to do actions on every virtualenv creation), although for basic usage, you could replace it with a single-line shell function.

Yet another tool that works alongside the classic toolset is pipx, which creates and manages virtual environments for apps written in Python. You tell it to pipx install Nikola, and it will create a virtual environment somewhere, install Nikola into it, and put a script for launching it in ~/.local/bin. While you could do it all yourself with venv and some symlinks, pipx can take care of this, and you don’t need to remember where the virtual environment is.

The scientific stack and conda

The scientific Python community have had their own tools for many years. The conda tool can manage environments and packages. It doesn’t use PyPI and wheels, but rather packages from conda channels (which are prebuilt, and expect an Anaconda-distributed Python). Back in the day, when there were no wheels, this was the easiest way to get things installed on Windows; this is not as much of a problem now with binary wheels on PyPI—but the Anaconda stack is still popular in the scientific world. Conda packages can be built with conda-build, which is separate, but closely related to conda itself. Conda packages are not compatible with pip in any way, they do not follow the packaging standards used by other tools. Is this good? No, because it makes integrating the two worlds harder, but also yes, because many problems that apply to scientific packages (and their C/C++ extension modules, and their high-performance numeric libraries, and other things) do not apply to other uses of Python, so having a separate tool lets people focusing the other uses simplify their workflows.

The new tools

A few years ago, new packaging tools appeared. Now, there were lots of “new fancy tools” introduced in the past, with setuptools extending distutils, then distribute forking setuptools, then distribute being merged back…

The earliest “new tool” was Pipenv. Pipenv had really terrible and misleading marketing, and it merged pip and venv, in that Pipenv would create a venv and install packages in it (from Pipfile or Pipfile.lock). Pipenv can place the venv in the project folder, or hide it somewhere in the project folder (the latter is the default). However, Pipenv does not handle any packages related to packaging your code, so it’s useful only for developing non-installable applications (Django sites, for example). If you’re a library developer, you need setuptools anyway.

The second new tool was Poetry. It manages environments and dependencies in a similar way to Pipenv, but it can also build .whl files with your code, and it can upload wheels and source distributions to PyPI. This means it has pretty much all the features the other tools have, except you need just one tool. However, Poetry is opinionated, and its opinions are sometimes incompatible with the rest of the packaging scene. Poetry uses the pyproject.toml standard, but it does not follow the standard specifying how metadata should be represented in a pyproject.toml file (PEP 621), instead using a custom [tool.poetry] table. This is partly because Poetry came out before PEP 621, but the PEP was accepted over 2 years ago—the biggest compatibility problem is Poetry’s node-inspired ~ and ^ dependency version markers, which are not compatible with PEP 508 (the dependency specification standard). Poetry can package C extension modules, although it uses setuptools’ infrastructure for this (and requires a custom build.py script).

Another similar tool is Hatch. This tool can also manage environments (it allows multiple environments per project, but it does not allow to put them in the project directory), and it can manage packages (but without lockfile support). Hatch can also be used to package a project (with PEP 621-compliant pyproject.toml files) and upload it to PyPI. It does not support C extension modules.

A tool that tries to be a simpler re-imagining of Setuptools is Flit. It can build and install a package using a pyproject.toml file. It also supports uploads to PyPI. It lacks support for C extension modules, and it expects you to manage environments on your own.

There’s one more interesting (albeit not popular or well-known) tool. This tool is PDM. It can manage venvs (but it defaults to the saner .venv location), manage dependencies, and it uses a standards-compliant pyproject.toml. There’s also a curious little feature called PEP 582 support, which we’ll talk about later.

How to use Python from a virtual environment?

There are two ways to do this. The first one is to activate it, by running the activate shell script installed in the environment’s bin directory. Another is to run the python executable (or any other script in the bin directory) directly from the venv. [2]

Activating venvs directly is more convenient for developers, but it also has some problems. Sometimes, activation fails to work, due to the shell caching the location of things in $PATH. Also, beginners are taught to activate and run python, which means they might be confused and try to use activate in scripts or cronjobs (but in those environments, you should not activate venvs, and instead use the Python executable directly). Virtual environment activation is more state you need to be aware of, and if you forget about it, or if it breaks, you might end up messing up your user-wide (or worse, system-wide) Python packages.

How are (system) Pythons and virtual environments related?

The virtual environment depends very tightly on the (system/global/pyenv-installed) Python used to create it. This is good for disk-space reasons (clean virtual environments don’t take up very much space), but this also makes the environment more fragile. If the Python used to create the environment is removed, the virtual environment stops working. If you fully manage your own Python, then it’s probably not going to happen, but if you depend on a system Python, upgrading packages on your OS might end up replacing Python 3.10 with Python 3.11. Some distributions (e.g. Ubuntu) would only make a jump like this on a new distribution release (so you can plan ahead), some of them (e.g. Arch) are rolling-release and a regular system upgrade may include a new Python, whereas some (e.g. Homebrew) make it even worse by using paths that include the patch Python version (3.x.y), which cause virtual environments to break much more often.

How to manage virtual environments?

The original virtualenv tool, and its simplified standard library rewrite venv, allow you to put a virtual environment anywhere in the file system, as long as you have write privileges there. This has led to people and tools inventing their own standards. Virtualenvwrapper stores environments in a central location, and does not care about their contents. Pipenv and poetry allow you to choose (either a central location or the .venv directory in the project), and environments are tied to a project (they will use the project-specific environment if you’re in the project directory). Hatch stores environments in a central location, and it allows you to have multiple environments per project (but there is no option to share environments between projects).

Brett Cannon has recently done a survey, and it has shown the community is split on their workflows: some people use a central location, some put them in the project directory, some people have multiple environments with different Python versions, some people reuse virtualenvs between projects… Everyone has different needs, and different opinions. For example, I use a central directory (~/virtualenvs) and reuse environments when working on Nikola (sharing the same environment between development and 4 Nikola sites). But on the other hand, when deploying web apps, the venv lives in the project folder, because this venv needs to be used by processes running as different users (me, root, or the service account for the web server, which might have interactive login disabled, or whose home directory may be set to something ephemeral).

So: does Python need virtual environments? Perhaps looking how other languages handle this problem can help us figure this out for Python?

JavaScript/Node.js (with npm)

There are two tools for dealing with packages in the Node world, namely npm and Yarn. The npm CLI tool is shipped with Node, so we’ll focus on it.

Let’s create a project:

$ mkdir mynpmproject
$ cd mynpmproject
$ npm init
…answer a few questions…
$ ls
package.json

We’ve got a package.json file, which has some metadata about our project (name, version, description, license). Let’s install a dependency:

$ npm install --save is-even

added 5 packages, and audited 6 packages in 2s

found 0 vulnerabilities

The mere existence of an is-even package is questionable; the fact that it includes four dependencies is yet another, and the fact that it depends on is-odd is even worse. But this post isn’t about is-even or the Node ecosystem’s tendency to use tiny packages for everything (but I wrote one about this topic before). Let’s look at what we have in the filesystem:

$ ls
node_modules/  package.json  package-lock.json
$ ls node_modules
is-buffer/  is-even/  is-number/  is-odd/  kind-of/

Let’s also take a peek at the package.json file:

{
  "name": "mynpmproject",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "is-even": "^1.0.0"
  }
}

Our package.json file now lists the dependency, and we’ve also got a lock file (package-lock.json), which records all the dependency versions used for this install. If this file is kept in the repository, any future attempts to npm install will use the dependency versions listed in this file, ensuring everything will work the same as it did originally (unless one of those packages were to get removed from the registry).

Let’s try writing a trivial program using the module and try running it:

$ cat index.js
var isEven = require('is-even');
console.log(isEven(0));

$ node index.js
true

Let’s try removing is-odd to demonstrate how badly designed this package is:

$ rm -rf node_modules/is-odd
$ node index.js
node:internal/modules/cjs/loader:998
  throw err;
  ^

Error: Cannot find module 'is-odd'
Require stack:
- /tmp/mynpmproject/node_modules/is-even/index.js
- /tmp/mynpmproject/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
    at Module._load (node:internal/modules/cjs/loader:841:27)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/tmp/mynpmproject/node_modules/is-even/index.js:10:13)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/tmp/mynpmproject/node_modules/is-even/index.js',
    '/tmp/mynpmproject/index.js'
  ]
}

Node.js v18.12.1

$ pwd
/tmp/mynpmproject
$ npm install

added 1 package, and audited 6 packages in 436ms

found 0 vulnerabilities
$ node /tmp/mynpmproject/index.js
true
$ cd ~
$ node /tmp/mynpmproject/index.js
true

$ su -
# node /tmp/mynpmproject/index.js
true
# sudo -u nginx node /tmp/mynpmproject/index.js
true

$ cd /tmp
$ mv mynpmproject mynodeproject
$ node /tmp/mynodeproject/index.js
true

C#/.NET (with dotnet CLI/MSBuild)

In modern .NET, the One True Tool is the dotnet CLI, which uses MSBuild for most of the heavy lifting. (In the classic .NET Framework, the duties were split between MSBuild and NuGet.exe, but let’s focus on the modern workflow.)

Let’s create a project:

$ mkdir mydotnetproject
$ cd mydotnetproject
$ dotnet new console
The template "Console App" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /tmp/mydotnetproject/mydotnetproject.csproj...
  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 92 ms).
Restore succeeded.
$ ls
mydotnetproject.csproj  obj/  Program.cs

We get three things: a mydotnetproject.csproj file, which defines a few properties of our project; Program.cs, which is a hello world program, and obj/, which contains a few files you don’t need to care about.

Let’s try adding a dependency. For a pointless example, but slightly more reasonable than the JS one, we’ll use AutoFixture, which brings in a dependency on Fare. If we run dotnet add package AutoFixture, we get some console output, and our mydotnetproject.csproj now looks like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AutoFixture" Version="4.17.0" />
  </ItemGroup>

</Project>

The first <PropertyGroup> specifies what our project is (Exe = something you can run), specifies the target framework (.NET 6.0 [3]), and enables a few opt-in features of C#. The second <ItemGroup> was inserted when we installed AutoFixture.

We can now write a pointless program in C#. Here’s our new Program.cs:

using AutoFixture;
var fixture = new Fixture();
var a = fixture.Create<int>();
var b = fixture.Create<int>();
var result = a + b == b + a;
Console.WriteLine(result ? "Math is working": "Math is broken");

(We could just use C#’s/.NET’s built-in random number generator, AutoFixture is complete overkill here—it’s meant for auto-generating test data, with support for arbitrary classes and other data structures, and we’re just getting two random ints here. I’m using AutoFixture for this example, because it’s simple to use and demonstrate, and because it gets us a transitive dependency.)

And now, we can run it:

$ dotnet run
Math is working

If we want something that can be run outside of the project, and possibly without .NET installed on the system, we can use dotnet publish. The most basic scenario:

$ dotnet publish
$ ls bin/Debug/net6.0/publish
AutoFixture.dll*  Fare.dll*  mydotnetproject*  mydotnetproject.deps.json  mydotnetproject.dll  mydotnetproject.pdb  mydotnetproject.runtimeconfig.json
$ du -h bin/Debug/net6.0/publish
424K    bin/Debug/net6.0/publish
$ bin/Debug/net6.0/publish/mydotnetproject
Math is working

You can see that we’ve got a few files related to our project, as well as AutoFixture.dll and Fare.dll, which are our dependencies (Fare.dll is a dependency of AutoFixture.dll). Now, let’s try to remove AutoFixture.dll from the published distribution:

$ rm bin/Debug/net6.0/publish/AutoFixture.dll
$ bin/Debug/net6.0/publish/mydotnetproject
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'. The system cannot find the file specified.

File name: 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'
[1]    45060 IOT instruction (core dumped)  bin/Debug/net6.0/publish/mydotnetproject

We can also try a more advanced scenario:

$ rm -rf bin obj  # clean up, just in case
$ dotnet publish --sc -r linux-x64 -p:PublishSingleFile=true -o myoutput
Microsoft (R) Build Engine version 17.0.1+b177f8fa7 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 4.09 sec).
  mydotnetproject -> /tmp/mydotnetproject/bin/Debug/net6.0/linux-x64/mydotnetproject.dll
  mydotnetproject -> /tmp/mydotnetproject/myoutput/
$ ls myoutput
mydotnetproject*  mydotnetproject.pdb
$ myoutput/mydotnetproject
Math is working
$ du -h myoutput/*
62M     myoutput/mydotnetproject
12K     myoutput/mydotnetproject.pdb
$ file -k myoutput/mydotnetproject
myoutput/mydotnetproject: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=47637c667797007d777f4322729d89e7fa53a870, for GNU/Linux 2.6.32, stripped, too many notes (256)\012- data
$ file -k myoutput/mydotnetproject.pdb
myoutput/mydotnetproject.pdb: Microsoft Roslyn C# debugging symbols version 1.0\012- data

We have a single output file that contains our program, its dependencies, and parts of the .NET runtime. We also get debugging symbols if we want to run our binary with a .NET debugger and see the associated source code. (There are ways to make the binary file smaller, and we can move most arguments of dotnet publish to the .csproj file, but this post is about Python, not .NET, so I’m not going to focus on them too much.)

Other languages and ecosystems

While we have explored two tools for two languages in depth, there are also other languages that deserve at least a mention. In the Java world, the two most commonly used tools are Maven and Gradle. Both tools can be used to manage dependencies and build artifacts that can be executed or distributed further (things like JAR files). Other tools with support for building Java projects exist, but most people just pick one of the two. The community of Scala, which is another JVM-based language, prefers sbt (which can be used for plain Java as well), but there are also Maven or Gradle users in that community. Finally, two new-ish languages which are quite popular in the recent times, Go and Rust, have first-party tooling integrated with the rest of the toolchain. The go command-line tool can accomplish many build/dependency/packaging tasks. Rust’s cargo, which ships with the standard distribution of Rust, handles dependencies, builds, running code and tests, as well as publishing your stuff to a registry.

Are those ecosystems’ tools perfect?

Not always, they have their deficiencies as well. In the Node ecosystem, packages may execute arbitrary code on install, which can be a security risk (there are some known examples, like a npm package wiping hard drives in Russia and Belarus, or another one stealing imaginary Internet money Bitcoin). Binary packages are not distributed on the npm registry directly, they’re either built with node-gyp, or have prebuilt packages downloaded via node-pre-gyp (which is a third-party tool).

In the .NET ecosystem, the tools also create an obj directory with temporary files. Those temporary files are tied to the environment they’re running in, and while the tooling will usually re-create them if something changes, it can sometimes fail and leave you with confusing errors (which can generally be solved by removing the bin and obj directories). If a package depends on native code (which is not already available on the target OS as part of a shared library), it must include binary builds in the NuGet package for all the platforms it supports, as there is no standard way to allow building something from source.

You can also find deficiencies in the tools for the other languages mentioned. Some people think Maven is terrible because it uses XML and Gradle is the way to go, and others think Gradle’s use of a Groovy-based DSL makes things much harder than they need to be and prefer Maven instead.

Enabling the future on your own machine

Let’s enable the future on my own machine. That will require one simple command:

$ eval "$(pdm --pep582)"

After that, we can initialize our project and install requests into it. Let’s try:

$ mkdir mypdmproject
$ cd mypdmproject
$ pdm init
Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /usr/bin/python (3.11)
1. /usr/bin/python3.11 (3.11)
2. /usr/bin/python2.7 (2.7)
Please select (0): 1
Using Python interpreter: /usr/bin/python3.11 (3.11)
Would you like to create a virtualenv with /usr/bin/python3.11? [y/n] (y): n
You are using the PEP 582 mode, no virtualenv is created.
For more info, please visit https://peps.python.org/pep-0582/
Is the project a library that will be uploaded to PyPI [y/n] (n): n
License(SPDX name) (MIT):
Author name (Chris Warrick):
Author email (…):
Python requires('*' to allow any) (>=3.11):
Changes are written to pyproject.toml.
$ ls
pyproject.toml
$ pdm add requests
Adding packages to default dependencies: requests
🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 5 to add, 0 to update, 0 to remove

  ✔ Install charset-normalizer 2.1.1 successful
  ✔ Install certifi 2022.12.7 successful
  ✔ Install idna 3.4 successful
  ✔ Install requests 2.28.1 successful
  ✔ Install urllib3 1.26.13 successful

🎉 All complete!

So far, so good (I’m not a fan of emoji in terminals, but that’s my only real complaint here.) Our pyproject.toml looks like this:

[tool.pdm]

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "Chris Warrick", email = "…"},
]
dependencies = [
    "requests>=2.28.1",
]
requires-python = ">=3.11"
license = {text = "MIT"}

If we try to look into our file structure, we have this:

$ ls
pdm.lock  __pypackages__/  pyproject.toml
$ ls __pypackages__
3.11/
$ ls __pypackages__/3.11
bin/  include/  lib/
$ ls __pypackages__/3.11/lib
certifi/             certifi-2022.12.7.dist-info/
idna/                idna-3.4.dist-info/
charset_normalizer/  charset_normalizer-2.1.1.dist-info/
requests/            requests-2.28.1.dist-info/
urllib3/             urllib3-1.26.13.dist-info/

We’ll write a simple Python program (let’s call it mypdmproject.py) that makes a HTTP request using requests. It will also print requests.__file__ so we’re sure it isn’t using some random system copy: [4]

import requests
print(requests.__file__)
r = requests.get("https://chriswarrick.com/")
print(r.text[:15])

$ python mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

Let’s finally try the tests we’ve done in the other languages. Requests is useless without urllib3, so let’s remove it [5] and see how well it works.

$ rm -rf __pypackages__/3.11/lib/urllib3*
$ python mypdmproject.py
Traceback (most recent call last):
  File "/tmp/mypdmproject/mypdmproject.py", line 1, in <module>
    import requests
  File "/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py", line 43, in <module>
    import urllib3
ModuleNotFoundError: No module named 'urllib3'

Finally, can we try with a different directory? How about a different user?

$ pdm install
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

  ✔ Install urllib3 1.26.13 successful

🎉 All complete!
$ pwd
/tmp/mypdmproject
$ cd ~
$ python /tmp/mypdmproject/mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>
# su -s /bin/bash -c 'eval "$(/tmp/pdmvenv/bin/pdm --pep582 bash)"; python /tmp/mypdmproject/mypdmproject.py' - nobody
su: warning: cannot change directory to /nonexistent: No such file or directory
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

This is looking pretty good. An independent project manages to do what the big Authority failed to do over so many years.

Is this the perfect thing?

Well, almost. There are two things that I have complaints about. The first one is the pdm --pep582 hack, but hopefully, the PyPA gets its act together and gets it into Python core soon. However, another important problem is the lack of separation from system site-packages. Avid readers of footnotes [6] might have noticed I had to use a Docker container in my PDM experiments, because requests is very commonly found in system site-packages (especially when using system Pythons, which have requests because of some random package, or because it was unbundled from pip). [7] This can break things in ways you don’t expect, because you might end up importing and depending on system-wide things, or mixing system-wide and local packages (if you don’t install an extra requirement, but those packages are present system-wide, then you might end up using an extra you haven’t asked for). This is an important problem—a good solution would be to disable system site-packages if a __pypackages__ directory is in use.

The part where the Steering Council kills it

In late March 2023, the Python Steering Council has announced the rejection of PEP 582. The reasons cited in the SC decision cited the limitations of the PEP (the __pypackages__ directory not always being enough, and the lack of specification on how it would behave in edge cases). Another argument is that it is possible to get __pypackages__ via “one of the many existing customization mechanisms for sys.path, like .pth files or a sitecustomize module” — things commonly considered hacks, not real solutions. While users certainly can do anything they want to the sys.path (often with tragic consequences), the point of having a common standard is to encourage tools to add support for it — if you use the aforementioned hacks, your IDE might end up not noticing the packages or considering them part of your code (trying to index them and search for things in them). Another reason cited for the rejection is the disagreement among the packaging community, which should not be surprising, especially in light of the next section.

The PEP 582/__pypackages__ mechanism may become official one day, and finally make Python packaging approachable. That would probably require someone to step up and write a new PEP that would make more people happy. Or Python might be stuck with all these incompatible tools, and invent 10 more in the next few years. (PDM is still there, and it still supports __pypackages__, even though its implementation isn’t exactly the same as suggested by the now-rejected PEP.) Python’s current trajectory, as demonstrated by this decision, and by many people still being forced to struggle with the needlessly complicated virtual environments, sounds an awful lot like the classic Onion headline: ‘No Way to Prevent This,’ Says Only Programming Community Where This Regularly Happens.

Application dependencies

Here is an example use case for Pipenv: I’m working on a website based on Django. I create ~/git/website and run pipenv install Django in that directory. Pipenv:

• automatically creates a virtualenv somewhere in my home directory

• writes a Pipfile, which lists Django as my dependency

• installs Django using pip

• proceeds to write Pipfile.lock, which stores the exact version and source file hash [2] of each package installed (including pytz, Django’s dependency).

The last part of the process was the most time consuming. At one point, while locking the dependency versions, Pipenv hangs for 46 seconds. This is one of Pipenv’s notable issues: it’s slow. Of course, this isn’t the only one, but it defintely doesn’t help. Losing 46 seconds isn’t much, but when we get to the longer waits in the timing test section later, we’ll see something that could easily discourage users from using a package.

Running scripts (badly)

But let’s continue with our workflow. pipenv run django-admin startproject foobanizer is what I must use now, which is rather unwieldy to type, and requires running pipenv even for the smallest things. (The manage.py script has /usr/bin/env python in its shebang.) I can run pipenv shell to get a new shell which runs the activate script by default, giving you the worst of both worlds when it comes to virtualenv activation: the unwieldiness of a new shell, and the activate script, which the proponents of the shell spawning dislike.

Using pipenv shell means spawning a new subshell, executing the shell startup scripts (eg. .bashrc), and requiring you to exit with exit or ^D. If you type deactivate, you are working with an extra shell, but now outside of the virtualenv. Or you can use the --fancy mode that manipulates $PATH before launching the subshell, but it requires a specific shell configuration, in which $PATH is not overridden in non-login shells — and also often changing the config of your terminal emulator to run a login shell, as many of the Linux terminals don’t do it.

Now, why does all this happen? Because a command cannot manipulate the environment of the shell it spawns. This means that Pipenv must pretend what it does is a reasonable thing instead of a workaround. This can be solved with manual activation using source $(pipenv --venv)/bin/activate (can be made into a neat alias), or shell wrappers (similar to what virtualenvwrapper does).

Finishing it all up

Anyway, I want a blog on my site. I want to write them in Markdown syntax, so I run pipenv install Markdown, and a few long seconds later, it’s added to both Pipfiles. Another thing I can do is pipenv install --dev ipython and get a handy shell for tinkering, but it will be marked as a development dependency — so, not installed in production. That last part is an important advantage of using Pipenv.

When I’m done working on my website, I commit both Pipfiles to my git repository, and push it to the remote server. Then I can clone it to, say, /srv/website. Now I can just pipenv install to get all the production packages installed (but not the development ones — Django, pytz, Markdown will be installed, but IPython and all its million dependencies won’t). There’s just one caveat: by default, the virtualenv will still be created in the current user’s home directory. This is a problem in this case, since it needs to be accessible by nginx and uWSGI, which do not have access to my (or root’s) home directory, and don’t have a home directory of their own. This can be solved with export PIPENV_VENV_IN_PROJECT=1. But note that I will now need to export this environment variable every time I work with the app in /srv via Pipenv. The tool supports loading .env files, but only when running pipenv shell and pipenv run. You can’t use it to configure Pipenv. And to run my app with nginx/uWSGI, I will need to know the exact virtualenv path anyway, since I can’t use pipenv run as part of uWSGI configuration.

Setup.py, source distributions, and wheels

Pipenv only concerns itself with managing dependencies. It isn’t a packaging tool. If you want your thing up on PyPI, Pipenv won’t help you with anything. You still need to write a setup.py with install_requires, because the Pipfile format only specifies the dependencies and runtime requirements (Python version), there is no place in it for the package name, and Pipenv does not mandate/expect you to install your project. It can come in handy to manage the development environment (as a requirements.txt replacement, or something used to write said file), but if your project has a setup.py, you still need to manually manage install_requires. Pipenv can’t create wheels on its own either. And pip freeze is going to be a lot faster than Pipenv ever will be.

Working outside of the project root

Another issue with Pipenv is the use of the working directory to select the virtual environment. [3] Let’s say I’m a library author. A user of my foobar library has just reported a bug and attached a repro.py file that lets me reproduce the issue. I download that file to ~/Downloads on my filesystem. With plain old virtualenv, I can easily confirm the reproduction in a spare shell with:

$ ~/virtualenvs/foobar/bin/python ~/Downloads/repro.py

And then I can launch my fancy IDE to fix the bug. I don’t have to cd into the project. But with Pipenv, I can’t really do that. If I put the virtualenv in .venv with the command line option, I can type ~/git/foobar/.venv/bin/python ~/Downloads/repro.py. If I use the centralized directory + hashes thing, Tab completion becomes mandatory, if I haven’t memorized the hash.

$ cd ~/git/foobar
$ pipenv run python ~/Downloads/repro.py

What if I had two .py files, or repro.py otherwise depended on being in the current working directory?

$ cd ~/git/foobar
$ pipenv shell
(foobar-Mwd1l2m9)$ cd ~/Downloads
(foobar-Mwd1l2m9)$ python repro.py
(foobar-Mwd1l2m9)$ exit  # (not deactivate!)

This is becoming ugly fairly quickly. Also, with virtualenvwrapper, I can do this:

$ cd ~/Downloads
$ workon foobar
(foobar)$ python repro.py
(foobar)$ deactivate

And let’s not forget that Pipenv doesn’t help me to write a setup.py, distribute code, or manage releases. It just manages dependencies. And it does it pretty badly.

Nikola

I’m a co-maintainer of a static site generator, Nikola. As part of this, I have the following places where I need to run Nikola:

• ~/git/nikola

• ~/git/nikola-site

• ~/git/nikola-plugins

• ~/git/nikola-themes

• ~/website (this blog)

• /Volumes/RAMDisk/n (demo site, used for testing and created when needed, on a RAM disk)

That list is long. End users of Nikola probably don’t have a list that long, but they might just have more than one Nikola site. For me, and for the aforementioned users, Pipenv does not work. To use Pipenv, all those repositories would need to live in one directory. I would also need to have a separate Pipenv environment for nikola-users, because that needs Django. Moreover, the Pipfile would have to be symlinked from ~/git/nikola if we were to make use of those in the project. So, I would have a ~/nikola directory just to make Pipenv happy, do testing/bug reproduction on a SSD (and wear it out faster), and so on… Well, I could also use the virtualenv directly. But in that case, Pipenv loses its usefulness, and makes my workflow more complicated. I can’t use virtualenvwrapper, because I would need to hack a fuzzy matching system onto it, or memorize the random string appended to my virtualenv name. All because Pipenv relies on the current directory too much.

Nikola end users who want to use Pipenv will also have a specific directory structure forced on them. What if the site serves as docs for a project, and lives inside another project’s repo? Two virtualenvs, 100 megabytes wasted. Or worse, Nikola ends up in the other project’s Pipfile, which is technically good for our download stats, but not really good for the other project’s contributors.

Pip-tools: locking and hashing, and that is all

Pip-tools contains two tools. The first one is pip-compile. It locks dependencies in requirements.txt files, and that’s all it does. It allows updating dependencies in the file based on what’s on PyPI. You can optionally add hashes to that file. The second tool is pip-sync. It will synchronize your virtualenv and the requirements file: it will delete packages not in that file, so that you don’t work with stuff not declared in requirements, and will ensure versions match the requirements file. pip-compile takes roughly 10-20 seconds to run in the Nikola repo with a clean pip-tools cache (but with the pip cache intact).

Its speed is fairly reasonable, and it does not try to be the be-all-end-all tool for development. It handles a specific task, does it well, and does not try to handle tasks it should not. Pip-tools lets you work with venvs in any way you like, and it does not require anything specific. Unlike Pipenv and Poetry, you can install it into the virtualenvs that need it, and not system-wide.

Poetry: better, but still not convincing

Poetry is somewhere in between. Its main aim is close to Pipenv, but it also makes it possible to distribute things to PyPI. It tries really hard to hide that it uses Pip behind the scenes. Its README comes with an extensive “What about Pipenv?” section, which I recommend reading — it has a few more examples of bad Pipenv features. Poetry claims to use the standardized (PEP 518) pyproject.toml file to replace the usual lot of files. Unfortunately, the only thing that is standardized is the file name and syntax. Poetry uses custom [tool.poetry] sections, which means that one needs Poetry to fully use the packages created with it, leading to vendor lock-in. There is a build feature to produce a sdist with setup.py and friends.

In February 2020, in a simple poetry add Nikola test, it took about a minute (55.1/50.8/53.6 s) to resolve dependencies (according to Poetry’s own count, Local environment, Poetry cache removed), complete with reassuring output and no quiet lockups. Not as good as pip, but it’s more reasonable than Pipenv. Also, the codebase and its layout are rather convoluted, and the docs are very sparse and lacking. Poetry produces packages instead of just managing dependencies, so it’s generally more useful than Pipenv. That said, I am not convinced by that tool either.