The classic stack

The classic Python packaging stack consists of many semi-related tools. Setuptools, probably the oldest tool of the group, and itself based on distutils, which is part of the standard library (although it will be removed in Python 3.12), is responsible for installing a single package. It previously used setup.py files to do its job, which required arbitrary code execution. It then added support for non-executable metadata specification formats: setup.cfg, and also pyproject.toml (partially still in beta). However, you aren’t supposed to use setup.py files directly these days, you’re supposed to be using pip. Pip installs packages, usually from the PyPI, but it can also support other sources (such as git repositories or the local filesystem). But where does pip install things? The default used to be to install globally and system-wide, which meant you could introduce conflicts between packages installed by pip and apt (or whatever the system package manager is). Even with a user-wide install (which pip is likely to attempt these days), you can still end up with conflicts, and you can also have conflicts in which package A requests X version 1.0.0, but package B expects X version 2.0.0—but A and B are not at all related and could live separately with their preferred version of X. Enter venv, a standard library descendant of virtualenv, which can create a lightweight virtual environment for packages to live in. This virtual environment gives you the separation from system packages and from different environments, but it is still tied to the system Python in some ways (and if the system Python disappears, the virtual environment stops working).

A few extra tools would be used in a typical packaging workflow. The wheel package enhances Setuptools with the ability to generate wheels, which are ready-to-install (without running setup.py). Wheels can either be pure-Python and be installed anywhere, or they can contain pre-compiled extension modules (things written in C) for a given OS and Python (and there’s even a standard that allows building and distributing one wheel for all typical Linux distros). The wheel package should be an implementation detail, something existing inside Setuptools and/or pip, but users need to be aware of it if they want to make wheels on their system, because virtual environments produced by venv do not have wheel installed. Regular users who do not maintain their own packages may sometimes be told that pip is using something legacy because wheel is not installed, which is not a good user experience. Package authors also need twine, whose sole task is uploading source distributions or wheels, created with other tools, to PyPI (and there’s not much more to say about that tool).

…and a few extensions

Over the years, there have been a few tools that are based on things from the classic stack. For example, pip-tools can simplify dependency management. While pip freeze lets you produce a file with everything installed in your environment, there is no way to specify the dependencies you need, and get a lock file with specific versions and transitive dependencies (without installing and freezing everything), there is no easy way to skip development dependencies (e.g. IPython) when you pip freeze, and there is no workflow to update all your dependencies with just pip. pip-tools adds two tools, pip-compile which takes in requirements.in files with the packages you care about, and produces a requrirements.txt with pinned versions of them and all transitive dependencies; and also pip-sync, which can install requirements.txt and removes things not listed in it.

Another tool that might come in useful is virtualenvwrapper, which can help you manage (create and activate) virtual environments in a central location. It has a few bells and whistles (such as custom hooks to do actions on every virtualenv creation), although for basic usage, you could replace it with a single-line shell function.

Yet another tool that works alongside the classic toolset is pipx, which creates and manages virtual environments for apps written in Python. You tell it to pipx install Nikola, and it will create a virtual environment somewhere, install Nikola into it, and put a script for launching it in ~/.local/bin. While you could do it all yourself with venv and some symlinks, pipx can take care of this, and you don’t need to remember where the virtual environment is.

The scientific stack and conda

The scientific Python community have had their own tools for many years. The conda tool can manage environments and packages. It doesn’t use PyPI and wheels, but rather packages from conda channels (which are prebuilt, and expect an Anaconda-distributed Python). Back in the day, when there were no wheels, this was the easiest way to get things installed on Windows; this is not as much of a problem now with binary wheels on PyPI—but the Anaconda stack is still popular in the scientific world. Conda packages can be built with conda-build, which is separate, but closely related to conda itself. Conda packages are not compatible with pip in any way, they do not follow the packaging standards used by other tools. Is this good? No, because it makes integrating the two worlds harder, but also yes, because many problems that apply to scientific packages (and their C/C++ extension modules, and their high-performance numeric libraries, and other things) do not apply to other uses of Python, so having a separate tool lets people focusing the other uses simplify their workflows.

The new tools

A few years ago, new packaging tools appeared. Now, there were lots of “new fancy tools” introduced in the past, with setuptools extending distutils, then distribute forking setuptools, then distribute being merged back…

The earliest “new tool” was Pipenv. Pipenv had really terrible and misleading marketing, and it merged pip and venv, in that Pipenv would create a venv and install packages in it (from Pipfile or Pipfile.lock). Pipenv can place the venv in the project folder, or hide it somewhere in the project folder (the latter is the default). However, Pipenv does not handle any packages related to packaging your code, so it’s useful only for developing non-installable applications (Django sites, for example). If you’re a library developer, you need setuptools anyway.

The second new tool was Poetry. It manages environments and dependencies in a similar way to Pipenv, but it can also build .whl files with your code, and it can upload wheels and source distributions to PyPI. This means it has pretty much all the features the other tools have, except you need just one tool. However, Poetry is opinionated, and its opinions are sometimes incompatible with the rest of the packaging scene. Poetry uses the pyproject.toml standard, but it does not follow the standard specifying how metadata should be represented in a pyproject.toml file (PEP 621), instead using a custom [tool.poetry] table. This is partly because Poetry came out before PEP 621, but the PEP was accepted over 2 years ago—the biggest compatibility problem is Poetry’s node-inspired ~ and ^ dependency version markers, which are not compatible with PEP 508 (the dependency specification standard). Poetry can package C extension modules, although it uses setuptools’ infrastructure for this (and requires a custom build.py script).

Another similar tool is Hatch. This tool can also manage environments (it allows multiple environments per project, but it does not allow to put them in the project directory), and it can manage packages (but without lockfile support). Hatch can also be used to package a project (with PEP 621-compliant pyproject.toml files) and upload it to PyPI. It does not support C extension modules.

A tool that tries to be a simpler re-imagining of Setuptools is Flit. It can build and install a package using a pyproject.toml file. It also supports uploads to PyPI. It lacks support for C extension modules, and it expects you to manage environments on your own.

There’s one more interesting (albeit not popular or well-known) tool. This tool is PDM. It can manage venvs (but it defaults to the saner .venv location), manage dependencies, and it uses a standards-compliant pyproject.toml. There’s also a curious little feature called PEP 582 support, which we’ll talk about later.

How to use Python from a virtual environment?

There are two ways to do this. The first one is to activate it, by running the activate shell script installed in the environment’s bin directory. Another is to run the python executable (or any other script in the bin directory) directly from the venv. [2]

Activating venvs directly is more convenient for developers, but it also has some problems. Sometimes, activation fails to work, due to the shell caching the location of things in $PATH. Also, beginners are taught to activate and run python, which means they might be confused and try to use activate in scripts or cronjobs (but in those environments, you should not activate venvs, and instead use the Python executable directly). Virtual environment activation is more state you need to be aware of, and if you forget about it, or if it breaks, you might end up messing up your user-wide (or worse, system-wide) Python packages.

How are (system) Pythons and virtual environments related?

The virtual environment depends very tightly on the (system/global/pyenv-installed) Python used to create it. This is good for disk-space reasons (clean virtual environments don’t take up very much space), but this also makes the environment more fragile. If the Python used to create the environment is removed, the virtual environment stops working. If you fully manage your own Python, then it’s probably not going to happen, but if you depend on a system Python, upgrading packages on your OS might end up replacing Python 3.10 with Python 3.11. Some distributions (e.g. Ubuntu) would only make a jump like this on a new distribution release (so you can plan ahead), some of them (e.g. Arch) are rolling-release and a regular system upgrade may include a new Python, whereas some (e.g. Homebrew) make it even worse by using paths that include the patch Python version (3.x.y), which cause virtual environments to break much more often.

How to manage virtual environments?

The original virtualenv tool, and its simplified standard library rewrite venv, allow you to put a virtual environment anywhere in the file system, as long as you have write privileges there. This has led to people and tools inventing their own standards. Virtualenvwrapper stores environments in a central location, and does not care about their contents. Pipenv and poetry allow you to choose (either a central location or the .venv directory in the project), and environments are tied to a project (they will use the project-specific environment if you’re in the project directory). Hatch stores environments in a central location, and it allows you to have multiple environments per project (but there is no option to share environments between projects).

Brett Cannon has recently done a survey, and it has shown the community is split on their workflows: some people use a central location, some put them in the project directory, some people have multiple environments with different Python versions, some people reuse virtualenvs between projects… Everyone has different needs, and different opinions. For example, I use a central directory (~/virtualenvs) and reuse environments when working on Nikola (sharing the same environment between development and 4 Nikola sites). But on the other hand, when deploying web apps, the venv lives in the project folder, because this venv needs to be used by processes running as different users (me, root, or the service account for the web server, which might have interactive login disabled, or whose home directory may be set to something ephemeral).

So: does Python need virtual environments? Perhaps looking how other languages handle this problem can help us figure this out for Python?

JavaScript/Node.js (with npm)

There are two tools for dealing with packages in the Node world, namely npm and Yarn. The npm CLI tool is shipped with Node, so we’ll focus on it.

Let’s create a project:

$ mkdir mynpmproject
$ cd mynpmproject
$ npm init
…answer a few questions…
$ ls
package.json

We’ve got a package.json file, which has some metadata about our project (name, version, description, license). Let’s install a dependency:

$ npm install --save is-even

added 5 packages, and audited 6 packages in 2s

found 0 vulnerabilities

The mere existence of an is-even package is questionable; the fact that it includes four dependencies is yet another, and the fact that it depends on is-odd is even worse. But this post isn’t about is-even or the Node ecosystem’s tendency to use tiny packages for everything (but I wrote one about this topic before). Let’s look at what we have in the filesystem:

$ ls
node_modules/  package.json  package-lock.json
$ ls node_modules
is-buffer/  is-even/  is-number/  is-odd/  kind-of/

Let’s also take a peek at the package.json file:

{
  "name": "mynpmproject",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "is-even": "^1.0.0"
  }
}

Our package.json file now lists the dependency, and we’ve also got a lock file (package-lock.json), which records all the dependency versions used for this install. If this file is kept in the repository, any future attempts to npm install will use the dependency versions listed in this file, ensuring everything will work the same as it did originally (unless one of those packages were to get removed from the registry).

Let’s try writing a trivial program using the module and try running it:

$ cat index.js
var isEven = require('is-even');
console.log(isEven(0));

$ node index.js
true

Let’s try removing is-odd to demonstrate how badly designed this package is:

$ rm -rf node_modules/is-odd
$ node index.js
node:internal/modules/cjs/loader:998
  throw err;
  ^

Error: Cannot find module 'is-odd'
Require stack:
- /tmp/mynpmproject/node_modules/is-even/index.js
- /tmp/mynpmproject/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
    at Module._load (node:internal/modules/cjs/loader:841:27)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/tmp/mynpmproject/node_modules/is-even/index.js:10:13)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/tmp/mynpmproject/node_modules/is-even/index.js',
    '/tmp/mynpmproject/index.js'
  ]
}

Node.js v18.12.1

$ pwd
/tmp/mynpmproject
$ npm install

added 1 package, and audited 6 packages in 436ms

found 0 vulnerabilities
$ node /tmp/mynpmproject/index.js
true
$ cd ~
$ node /tmp/mynpmproject/index.js
true

$ su -
# node /tmp/mynpmproject/index.js
true
# sudo -u nginx node /tmp/mynpmproject/index.js
true

$ cd /tmp
$ mv mynpmproject mynodeproject
$ node /tmp/mynodeproject/index.js
true

C#/.NET (with dotnet CLI/MSBuild)

In modern .NET, the One True Tool is the dotnet CLI, which uses MSBuild for most of the heavy lifting. (In the classic .NET Framework, the duties were split between MSBuild and NuGet.exe, but let’s focus on the modern workflow.)

Let’s create a project:

$ mkdir mydotnetproject
$ cd mydotnetproject
$ dotnet new console
The template "Console App" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /tmp/mydotnetproject/mydotnetproject.csproj...
  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 92 ms).
Restore succeeded.
$ ls
mydotnetproject.csproj  obj/  Program.cs

We get three things: a mydotnetproject.csproj file, which defines a few properties of our project; Program.cs, which is a hello world program, and obj/, which contains a few files you don’t need to care about.

Let’s try adding a dependency. For a pointless example, but slightly more reasonable than the JS one, we’ll use AutoFixture, which brings in a dependency on Fare. If we run dotnet add package AutoFixture, we get some console output, and our mydotnetproject.csproj now looks like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AutoFixture" Version="4.17.0" />
  </ItemGroup>

</Project>

The first <PropertyGroup> specifies what our project is (Exe = something you can run), specifies the target framework (.NET 6.0 [3]), and enables a few opt-in features of C#. The second <ItemGroup> was inserted when we installed AutoFixture.

We can now write a pointless program in C#. Here’s our new Program.cs:

using AutoFixture;
var fixture = new Fixture();
var a = fixture.Create<int>();
var b = fixture.Create<int>();
var result = a + b == b + a;
Console.WriteLine(result ? "Math is working": "Math is broken");

(We could just use C#’s/.NET’s built-in random number generator, AutoFixture is complete overkill here—it’s meant for auto-generating test data, with support for arbitrary classes and other data structures, and we’re just getting two random ints here. I’m using AutoFixture for this example, because it’s simple to use and demonstrate, and because it gets us a transitive dependency.)

And now, we can run it:

$ dotnet run
Math is working

If we want something that can be run outside of the project, and possibly without .NET installed on the system, we can use dotnet publish. The most basic scenario:

$ dotnet publish
$ ls bin/Debug/net6.0/publish
AutoFixture.dll*  Fare.dll*  mydotnetproject*  mydotnetproject.deps.json  mydotnetproject.dll  mydotnetproject.pdb  mydotnetproject.runtimeconfig.json
$ du -h bin/Debug/net6.0/publish
424K    bin/Debug/net6.0/publish
$ bin/Debug/net6.0/publish/mydotnetproject
Math is working

You can see that we’ve got a few files related to our project, as well as AutoFixture.dll and Fare.dll, which are our dependencies (Fare.dll is a dependency of AutoFixture.dll). Now, let’s try to remove AutoFixture.dll from the published distribution:

$ rm bin/Debug/net6.0/publish/AutoFixture.dll
$ bin/Debug/net6.0/publish/mydotnetproject
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'. The system cannot find the file specified.

File name: 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'
[1]    45060 IOT instruction (core dumped)  bin/Debug/net6.0/publish/mydotnetproject

We can also try a more advanced scenario:

$ rm -rf bin obj  # clean up, just in case
$ dotnet publish --sc -r linux-x64 -p:PublishSingleFile=true -o myoutput
Microsoft (R) Build Engine version 17.0.1+b177f8fa7 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 4.09 sec).
  mydotnetproject -> /tmp/mydotnetproject/bin/Debug/net6.0/linux-x64/mydotnetproject.dll
  mydotnetproject -> /tmp/mydotnetproject/myoutput/
$ ls myoutput
mydotnetproject*  mydotnetproject.pdb
$ myoutput/mydotnetproject
Math is working
$ du -h myoutput/*
62M     myoutput/mydotnetproject
12K     myoutput/mydotnetproject.pdb
$ file -k myoutput/mydotnetproject
myoutput/mydotnetproject: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=47637c667797007d777f4322729d89e7fa53a870, for GNU/Linux 2.6.32, stripped, too many notes (256)\012- data
$ file -k myoutput/mydotnetproject.pdb
myoutput/mydotnetproject.pdb: Microsoft Roslyn C# debugging symbols version 1.0\012- data

We have a single output file that contains our program, its dependencies, and parts of the .NET runtime. We also get debugging symbols if we want to run our binary with a .NET debugger and see the associated source code. (There are ways to make the binary file smaller, and we can move most arguments of dotnet publish to the .csproj file, but this post is about Python, not .NET, so I’m not going to focus on them too much.)

Other languages and ecosystems

While we have explored two tools for two languages in depth, there are also other languages that deserve at least a mention. In the Java world, the two most commonly used tools are Maven and Gradle. Both tools can be used to manage dependencies and build artifacts that can be executed or distributed further (things like JAR files). Other tools with support for building Java projects exist, but most people just pick one of the two. The community of Scala, which is another JVM-based language, prefers sbt (which can be used for plain Java as well), but there are also Maven or Gradle users in that community. Finally, two new-ish languages which are quite popular in the recent times, Go and Rust, have first-party tooling integrated with the rest of the toolchain. The go command-line tool can accomplish many build/dependency/packaging tasks. Rust’s cargo, which ships with the standard distribution of Rust, handles dependencies, builds, running code and tests, as well as publishing your stuff to a registry.

Are those ecosystems’ tools perfect?

Not always, they have their deficiencies as well. In the Node ecosystem, packages may execute arbitrary code on install, which can be a security risk (there are some known examples, like a npm package wiping hard drives in Russia and Belarus, or another one stealing imaginary Internet money Bitcoin). Binary packages are not distributed on the npm registry directly, they’re either built with node-gyp, or have prebuilt packages downloaded via node-pre-gyp (which is a third-party tool).

In the .NET ecosystem, the tools also create an obj directory with temporary files. Those temporary files are tied to the environment they’re running in, and while the tooling will usually re-create them if something changes, it can sometimes fail and leave you with confusing errors (which can generally be solved by removing the bin and obj directories). If a package depends on native code (which is not already available on the target OS as part of a shared library), it must include binary builds in the NuGet package for all the platforms it supports, as there is no standard way to allow building something from source.

You can also find deficiencies in the tools for the other languages mentioned. Some people think Maven is terrible because it uses XML and Gradle is the way to go, and others think Gradle’s use of a Groovy-based DSL makes things much harder than they need to be and prefer Maven instead.

Enabling the future on your own machine

Let’s enable the future on my own machine. That will require one simple command:

$ eval "$(pdm --pep582)"

After that, we can initialize our project and install requests into it. Let’s try:

$ mkdir mypdmproject
$ cd mypdmproject
$ pdm init
Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /usr/bin/python (3.11)
1. /usr/bin/python3.11 (3.11)
2. /usr/bin/python2.7 (2.7)
Please select (0): 1
Using Python interpreter: /usr/bin/python3.11 (3.11)
Would you like to create a virtualenv with /usr/bin/python3.11? [y/n] (y): n
You are using the PEP 582 mode, no virtualenv is created.
For more info, please visit https://peps.python.org/pep-0582/
Is the project a library that will be uploaded to PyPI [y/n] (n): n
License(SPDX name) (MIT):
Author name (Chris Warrick):
Author email (…):
Python requires('*' to allow any) (>=3.11):
Changes are written to pyproject.toml.
$ ls
pyproject.toml
$ pdm add requests
Adding packages to default dependencies: requests
🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 5 to add, 0 to update, 0 to remove

  ✔ Install charset-normalizer 2.1.1 successful
  ✔ Install certifi 2022.12.7 successful
  ✔ Install idna 3.4 successful
  ✔ Install requests 2.28.1 successful
  ✔ Install urllib3 1.26.13 successful

🎉 All complete!

So far, so good (I’m not a fan of emoji in terminals, but that’s my only real complaint here.) Our pyproject.toml looks like this:

[tool.pdm]

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "Chris Warrick", email = "…"},
]
dependencies = [
    "requests>=2.28.1",
]
requires-python = ">=3.11"
license = {text = "MIT"}

If we try to look into our file structure, we have this:

$ ls
pdm.lock  __pypackages__/  pyproject.toml
$ ls __pypackages__
3.11/
$ ls __pypackages__/3.11
bin/  include/  lib/
$ ls __pypackages__/3.11/lib
certifi/             certifi-2022.12.7.dist-info/
idna/                idna-3.4.dist-info/
charset_normalizer/  charset_normalizer-2.1.1.dist-info/
requests/            requests-2.28.1.dist-info/
urllib3/             urllib3-1.26.13.dist-info/

We’ll write a simple Python program (let’s call it mypdmproject.py) that makes a HTTP request using requests. It will also print requests.__file__ so we’re sure it isn’t using some random system copy: [4]

import requests
print(requests.__file__)
r = requests.get("https://chriswarrick.com/")
print(r.text[:15])

$ python mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

Let’s finally try the tests we’ve done in the other languages. Requests is useless without urllib3, so let’s remove it [5] and see how well it works.

$ rm -rf __pypackages__/3.11/lib/urllib3*
$ python mypdmproject.py
Traceback (most recent call last):
  File "/tmp/mypdmproject/mypdmproject.py", line 1, in <module>
    import requests
  File "/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py", line 43, in <module>
    import urllib3
ModuleNotFoundError: No module named 'urllib3'

Finally, can we try with a different directory? How about a different user?

$ pdm install
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

  ✔ Install urllib3 1.26.13 successful

🎉 All complete!
$ pwd
/tmp/mypdmproject
$ cd ~
$ python /tmp/mypdmproject/mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>
# su -s /bin/bash -c 'eval "$(/tmp/pdmvenv/bin/pdm --pep582 bash)"; python /tmp/mypdmproject/mypdmproject.py' - nobody
su: warning: cannot change directory to /nonexistent: No such file or directory
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

This is looking pretty good. An independent project manages to do what the big Authority failed to do over so many years.

Is this the perfect thing?

Well, almost. There are two things that I have complaints about. The first one is the pdm --pep582 hack, but hopefully, the PyPA gets its act together and gets it into Python core soon. However, another important problem is the lack of separation from system site-packages. Avid readers of footnotes [6] might have noticed I had to use a Docker container in my PDM experiments, because requests is very commonly found in system site-packages (especially when using system Pythons, which have requests because of some random package, or because it was unbundled from pip). [7] This can break things in ways you don’t expect, because you might end up importing and depending on system-wide things, or mixing system-wide and local packages (if you don’t install an extra requirement, but those packages are present system-wide, then you might end up using an extra you haven’t asked for). This is an important problem—a good solution would be to disable system site-packages if a __pypackages__ directory is in use.

The part where the Steering Council kills it

In late March 2023, the Python Steering Council has announced the rejection of PEP 582. The reasons cited in the SC decision cited the limitations of the PEP (the __pypackages__ directory not always being enough, and the lack of specification on how it would behave in edge cases). Another argument is that it is possible to get __pypackages__ via “one of the many existing customization mechanisms for sys.path, like .pth files or a sitecustomize module” — things commonly considered hacks, not real solutions. While users certainly can do anything they want to the sys.path (often with tragic consequences), the point of having a common standard is to encourage tools to add support for it — if you use the aforementioned hacks, your IDE might end up not noticing the packages or considering them part of your code (trying to index them and search for things in them). Another reason cited for the rejection is the disagreement among the packaging community, which should not be surprising, especially in light of the next section.

The PEP 582/__pypackages__ mechanism may become official one day, and finally make Python packaging approachable. That would probably require someone to step up and write a new PEP that would make more people happy. Or Python might be stuck with all these incompatible tools, and invent 10 more in the next few years. (PDM is still there, and it still supports __pypackages__, even though its implementation isn’t exactly the same as suggested by the now-rejected PEP.) Python’s current trajectory, as demonstrated by this decision, and by many people still being forced to struggle with the needlessly complicated virtual environments, sounds an awful lot like the classic Onion headline: ‘No Way to Prevent This,’ Says Only Programming Community Where This Regularly Happens.

The obligatory hello world program

Console.WriteLine("Hello, world!");

Just one line is enough. This program requires C# 10 and .NET 6, the latest versions of the language and the framework (and the implicit usings feature enabled).

The slightly less cool version of the hello world program

using System;

class Program
{
  static void Main(string[] args)
  {
    Console.WriteLine("Hello, world!");
  }
}

We’ve got four lines of code (plus four lines of braces [1] ). We can see the using directive to import everything from the System namespace, the definition of a class, a Main method, and a call to Console.WriteLine.

We’ll talk more about C# later, highlighting some of the cooler things seen in Gitco.NET.

Snippet 1

public static List<Branch> ExtractBranchListFromGitOutput(string gitOutput)
  => gitOutput
    .TrimEnd()
    .ReplaceLineEndings("\n")
    .Split('\n')
    .Select(branchLine =>
    {
      var isCurrent = branchLine.StartsWith('*');
      var branch = branchLine[2..];
      var isRemote = false;

      if (branch.StartsWith(remotePrefix))
      {
        isRemote = true;
        branch = string.Join(
          '/',
          branch.Split(" ").First().Split("/").Skip(2));
      }

      return new Branch(branch, isRemote, isCurrent);
    })
    .OrderBy(b => b.Name)
    .ThenBy(b => b.IsRemote)
    .DistinctBy(b => b.Name)
    .ToList();

This snippet defines a fairly standard pipeline that goes from git output (a single string) to a list of parsed objects. This pipeline is a function (or a static method, to be more precise). This function uses expression-bodied members: since we can fit the entire pipeline in a single expression, we can skip the braces and the return keyword, and instead use a more compact syntax with an arrow (=>). After some cleanups and sanitization of the string, we split the string by the \n character, and the type of our pipeline changes from string to string[] (an array of strings). We then use five operations from the System.Linq namespace. Those operations are extension methods for enumerables (IEnumerable<T>) — adding using System.Linq; at the top of your program adds those methods to any enumerables (including arrays, lists, dictionaries, sets…).

The first operation is a Select. LINQ methods are inspired by SQL; the more typical name for this one would be map. (Similarly, Where is LINQ’s name for filter.) The logic inside Select is written in a multi-line anonymous function (lambda), with braces (so there’s a return) [2]. Inside that anonymous function, there are a few niceties, such as var (type inference for variables), slicing ([2..]), as well as some more LINQ in string manipulations (.First() and .Skip(2), which do what they say on the tin).

The next three operations are fairly straightforward sorting, and extracting unique values. Those use single-expression lambdas, which don’t use return. The pipeline ends with converting IEnumerable<Branch> (which appeared at the .Select() stage) into a List<Branch>.

Snippet 2

public static IEnumerable<BranchDisplay> FilterAndNumberBranches(
 List<Branch> branches, string? filter)
{
  var branchWidth = branches.Count
    .ToString(CultureInfo.InvariantCulture).Length;
  var numberFormatString = $"{{0,{branchWidth}}}. ";

  return branches.Select(
    (branch, index) =>
      new BranchDisplay(
        Number: string.Format(numberFormatString, index + 1),
        BranchName: branch.Name,
        IsRemote: branch.IsRemote,
        IsCurrent: branch.IsCurrent
      )
    ).Where(branchDisplay =>
      filter == null || branchDisplay.BranchName.Contains(filter));
}

This function adds numbers to the branch list, and then filters branches based on the user’s query. The first thing to notice is the second argument: string? filter. C# has support for nullable types, which means the compiler warns you if you use a possibly null value somewhere it isn’t expected [3]. numberFormatString uses an interpolated string, in which {branchWidth} will be replaced with the variable defined before. In the LINQ expression, you can see two interesting things: one is a two-argument lambda for Select, and argument names, which can be optionally passed to functions and constructors for readability or to set parameters out of order.

Snippet 3

How much boilerplate do you need to define an immutable data class with a constructor, value equality, and a string representation?

Exactly zero:

public record Branch(
 string Name,
 bool IsRemote = false,
 bool IsCurrent = false);

public record BranchDisplay(
 string Number,
 string BranchName,
 bool IsRemote = false,
 bool IsCurrent = false);

(If you want things to be mutable, you do need to write some more code. Still, all you need for encapsulated properties is int Foo { get; set; }, which is miles better than having to write out getters and setters by hand, as you would do in Java.)

dotnet CLI

.NET has a CLI for performing typical build and project configuration tasks. You can dotnet build a project, you can dotnet run it, you can dotnet test things, and you can dotnet publish. The CLI figures out what to do, it restores the dependencies if needed, it handles the compilation of your code. If you type dotnet test in a directory with your solution file (.sln), it will restore dependencies, build the code, and then find tests and run them.

IDE

What IDE should you use? There are a few options:

• Visual Studio Code. The quite advanced text editor supports pretty much any language. C# support works okay, with all the IDE features available, but in my experience, it can sometimes get confused (requiring a restart of the IDE). You will probably need to spend some more time with the dotnet CLI than you would with the other options.

• Visual Studio for Windows. The IDE with the purple icon is an option, although VS can feel arcane to people used to other IDEs/editors, and the Vim bindings are quite bad (especially if you select things with a mouse sometimes). It’s free for personal and very-small-business use, but for anything even slightly serious, you’ll need paid licenses.

• Visual Studio for Mac. A completely separate product, works reasonably well, same pricing as with the Windows version.

• Visual Studio for Windows + ReSharper Ultimate. Adding this (paid) extension makes VS much smarter, although it can also affect performance negatively.

• JetBrains Rider. This is an IDE based on the IntelliJ platform, with all the magic seen in ReSharper (as well as other JetBrains IDEs), but none of the performance issues and Visual Studio being Visual Studio (although if you do prefer VS behaviors and keyboard shortcuts, you can configure those as well). This is probably your best bet if you’re willing to invest some money (or your employer is).

Python

• C# is statically typed. Modern Python’s static typing (via things like mypy) is quite cool, but not all libraries and ecosystems have adopted it. Statically typed languages are safer, and allow IDEs to be smarter.

• C# has a better approach to functional programming. Python has ugly and single-expression lambdas (with a pointless lambda keyword), C# has inline functions that can contain multiple statements.

• C# has much better package management.

• C# is trivial to compile to a single-file executable.

• C# is much faster than Python.

• ~Nobody does machine learning and data science in C#, which is a plus in my book.

Java

• C# has a lot more developer-quality-of-life features and less boilerplate. For example, Lists and Dictionaries can be accessed using brackets, and properties are accessible via dot notation instead of having to explicitly call getter and setter methods.

• C#’s generic are more flexible, as they aren’t erased on compilation.

• C# has null safety. It also has the safe navigation ?. operator, and the null coalescing ?? operator, both of which make working with nullable values easier.

• C# has easy concurrency via async and await.

• Web stuff: Spring is painful, Spring Boot doesn’t make it much better. ASP.NET Core is much nicer.

Additional reading: Wikipedia has a very nice and detailed Comparison of C# and Java.

Also…

• C# is a high-level language with automated memory management, which is very convenient in many use-cases.

• C# has exceptions.

• There are quite a lot of jobs for C# developers, although not necessarily in Silicon Valley.

But on the other hand…

• C# can still sometimes feel a bit Windows-oriented.

• C# jobs tend to be enterprisey.

• Python is a great language to learn as a beginner. It’s also great for one-off things, interactive work, and scripting.

• The non-Windows desktop GUI story isn’t too great, although it is getting better with MAUI (which supports macOS).

• If you’re targeting mobile, I would probably focus on the native APIs and languages for the best user experience (Swift and Cocoa Touch for iOS; Kotlin and the Android Platform APIs for Android). That said, MAUI might be worth a go as well.

• If you’re doing very low-level stuff, C# probably won’t cut it.

• If you want real functional programming, go with F#. You might also prefer Scala or Haskell or such.

• And if you’re making web front-end stuff, TypeScript (or plain JavaScript) is still your best bet. C# has Blazor, but I’d prefer for web apps not to embed all of .NET via WebAssembly.

But for console apps, Windows desktop, and web back-end services? Do give C# a try, it might just win you over. It is a pretty good language, but one that was held back by the Windows association for a long time. But now it’s part of a modern, multi-platform, developer-friendly ecosystem.