(Also, Docker Desktop demands payment for enterprise use; if your company fits the enterprise criteria but cannot afford $16/developer/month, run.)

Console command conventions: $ means Linux, > means Windows (PowerShell).
Path conventions: / means Linux, \ means Windows.

Linux setup

You need to have WSL and a distribution installed, and you must be using WSL 2. Read the install docs if you need to do it.

➡️ Start by installing Docker in the usual way for your Linux distribution of choice. I’m using Ubuntu with the official Docker apt repository. Make sure to add yourself to the Docker group.

➡️ Enable systemd support in WSL. Put those two lines into /etc/wsl.conf:

[boot]
systemd=true

You can do it like so (if you don’t have a wsl.conf file already):

$ printf '[boot]\nsystemd=true\n' | sudo tee /etc/wsl.conf

➡️ Next, enable the systemd services:

$ sudo systemctl enable containerd.service
$ sudo systemctl enable docker.service

Exposing Docker to Windows

By default, Docker only listens on a Unix domain socket. To work with Docker from Windows, it needs to listen on a TCP socket. If you’re fine with using Docker only from the WSL terminal, and won’t run any Docker-compatible tools on the Windows side, you can basically stop reading here.

Adding a socket can be done in two ways: by adding command-line arguments, or by modifying /etc/docker/daemon.json. If your Docker systemd service is passing in arguments, you need to edit the service, and can’t do it using the daemon.json file. Let’s edit the service, as this is more likely to be what you need.

➡️ Run sudo systemctl edit docker.service and override the ExecStart command. The first ExecStart= line clears out the existing configuration defined in the system-provided service. The second line is the new command. To avoid breaking things, review the ExecStart command that is shown as the current content of your docker.service, and make sure you have both -H arguments as in my example below.

➡️ We’re done with the WSL setup, so let’s restart WSL and make sure Docker is working:

Windows setup

➡️ Download the latest version of the Docker binaries. Extract docker.exe and put it somewhere on your Windows PATH. I have a ~\Tools\bin folder for things like that.

We need to tell the Docker CLI where to look for Docker. The easiest way to do this is to set the DOCKER_HOST environment variable. This might not be the cleanest way if you want to work with Windows containers or Docker Desktop, but you don’t, so we don’t need to bother with anything nicer.

➡️ Press the Start key, type environment variables (or the equivalent phrase in your Windows language) and open the environment variables editor. Add an environment variable named DOCKER_HOST with the value tcp://127.0.0.1:2375.

➡️ Restart your Terminal. In one tab, run a WSL shell. In another, run PowerShell (or cmd):

Optional: Docker Compose

Docker Compose is a plugin for Docker CLI, and it not included in the Docker binary we’ve installed on Windows. It is quite easy to add it in if you want.

➡️ Download the latest version of the Docker Compose binary (docker-compose-windows-x86_64.exe). Move it to ~\.docker\cli-plugins\docker-compose.exe.

> mkdir ~\.docker\cli-plugins
> mv ~\Downloads\docker-compose-windows-x86_64.exe ~\.docker\cli-plugins\docker-compose.exe

Let’s test it out:

> docker compose version
Docker Compose version v5.1.0

Optional: WSL VM management

WSL does not automatically start the Linux VM. And it kills it if no user process is running inside of it.

If you’re fine with having a WSL terminal open while you’re working with Docker, you don’t need to do anything. But if you don’t want to see a WSL terminal, make Windows start WSL when you log in and set a really large VM Idle Timeout in the Optional Features tab of the WSL Settings app. The maximum value accepted by the GUI¹ is the maximum value a 32-bit integer can represent, i.e. 2147483647 ms (24 days). You can also manually add the option to ~\.wslconfig:

[wsl2]
vmIdleTimeout=2147483647

Why would I do that? Why would I use C# for it? And how fast is it? Continue reading to find out.

OK, but why?

You might have noticed I’m not happy with the Python packaging ecosystem. But the language itself is no longer fun for me to code in either. It is especially not fun to maintain projects in. Elementary quality-of-life features get bogged down in months of discussions and design-by-committee. At the same time, there’s a new release every year, full of removed and deprecated features. A lot of churn, without much benefit. I just don’t feel like doing it anymore.

Python is praised for being fast to develop in. That’s certainly true, but a good high-level statically-typed language can yield similar development speed with more correctness from day one. For example, I coded an entire table-of-contents-sidebar feature in one evening (and one more evening of CSS wrangling to make it look good). This feature extracts headers from either the Markdown AST or the HTML fragment. I could do it in Python, but I’d need to jump through hoops to get Python-Markdown to output headings with IDs. In C#, introspecting what a class can do is easier thanks to great IDE support and much less dynamic magic happening at runtime. There are also decompiler tools that make it easy to look under the hood and see what a library is doing.

Writing a static site generator is also a learning experience. A competent SSG needs to ingest content in various formats (as nobody wants to write blog posts in HTML by hand) and generate HTML (usually from templates) and XML (which you could, in theory, do from templates, but since XML parsers are not at all lenient, you don’t want to). Image processing to generate thumbnails is needed too. And to generate correct RSS feeds, you need to parse HTML to rewrite links. The list of small-but-useful things goes on.

Is C#/.NET a viable technology stack for a static site generator?

C#/.NET is certainly not the most popular technology stack for static site generators. JamStack.org have gathered a list of 377 SSGs. Grouping by language, there are 154 generators written in JavaScript or TypeScript, 55 generators written in Python, and 28 written in PHP of all languages. C#/.NET is in sixth place with 13 (not including YABG; I’m probably not submitting it).

However, it is a pretty good choice. Language-level support for concurrency with async/await (based on a thread pool) and JIT compilation help to make things fast. But it is still a high-level, object-oriented language where you don’t need to manually manage memory (hi Rustaceans!).

The library ecosystem is solid too. There are plenty of good libraries for working with data serialization formats: CsvHelper, YamlDotNet, Microsoft.Data.Sqlite, and the built-in System.Text.Json and System.Xml.Linq. Markdig handles turning Markdown into HTML. Fluid is an excellent templating library that implements the Liquid templating language. HtmlAgilityPack is solid for manipulating HTML, and Magick.NET wraps the ImageMagick library.

There’s one major thing missing from the above list: code highlighting. There are a few highlighting libraries on NuGet, but I decided to stick with Pygments. I still need the Pygments stylesheets around since I’m not converting old reStructuredText posts to Markdown (I’m copying them as HTML directly from Nikola’s cache), so using Pygments for new content keeps things consistent. Staying with Pygments means I still maintain a bit of Python code, but much less: 230 LoC in pygments_better_html and 89 in yabg_pygments_adapter, with just one third-party dependency. Calling a subprocess while rendering listings is slow, but it’s a price worth paying.

Paid libraries in the .NET ecosystem

All the above libraries are open source (MIT, Apache 2.0, BSD-2-Clause). However, one well-known issue of the .NET ecosystem is the number of packages that suddenly become commercial. This trend was started by ImageSharp, a popular 2D image manipulation library. I could probably use it, since it’s licensed to open-source projects under Apache 2.0, but I’d rather not. I initially tried SkiaSharp, but it has terrible image scaling algorithms, so I settled on Magick.NET.

Open-source sustainability is hard, maybe impossible. But I don’t think transitioning from open-source to pay-for-commercial-use is the answer. In practice, many businesses just use the last free version or switch to a different library. I’d rather support open-source projects developed by volunteers in their spare time. They might not be perfect or always do exactly what I want, but I’m happy to contribute fixes and improve things for everyone. I will avoid proprietary or dual-licensed libraries, even for code that never leaves my computer. Some people complain when Microsoft creates a library that competes with a third-party open-source library (e.g. Microsoft.AspNetCore.OpenApi, which was built to replace Swashbuckle.AspNetCore), but I am okay with that, since libraries built or backed by large corporations (like Microsoft) tend to be better maintained.

But at least sometimes trash libraries take themselves out.

Is it fast?

One of the things that set Nikola apart from other Python static site generators is that it only rebuilds files that need to be rebuild. This does make Nikola fast when rebuilding things, but it comes at a cost: Nikola needs to track all dependencies very closely. Also, some features that are present in other SSGs are not easy to achieve in Nikola, because they would cause many pages to be rebuilt.

YetAnotherBlogGenerator has almost no caching. The only thing currently cached is code listings, since they’re rendered using Pygments in a subprocess. Additionally, the image scaling service checks the file modification date to skip regenerating thumbnails if the source image hasn’t changed. And yet, even if it rewrites everything, YABG finishes faster than Nikola when the site is fully up-to-date (there is nothing to do).

I ran some quick benchmarks comparing the performance of rendering the final Nikola version of this blog against the first YABG version (before the Bootstrap 5 redesign).

Testing methodology

Here’s the testing setup:

• AWS EC2 instances
  • c7a.xlarge (4 vCPU, 8 GB RAM)
  • 30 GB io2 SSD (30000 IOPS)
  • Total cost: $2.95 + tax for about an hour’s usage ($2.66 of which were storage costs)
• Fedora 43 from official Fedora AMI
  • Python 3.14.2 (latest available in the repos)
  • .NET SDK 10.0.102 / .NET 10.0.2 (latest available in the repos)
  • setenforce 0, SELINUX=disabled
• Windows Server 2025
  • Python 3.14.3 (latest available in winget)
  • .NET SDK 10.0.103 / .NET 10.0.3 (latest available in winget)
  • Windows Defender disabled

I ran three tests. Each test was run 11 times. The first attempt was discarded (as a warmup and to let me verify the log). The other ten attempts were averaged as the final result. I used PowerShell’s Measure-Command cmdlet for measurements.

The tests were as follows:

1. Clean build (no cache, no output)
   • Removing .doit.db, cache, and output from the Nikola site, so that everything has to be rebuilt from scratch.
   • Removing .yabg_cache.sqlite3 and output from the YABG site, so that everything has to be reuilt from scratch, most notably the Pygments code listings have to be regenerated via a subprocess.
2. Build with cache, but no output
   • Removing output from the Nikola site, so that posts rendered to HTML by docutils/Python-Markdown are cached, but the final HTML still need to be built.
   • Removing output from the YABG site, so that the code listings rendered to HTML by Pygments are cached, but everything else needs to be built.
3. Rebuild (cache and output intact)
   • Not removing anything from the Nikola site, so that there is nothing to do.
   • Not removing anything from the YABG site. Things are still rebuilt, except for Pygments code listings and thumbnails.

For YetAnotherBlogGenerator, I tested two builds: one in Release mode (standard), and another in ReadyToRun mode, trading build time and executable size for faster execution.

All the scripts I used for setup and testing can be found in listings.

Test results

Design details and highlights

Here are some fun tidbits from development.

Everything is an item

In Nikola, there are several different entities that can generate HTML files. Posts and Pages are both Post objects. Listings and galleries each have their own task generators. There’s no Listing class, everything is handled within the listing plugin. Galleries can optionally have a Post object attached (though that Post is not picked up by the file scanner, and it is not part of the timeline). The listings and galleries task generators both have ways to build directory trees.

In YABG, all of the above are Items. Specifically, they start as SourceItems and become Items when rendered. For listings, the source is just the code and the rendered content is Pygments-generated HTML. For galleries, the source is a TSV file with a list of included gallery images (order, filenames, and descriptions), and the generated content comes from a meta field named galleryIntroHtml. Gallery objects have a GalleryData object attached to their Item object as RichItemData.

This simplifies the final rendering pipeline design. Only four classes (actual classes, not temporary structures in some plugin) can render to HTML: Item, ItemGroup (tags, categories, yearly archives, gallery indexes), DirectoryTreeGroup (listings), and LinkGroup (archive and tag indexes). Each has a corresponding template model. Nikola’s sitemap generator recurses through the output directory to find files, but YABG can just use the lists of items and groups. The sitemap won’t include HTML files from the files folder, but I don’t need them there (though I could add them if needed).

Windows first, Linux in zero time

I developed YABG entirely on Windows. This forced me to think about paths and URLs as separate concepts. I couldn’t use most System.IO.Path facilities for URLs, since they would produce backslashes. As a result, there are zero bugs where backslashes leak into output on Windows. Nikola has such bugs pop up occasionally; indeed, I fixed one yesterday.

But when YABG was nearly complete, I ran it on Linux. And it just worked. No code changes needed. No output differences. (I had to add SkiaSharp.NativeAssets.Linux and apt install libfontconfig1 since I was stilll using SkiaSharp at that point, but that’s no longer needed with Magick.NET.)

Not everything is perfect, though. I added a --watch mode based on FileSystemWatcher, but it doesn’t work on Linux. I don’t need it there; I’d have to switch to polling to make it work.

Dependency injection everywhere

A good principle used in object-oriented development (though not very often in Python) is dependency injection. I have several grouping services, all implementing either IPostGrouper or IItemGrouper. They’re registered in the DI container as implementations of those interfaces. The GroupEngine doesn’t need to know about specific group types, it just gets them from the container and passes the post and item arrays.

The ItemRenderEngine has a slightly different challenge: it needs to pick the correct renderer for the post (Gallery, HTML, Listing, Markdown). The renderers are registered as keyed services. The render engine does not need to know anything about the specific renderer types, it just gets the renderer name from the SourceItem’s ScanPattern (so ultimately from the configuration file) and asks the DI container to provide it with the right implementation.

In total, there are 37 specific service implementations registered (plus system services like TimeProvider and logging). Beyond these two examples, the main benefit is testability. I can write unit tests without dependencies on unrelated services, and without monkey-patching random names. (In Python, unittest.mock does both monkey-patching and mocking.)

Okay, I haven’t written very many tests, but I could easily ask an LLM to do it.

Immutable data structures and no global state

All classes are immutable. This helps in several ways. It’s easier to reason about state when SourceItem becomes Item during rendering, compared to a single class with a nullable Content property. Immutability also makes concurrency safer. But the biggest win is how easy it was to develop the --watch mode. Every service has Scoped lifetime, and main logic lives in IMainEngine. I can just create a new scope, get the engine, and run it without state leaking between executions. No subprocess launching, no state resetting — everything disappears when the scope is disposed.

Can anyone use it?

On one hand, it’s open source under the 3-clause BSD license and available on GitHub.

On the other hand, it’s more of a source-available project. There are no docs, and it was designed specifically for this site (so some things are probably too hardcoded for your needs). In fact, this blog’s configuration and templates were directly hardcoded in the codebase until the day before launch. But I’m happy to answer questions and review pull requests!

You can jump to the scripts and configuration files if you don’t care about the theory and just want to see the end result.

Why Docker?

Docker is a technology that has taken the software engineering world by storm. The main promise is isolation: a Docker container that works on an x86_64 Linux machine will work on any x86_64 Linux machine in the same way. Want to quickly set up PostgreSQL for testing? Just run docker run --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d --restart=unless-stopped postgres and wait a few seconds. Docker is great for deployment as well as production deployments, and it even supports Windows Server containers these days (whether or not this is a pleasant experience is a different question).

Of course, there is a trade-off: running something in a Docker container requires more disk space than running the same software outside of Docker would. This is because Docker containers have their own copies of everything: the C library, the shell, core commands, and the runtime of your favorite language. But this is not a bug, it’s a feature.

If you read the nginx/uWSGI Emperor guide I wrote ten years ago, you might notice there are many cases where the configuration differs depending on the Linux distribution in use. Some distributions made logging an optional feature, others have it built in. Each distribution has a slightly different directory structure, and different users and groups for Web services. Some distros did not ship systemd service files. Red Hat is still pushing SELinux.

But uWSGI is not the only pain point. There’s also the system Python. Every distribution treats it differently and applies different customizations. Arch maps python to python3, but other distributions do not. Arch and Fedora ship a single Python package, while Debian/Ubuntu have many split packages. Taking a dependency on the system Python also makes distro upgrades harder: if the system Python is upgraded, the virtual environment needs to be recreated. (Hope you have an up-to-date requirements.txt!)

Should everything be in Docker?

It depends on your definition of everything. I ended up with only one non-dockerized application (which requires access to resources that cannot easily be provided to it when it is in a container): an ASP.NET Core Minimal API compiled with Native AOT, so its maintenance burden is essentially zero. Except for that app, all web applications on my VPS, whether written in PHP, Python, or .NET, run in Docker. I don't need to figure out how to make PHP happy; I use a container image in which someone else had done the hard parts for me. I don't need to maintain Python virtual environments; they can just live inside containers. The maintenance burden for .NET is smaller, but containers still prevent issues that could be caused by removing an old version of .NET, for example. (Those apps are ASP.NET Core MVC, so they are not compatible with Native AOT.)

But I am not running the Web-facing nginx instance in Docker. Similarly, I’ve kept PostgreSQL outside of Docker, and I’ve just un-dockerized a MariaDB instance. The main difference here is that I can apt install nginx and get a supported and secure build, with default configuration that might be more reasonable than the default. (But then again, apt install python3 gets you a fairly mediocre build.)

For the Python dockerization project, my requirements are fairly simple. The Docker container only needs Python, a venv with dependencies installed, and all data files of the app. The database exists outside of Docker, and I’ve already configured PostgreSQL to listen on an address accessible from within Docker (but of course, not from the public Internet). Because Django and Python have abysmal performance, static files must be served by nginx. Since I don’t want a dedicated nginx in a Docker container, the easiest solution is to mount a folder as a volume: Django runs collectstatic inside the container to write static files there, and the host nginx serves them.

What should be in Docker?

There are four things that we need to put in our Docker container: Linux, Python, a venv, and a WSGI server.

Linux and Python (base image)

The choice of a base image can affect a few things about the Docker experience. For Python, the most commonly used image is the one prepared by Docker, Inc. simply named python. That image has three versions, or tags in Docker parlance:

• python:3.14 (the default), which is based on Debian (1.12GB)
• python:3.14-slim, which is based on Debian but with less cruft (119MB)
• python:3.14-alpine, which is bvased on Alpine Linux (47.4MB)

Alpine Linux images are really small, but with the caveat that they are based on the musl libc instead of GNU glibc, which means most binaries, including manylinux binary wheels, do not work, and special builds (musllinux binary wheels in this case) are required. I started with the Debian-based images (the default one for build, the slim one for production), but I switched to python:3.14-alpine, as the only binary dependency I have is psycopg2, which I build from source, but there are musllinux wheels of psycopg2-binary available.

Virtual environments and dependency management

Python packaging is still a mess. I don’t feel like using the VC-backed uv, and the other big tools (like poetry or pipenv) introduce too much magic and bloat. All I need is a requirements.txt file I can install with pip into a venv, but without having to manually track version numbers. For that, pip-compile from pip-tools is a great option. It takes a requirements.in file with package names and optional version ranges, and produces a requirements.txt with all dependencies (including transitive dependencies) pinned to exact versions. It doesn’t get much simpler than that.

There is one issue: the requirements.txt files produced by pip-compile are specific to the environment in which it was executed. So if you want a requirements.txt for Linux and Python 3.14, you must run pip-compile on Linux with Python 3.14. While this does not matter much for this project (it has very simple dependencies), I wanted to ensure the tool runs with the same Python version the container will use, and to allow building the image without having pip-compile installed on the development machine.

So, I quickly hacked together a tool unoriginally named docker-pip-compile. It’s a five-line Dockerfile and a slightly longer shell script to help run it. That way, dependencies can be updated and the entire project can be built even without a functional system Python. The only catch here (and the reason for the hackiest line in the Dockerfile) is the fact that the package must be buildable in the environment where pip-compile runs, so I had to install libpq (the PostgreSQL client library) there.

WSGI server

The old post used uWSGI (it’s even mentioned in the title). Sadly, uWSGI has been in maintenance mode since 2022. On the other hand, gunicorn is doing pretty well, so I used that. I also decided to add uvicorn to the mix, because why not.

Scripts and configuration files

You might be able to find a more up-to-date version of those scripts in the nikola-users repository.

Dockerfile

The Dockerfile is pretty straightforward. To save a little disk space, I use a multi-stage build. The build stage sets up a virtual environment, installs packages into it, and copies files into /app. The final stage installs libpq (the PostgreSQL client library, needed for psycopg2) and copies /venv and /app from the build stage. (The disk space savings come from not having build-base and libpq-dev in the final image.)

docker-entrypoint.sh

The entrypoint is fairly simple as well. Before starting gunicorn, we need to run migrations and collect static files. To avoid repeating this on container restarts, we create a marker file inside the container to indicate whether setup has already completed. I use three workers, which should be enough for those sites.

docker-compose.yml

I mentioned that only one service runs inside Docker, since nginx and PostgreSQL live outside. Docker Compose may feel unnecessary with just one service, but it is still a convenient way to keep the required configuration in a file.

nginx

The nginx configuration is as simple as it gets:

Conclusion

In just two evenings, I got rid of the venv and system Python maintenance burden. When I upgrade to Ubuntu 26.04 in a few months, my Python apps will just work with no extra steps needed. Docker is a lot of fun.

Why switch?

Fedora is a relatively popular distro, so it’s well supported by software vendors. Its packagers adopt a no-nonsense approach, making very little changes that deviate from the upstream.

Ubuntu is not my favorite distro, far from it. While it is perhaps the most popular distro out there, its packages contain many more patches compared to Fedora, and Canonical (the company behind Ubuntu) are famous for betting on the wrong horse (Unity, upstart, Mir…). But one thing Ubuntu does well is stability. Fedora makes releases every 6 months, and those releases are supported for just 13 months, which means upgrading at least every year. Every upgrade may introduce incompatibilities, almost every upgrade requires recreating Python venvs. That gets boring fast, and it does not necessarily bring benefits. Granted, the Fedora system upgrade works quite well, and I upgraded through at least eight releases without a re-install, but I would still prefer to avoid it. That’s why I went with Ubuntu LTS, which is supported for five years, with a new release every two years, but which still comes with reasonably new software (and with many third-party repositories if something is missing or outdated).

Test your backups

I have a backup “system” that’s a bunch of Bash scripts. After upgrading one of the services that is being backed up, the responsible script started crashing, and thus backups stopped working. Another thing that broke was e-mails from cron, so I didn’t know anything was wrong.

While I do have full disk backups enabled at Hetzner (disclaimer: referral link), my custom backups are more fine-grained (e.g. important configuration files, database dumps, package lists), so they are quite useful in migrating between OSes.

So, here’s a reminder not only to test your backups regularly, but also to make sure they are being created at all, and to make sure cron can send you logs somewhere you can see them.

Bonus cron tip: set MAILFROM= and MAILTO= in your crontab if your SMTP server does not like the values cron uses by default.

Think about IP address reassignment (or pray to the DNS gods)

A new VPS or cloud server probably means a new IP address. But if you get a new IP address, that might complicate the migration of your publicly accessible applications. If you’re proxying all your Web properties through Cloudflare or something similar, that’s probably not an issue. But if you have a raw A record somewhere, things can get complicated. DNS servers and operating systems do a lot of caching. The conventional wisdom is to wait 24 or even 48 hours after changing DNS values. This might be true if your TTL is set to a long value, but if your TTL is short, the only worry are DNS servers that ignore TTL values and cache records for longer. If you plan a migration, it’s good to check your TTL well in advance, and not worry too much about broken DNS servers.

But you might not need a new IP. Carefully review your cloud provider’s IP management options before making any changes. Hetzner is more flexible than other hosts in this regard, as it is possible to move primary public IP addresses (not “floating” or “elastic” IPs) between servers, as long as you’re okay with a few minutes’ downtime (you will need to shut down the source and destination servers).

If you’re not okay with any downtime, you would probably want to leverage the floating/elastic IP feature, or hope DNS propagates quickly enough.

Trim the fat

My VPS ran a lot of services I don’t need anymore, but never really got around to decommissioning. For example, I had a full Xfce install with VNC access (the VNC server was only running when needed). I haven’t actually used the desktop for ages, so I just dropped it.

I also had an OpenVPN setup. It was useful years ago, when mobile data allowances were much smaller and speeds much worse. These days, I don’t use public WiFi networks at all, unless I’m going abroad, and I just buy one month of Mullvad VPN for €5 whenever that happens. So, add another service to the “do not migrate” list.

One thing that I could not just remove was the e-mail server. Many years ago, I ran a reasonably functional e-mail server on my VPS. I’ve since then migrated to Zoho Mail (which costs €10.80/year), in part due to IP reputation issues after changing hosting providers, and also to avoid having to fight spam. When I did that, I kept Postfix around, but as a local server for things like cron or Django to send e-mail with, and I configured it to send all e-mails via Zoho. But I did not really want to move over all the configuration, hoping that Ubuntu’s Postfix packages can work with my hacked together config from Fedora. So I replaced the server with OpenSMTPD (from the OpenBSD project), and all the Postfix configuration files with just one short configuration file:

Dockerize everything…

My server runs a few different apps, some of which are exposed on the public Internet, while some do useful work in the background. The services I have set up most recently are containerized with the help of Docker. The only Docker-based service that was stateful (and did not just use folders mounted as volumes) was a MariaDB database. Migrating that is straightforward with a simple dump-and-restore.

Of course, not everything on my server is in Docker. The public-facing nginx install isn’t, and neither is PostgreSQL (but that was also a quick dump-and-restore migration with some extra steps).

…especially Python

But then, there are the Python apps. Python the language is cool (if a little slow), but the packaging story is a total dumpster fire.

By the way, here’s a quick recap of 2024/2025 in Python packaging: the most hyped Python package manager (uv) is written in Rust, which screams “Python is a toy language in which you can’t even write a tool as simple as a package manager”. (I know, dependency resolution is computationally expensive, so doing that in Rust makes sense, but everything else could easily be done in pure Python. And no, the package manager should not manage Python installs.) Of course, almost all the other contenders listed in my 2023 post are still being developed. On the standards front, the community finally produced a lockfile standard after years of discussions.

Anyway, I have three Python apps. One of them is Isso, which is providing the comments box below this post. I used to run a modified version of Isso a long time ago, but I don’t need to anymore. I looked at the docs, and they offer a pre-built Docker image, which means I could just quickly deploy it on my server with Docker and skip the pain of managing Python environments.

The other two apps are Django projects built by yours truly. They are not containerized, they exist in venvs created using the system Python. Moving venvs between machines is generally impossible, so I had to re-create them. Of course, I hit a deprecation, because the Python maintainers (especially in the packaging world) does not understand their responsibility as maintainers of the most popular programming language. This time, it was caused by an old editable install with setuptools (using setup.py develop, not PEP 660), and installs with more recent pip/setuptools versions would not have this error… although some people want to remove the fallback to setuptools if there is no pyproject.toml, so you need to stay up to date with the whims of the Python packaging industry if you want to use Python software.

Update 2026-02-06: I migrated the two Django apps to Docker and wrote a post about it.

Don’t bother with ufw

Ubuntu ships with ufw, the “uncomplicated firewall”, in the default install. I was previously using firewalld, a Red Hat-adjacent project, but I decided to give ufw a try. Since if it’s part of the default install, it might be supported better by the system.

It turns out that Docker and ufw don’t play together. Someone has built a set of rules that are supposed to fix it, but that did not work for me.

Docker does integrate with firewalld, and Ubuntu has packages for it, so I just installed it, enabled the services that need to be publicly available and things were working again.

Update (2025-11-23): The iptables integration was not very stable on my Ubuntu system, so I disabled the iptables integration and switched to a simpler config in firewalld only.

Kill the ads (and other nonsense too)

Red Hat makes money by selling a stable OS with at least 10 years of support to enterprises, and their free offering is Fedora, with just 13 months of support; RHEL releases are branched off from Fedora. SUSE also sells SUSE Linux Enterprise and has openSUSE as the free offering (but the relationship between the paid and free version is more complicated).

Ubuntu chose a different monetization strategy: the enterprise offering is the same OS as the free offering, but it gets extra packages and extra updates. The free OS advertises the paid services. It is fairly simple to get rid of them all:

sudo apt autoremove ubuntu-pro-client
sudo chmod -x /etc/update-motd.d/*

Also, Ubuntu installs snap by default. Snap is a terrible idea. Luckily, there are no snaps installed by default on a Server install, so we can just remove snapd. We’ll also remove lxd-installer to save ~25 kB of disk space, since the installer requires snap, and lxd is another unsuccessful Canonical project.

sudo apt autoremove snapd lxd-installer

The cost of downgrading

Going from Fedora 42 (April 2025) to Ubuntu 24.04 (April 2024) means some software will be downgraded in the process. In general, this does not matter, as most software does not mind downgrades as much. One notable exception is WeeChat, the IRC client, whose config files are versioned, and Ubuntu’s version is not compatible with the one in Fedora. But here’s where Ubuntu’s popularity shines: WeeChat has its own repositories for Debian and Ubuntu, so I could just get the latest version without building it myself or trying to steal packages from a newer version.

Other than WeeChat, I haven’t experienced any other issues with software due to a downgrade. Some of it is luck (or not using new/advanced features), some of it is software caring about backwards compatibility.

Conclusion

Was it worth it? Time will tell. Upgrading Fedora itself was not that hard, and I expect Ubuntu upgrades to be OK too — the annoying part was cleaning up and getting things to work after the upgrade, and the switch means I will have to do it only every 2-4 years instead of every 6-12 months.

The switchover took a few hours, especially since I didn’t have much up-to-date documentation of what is actually installed and running, and there are always the minor details where distros differ that may require adjusting to. I think a migration like this is worth trying if rolling-release or frequently-released distros are too unstable for your needs.

Getting Started

PowerShell is so nice, Microsoft made it twice.

Specifically, there concurrently exist two products named PowerShell:

• Windows PowerShell (5.1) is a built-in component of Windows. It is proprietary, Windows-only, and is based on the equally proprietary and equally Windows-only .NET Framework 4.x. It has a blue icon.
• PowerShell (7.x), formerly known as PowerShell Core, is a stand-alone application. It is MIT-licensed (developed on GitHub), available for Windows, Linux, and macOS, and is based on the equally MIT-licensed and equally multi-platform .NET (formerly .NET Core). It has a black icon.

Windows PowerShell development stopped when PowerShell (Core) came out. There are some niceties and commands missing in it, but it is still a fine option for trying it out or for when one can’t install PowerShell on a Windows system but need to solve something with code.

All examples in this post should work in either version of PowerShell on any OS (unless explicitly noted otherwise).

Install the modern PowerShell: Windows, Linux, macOS.

Objects? In my shell?

Let’s try getting a directory listing. This is Microsoft land, so let’s try the DOS command for a directory listing — that would be dir:

This looks like a typical (if slightly verbose) file listing.

Now, let’s try to do something useful with this. Let’s get the total size of all .txt files.

In a Unix shell, one option is du -bc *.txt. The arguments: -b (--bytes) gives the real byte size, and -c (--summarize) produces a total. The result is this:

But how to get just the number? This requires text manipulation (getting the first word of the last line). Something like du -bc *.txt | tail -n 1 | cut -f 1 will do. There’s also wc --total=only --bytes *.txt — but this is specific to GNU wc, so it won’t cut it on *BSD or macOS. Another option would be to parse the output of ls -l — but that might not always be easy, and the output may contain something unexpected added by the specific ls version or the user’s specific shell configuration.

Let’s try something in PowerShell. If we do $x = dir, we’ll have the output of the dir command in $x. Let’s try to analyse it further, is the first character a newline?

That’s interesting, we didn’t get the first character or the first line, we got the first file. And if we try $x[1]?

What if we try getting the Length property out of that?

PS C:\tmp\hello> $x[1].Length
23

It turns out that dir returns an array of objects, and PowerShell knows how to format this array (and a single item from the array) into a nice table. What can we do with it? This:

We can iterate over all file objects, get their length (using ForEach-Object and a lambda), and then use Measure-Object to compute the sum (Measure-Object returns an object, we need to get its Sum property). We can replace the ForEach-Object call with the -Property argument in Measure-Object. And if we want to look into subdirectories, we can easily add -Recurse to Get-ChildItem. We get actual integers we can do math on.

You might have noticed I used Get-ChildItem instead of dir in the previous example. Get-ChildItem is the full name of the command (cmdlet). dir is one of its aliases, alongside gci and ls (Windows-only to avoid shadowing /bin/ls). Many common commands have aliases defined for easier typing and ease of use — Copy-Item can be written as cp (for compatibility with Unix), copy (for compatibility with MS-DOS), and ci. In our examples, we could also use measure for Measure-Object and foreach or % for ForEach-Object. Those aliases are a nice thing to have for interactive use, but for scripts, it’s best to use the full names for readability, and to avoid depending on the environment for those aliases.

More filesystem operations

Files per folder

There’s a photo collection in a Photos folder, grouped into folders. The objective is to see how many .jpg files are in each folder. Here’s the PowerShell solution:

In Unix land, StackOverflow has a lot of solutions. The top solution is du -a | cut -d/ -f2 | sort | uniq -c | sort -nr — a lot of tools mashed together, starting with a tool to check disk usage, and a lot of string manipulation. The second solution uses find, read, and shell globbing. The PowerShell solution is quite simple and obvious to anyone who has ever touched SQL.

The above example works for one level of nesting. For more levels, given Photos\one\two\three.jpg, use Get-ChildItem -Filter '*.jpg' -Recurse Photos, and:

• Group by $_.Directory.Name (same as before) to get two
• Group by Split-Path -Parent ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)) to get one/two
• Group by ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)).Split([System.IO.Path]::DirectorySeparatorChar)[0] to get one

(All of the above examples work for a single folder as well. The latter two examples don’t work on Windows PowerShell.)

Duplicate finder

Let’s build a simple tool to detect byte-for-byte duplicated files. Get-FileHash is a shell built-in. We can use Group-Object again, and Where-Object to filter only matching objects. Computing the hash of every file is quite inefficient, so we’ll group by the file length first, and then ensure the hashes match. This gives us a nice pipeline of 6 commands:

Serious Scripting: Software Bill of Materials

Software Bills of Materials (SBOMs) and supply chain security are all the rage these days. The boss wants to have something like that, i.e. a CSV file with a list of packages and versions, and only the direct production dependencies. Sure, there exist standards like SPDX, but the boss does not like those pesky “standards”. The backend is written in C#, and the frontend is written in Node.js. Since we care only about the production dependencies, we can look at the .csproj and package.json files. For Node packages, we’ll also try to fetch the license name from the npm API (the API is a bit more complicated for NuGet, so we’ll keep it as a TODO in this example).

Just like every well-written shell script starts with set -euo pipefail, every PowerShell script should start with $ErrorActionPreference = "Stop" so that execution is stopped as soon as something goes wrong. Note that this does not affect native commands, you still need to check $LASTEXITCODE. Another useful early command is Set-StrictMode -Version 3.0 to catch undefined variables.

For .csproj files, which are XML, we look for PackageReference elements using XPath, and then build a PSCustomObject out of a hashmap — extracting the appropriate attributes from the PackageReference nodes.

For package.json, we read the file, parse the JSON, and extract the properties of the dependencies object (it’s a map of package names to versions). To get the license, we use Invoke-RestMethod, which takes care of parsing JSON for us.

In the main body of the script, we look for the appropriate files (skipping things under node_modules) and call our parser functions. After retrieving all data, we concatenate the two arrays, convert to CSV, and use Tee-Object to output to a file and to standard output. We get this:

Could it be done in a different language? Certainly, but PowerShell is really easy to integrate with CI, e.g. GitHub Actions or Azure Pipelines. On Linux, you might be tempted to use Python — and you could get something done equally simply, as long as you don’t mind using the ugly urllib.request library, or alternatively ensuring requests is installed (and then you get into the hell that is Python package management).

Using .NET classes

PowerShell is built on top of .NET. This isn’t just the implementation technology — PowerShell gives access to everything the .NET standard library offers. For example, the alternate ways to group photos in multiple subdirectories we’ve explored above involve a call to a static method of the .NET System.IO.Path class.

Other .NET types are also available. Need a HashSet? Here goes:

It is also possible to load any .NET DLL into PowerShell (as long as it’s compatible with the .NET version PowerShell is built against) and use it as usual from C# (although possibly with slightly ugly syntax).

Sick Windows Tricks

Microsoft supposedly killed off Internet Explorer last year. Attempting to launch iexplore.exe will bring up Microsoft Edge. But you see, Internet Explorer is a crucial part of Windows, and has been so for over two decades. Software vendors have built software that depends on IE being there and being able to show web content. Some of them are using web views, but some of them prefer something else: COM.

COM, or Component Object Model, is Microsoft’s thing for interoperability between different applications and/or components. COM is basically a way for classes offered by different vendors and potentially written in different languages to talk to one another. Under the hood, COM is C++ vtables plus standard reference counting and class loading/discovery mechanisms. The .NET Framework, and its successor .NET, have always included COM interoperability. The modern WinRT platform is COM on steroids.

Coming back to Internet Explorer, it exposes some COM classes. They were not removed with iexplore.exe. This means you can bring up a regular Internet Explorer window in just two lines of PowerShell:

$ie = New-Object -ComObject InternetExplorer.Application
$ie.Visible = $true

Why would you do that? The InternetExplorer.Application object lets you control the browser, e.g. you can use $ie.Navigate("https://example.com/") to go to a page. Why would you want to launch IE in 2024? I don’t know, I guess you can use it to laugh in the faces of the Microsoft developers who removed the user-accessible shortcuts? But there definitely exist some legacy applications that expect a COM-controllable IE.

We have already explored the possibility of using classes from .NET. .NET comes with a GUI framework named Windows Forms, which can be loaded from PowerShell and used to build a GUI. There is no form designer, so it requires manually defining and positioning controls, but it actually works.

PowerShell can also do various Windows management tasks. It can manage boot settings, BitLocker, Hyper-V, networking, storage… For example, to get the percentage of disk space remaining:

$c = Get-Volume C
"$(($c.SizeRemaining / $c.Size) * 100)%"

Getting out of PowerShell land

As a shell, PowerShell can obviously launch subprocesses. Unlike something like Python, running a subprocess is as simple as running anything else. If you need to git pull, you just type that. Or you can make PowerShell interact with non-PowerShell commands, reading output and passing arguments:

I chose to compute untracked files with the help of standard .NET string manipulation methods, but there’s also a regex option. On a related note, there are three content check operators: -match uses regex, -like uses wildcards, and -contains checks collection membership.

Profile script

I use a fairly small profile script that adds some behaviours I’m used to from Unix, and to make Tab completion show a menu. Here are the most basic bits:

Apart from that, I use a few aliases and a pretty prompt with the help of oh-my-posh.

The unusual and sometimes confusing parts

PowerShell can be verbose. Some of its syntax is a little quirky, compared to other languages, e.g. the equality and logic operators (for example, -eq, -le, -and). The aliases usually help with remembering commands, but they can’t always be depended on — ls is defined as an alias only on Windows, and Windows PowerShell aliases wget and curl to Invoke-WebRequest, even though all three have completely different command line arguments and outputs (this was removed in PowerShell).

Moreover, the Unix/DOS aliases do not change the argument handling. rm -rf foo is invalid. rm -r foo is, since argument names can be abbreviated as long as the abbreviation is unambiguous. rm -r -f foo is not valid, because -f can be an abbreviation of -Filter or -Force (so rm -r -fo foo) will do. rm foo bar does not work, an array is needed: rm foo,bar.

C:\Windows\regedit.exe launches the Registry editor. "C:\Program Files\Mozilla Firefox\firefox.exe" is a string. Launching something with spaces in its name requires the call operator: & "C:\Program Files\Mozilla Firefox\firefox.exe". PowerShell’s tab completion will add the & if necessary.

There are two function call syntaxes. Calling a function/cmdlet uses the shell-style syntax with argument names: Some-Function -Arg1 value1 -Arg2 value2, and argument names can be abbreviated, and can sometimes be omitted. Calling a method requires a more traditional syntax: $obj.SomeMethod(value1, value2). Names are case-insensitive in either case.

The escape character is the backtick. The backslash is the path separator in Windows, so making it an escape character would make everything painful on Windows. At least it makes it easy to write regex.

The ugliest part

The ugliest and the least intuitive part of PowerShell is the handling of single-element arrays. PowerShell really wants to unpack them to a scalar. The command (Get-ChildItem).Length will produce the number of files in the current directory — unless there is exactly one file, in which case it will produce the single file’s size in bytes. And if there are zero items, instead of an empty array, PowerShell produces $null. Sometimes, things will work out in the end (since many cmdlets are happy to get either as inputs), but sometimes, PowerShell must be asked to stop this madness and return an array: @(Get-ChildItem).Length.

The previous example with git status leverages its --null argument to get zero-delimited data, so we expect either $null or a single string according to the rules. If we didn’t want to use --null, we would need to use @(git status --porcelain) to always get an array (but we would also need to remove quotes that git adds to paths that contain spaces).

Conclusion

PowerShell is a fine interactive shell and scripting language. While it does have some warts, it is more powerful than your usual Unix shell, and its strongly-typed, object-oriented code beats stringly-typed sh spaghetti any day.

Join me on a journey through packaging in Python and elsewhere. We’ll start by describing the classic packaging stack (involving setuptools and friends), the scientific stack (with conda), and some of the modern/alternate tools, such as Pipenv, Poetry, Hatch, or PDM. We’ll also look at some examples of packaging and dependency-related workflows seen elsewhere (Node.js and .NET). We’ll also take a glimpse at a possible future (with a venv-less workflow with PDM), and see if the PyPA agrees with the vision and insights of eight thousand users.

$ mkdir mynpmproject
$ cd mynpmproject
$ npm init
…answer a few questions…
$ ls
package.json

$ npm install --save is-even

added 5 packages, and audited 6 packages in 2s

found 0 vulnerabilities

$ ls
node_modules/  package.json  package-lock.json
$ ls node_modules
is-buffer/  is-even/  is-number/  is-odd/  kind-of/

{
  "name": "mynpmproject",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "is-even": "^1.0.0"
  }
}

$ cat index.js
var isEven = require('is-even');
console.log(isEven(0));

$ node index.js
true

$ rm -rf node_modules/is-odd
$ node index.js
node:internal/modules/cjs/loader:998
  throw err;
  ^

Error: Cannot find module 'is-odd'
Require stack:
- /tmp/mynpmproject/node_modules/is-even/index.js
- /tmp/mynpmproject/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
    at Module._load (node:internal/modules/cjs/loader:841:27)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/tmp/mynpmproject/node_modules/is-even/index.js:10:13)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/tmp/mynpmproject/node_modules/is-even/index.js',
    '/tmp/mynpmproject/index.js'
  ]
}

Node.js v18.12.1

$ pwd
/tmp/mynpmproject
$ npm install

added 1 package, and audited 6 packages in 436ms

found 0 vulnerabilities
$ node /tmp/mynpmproject/index.js
true
$ cd ~
$ node /tmp/mynpmproject/index.js
true

$ su -
# node /tmp/mynpmproject/index.js
true
# sudo -u nginx node /tmp/mynpmproject/index.js
true

$ cd /tmp
$ mv mynpmproject mynodeproject
$ node /tmp/mynodeproject/index.js
true

$ mkdir mydotnetproject
$ cd mydotnetproject
$ dotnet new console
The template "Console App" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /tmp/mydotnetproject/mydotnetproject.csproj...
  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 92 ms).
Restore succeeded.
$ ls
mydotnetproject.csproj  obj/  Program.cs

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AutoFixture" Version="4.17.0" />
  </ItemGroup>

</Project>

using AutoFixture;
var fixture = new Fixture();
var a = fixture.Create<int>();
var b = fixture.Create<int>();
var result = a + b == b + a;
Console.WriteLine(result ? "Math is working": "Math is broken");

$ dotnet run
Math is working

$ dotnet publish
$ ls bin/Debug/net6.0/publish
AutoFixture.dll*  Fare.dll*  mydotnetproject*  mydotnetproject.deps.json  mydotnetproject.dll  mydotnetproject.pdb  mydotnetproject.runtimeconfig.json
$ du -h bin/Debug/net6.0/publish
424K    bin/Debug/net6.0/publish
$ bin/Debug/net6.0/publish/mydotnetproject
Math is working

$ rm bin/Debug/net6.0/publish/AutoFixture.dll
$ bin/Debug/net6.0/publish/mydotnetproject
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'. The system cannot find the file specified.

File name: 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'
[1]    45060 IOT instruction (core dumped)  bin/Debug/net6.0/publish/mydotnetproject

$ rm -rf bin obj  # clean up, just in case
$ dotnet publish --sc -r linux-x64 -p:PublishSingleFile=true -o myoutput
Microsoft (R) Build Engine version 17.0.1+b177f8fa7 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 4.09 sec).
  mydotnetproject -> /tmp/mydotnetproject/bin/Debug/net6.0/linux-x64/mydotnetproject.dll
  mydotnetproject -> /tmp/mydotnetproject/myoutput/
$ ls myoutput
mydotnetproject*  mydotnetproject.pdb
$ myoutput/mydotnetproject
Math is working
$ du -h myoutput/*
62M     myoutput/mydotnetproject
12K     myoutput/mydotnetproject.pdb
$ file -k myoutput/mydotnetproject
myoutput/mydotnetproject: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=47637c667797007d777f4322729d89e7fa53a870, for GNU/Linux 2.6.32, stripped, too many notes (256)\012- data
$ file -k myoutput/mydotnetproject.pdb
myoutput/mydotnetproject.pdb: Microsoft Roslyn C# debugging symbols version 1.0\012- data

$ eval "$(pdm --pep582)"

$ mkdir mypdmproject
$ cd mypdmproject
$ pdm init
Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /usr/bin/python (3.11)
1. /usr/bin/python3.11 (3.11)
2. /usr/bin/python2.7 (2.7)
Please select (0): 1
Using Python interpreter: /usr/bin/python3.11 (3.11)
Would you like to create a virtualenv with /usr/bin/python3.11? [y/n] (y): n
You are using the PEP 582 mode, no virtualenv is created.
For more info, please visit https://peps.python.org/pep-0582/
Is the project a library that will be uploaded to PyPI [y/n] (n): n
License(SPDX name) (MIT):
Author name (Chris Warrick):
Author email (…):
Python requires('*' to allow any) (>=3.11):
Changes are written to pyproject.toml.
$ ls
pyproject.toml
$ pdm add requests
Adding packages to default dependencies: requests
🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 5 to add, 0 to update, 0 to remove

  ✔ Install charset-normalizer 2.1.1 successful
  ✔ Install certifi 2022.12.7 successful
  ✔ Install idna 3.4 successful
  ✔ Install requests 2.28.1 successful
  ✔ Install urllib3 1.26.13 successful

🎉 All complete!

[tool.pdm]

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "Chris Warrick", email = "…"},
]
dependencies = [
    "requests>=2.28.1",
]
requires-python = ">=3.11"
license = {text = "MIT"}

$ ls
pdm.lock  __pypackages__/  pyproject.toml
$ ls __pypackages__
3.11/
$ ls __pypackages__/3.11
bin/  include/  lib/
$ ls __pypackages__/3.11/lib
certifi/             certifi-2022.12.7.dist-info/
idna/                idna-3.4.dist-info/
charset_normalizer/  charset_normalizer-2.1.1.dist-info/
requests/            requests-2.28.1.dist-info/
urllib3/             urllib3-1.26.13.dist-info/

import requests
print(requests.__file__)
r = requests.get("https://chriswarrick.com/")
print(r.text[:15])

$ python mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

$ rm -rf __pypackages__/3.11/lib/urllib3*
$ python mypdmproject.py
Traceback (most recent call last):
  File "/tmp/mypdmproject/mypdmproject.py", line 1, in <module>
    import requests
  File "/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py", line 43, in <module>
    import urllib3
ModuleNotFoundError: No module named 'urllib3'

$ pdm install
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

  ✔ Install urllib3 1.26.13 successful

🎉 All complete!
$ pwd
/tmp/mypdmproject
$ cd ~
$ python /tmp/mypdmproject/mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>
# su -s /bin/bash -c 'eval "$(/tmp/pdmvenv/bin/pdm --pep582 bash)"; python /tmp/mypdmproject/mypdmproject.py' - nobody
su: warning: cannot change directory to /nonexistent: No such file or directory
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

PyPA versus reality: packaging survey results and PyPA reaction

Some time ago, the PSF ran a survey on packaging. Over 8000 people responded. The users have spoken:

• Most people think packaging is too complex.

• An overwhelming majority prefers using just a single tool.

• Most people also think the existence of multiple tools is not beneficial for the Python packaging ecosystem.

• Virtually everyone would prefer a clearly defined official workflow.

• Over 50% of responses think tools for other ecosystems are better at managing dependencies and installing packages.

The next step after this survey was for the packaging community to discuss its results and try to come up with a new packaging strategy. The first post from Shamika Mohanan (the Packaging Project Manager at PSF) that triggered the discussion also focused heavily on the users’ vision to unify packaging tools and to have One True Tool. This discussion was open to people involved with the packaging world; many participants of the discussion are involved with PyPA, and I don’t think I’ve seen a single comment from the people behind Poetry or PDM.

Most of the thread ended up being discussion of binary extensions, including discussions of how to help tool proliferation by making it possible for tools that aren’t setuptools to build binary extensions. There was also a lot of focus on the scientific community’s issues with libraries with native code, heavily rooted in C/C++, and with attempts to replace Conda with new PyPA-approved tools. The “unified tool” for everyone else was mentioned in some posts, but they were certainly the minority.

Some PyPA members talked about a UX analysis, and that they expect the unified tool to be re-exporting functionality from existing tools—which immediately raises the question: which tools should it export functionality from and why? Is pip install unified-packaging-tool going to bring in all fourteen? Is the fact that users are unhappy with what they have, and many of them would be happy with something lke npm/dotnet/cargo, not enough to determine the UX direction of the unified tool?

Some of them are also against breaking existing workflows. Is a unified packaging tool going to work for every single user? Definitely not. But are there that many distinct basic workflows? If we ignore things that border on bikeshedding, such as src vs no-src, or venv locations, are there that many workflows to consider? Someone making a library and someone making an application do have different needs (e.g. with regard to publishing the package or acceptable dependency versions). Someone working with C extensions (or extensions using something like Cython) may have different needs, but their needs would usually be a superset of the needs of someone working on a pure-Python project. The scientific community might have more specialized needs, related to complex non-Python parts, but I am positive many of their points could be solved by the unified tool as well, even if it’s not by the time this tool reaches v1.0. It is also possible that the scientific community might prefer to stay with Conda, or with some evolution of it that brings it closer in line with the Unified Packaging Tool but also solves the scientists’ needs better than a tool also solving the non-scientists’ needs can.

Then there’s a discussion about the existing tools and which one is the tool for the future. The maintainer of Hatch (Ofek Lev) says that Hatch can provide the “unified UX”. But do the maintainers of Poetry or PDM agree? Poetry seems to be far more active than Hatch, going by GitHub issues, and it’s also worth noting that Hatch’s bus factor is 1 (with Ofek Lev responsible for 542 out of 576 commits to the master branch). Russell Keith-Magee from BeeWare has highlighted the fact that tooling aside, the PyPA does a bad job at communicating things. Russell mentioned that one of PyPA tutorials now uses Hatch, but there is no way to know if the PyPA considers Hatch to be the future, are people supposed to migrate onto Hatch, and is Flit, another recent PyPA tool, now useless? Russell also makes good points about focusing efforts: should people focus on helping Hatch support extension modules (which, according to the Hatch maintainer, is the last scenario requiring setuptools; other participants note that you can already build native code without setuptools), or should people focus on improving setuptools compatibility with PEP 517?

There were also some people stating their opinions on unifying things in various ways—and many of them are against unifying things. There were some voices of reason, like that of Russell Keith-Magee, or of Simon Notley, who correctly noticed the thread fails to resolve problems of developers, who are confused about packaging, and don’t understand the different choices available and how they interoperate. Simon does agree that native dependencies are important and happen often in Python projects (and so do I), but the users who responded to the survey had something else in mind — as exemplified by the discussion opening post, mentioning the user expecting the simplicity of Rust’s cargo, and by the survey results. 70% of the survey respondents also use npm, so many Python users have already seen the simpler workflows. The survey respondents were also asked to rank a few focus areas based on importance. “Making Python packaging better serve common use cases and workflows” was ranked first out of the provided options [8] by 3248 participants. “Supporting a wider range of use cases (e.g. edge cases, etc.)” was ranked first by 379 people, and it was the least important in the minds of 2989 people.

One more point that highlights the detachment of packaging folk from reality was mentioned by Anderson Bravalheri. To Anderson, a new unified tool would be disrespectful of the work the maintainers of the existing tools put into maintaining them, and disrespectful of users who had to adapt to the packaging mess. This point is completely absurd. Was the replacement of MS-DOS/Windows 9x and Classic Mac OS with Windows NT and Mac OS X OS X macOS disrespectful to their respective designers, and the users who had to adapt to manually configuring minutiae, figuring out how to get all your software and hardware to run with weird limitations that were necessary in the 1980s, and the system crashing every once in a while? Was the replacement of horses with cars disrespectful to horses, and the people who were removing horse manure from the streets? Was the replacement of the Ford Model T with faster, safer, more efficient, and easier to use cars disrespectful to Henry Ford? Technology comes and goes, and sometimes, getting an improvement means we need to get rid of the old stuff. This applies outside of technology, too—you could come up with many examples of change in the world, which might have put some people out of power, but has greatly improved the lives of millions of people (the fall of communism in Europe, for example). Also, going back to the technology world of today, this sentiment suggests Anderson is far too attached to the software they write—is this a healthy approach?

Nobody raised PEP 582 or the complexity of virtual environments. It might not be visible from the ivory towers of packaging tool maintainers, who have years of experience dealing with them, but it certainly does exist for regular people, for people who think the Python provided by their Linux distro is good enough, and especially for people for whom Python is their introduction to programming.

I would like to once again highlight: that’s not just the opinion of one random rambling Chris. The opinion that Python packaging needs to be simplified and unified is held by about half of the 8774 people who took the survey.

But here’s one more interesting thing: Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be always accurate, but if we assume it is, the link to the results summary was clicked only 14 times (as of 2023-01-14 21:20 UTC). The discussion has 28 participants and 2.2k views. If we believe the link click counter, half of the discussion participants did not even bother reading what the people think.

Many software development workflows involve virtualization. WSL, Docker for Windows, and the Android Emulator are some examples of common virtualization-based tools. Then there are general virtualization tools/hypervisors, such as VMware Workstation, Hyper-V or VirtualBox. All these tools require hardware virtualization extensions (Intel VT-x, AMD-V) or at least are very slow without them. Virtualization extensions are not enabled by default in the CPU, they must be enabled by something. On typical PCs, this is often a firmware-level setting (that might be disabled by default), or it might be unconditionally enabled by the firmware. On a Mac, however, enabling VT-x is done by macOS, as part of the boot process. This means that Windows running in Boot Camp will start without virtualization, unless you want to boot into macOS first and then reboot into Windows. That setup isn’t quite ergonomic (and what if macOS refuses to shut down, as it often does for me?).

Instead, we’re going to use rEFInd, a boot manager for EFI-based systems that can boot into various OSes and also handle other parts of the boot process. But first, let’s prepare our system for this.

DISCLAIMER: Those steps may make your Mac fail to boot. I don’t take any responsibility whatsoever if that happens. Prepare for the worst — make backups, perhaps have install media ready, plan some downtime.

sudo chflags hidden /Volumes/rEFInd

./refind-install --ownhfs disk9s9

In this post, I am not going to focus in the differences between x86_64 and ARM, RISC and CISC, and all the benchmarks. Let’s assume that Apple manages to offer ARM-based CPUs that can match performance of most Intel processors in Apple’s lineup, and let’s even assume they can make an ARM Mac Pro. (A note on naming: Apple Silicon is the official name, but it sounds ugly. I’ll just call it ARM. For Intel, I’ll use either Intel or x86(_64).)

For many users, the transition will be more-or-less transparent. Sure, they’ll lose some apps, just like they probably did with Catalina (which dropped support for 32-bit Intel apps), or some apps will not be available/will be buggy in the first few months of the transition (though it will be easier than the PowerPC transition, because Apple uses little-endian byte order on ARM).