<?xml version="1.0" encoding="utf-8"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Posts about Python</title><link>https://chriswarrick.com/</link><atom:link href="https://chriswarrick.com/blog/tags/cat_python.xml" rel="self" type="application/rss+xml" /><description>A rarely updated blog, mostly about programming.</description><lastBuildDate>Fri, 06 Feb 2026 19:45:00 GMT</lastBuildDate><generator>https://github.com/Kwpolska/YetAnotherBlogGenerator</generator><item><title>Deploying Python Web Applications with Docker</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/</link><pubDate>Fri, 06 Feb 2026 19:45:00 GMT</pubDate><guid>https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/</guid><description>Ten years ago, almost to the day, I wrote a very long blog post titled Deploying Python Web Applications with nginx and uWSGI Emperor. This week, I’ve migrated the Python web applications hosted on my VPS to Docker containers. Here are some reasons why, and all my Docker files to help you do this on your server.
</description><content:encoded><![CDATA[<p>Ten years ago, almost to the day, I wrote a very long blog post titled <a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/">Deploying Python Web Applications with nginx and uWSGI Emperor</a>. This week, I’ve migrated the Python web applications hosted on my VPS to Docker containers. Here are some reasons why, and all my Docker files to help you do this on your server.</p>



<p>You can jump to the <a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#scripts-and-configuration-files">scripts and configuration files</a> if you don’t care about the theory and just want to see the end result.</p>
<h2 id="why-docker">Why Docker?</h2>
<p>Docker is a technology that has taken the software engineering world by storm. The main promise is isolation: a Docker container that works on an x86_64 Linux machine will work on any x86_64 Linux machine in the same way. Want to quickly set up PostgreSQL for testing? Just run <code>docker run --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d --restart=unless-stopped postgres</code> and wait a few seconds. Docker is great for deployment as well as production deployments, and it even supports Windows Server containers these days (whether or not this is a pleasant experience is a different question).</p>
<p>Of course, there is a trade-off: running something in a Docker container requires more disk space than running the same software outside of Docker would. This is because Docker containers have their own copies of <em>everything</em>: the C library, the shell, core commands, and the runtime of your favorite language. But this is not a bug, it’s a feature.</p>
<p>If you read <a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/">the nginx/uWSGI Emperor guide</a> I wrote ten years ago, you might notice there are many cases where the configuration differs depending on the Linux distribution in use. Some distributions made logging an optional feature, others have it built in. Each distribution has a slightly different directory structure, and different users and groups for Web services. Some distros did not ship systemd service files. Red Hat is still pushing SELinux.</p>
<p>But uWSGI is not the only pain point. There’s also the system Python. Every distribution treats it differently and applies different customizations. Arch maps <code>python</code> to <code>python3</code>, but other distributions do not. Arch and Fedora ship a single Python package, while Debian/Ubuntu have many split packages. Taking a dependency on the system Python also makes distro upgrades harder: if the system Python is upgraded, the virtual environment needs to be recreated. (Hope you have an up-to-date <code>requirements.txt</code>!)</p>
<h2 id="should-everything-be-in-docker">Should everything be in Docker?</h2>
<p>It depends on your definition of <em>everything</em>. I ended up with only one non-dockerized application (which requires access to resources that cannot easily be provided to it when it is in a container): an <a href="https://dotnet.microsoft.com/en-us/apps/aspnet">ASP.NET Core</a> Minimal API compiled with <a href="https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/">Native AOT</a>, so its maintenance burden is essentially zero. Except for that app, all web applications on my VPS, whether written in PHP, Python, or .NET, run in Docker. I don't need to figure out how to make PHP happy; I use a container image in which someone else had done the hard parts for me. I don't need to maintain Python virtual environments; they can just live inside containers. The maintenance burden for .NET is smaller, but containers still prevent issues that could be caused by removing an old version of .NET, for example. (Those apps are ASP.NET Core MVC, so they are not compatible with Native AOT.)</p>
<p>But I am not running the Web-facing nginx instance in Docker. Similarly, I’ve kept PostgreSQL outside of Docker, and I’ve just un-dockerized a MariaDB instance. The main difference here is that I can <code>apt install nginx</code> and get a supported and secure build, with default configuration that might be more reasonable than the default. (But then again, <code>apt install python3</code> gets you a fairly mediocre build.)</p>
<p>For the Python dockerization project, my requirements are fairly simple. The Docker container only needs Python, a venv with dependencies installed, and all data files of the app. The database exists outside of Docker, and I’ve already configured PostgreSQL to listen on an address accessible from within Docker (but of course, not from the public Internet). Because Django and Python have abysmal performance, static files must be served by nginx. Since I don’t want a dedicated nginx in a Docker container, the easiest solution is to mount a folder as a volume: Django runs <code>collectstatic</code> inside the container to write static files there, and the host nginx serves them.</p>
<h2 id="what-should-be-in-docker">What should be in Docker?</h2>
<p>There are four things that we need to put in our Docker container: Linux, Python, a venv, and a WSGI server.</p>
<h3 id="linux-and-python-base-image">Linux and Python (base image)</h3>
<p>The choice of a base image can affect a few things about the Docker experience. For Python, the most commonly used image is the one prepared by Docker, Inc. simply named <code>python</code>. That image has three versions, or tags in Docker parlance:</p>
<ul>
<li><code>python:3.14</code> (the default), which is based on Debian (1.12GB)</li>
<li><code>python:3.14-slim</code>, which is based on Debian but with less cruft (119MB)</li>
<li><code>python:3.14-alpine</code>, which is bvased on Alpine Linux (47.4MB)</li>
</ul>
<p>Alpine Linux images are really small, but with the caveat that they are based on the <code>musl</code> libc instead of GNU <code>glibc</code>, which means most binaries, including <code>manylinux</code> binary wheels, do not work, and special builds (<code>musllinux</code> binary wheels in this case) are required. I started with the Debian-based images (the default one for build, the slim one for production), but I switched to <code>python:3.14-alpine</code>, as the only binary dependency I have is <code>psycopg2</code>, which I build from source, but there are <code>musllinux</code> wheels of <code>psycopg2-binary</code> available.</p>
<h3 id="virtual-environments-and-dependency-management">Virtual environments and dependency management</h3>
<p>Python packaging is still a mess. I don’t feel like using the VC-backed <code>uv</code>, and the other big tools (like <code>poetry</code> or <code>pipenv</code>) introduce too much magic and bloat. All I need is a <code>requirements.txt</code> file I can install with pip into a venv, but without having to manually track version numbers. For that, <a href="https://github.com/jazzband/pip-tools"><code>pip-compile</code> from <code>pip-tools</code></a> is a great option. It takes a <code>requirements.in</code> file with package names and optional version ranges, and produces a <code>requirements.txt</code> with all dependencies (including transitive dependencies) pinned to exact versions. It doesn’t get much simpler than that.</p>
<p>There is one issue: the <code>requirements.txt</code> files produced by <code>pip-compile</code> are specific to the environment in which it was executed. So if you want a <code>requirements.txt</code> for Linux and Python 3.14, you must run <code>pip-compile</code> on Linux with Python 3.14. While this does not matter much for this project (it has very simple dependencies), I wanted to ensure the tool runs with the same Python version the container will use, and to allow building the image without having <code>pip-compile</code> installed on the development machine.</p>
<p>So, I quickly hacked together a tool unoriginally named <a href="https://github.com/Kwpolska/docker-pip-compile"><code>docker-pip-compile</code></a>. It’s a five-line <code>Dockerfile</code> and a slightly longer shell script to help run it. That way, dependencies can be updated and the entire project can be built even without a functional system Python. The only catch here (and the reason for the hackiest line in the Dockerfile) is the fact that the package must be buildable in the environment where <code>pip-compile</code> runs, so I had to install <code>libpq</code> (the PostgreSQL client library) there.</p>
<h3 id="wsgi-server">WSGI server</h3>
<p>The old post used uWSGI (it’s even mentioned in the title). Sadly, <a href="https://github.com/unbit/uwsgi/commit/5838086dd4490b8a55ff58fc0bf0f108caa4e079">uWSGI has been in maintenance mode since 2022</a>. On the other hand, <a href="https://github.com/benoitc/gunicorn">gunicorn</a> is doing pretty well, so I used that. I also decided to add <a href="https://uvicorn.dev/">uvicorn</a> to the mix, because why not.</p>
<h2 id="scripts-and-configuration-files">Scripts and configuration files</h2>
<p>You might be able to find a more up-to-date version of those scripts in the <a href="https://github.com/getnikola/nikola-users/tree/master/docker">nikola-users</a> repository.</p>
<h3 id="dockerfile">Dockerfile</h3>
<p>The Dockerfile is pretty straightforward. To save a little disk space, I use a multi-stage build. The build stage sets up a virtual environment, installs packages into it, and copies files into <code>/app</code>. The final stage installs <code>libpq</code> (the PostgreSQL client library, needed for <code>psycopg2</code>) and copies <code>/venv</code> and <code>/app</code> from the build stage. (The disk space savings come from not having <code>build-base</code> and <code>libpq-dev</code> in the final image.)</p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-1"><code data-line-number=" 1"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-1" name="code_381600d5da99646e75aaa66100a72b972eb5088c-1"></a><span class="k">FROM</span><span class="w"> </span><span class="s">python:3.14-alpine</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">build</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-2"><code data-line-number=" 2"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-2" name="code_381600d5da99646e75aaa66100a72b972eb5088c-2"></a><span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-3"><code data-line-number=" 3"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-3" name="code_381600d5da99646e75aaa66100a72b972eb5088c-3"></a><span class="k">RUN</span><span class="w"> </span>apk<span class="w"> </span>add<span class="w"> </span>build-base<span class="w"> </span>libpq<span class="w"> </span>libpq-dev
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-4"><code data-line-number=" 4"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-4" name="code_381600d5da99646e75aaa66100a72b972eb5088c-4"></a><span class="k">RUN</span><span class="w"> </span>python<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span>/venv<span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span>/venv/bin/python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>--no-cache-dir<span class="w"> </span>-U<span class="w"> </span>pip
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-5"><code data-line-number=" 5"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-5" name="code_381600d5da99646e75aaa66100a72b972eb5088c-5"></a><span class="k">COPY</span><span class="w"> </span>requirements.txt<span class="w"> </span>/app
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-6"><code data-line-number=" 6"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-6" name="code_381600d5da99646e75aaa66100a72b972eb5088c-6"></a><span class="k">RUN</span><span class="w"> </span>/venv/bin/pip<span class="w"> </span>install<span class="w"> </span>--no-cache-dir<span class="w"> </span>-r<span class="w"> </span>requirements.txt
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-7"><code data-line-number=" 7"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-7" name="code_381600d5da99646e75aaa66100a72b972eb5088c-7"></a><span class="k">COPY</span><span class="w"> </span>nikolausers<span class="w"> </span>/app/nikolausers
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-8"><code data-line-number=" 8"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-8" name="code_381600d5da99646e75aaa66100a72b972eb5088c-8"></a><span class="k">COPY</span><span class="w"> </span>sites<span class="w"> </span>/app/sites
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-9"><code data-line-number=" 9"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-9" name="code_381600d5da99646e75aaa66100a72b972eb5088c-9"></a><span class="k">COPY</span><span class="w"> </span>templates<span class="w"> </span>/app/templates
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-10"><code data-line-number="10"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-10" name="code_381600d5da99646e75aaa66100a72b972eb5088c-10"></a><span class="k">COPY</span><span class="w"> </span>manage.py<span class="w"> </span>docker/docker-entrypoint.sh<span class="w"> </span>/app/
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-11"><code data-line-number="11"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-11" name="code_381600d5da99646e75aaa66100a72b972eb5088c-11"></a><span class="k">RUN</span><span class="w"> </span>chmod<span class="w"> </span>+x<span class="w"> </span>/app/docker-entrypoint.sh
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-12"><code data-line-number="12"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-12" name="code_381600d5da99646e75aaa66100a72b972eb5088c-12"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-13"><code data-line-number="13"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-13" name="code_381600d5da99646e75aaa66100a72b972eb5088c-13"></a><span class="k">FROM</span><span class="w"> </span><span class="s">python:3.14-alpine</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-14"><code data-line-number="14"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-14" name="code_381600d5da99646e75aaa66100a72b972eb5088c-14"></a><span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-15"><code data-line-number="15"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-15" name="code_381600d5da99646e75aaa66100a72b972eb5088c-15"></a><span class="k">RUN</span><span class="w"> </span>apk<span class="w"> </span>add<span class="w"> </span>libpq
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-16"><code data-line-number="16"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-16" name="code_381600d5da99646e75aaa66100a72b972eb5088c-16"></a><span class="k">COPY</span><span class="w"> </span>--from<span class="o">=</span>build<span class="w"> </span>/venv<span class="w"> </span>/venv
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-17"><code data-line-number="17"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-17" name="code_381600d5da99646e75aaa66100a72b972eb5088c-17"></a><span class="k">COPY</span><span class="w"> </span>--from<span class="o">=</span>build<span class="w"> </span>/app<span class="w"> </span>/app
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_381600d5da99646e75aaa66100a72b972eb5088c-18"><code data-line-number="18"></code></a></td><td class="code"><code><a id="code_381600d5da99646e75aaa66100a72b972eb5088c-18" name="code_381600d5da99646e75aaa66100a72b972eb5088c-18"></a><span class="k">ENTRYPOINT</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;/app/docker-entrypoint.sh&quot;</span><span class="p">]</span>
</code></td></tr></table></div>
<h3 id="docker-entrypoint.sh">docker-entrypoint.sh</h3>
<p>The entrypoint is fairly simple as well. Before starting gunicorn, we need to run migrations and collect static files. To avoid repeating this on container restarts, we create a marker file inside the container to indicate whether setup has already completed. I use three workers, which should be enough for those sites.</p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-1"><code data-line-number="1"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-1" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-1"></a><span class="ch">#!/bin/sh</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-2"><code data-line-number="2"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-2" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-2"></a><span class="nv">statefile</span><span class="o">=</span>/tmp/migrated
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-3"><code data-line-number="3"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-3" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-3"></a><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-f<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$statefile</span><span class="s2">&quot;</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-4"><code data-line-number="4"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-4" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-4"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>/venv/bin/python<span class="w"> </span>manage.py<span class="w"> </span>collectstatic<span class="w"> </span>--noinput
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-5"><code data-line-number="5"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-5" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-5"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>/venv/bin/python<span class="w"> </span>manage.py<span class="w"> </span>migrate<span class="w"> </span>--noinput
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-6"><code data-line-number="6"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-6" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-6"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>touch<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$statefile</span><span class="s2">&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-7"><code data-line-number="7"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-7" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-7"></a><span class="k">fi</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_77ec481338d658dd68a97d1b6cf252576c12e963-8"><code data-line-number="8"></code></a></td><td class="code"><code><a id="code_77ec481338d658dd68a97d1b6cf252576c12e963-8" name="code_77ec481338d658dd68a97d1b6cf252576c12e963-8"></a><span class="nb">exec</span><span class="w"> </span>/venv/bin/python<span class="w"> </span>-m<span class="w"> </span>gunicorn<span class="w"> </span>--bind<span class="w"> </span><span class="m">0</span>.0.0.0:6868<span class="w"> </span>nikolausers.asgi:application<span class="w"> </span>-k<span class="w"> </span>uvicorn_worker.UvicornWorker<span class="w"> </span>-w<span class="w"> </span><span class="m">3</span>
</code></td></tr></table></div>
<h3 id="docker-compose.yml">docker-compose.yml</h3>
<p>I mentioned that only one service runs inside Docker, since nginx and PostgreSQL live outside. Docker Compose may feel unnecessary with just one service, but it is still a convenient way to keep the required configuration in a file.</p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-1"><code data-line-number=" 1"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-1" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-1"></a>services:
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-2"><code data-line-number=" 2"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-2" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-2"></a><span class="w">&nbsp;&nbsp;</span>nikolausers:
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-3"><code data-line-number=" 3"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-3" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-3"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>restart:<span class="w"> </span>unless-stopped
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-4"><code data-line-number=" 4"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-4" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-4"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>image:<span class="w"> </span>nikolausers:latest
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-5"><code data-line-number=" 5"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-5" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-5"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>ports:
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-6"><code data-line-number=" 6"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-6" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-6"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span>-<span class="w"> </span><span class="m">127</span>.0.0.1:6868:6868
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-7"><code data-line-number=" 7"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-7" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-7"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>user:<span class="w"> </span><span class="s2">&quot;33:33&quot;</span><span class="w"> </span><span class="c1"># www-data on Debian</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-8"><code data-line-number=" 8"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-8" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-8"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>environment:
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-9"><code data-line-number=" 9"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-9" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-9"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1">#SECRET_KEY: &quot;&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-10"><code data-line-number="10"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-10" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-10"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1">#DB_NAME: &quot;&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-11"><code data-line-number="11"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-11" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-11"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1">#DB_USER: &quot;&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-12"><code data-line-number="12"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-12" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-12"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1">#DB_PASSWORD: &quot;&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-13"><code data-line-number="13"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-13" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-13"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1">#DB_HOST: &quot;&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-14"><code data-line-number="14"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-14" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-14"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1">#EMAIL_HOST: &quot;&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-15"><code data-line-number="15"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-15" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-15"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span>volumes:
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_0df510911c34864d8036ab1906f2ab9cb403c19f-16"><code data-line-number="16"></code></a></td><td class="code"><code><a id="code_0df510911c34864d8036ab1906f2ab9cb403c19f-16" name="code_0df510911c34864d8036ab1906f2ab9cb403c19f-16"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span>-<span class="w"> </span>./static:/app/static
</code></td></tr></table></div>
<h3 id="nginx">nginx</h3>
<p>The nginx configuration is as simple as it gets:</p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-1"><code data-line-number=" 1"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-1" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-1"></a><span class="k">server</span><span class="w"> </span><span class="p">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-2"><code data-line-number=" 2"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-2" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-2"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1"># skip standard host config...</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-3"><code data-line-number=" 3"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-3" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-3"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-4"><code data-line-number=" 4"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-4" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-4"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-5"><code data-line-number=" 5"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-5" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-5"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://127.0.0.1:6868/</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-6"><code data-line-number=" 6"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-6" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-6"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">include</span><span class="w"> </span><span class="s">proxy_params</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-7"><code data-line-number=" 7"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-7" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-7"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="p">}</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-8"><code data-line-number=" 8"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-8" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-8"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-9"><code data-line-number=" 9"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-9" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-9"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">location</span><span class="w"> </span><span class="s">/static</span><span class="w"> </span><span class="p">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-10"><code data-line-number="10"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-10" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-10"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">alias</span><span class="w"> </span><span class="s">/srv/users.getnikola.com/static</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-11"><code data-line-number="11"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-11" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-11"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="p">}</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/#code_8c7aabea1826b4da5011effecea7431edf34cb4e-12"><code data-line-number="12"></code></a></td><td class="code"><code><a id="code_8c7aabea1826b4da5011effecea7431edf34cb4e-12" name="code_8c7aabea1826b4da5011effecea7431edf34cb4e-12"></a><span class="p">}</span>
</code></td></tr></table></div>
<h2 id="conclusion">Conclusion</h2>
<p>In just two evenings, I got rid of the venv and system Python maintenance burden. When I upgrade to Ubuntu 26.04 in a few months, my Python apps will just work with no extra steps needed. Docker is a lot of fun.</p>
]]></content:encoded><category>Python</category><category>Django</category><category>Docker</category><category>gunicorn</category><category>Internet</category><category>Linux</category><category>nginx</category><category>Python</category></item><item><title>Python Packaging, One Year Later: A Look Back at 2023 in Python Packaging</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/</link><pubDate>Mon, 15 Jan 2024 18:50:00 GMT</pubDate><guid>https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/</guid><description>
A year ago, I wrote about the sad state of Python packaging. The large number of tools in the space, the emphasis on writing vague standards instead of rallying around the One True Tool, and the complicated venv-based ecosystem instead of a solution similar to node_modules. What has changed in the past year? Has anything improved, is everything the same, or are things worse than they were before?
</description><content:encoded><![CDATA[
<p>A year ago, I wrote about the sad state of Python packaging. The large number of tools in the space, the emphasis on writing vague standards instead of rallying around the One True Tool, and the complicated <code class="docutils literal">venv</code>-based ecosystem instead of a solution similar to <code class="docutils literal">node_modules</code>. What has changed in the past year? Has anything improved, is everything the same, or are things worse than they were before?</p>



<section id="the-tools">
<h1>The tools</h1>
<p><a class="reference external" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/">The original post</a> listed a bunch of packaging tools, calling <em>fourteen tools at least twelve too many</em>. My idea with that was that most people would be happy with one tool that does everything, but the scientific-Python folks might have special requirements that would work best as a second tool.</p>
<p>Out of the tools named in last year’s post, all of them still seem to be maintained. Except for Flit (zero new commits in the past 30 days) and virtualenv (only automated and semi-automated version bumps), the tools have recent commits, pull requests, and issues.</p>
<p>All of those tools are still in use. <a class="reference external" href="https://framapiaf.org/&#64;fcodvpt/111540079686191842">Françoise Conil analysed all PyPI packages</a> and checked their PEP 517 build backends: setuptools is the most popular (at 50k packages), Poetry is second at 41k, Hatchling is third at 8.1k. Other tools to cross 500 users include Flit (4.4k), PDM (1.3k), Maturin (1.3k, build backend for Rust-based packages).</p>
<p>There are some new tools, of course. Those that crossed my radar are <a class="reference external" href="https://github.com/njsmith/posy">Posy</a> and <a class="reference external" href="https://github.com/mitsuhiko/rye">Rye</a>. Posy is a project of Nathaniel J. Smith (of trio fame), Rye is a project of Armin Ronacher (of Flask fame). The vision for both of them is to manage Python interpreters and projects, but not have a custom build backend (instead using something like hatchling). Posy is built on top of PyBI (a format for distributing binaries of Python interpreters, proposed by Smith in draft <a class="reference external" href="https://peps.python.org/pep-0711/">PEP 711</a>), Rye uses Gregory Szorc’s pre-built Pythons. Rye seems to be fairly complete and usable, Posy is right now a PoC of the PyBI format, and only offers a REPL with pre-installed packages.</p>
<p>Both Posy and Rye are written in Rust. On the one hand, it makes sense that the part that manages Python interpreters is not written in Python, because that would require a separate Python, not managed by Posy/Rye, to run those tools. But Rye also has its own pyproject.toml parser in Rust, and many of its commands are implemented mostly or largely using Rust (sometimes also calling one-off Python scripts; although the main tasks of creating venvs, installing packages, and working with lockfiles are handed off to <code class="docutils literal">venv</code>, <code class="docutils literal">pip</code>, and <code class="docutils literal"><span class="pre">pip-tools</span></code> respectively).</p>
<p>Speaking of Rust and Python, there’s been another project in that vein that has grown a lot <a class="reference external" href="https://astral.sh/blog/announcing-astral-the-company-behind-ruff">(and gathered a lot of funding)</a> in the past year. That project is <a class="reference external" href="https://github.com/astral-sh/ruff">Ruff</a>, which is a linter and code formatter. Ruff formats Python code, and is written in Rust. This means it’s 10–100× faster than existing tools written in Python (according to Ruff’s own benchmarks). Fast is good, I guess, but what does this say about Python? Is the fact that package tools (which aren’t rocket science, maybe except for fast dependency solvers, and which often need access to Python internals to do their job) and code formatters (which require a deep understanding of Python syntax, and parsing Python sources to ASTs, something easy by the <code class="docutils literal">ast</code> Python module) are written in another language? Does this trend make Python a toy language (as it is also often considered <em>a glue language</em> for NumPy and friends)? Also, why should contributing to a tool important to many Python developers require learning Rust?</p>
</section>
<section id="the-standards">
<h1>The standards</h1>
<p>Last time we looked at packaging standards, we focused on <a class="reference external" href="https://peps.python.org/pep-0582/">PEP 582</a>. It proposed the introduction of <code class="docutils literal">__pypackages__</code>, which would be a place for third-party packages to be installed to locally, on a per-project basis, without involving virtual environments, similarly to what <code class="docutils literal">node_modules</code> is for node. The PEP was ultimately <a class="reference external" href="https://discuss.python.org/t/pep-582-python-local-packages-directory/963/430">rejected</a> in March 2023. The PEP wasn’t perfect, and some of its choices were questionable or insufficient (such as not recursively searching for <code class="docutils literal">__pypackages__</code> in parent directories, or focusing on simple use-cases only). No new standards for something in that vein (with a better design) were proposed to this day.</p>
<p>Another contentious topic is lock files. Lock files for packaging systems are useful for reproducible dependency installations. The lock file records all installed packages (i.e. includes transitive dependencies) and their versions. Lock files often include checksums (like sha512) of the installed packages, and they often support telling apart packages installed via different groups of dependencies (runtime, buildtime, optional, development, etc.).</p>
<p>The classic way of achieving this goal are <code class="docutils literal">requirements.txt</code> files. They are specific to pip, and they only contain a list of packages, versions, and possibly checksums. Those files can be generated by <code class="docutils literal">pip freeze</code>, or the third-party <code class="docutils literal"><span class="pre">pip-compile</span></code> from <code class="docutils literal"><span class="pre">pip-tools</span></code>. <code class="docutils literal">pip freeze</code> is very basic, <code class="docutils literal"><span class="pre">pip-compile</span></code> can’t handle different groups of dependencies other than making multiple <code class="docutils literal">requirements.in</code> files, compiling them, and hoping there are no conflicts.</p>
<p>Pipenv, Poetry, and PDM have their own lockfile implementations, incompatible with one another. Rye piggybacks on top of <code class="docutils literal"><span class="pre">pip-tools</span></code>. Hatch doesn’t have anything in core; they’re waiting for a standard implementation (there are some plugins though). <a class="reference external" href="https://peps.python.org/pep-0665/">PEP 665</a> was <a class="reference external" href="https://discuss.python.org/t/pep-665-take-2-a-file-format-to-list-python-dependencies-for-reproducibility-of-an-application/11736/140">rejected</a> in January 2022. Its author, Brett Cannon, <a class="reference external" href="https://snarky.ca/state-of-standardized-lock-files-for-python-august-2023/">is working on a PoC</a> of something that <em>might</em> become a standard (named <a class="reference external" href="https://github.com/brettcannon/mousebender">mousebender</a>).</p>
<p>This is the danger of the working model adopted by the Python packaging world. Even for something as simple as lock files, there are at least four incompatible standards. An attempt at a specification was rejected due to “lukewarm reception”, even though there exist at least four implementations which are achieving roughly the same goals, and other ecosystems also went through this before.</p>
<p>Another thing important to Python are extension modules. Extension modules are written in C, and they are usually used to interact with libraries written in other languages (and also sometimes for performance). Poetry, PDM, and Hatchling don’t really support building extension modules. Setuptools does; <a class="reference external" href="https://numpy.org/doc/stable/reference/distutils_status_migration.html">SciPy and NumPy migrated from their custom numpy.distutils to Meson</a>. The team behind the PyO3 Rust bindings for Python develops <a class="reference external" href="https://github.com/PyO3/maturin">Maturin</a>, which allows for building Rust-based extension modules — but it’s not useful if you’re working with C.</p>
<p>There weren’t many packaging-related standards that were accepted in 2023. A standard worth mentioning is <a class="reference external" href="https://peps.python.org/pep-0668/">PEP 668</a>, which allows distributors to prevent <cite>pip</cite> from working (to avoid breaking distro-owned site packages) by adding an <code class="docutils literal"><span class="pre">EXTERNALLY-MANAGED</span></code> file. It was accepted in June 2022, but pip only implemented support for it in January 2023, and many distros already have enabled this feature in 2023. Preventing broken systems is a good thing.</p>
<p>But some standards did make it through. Minor and small ones aside, the most prominent 2023 standard would be <a class="reference external" href="https://peps.python.org/pep-0723/">PEP 723</a>: inline script metadata. It allows to add a comment block at the top of the file, that specifies the dependencies and the minimum Python version in a way that can be consumed by tools. Is it super useful? I don’t think so; setting up a project with pyproject.toml would easily allow things to grow. If you’re sending something via a GitHub gist, just make a repo. If you’re sending something by e-mail, just tar the folder. That approach promotes messy programming without source control.</p>
<section id="learning-curves-and-the-deception-of-simple">
<h2>Learning curves and the deception of “simple”</h2>
<p>Microsoft Word is simple, and a great beginner’s writing tool. You can make text bold with a single click. You can also make it blue in two clicks. But it’s easy to make an inconsistent mess. To make section headers, many users may just make the text bold and a bit bigger, without any consistency or semantics <a class="brackets" href="https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/#footnote-1" id="footnote-reference-1" role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a>. Making a consistent document with semantic formatting is hard in Word. Adding <a class="reference external" href="https://www.techrepublic.com/article/how-to-create-multilevel-numbered-headings-in-word-2016/">section numbering</a> requires you to select a heading and turn it into a list. There’s also supposedly some magic involved, that magic doesn’t work for me, and I have to tell Word to update the heading style. Even if you try doing things nicely, Word will randomly break, mess up the styles, mix up styles and inline ad-hoc formatting, and your document may look differently on different computers.</p>
<p>LaTeX is very confusing to a beginner, and has a massive learning curve. And you can certainly write <code class="docutils literal">\textbf{hello}</code> everywhere. But with some learning, you’ll be producing beautiful documents. You’ll define a <code class="docutils literal">\code{}</code> command that makes code monospace and adds a border today, but it might change the background and typeset in Comic Sans tomorrow if you so desire. You’ll use packages that can render code from external files with syntax highlighting. Heading numbering is on by default, but it can easily be disabled for a section. LaTeX can also automatically put new sections on new pages, for example. LaTeX was built for scientific publishing, so it has stellar support for maths and bibliographies, among other things.</p>
<p>Let’s now talk about programming. Python is simple, and a great beginner’s programming language. You can write <em>hello world</em> in a single line of code. The syntax is simpler, there are no confusing leftovers from C (like the index-based <code class="docutils literal">for</code> loop) or machine-level code (like <code class="docutils literal">break</code> in <code class="docutils literal">switch</code>), no pointers in sight. You also don’t need to write classes at all; you don’t need to write a class only to put a <code class="docutils literal">public static void main(String[] args)</code> method there <a class="brackets" href="https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/#footnote-2" id="footnote-reference-2" role="doc-noteref"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></a>. You don’t need an IDE, you can just write code using any editor (even notepad.exe will do for the first day or so), you can save it as a .py  file and run it using <code class="docutils literal">python whatever.py</code>.</p>
<p>Your code got more complicated? No worry, you can split it into multiple <code class="docutils literal">.py</code> files, use <code class="docutils literal">import name_of_other_file_without_py</code> and it will just work. Do you need more structure, grouping into folders perhaps? Well, forget about <code class="docutils literal">python whatever.py</code>, you must use <code class="docutils literal">python <span class="pre">-m</span> whatever</code>, and you must <code class="docutils literal">cd</code> to where your code is, or mess with <code class="docutils literal">PYTHONPATH</code>, or install your thing with <code class="docutils literal">pip</code>. This simple yet common action (grouping things into folders) has massively increased complexity.</p>
<p>The standard library is not enough <a class="brackets" href="https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/#footnote-3" id="footnote-reference-3" role="doc-noteref"><span class="fn-bracket">[</span>3<span class="fn-bracket">]</span></a> and you need a third-party dependency? You find some tutorial that tells you to <code class="docutils literal">pip install</code>, but <code class="docutils literal">pip</code> will now tell you to use <code class="docutils literal">apt</code>. And <code class="docutils literal">apt</code> may work, but it may give you an ancient version that does not match the tutorial you’re reading. Or it may not have the package. Or the Internet will tell you not to use Python packages from <code class="docutils literal">apt</code>. So now you need to <a class="reference external" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/">learn about venvs</a> (which add more complexity, more things to remember; most tutorials teach activation, venvs are easy to mess up via basic operations like renaming a folder, and you may end up with a venv in git or your code in a venv). Or you need to pick one of the many one-stop-shop tools to manage things.</p>
<p>In other ecosystems, an IDE is often a necessity, even for beginners. The IDE will force you into a project system (maybe not the best or most common one by default, but it will still be a coherent project system). Java will force you to make more than one file with the “1 public class = 1 file” rule, and it will be easy to do so, you won’t even need an <code class="docutils literal">import</code>.</p>
<p>Do you want folders? In Java or C#, you just create a folder in the IDE, and create a class there. The new file may have a different <code class="docutils literal">package</code>/<code class="docutils literal">namespace</code>, but the IDE will help you to add the correct <code class="docutils literal">import</code>/<code class="docutils literal">using</code> to the codebase, and there is no risk of you using too many directories (including something like <code class="docutils literal">src</code>) or using too few (not making a top-level package for all your code) that will require correcting all imports. The disruption from adding a folder in Java or C# is minimal.</p>
<p>The project system will also handle third-party packages without you needing to think about where they’re downloaded or what a virtual environment is and how to activate it from different contexts. A few clicks and you’re done. And if you don’t like IDEs? Living in the CLI is certainly possible in many ecosystems, they have reasonable CLI tools for common management tasks, as well as building and running your project.</p>
<p>PEP 723 solves a very niche problem: dependency management for single-file programs. Improving life for one-off things and messy code was apparently more important to the packaging community than any other improvements for big projects.</p>
<p>By the way, you could adapt this lesson to static and dynamic typing. Dynamic typing is easier to get started with and requires less typing, but compile-type checking can prevent many bugs — bugs that require higher test coverage to catch with dynamic typing. That’s why the JS world has TypeScript, that’s why mypy/pyright/typing has gained a lot of mindshare in the Python world.</p>
</section>
</section>
<section id="the-future">
<h1>The future…</h1>
<p>Looking at the <a class="reference external" href="https://discuss.python.org/c/packaging/14">Python Packaging Discourse</a>, there were some discussions about ways to improve things.</p>
<p>For example, this <a class="reference external" href="https://discuss.python.org/t/user-experience-with-porting-off-setup-py/37502">discussion about porting off setup.py</a> was started by Gregory Szorc, who had <a class="reference external" href="https://gregoryszorc.com/blog/2023/10/30/my-user-experience-porting-off-setup.py/">a long list of complaints</a>, pointing out the issues with the communication from the packaging world, and documentation mess (his post is worth a read, or at least a skim, because it’s long and full of packaging failures). There’s one page which recommends setuptools, another which has four options with Hatchling as a default, and another still promoting Pipenv. We’ve seen this a year ago, nothing changed in that regard. Some people tried finding solutions, some people shared their opinions… and then the Discourse moderator decided to protect his PyPA friends from having to read user feedback and locked the thread.</p>
<p>Many other threads about visions were had, like the one about <a class="reference external" href="https://discuss.python.org/t/the-10-year-view-on-python-packaging-whats-yours/31834">10-year views</a> or about <a class="reference external" href="https://discuss.python.org/t/wanting-a-singular-packaging-tool-vision/21141">singular packaging tools</a>. The strategy discussions, based on the user survey, had <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-2/23442">a second part</a> (the <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420">first one</a> concluded in January 2023), but it saw less posts than the first one, and discussions did not continue (and there were <a class="reference external" href="https://discuss.python.org/t/structure-of-the-packaging-strategy-discussions/23478">discussions about how to hold the discussions</a>). There are plans to <a class="reference external" href="https://discuss.python.org/t/draft-update-to-python-packaging-governance/31608">create a packaging council</a> — design-by-committee at its finest.</p>
<p>But all those discussions, even when not locked by an overzealous moderator, haven’t had any meaningful effect. The packaging ecosystem is still severely fragmented and confusing. <a class="reference external" href="https://packaging.python.org/en/latest/tutorials/">The PyPA docs and tutorials</a> still contradict each other. The PyPA-affiliated tools still have less features than the unaffiliated competition (even the upstart Rye has some form of lockfiles, unlike Hatch or Flit), and going by the PEP 517 build backend usage statistics, they are more popular than the modern PyPA tools. The authors of similar yet competing tools have not joined forces to produce the One True Packaging Tool.</p>
<section id="is-looking-pretty-bleak">
<h2>…is looking pretty bleak</h2>
<p>On the other hand, if you look at the 2023 contribution graphs for most packaging tools, you might be worried about the state of the packaging ecosystem.</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/pypa/pip/graphs/contributors?from=2023-01-01&amp;to=2023-12-31&amp;type=c">Pip</a> has had a healthy mix of contributors and a lot of commits going into it.</p></li>
<li><p><a class="reference external" href="https://github.com/pypa/pipenv/graphs/contributors?from=2023-01-01&amp;to=2023-12-31&amp;type=c">Pipenv</a> and <a class="reference external" href="https://github.com/pypa/setuptools/graphs/contributors?from=2023-01-01&amp;to=2023-12-31&amp;type=c">setuptools</a> have two lead committers, but still a healthy amount of commits.</p></li>
<li><p><a class="reference external" href="https://github.com/pypa/hatch/graphs/contributors?from=2023-01-01&amp;to=2023-12-31&amp;type=c">Hatch</a>, however, is a <strong>one-man-show</strong>: Ofek Lev (the project founder) made 184 commits, the second place belongs to Dependabot with 6 commits, and the third-place contributor (who is a human) has five commits.  The bus factor of Hatch and Hatchling is 1.</p></li>
</ul>
<p>The non-PyPA tools aren’t doing much better:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/python-poetry/poetry/graphs/contributors?from=2023-01-01&amp;to=2023-12-31&amp;type=c">Poetry</a> has two top contributors, but at least there are four human contributors with a double-digit number of commits.</p></li>
<li><p><a class="reference external" href="https://github.com/pdm-project/pdm/graphs/contributors?from=2023-01-01&amp;to=2023-12-31&amp;type=c">PDM</a> is a one-man-show, like Hatch.</p></li>
<li><p><a class="reference external" href="https://github.com/mitsuhiko/rye/graphs/contributors?from=2023-04-23&amp;to=2023-12-31&amp;type=c">Rye</a> has one main contributor, and three with a double-digit number of commits; note it’s pretty new (started in late April 2023) and it’s not as popular as the others.</p></li>
</ul>
</section>
</section>
<section id="conclusion">
<h1>Conclusion</h1>
<p>I understand the PyPA is a loose association of volunteers. It is sometimes said the name <em>Python Packaging Authority</em> was <a class="reference external" href="https://discuss.python.org/t/remove-the-authority-from-packaging/1993">originally a joke</a>. However, they are also the group that maintains all the packaging standards, so they <em>are</em> the authority when it comes to packaging. For example, <a class="reference external" href="https://peps.python.org/pep-0668/">PEP 668</a> starts with a warning block saying it’s a historical document, and <a class="reference external" href="https://packaging.python.org/en/latest/specifications/externally-managed-environments/">the up-to-date version of the specification is on PyPA’s site</a> (as well as a bunch of other <a class="reference external" href="https://packaging.python.org/en/latest/specifications/">packaging specs</a>).</p>
<p><strong>The PyPA should shut down or merge some duplicate projects, and work with the community (including maintainers of non-PyPA projects) to build One True Packaging Tool.</strong> To make things easier. To avoid writing code that does largely the same thing 5 times. To make sure thousands of projects don’t depend on tools with a bus factor of 1 or 2. To turn packaging from a problem and an insurmountable obstacle to something that <em>just works™</em>, something that an average developer doesn’t need to think about.</p>
<p>It’s not rocket science. Tons of languages, big and small, have a coherent packaging ecosystem (just read <a class="reference external" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/">last year’s post</a> for some examples of how simple it can be). Instead of focusing on specifications and governance, focus on producing one comprehensive, usable, user-friendly tool.</p>
<p>Discuss below or <a class="reference external" href="https://news.ycombinator.com/item?id=39004600">on Hacker News</a>.</p>
</section>
<section id="footnotes">
<h1>Footnotes</h1>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="footnote-1" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/#footnote-reference-1">1</a><span class="fn-bracket">]</span></span>
<p>Modern Word at least makes this easier, because the heading styles get top billing on the ribbon; they were hidden behind a completely non-obvious combo box that said <em>Normal</em> in Word 2003 and older.</p>
</aside>
<aside class="footnote brackets" id="footnote-2" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/#footnote-reference-2">2</a><span class="fn-bracket">]</span></span>
<p>C# 10 removed the requirement to make a class with a <code class="docutils literal">Main</code> method, it can pick up one file with top-level statements and make it the entrypoint.</p>
</aside>
<aside class="footnote brackets" id="footnote-3" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/#footnote-reference-3">3</a><span class="fn-bracket">]</span></span>
<p>The Python standard library gets a lot of praise. It <em>is</em> large compared to C, but nothing special compared to Java or C#. It is also full of low-quality libraries, like <code class="docutils literal">http.server</code> or <code class="docutils literal">urllib.request</code>, yet some people insist on only using the standard library. The standard library is also less stable and dependable (with constant deprecations and removals, and with new features requiring upgrading all of Python). All the “serious” use-cases, like web development or ML/AI/data science are impossible with just the standard library.</p>
</aside>
</aside>
</section>
]]></content:encoded><category>Python</category><category>packaging</category><category>PDM</category><category>pip</category><category>PyPA</category><category>Python</category><category>virtual environments</category></item><item><title>How to improve Python packaging, or why fourteen tools are at least twelve too many</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/</link><pubDate>Sun, 15 Jan 2023 13:45:00 GMT</pubDate><guid>https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/</guid><description>
There is an area of Python that many developers have problems with. This is an area that has seen many different solutions pop up over the years, with many different opinions, wars, and attempts to solve it. Many have complained about the packaging ecosystem and tools making their lives harder. Many beginners are confused about virtual environments. But does it have to be this way? Are the current solutions to packaging problems any good? And is the organization behind most of the packaging tools and standards part of the problem itself?
Join me on a journey through packaging in Python and elsewhere. We’ll start by describing the classic packaging stack (involving setuptools and friends), the scientific stack (with conda), and some of the modern/alternate tools, such as Pipenv, Poetry, Hatch, or PDM. We’ll also look at some examples of packaging and dependency-related workflows seen elsewhere (Node.js and .NET). We’ll also take a glimpse at a possible future (with a venv-less workflow with PDM), and see if the PyPA agrees with the vision and insights of eight thousand users.
</description><content:encoded><![CDATA[
<p>There is an area of Python that many developers have problems with. This is an area that has seen many different solutions pop up over the years, with many different opinions, wars, and attempts to solve it. Many have complained about the packaging ecosystem and tools making their lives harder. Many beginners are confused about virtual environments. But does it have to be this way? Are the current solutions to packaging problems any good? And is the organization behind most of the packaging tools and standards part of the problem itself?</p>
<p>Join me on a journey through packaging in Python and elsewhere. We’ll start by describing the classic packaging stack (involving setuptools and friends), the scientific stack (with conda), and some of the modern/alternate tools, such as Pipenv, Poetry, Hatch, or PDM. We’ll also look at some examples of packaging and dependency-related workflows seen elsewhere (Node.js and .NET). We’ll also take a glimpse at a possible future (with a venv-less workflow with PDM), and see if the PyPA agrees with the vision and insights of eight thousand users.</p>



<section id="the-plethora-of-tools">
<h1>The plethora of tools</h1>
<p>There are many packaging-related tools in Python. All of them with different authors, lineages, and often different opinions, although most of them are now unified under the Python Packaging Authority (PyPA) umbrella. Let’s take a look at them.</p>
<section id="the-classic-stack">
<h2>The classic stack</h2>
<p>The classic Python packaging stack consists of many semi-related tools. Setuptools, probably the oldest tool of the group, and itself based on <code class="docutils literal">distutils</code>, which is part of the standard library (although it will be removed in Python 3.12), is responsible for installing a single package. It previously used <code class="docutils literal">setup.py</code> files to do its job, which required arbitrary code execution. It then added support for non-executable metadata specification formats: <code class="docutils literal">setup.cfg</code>, and also <code class="docutils literal">pyproject.toml</code> (partially still in beta). However, you aren’t supposed to use <code class="docutils literal">setup.py</code> files directly these days, you’re supposed to be using pip. Pip installs packages, usually from the PyPI, but it can also support other sources (such as git repositories or the local filesystem). But where does pip install things? The default used to be to install globally and system-wide, which meant you could introduce conflicts between packages installed by pip and apt (or whatever the system package manager is). Even with a user-wide install (which pip is likely to attempt these days), you can still end up with conflicts, and you can also have conflicts in which package A requests X version 1.0.0, but package B expects X version 2.0.0—but A and B are not at all related and could live separately with their preferred version of X. Enter <code class="docutils literal">venv</code>, a standard library descendant of <code class="docutils literal">virtualenv</code>, which can create a lightweight virtual environment for packages to live in. This virtual environment gives you the separation from system packages and from different environments, but it is still tied to the system Python in some ways (and if the system Python disappears, the virtual environment stops working).</p>
<p>A few extra tools would be used in a typical packaging workflow. The <code class="docutils literal">wheel</code> package enhances Setuptools with the ability to generate wheels, which are ready-to-install (without running <code class="docutils literal">setup.py</code>). Wheels can either be pure-Python and be installed anywhere, or they can contain pre-compiled extension modules (things written in C) for a given OS and Python (and there’s even a standard that allows building and distributing one wheel for all typical Linux distros). The <code class="docutils literal">wheel</code> package should be an implementation detail, something existing inside Setuptools and/or pip, but users need to be aware of it if they want to make wheels on their system, because virtual environments produced by <code class="docutils literal">venv</code> do not have <code class="docutils literal">wheel</code> installed. Regular users who do not maintain their own packages may sometimes be told that pip is using something legacy because <code class="docutils literal">wheel</code> is not installed, which is not a good user experience. Package authors also need <code class="docutils literal">twine</code>, whose sole task is uploading source distributions or wheels, created with other tools, to PyPI (and there’s not much more to say about that tool).</p>
</section>
<section id="and-a-few-extensions">
<h2>…and a few extensions</h2>
<p>Over the years, there have been a few tools that are based on things from the classic stack. For example, <code class="docutils literal"><span class="pre">pip-tools</span></code> can simplify dependency management. While <code class="docutils literal">pip freeze</code> lets you produce a file with everything installed in your environment, there is no way to specify the dependencies you need, and get a lock file with specific versions and transitive dependencies (without installing and freezing everything), there is no easy way to skip development dependencies (e.g. IPython) when you <code class="docutils literal">pip freeze</code>, and there is no workflow to update all your dependencies with just pip. <code class="docutils literal"><span class="pre">pip-tools</span></code> adds two tools, <code class="docutils literal"><span class="pre">pip-compile</span></code> which takes in <code class="docutils literal">requirements.in</code> files with the packages you care about, and produces a <code class="docutils literal">requrirements.txt</code> with pinned versions of them and all transitive dependencies; and also <code class="docutils literal"><span class="pre">pip-sync</span></code>, which can install <code class="docutils literal">requirements.txt</code> and removes things not listed in it.</p>
<p>Another tool that might come in useful is <code class="docutils literal">virtualenvwrapper</code>, which can help you manage (create and activate) virtual environments in a central location. It has a few bells and whistles (such as custom hooks to do actions on every virtualenv creation), although for basic usage, you could replace it with a single-line shell function.</p>
<p>Yet another tool that works alongside the classic toolset is <code class="docutils literal">pipx</code>, which creates and manages virtual environments for apps written in Python. You tell it to <code class="docutils literal">pipx install Nikola</code>, and it will create a virtual environment somewhere, install Nikola into it, and put a script for launching it in <code class="docutils literal"><span class="pre">~/.local/bin</span></code>. While you could do it all yourself with venv and some symlinks, pipx can take care of this, and you don’t need to remember where the virtual environment is.</p>
</section>
<section id="the-scientific-stack-and-conda">
<h2>The scientific stack and conda</h2>
<p>The scientific Python community have had their own tools for many years. The conda tool can manage environments and packages. It doesn’t use PyPI and wheels, but rather packages from conda channels (which are prebuilt, and expect an Anaconda-distributed Python). Back in the day, when there were no wheels, this was the easiest way to get things installed on Windows; this is not as much of a problem now with binary wheels on PyPI—but the Anaconda stack is still popular in the scientific world. Conda packages can be built with <code class="docutils literal"><span class="pre">conda-build</span></code>, which is separate, but closely related to <code class="docutils literal">conda</code> itself. Conda packages are not compatible with <code class="docutils literal">pip</code> in any way, they do not follow the packaging standards used by other tools. Is this good? No, because it makes integrating the two worlds harder, but also yes, because many problems that apply to scientific packages (and their C/C++ extension modules, and their high-performance numeric libraries, and other things) do not apply to other uses of Python, so having a separate tool lets people focusing the other uses simplify their workflows.</p>
</section>
<section id="the-new-tools">
<h2>The new tools</h2>
<p>A few years ago, new packaging tools appeared. Now, there were lots of “new fancy tools” introduced in the past, with setuptools extending distutils, then distribute forking setuptools, then distribute being merged back…</p>
<p>The earliest “new tool” was Pipenv. Pipenv had really terrible and misleading marketing, and it merged pip and venv, in that Pipenv would create a venv and install packages in it (from <code class="docutils literal">Pipfile</code> or <code class="docutils literal">Pipfile.lock</code>). Pipenv can place the venv in the project folder, or hide it somewhere in the project folder (the latter is the default). However, Pipenv does not handle any packages related to packaging your code, so it’s useful only for developing non-installable applications (Django sites, for example). If you’re a library developer, you need setuptools anyway.</p>
<p>The second new tool was Poetry. It manages environments and dependencies in a similar way to Pipenv, but it can also build <code class="docutils literal">.whl</code> files with your code, and it can upload wheels and source distributions to PyPI. This means it has pretty much all the features the other tools have, except you need just one tool. However, Poetry is opinionated, and its opinions are sometimes incompatible with the rest of the packaging scene. Poetry uses the <code class="docutils literal">pyproject.toml</code> standard, but it does not follow the standard specifying how metadata should be represented in a <code class="docutils literal">pyproject.toml</code> file (PEP 621), instead using a custom <code class="docutils literal">[tool.poetry]</code> table. This is partly because Poetry came out before PEP 621, but the PEP was accepted over 2 years ago—the biggest compatibility problem is Poetry’s node-inspired <code class="docutils literal">~</code> and <code class="docutils literal">^</code> dependency version markers, which are not compatible with PEP 508 (the dependency specification standard). Poetry can package C extension modules, although it uses setuptools’ infrastructure for this (and requires a custom <code class="docutils literal">build.py</code> script).</p>
<p>Another similar tool is Hatch. This tool can also manage environments (it allows multiple environments per project, but it does not allow to put them in the project directory), and it can manage packages (but without lockfile support). Hatch can also be used to package a project (with PEP 621-compliant <code class="docutils literal">pyproject.toml</code> files) and upload it to PyPI. It does not support C extension modules.</p>
<p>A tool that tries to be a simpler re-imagining of Setuptools is Flit. It can build and install a package using a <code class="docutils literal">pyproject.toml</code> file. It also supports uploads to PyPI. It lacks support for C extension modules, and it expects you to manage environments on your own.</p>
<p>There’s one more interesting (albeit not popular or well-known) tool. This tool is PDM. It can manage venvs (but it defaults to the saner <code class="docutils literal">.venv</code> location), manage dependencies, and it uses a standards-compliant <code class="docutils literal">pyproject.toml</code>. There’s also a curious little feature called PEP 582 support, which we’ll talk about later.</p>
</section>
</section>
<section id="tooling-proliferation-and-the-python-package-authority">
<h1>Tooling proliferation and the Python Package Authority</h1>
<p>The previous sections mentioned 14 (fourteen!) distinct tools. As we’ll discover soon, that’s at least 12 too many. Let’s try to compare them.</p>
<p>First, let’s define nine things that we would expect packaging tools to do:</p>
<ol class="arabic simple">
<li><p>Manage environments</p></li>
<li><p>Install packages</p></li>
<li><p>Package/develop apps</p></li>
<li><p>Package libraries</p></li>
<li><p>Package C extension modules</p></li>
<li><p>Install in editable mode</p></li>
<li><p>Lock dependencies</p></li>
<li><p>Support pyproject.toml files</p></li>
<li><p>Upload to PyPI</p></li>
</ol>
<div style="font-size: 90%"><table class="table table-hover">
<thead>
<tr><th class="head"><p>Tool</p></th>
<th class="head"><p>Maintainer</p></th>
<th class="head"><p>Use-case</p></th>
<th class="head"><p># of supported features</p></th>
<th class="head"><p># of partially supported features</p></th>
<th class="head"><p># of unsupported features</p></th>
</tr>
</thead>
<tbody>
<tr><td><p>setuptools</p></td>
<td><p>PyPA</p></td>
<td><p>Making things installable</p></td>
<td><p>4</p></td>
<td><p>2 (pyproject.toml partially in beta, installing—only setuptools-based sdists)</p></td>
<td><p>3</p></td>
</tr>
<tr><td><p>pip</p></td>
<td><p>PyPA</p></td>
<td><p>Installing packages</p></td>
<td><p>2</p></td>
<td><p>1 (Locking dependencies only manually)</p></td>
<td><p>6</p></td>
</tr>
<tr><td><p>venv</p></td>
<td><p>PyPA</p></td>
<td><p>Creating virtual environments</p></td>
<td><p>1 (Creating environments)</p></td>
<td><p>0</p></td>
<td><p>8</p></td>
</tr>
<tr><td><p>wheel</p></td>
<td><p>PyPA</p></td>
<td><p>Building wheels in setuptools</p></td>
<td><p>0</p></td>
<td><p>1 (Building wheels in setuptools)</p></td>
<td><p>8</p></td>
</tr>
<tr><td><p>Twine</p></td>
<td><p>PyPA</p></td>
<td><p>Uploading to PyPI</p></td>
<td><p>1 (Uploading to PyPI)</p></td>
<td><p>0</p></td>
<td><p>8</p></td>
</tr>
<tr><td><p>pip-tools</p></td>
<td><p>Jazzband</p></td>
<td><p>Managing requirements files</p></td>
<td><p>2 (Locking dependencies, installing packages)</p></td>
<td><p>0</p></td>
<td><p>7</p></td>
</tr>
<tr><td><p>virtualenvwrapper</p></td>
<td><p>Doug Hellmann</p></td>
<td><p>Managing virtual environments</p></td>
<td><p>1 (Managing environments)</p></td>
<td><p>0</p></td>
<td><p>8</p></td>
</tr>
<tr><td><p>pipx</p></td>
<td><p>PyPA</p></td>
<td><p>Installing Python command-line tools</p></td>
<td><p>2 (Installing packages, editable installs)</p></td>
<td><p>1 (Managing environments)</p></td>
<td><p>6</p></td>
</tr>
<tr><td><p>conda</p></td>
<td><p>Anaconda, Inc.</p></td>
<td><p>Managing environments and dependencies</p></td>
<td><p>3 (Managing environments, installing things)</p></td>
<td><p>4 (Manual locking, packaging requires conda-build)</p></td>
<td><p>2 (pyproject.toml and PyPI)</p></td>
</tr>
<tr><td><p>Pipenv</p></td>
<td><p>PyPA</p></td>
<td><p>Managing dependencies for apps</p></td>
<td><p>3 (Managing environments, installing and locking)</p></td>
<td><p>1 (Developing apps)</p></td>
<td><p>5</p></td>
</tr>
<tr><td><p>Poetry</p></td>
<td><p>Sébastien Eustace et al.</p></td>
<td><p>Packaging and managing dependencies</p></td>
<td><p>7</p></td>
<td><p>2 (pyproject.toml, C extensions)</p></td>
<td><p>0</p></td>
</tr>
<tr><td><p>Flit</p></td>
<td><p>PyPA</p></td>
<td><p>Packaging pure-Python projects</p></td>
<td><p>5</p></td>
<td><p>1 (Installing only flit packages)</p></td>
<td><p>3</p></td>
</tr>
<tr><td><p>Hatch</p></td>
<td><p>PyPA</p></td>
<td><p>Packaging and managing dependencies</p></td>
<td><p>7</p></td>
<td><p>0</p></td>
<td><p>2 (C extensions, locking dependencies)</p></td>
</tr>
<tr><td><p>PDM</p></td>
<td><p>Frost Ming</p></td>
<td><p>Packaging and managing dependencies</p></td>
<td><p>8</p></td>
<td><p>0</p></td>
<td><p>1 (C extensions)</p></td>
</tr>
</tbody>
</table>
</div>
<details style="margin-bottom: 1rem">
<summary style="background: rgba(0, 170, 221, 10%); padding: .25rem; border-radius: .25rem">Expand table with more details about support for each feature</summary>
<div style="font-size: 90%; margin-top: .5rem"><table class="table table-hover">
<thead>
<tr><th class="head"><p>Tool</p></th>
<th class="head"><p>F1 (Envs)</p></th>
<th class="head"><p>F2 (Install)</p></th>
<th class="head"><p>F3 (Apps)</p></th>
<th class="head"><p>F4 (Libraries)</p></th>
<th class="head"><p>F5 (Extensions)</p></th>
<th class="head"><p>F6 (Editable)</p></th>
<th class="head"><p>F7 (Lock)</p></th>
<th class="head"><p>F8 (pyproject.toml)</p></th>
<th class="head"><p>F9 (Upload)</p></th>
</tr>
</thead>
<tbody>
<tr><td><p>setuptools</p></td>
<td><p>No</p></td>
<td><p>Only if authoring the package, direct use not recommended</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Beta</p></td>
<td><p>No (can build sdist)</p></td>
</tr>
<tr><td><p>pip</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>Manually</p></td>
<td><p>N/A</p></td>
<td><p>No</p></td>
</tr>
<tr><td><p>venv</p></td>
<td><p>Only creating environments</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr><td><p>wheel</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No (can build wheels)</p></td>
</tr>
<tr><td><p>Twine</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
<tr><td><p>pip-tools</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr><td><p>virtualenvwrapper</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr><td><p>pipx</p></td>
<td><p>Sort of</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr><td><p>conda</p></td>
<td><p>Yes</p></td>
<td><p>Yes (from conda channels)</p></td>
<td><p>Develop (conda-build is a separate tool)</p></td>
<td><p>With conda-build</p></td>
<td><p>With conda-build</p></td>
<td><p>Yes</p></td>
<td><p>Manually</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr><td><p>Pipenv</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Only develop</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr><td><p>Poetry</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Sort of (custom build.py script)</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes, but using custom fields</p></td>
<td><p>Yes</p></td>
</tr>
<tr><td><p>Flit</p></td>
<td><p>No</p></td>
<td><p>Only if authoring the package</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr><td><p>Hatch</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr><td><p>PDM</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
</tbody>
</table>
</div>
</details><p>You should pay close attention to the Maintainer column in the table. The vast majority of them are maintained by PyPA, the Python Packaging Authority. Even more curiously, the two tools that have the most “Yes” values (Poetry and PDM) are not maintained by the PyPA, but instead other people completely independent of them and not participating in the working group. So, is the working group successful, if it cannot produce one fully-featured tool? Is the group successful if it has multiple projects with overlapping responsibilities? Should the group focus their efforts on standards like <a class="reference external" href="https://peps.python.org/pep-0517/">PEP 517</a>, which is a common API for packaging tools, and which also encourages the creation of even more incompatible and competing tools?</p>
<p>Most importantly: which tool should a beginner use? The PyPA has a few guides and tutorials, one is <a class="reference external" href="https://packaging.python.org/en/latest/tutorials/installing-packages/">using pip + venv</a>, another is <a class="reference external" href="https://packaging.python.org/en/latest/tutorials/managing-dependencies/">using pipenv</a> (why would you still do that?), and <a class="reference external" href="https://packaging.python.org/en/latest/tutorials/packaging-projects/">another tutorial</a> that lets you pick between Hatchling (hatch’s build backend), setuptools, Flit, and PDM, without explaining the differences between them—and without using any environment tools, and without using Hatch’s/PDM’s build and PyPI upload features (instead opting to use <code class="docutils literal">python <span class="pre">-m</span> build</code> and <code class="docutils literal">twine</code>). The concept of virtual environments can be very confusing for beginners, and managing virtual environments is difficult if everyone has incompatible opinions about it.</p>
<p>It is also notable that <a class="reference external" href="https://peps.python.org/pep-0020/">PEP 20</a>, the Zen of Python, states this:</p>
<blockquote>
<p><em>There should be one-- and preferably only one --obvious way to do it.</em></p>
</blockquote>
<p>Python packaging definitely does not follow it <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-1" id="footnote-reference-1" role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a>. There are 14 ways, and none of them is obvious or the only good one. All in all, this is an unsalvageable mess. Why can’t Python pick one tool? What does the competition do? We’ll look at this in a minute. But first, let’s talk about the elephant in the room: Python virtual environments.</p>
</section>
<section id="does-python-really-need-virtual-environments">
<h1>Does Python really need virtual environments?</h1>
<p>Python relies on virtual environments for separation between projects. Virtual environments (aka virtualenvs or venvs) are folders with symlinks to a system-installed Python, and their own set of site-packages. There are a few problems with them:</p>
<section id="how-to-use-python-from-a-virtual-environment">
<h2>How to use Python from a virtual environment?</h2>
<p>There are two ways to do this. The first one is to activate it, by running the activate shell script installed in the environment’s bin directory. Another is to run the python executable (or any other script in the bin directory) directly from the venv. <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-2" id="footnote-reference-2" role="doc-noteref"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></a></p>
<p>Activating venvs directly is more convenient for developers, but it also has some problems. Sometimes, activation fails to work, due to the shell caching the location of things in <code class="docutils literal">$PATH</code>. Also, beginners are taught to <code class="docutils literal">activate</code> and run <code class="docutils literal">python</code>, which means they might be confused and try to use activate in scripts or cronjobs (but in those environments, you should not activate venvs, and instead use the Python executable directly). Virtual environment activation is more state you need to be aware of, and if you forget about it, or if it breaks, you might end up messing up your user-wide (or worse, system-wide) Python packages.</p>
</section>
<section id="how-are-system-pythons-and-virtual-environments-related">
<h2>How are (system) Pythons and virtual environments related?</h2>
<p>The virtual environment depends very tightly on the (system/global/pyenv-installed) Python used to create it. This is good for disk-space reasons (clean virtual environments don’t take up very much space), but this also makes the environment more fragile. If the Python used to create the environment is removed, the virtual environment stops working. If you fully manage your own Python, then it’s probably not going to happen, but if you depend on a system Python, upgrading packages on your OS might end up replacing Python 3.10 with Python 3.11. Some distributions (e.g. Ubuntu) would only make a jump like this on a new distribution release (so you can plan ahead), some of them (e.g. Arch) are rolling-release and a regular system upgrade may include a new Python, whereas some (e.g. Homebrew) make it even worse by using paths that include the patch Python version (3.x.y), which cause virtual environments to break much more often.</p>
</section>
<section id="how-to-manage-virtual-environments">
<h2>How to manage virtual environments?</h2>
<p>The original virtualenv tool, and its simplified standard library rewrite venv, allow you to put a virtual environment anywhere in the file system, as long as you have write privileges there. This has led to people and tools inventing their own standards. Virtualenvwrapper stores environments in a central location, and does not care about their contents. Pipenv and poetry allow you to choose (either a central location or the .venv directory in the project), and environments are tied to a project (they will use the project-specific environment if you’re in the project directory). Hatch stores environments in a central location, and it allows you to have multiple environments per project (but there is no option to share environments between projects).</p>
<p>Brett Cannon has recently done <a class="reference external" href="https://snarky.ca/classifying-python-virtual-environment-workflows/">a survey</a>, and it has shown the community is split on their workflows: some people use a central location, some put them in the project directory, some people have multiple environments with different Python versions, some people reuse virtualenvs between projects… Everyone has different needs, and different opinions. For example, I use a central directory (~/virtualenvs) and reuse environments when working on Nikola (sharing the same environment between development and 4 Nikola sites). But on the other hand, when deploying web apps, the venv lives in the project folder, because this venv needs to be used by processes running as different users (me, root, or the service account for the web server, which might have interactive login disabled, or whose home directory may be set to something ephemeral).</p>
<p>So: <strong>does Python need virtual environments?</strong> Perhaps looking how other languages handle this problem can help us figure this out for Python?</p>
</section>
</section>
<section id="how-everyone-else-is-doing-it">
<h1>How everyone else is doing it</h1>
<p>We’ll look at two ecosystems. We’ll start with <a class="reference internal" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#javascript-node-js-with-npm">JavaScript/Node.js (with npm)</a>, and then we’ll look at the <a class="reference internal" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#c-net-with-dotnet-cli-msbuild">C#/.NET (with dotnet CLI/MSBuild)</a> ecosystem for comparison. We’ll demonstrate a sample flow of making a project, installing dependencies in it, and running things. If you’re familiar with those ecosystems and want to skip the examples, continue with <a class="reference internal" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#how-is-node-better-than-python">How is Node better than Python?</a> and <a class="reference internal" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#are-those-ecosystems-tools-perfect">Are those ecosystems’ tools perfect?</a>. Otherwise, read on.</p>
<section id="javascript-node-js-with-npm">
<h2>JavaScript/Node.js (with npm)</h2>
<p>There are two tools for dealing with packages in the Node world, namely npm and Yarn. The npm CLI tool is shipped with Node, so we’ll focus on it.</p>
<p>Let’s create a project:</p>
<div class="code"><pre class="code text"><a id="rest_code_21bf981bce0e4148bc3f2129aa4786ec-1" name="rest_code_21bf981bce0e4148bc3f2129aa4786ec-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_21bf981bce0e4148bc3f2129aa4786ec-1"></a>$ mkdir mynpmproject
<a id="rest_code_21bf981bce0e4148bc3f2129aa4786ec-2" name="rest_code_21bf981bce0e4148bc3f2129aa4786ec-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_21bf981bce0e4148bc3f2129aa4786ec-2"></a>$ cd mynpmproject
<a id="rest_code_21bf981bce0e4148bc3f2129aa4786ec-3" name="rest_code_21bf981bce0e4148bc3f2129aa4786ec-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_21bf981bce0e4148bc3f2129aa4786ec-3"></a>$ npm init
<a id="rest_code_21bf981bce0e4148bc3f2129aa4786ec-4" name="rest_code_21bf981bce0e4148bc3f2129aa4786ec-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_21bf981bce0e4148bc3f2129aa4786ec-4"></a>…answer a few questions…
<a id="rest_code_21bf981bce0e4148bc3f2129aa4786ec-5" name="rest_code_21bf981bce0e4148bc3f2129aa4786ec-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_21bf981bce0e4148bc3f2129aa4786ec-5"></a>$ ls
<a id="rest_code_21bf981bce0e4148bc3f2129aa4786ec-6" name="rest_code_21bf981bce0e4148bc3f2129aa4786ec-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_21bf981bce0e4148bc3f2129aa4786ec-6"></a>package.json
</pre></div>
<p>We’ve got a package.json file, which has some metadata about our project (name, version, description, license). Let’s install a dependency:</p>
<div class="code"><pre class="code text"><a id="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-1" name="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_46c57d5e926a4f7f9dd6ae4afd442928-1"></a>$ npm install --save is-even
<a id="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-2" name="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_46c57d5e926a4f7f9dd6ae4afd442928-2"></a>
<a id="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-3" name="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_46c57d5e926a4f7f9dd6ae4afd442928-3"></a>added 5 packages, and audited 6 packages in 2s
<a id="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-4" name="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_46c57d5e926a4f7f9dd6ae4afd442928-4"></a>
<a id="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-5" name="rest_code_46c57d5e926a4f7f9dd6ae4afd442928-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_46c57d5e926a4f7f9dd6ae4afd442928-5"></a>found 0 vulnerabilities
</pre></div>
<p>The mere existence of an <code class="docutils literal"><span class="pre">is-even</span></code> package is questionable; the fact that it includes four dependencies is yet another, and the fact that it depends on <code class="docutils literal"><span class="pre">is-odd</span></code> is even worse. But this post isn’t about <code class="docutils literal"><span class="pre">is-even</span></code> or the Node ecosystem’s tendency to use tiny packages for everything (but I wrote one about this topic <a class="reference external" href="https://chriswarrick.com/blog/2019/02/15/modern-web-development-where-you-need-500-packages-to-build-bootstrap/">before</a>). Let’s look at what we have in the filesystem:</p>
<div class="code"><pre class="code text"><a id="rest_code_b7b59a6104744d158a65decb4d8b848f-1" name="rest_code_b7b59a6104744d158a65decb4d8b848f-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_b7b59a6104744d158a65decb4d8b848f-1"></a>$ ls
<a id="rest_code_b7b59a6104744d158a65decb4d8b848f-2" name="rest_code_b7b59a6104744d158a65decb4d8b848f-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_b7b59a6104744d158a65decb4d8b848f-2"></a>node_modules/  package.json  package-lock.json
<a id="rest_code_b7b59a6104744d158a65decb4d8b848f-3" name="rest_code_b7b59a6104744d158a65decb4d8b848f-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_b7b59a6104744d158a65decb4d8b848f-3"></a>$ ls node_modules
<a id="rest_code_b7b59a6104744d158a65decb4d8b848f-4" name="rest_code_b7b59a6104744d158a65decb4d8b848f-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_b7b59a6104744d158a65decb4d8b848f-4"></a>is-buffer/  is-even/  is-number/  is-odd/  kind-of/
</pre></div>
<p>Let’s also take a peek at the <code class="docutils literal">package.json</code> file:</p>
<div class="code"><pre class="code json"><a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-1" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-1"></a><span class="p">{</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-2" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-2"></a><span class="w">  </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;mynpmproject&quot;</span><span class="p">,</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-3" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-3"></a><span class="w">  </span><span class="nt">&quot;version&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;1.0.0&quot;</span><span class="p">,</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-4" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-4"></a><span class="w">  </span><span class="nt">&quot;description&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;&quot;</span><span class="p">,</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-5" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-5"></a><span class="w">  </span><span class="nt">&quot;main&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;index.js&quot;</span><span class="p">,</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-6" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-6"></a><span class="w">  </span><span class="nt">&quot;scripts&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-7" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-7"></a><span class="w">    </span><span class="nt">&quot;test&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;echo \&quot;Error: no test specified\&quot; &amp;&amp; exit 1&quot;</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-8" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-8"></a><span class="w">  </span><span class="p">},</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-9" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-9"></a><span class="w">  </span><span class="nt">&quot;author&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;&quot;</span><span class="p">,</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-10" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-10"></a><span class="w">  </span><span class="nt">&quot;license&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;ISC&quot;</span><span class="p">,</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-11" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-11"></a><span class="w">  </span><span class="nt">&quot;dependencies&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-12" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-12"></a><span class="w">    </span><span class="nt">&quot;is-even&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;^1.0.0&quot;</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-13" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-13" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-13"></a><span class="w">  </span><span class="p">}</span>
<a id="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-14" name="rest_code_2d75fbb0bc1d4353b74ed823295e39d5-14" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_2d75fbb0bc1d4353b74ed823295e39d5-14"></a><span class="p">}</span>
</pre></div>
<p>Our <code class="docutils literal">package.json</code> file now lists the dependency, and we’ve also got a lock file (<code class="docutils literal"><span class="pre">package-lock.json</span></code>), which records all the dependency versions used for this install. If this file is kept in the repository, any future attempts to <code class="docutils literal">npm install</code> will use the dependency versions listed in this file, ensuring everything will work the same as it did originally (unless one of those packages were to get removed from the registry).</p>
<p>Let’s try writing a trivial program using the module and try running it:</p>
<div class="code"><pre class="code text"><a id="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-1" name="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-1"></a>$ cat index.js
<a id="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-2" name="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-2"></a>var isEven = require(&#39;is-even&#39;);
<a id="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-3" name="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-3"></a>console.log(isEven(0));
<a id="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-4" name="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-4"></a>
<a id="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-5" name="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-5"></a>$ node index.js
<a id="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-6" name="rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_e7fb3d8f089d440f9b0f5b9eff4e3e16-6"></a>true
</pre></div>
<p>Let’s try removing <code class="docutils literal"><span class="pre">is-odd</span></code> to demonstrate how badly designed this package is:</p>
<div class="code"><pre class="code text"><a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-1" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-1"></a>$ rm -rf node_modules/is-odd
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-2" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-2"></a>$ node index.js
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-3" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-3"></a>node:internal/modules/cjs/loader:998
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-4" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-4"></a>  throw err;
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-5" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-5"></a>  ^
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-6" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-6"></a>
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-7" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-7"></a>Error: Cannot find module &#39;is-odd&#39;
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-8" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-8"></a>Require stack:
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-9" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-9"></a>- /tmp/mynpmproject/node_modules/is-even/index.js
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-10" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-10"></a>- /tmp/mynpmproject/index.js
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-11" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-11"></a>    at Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-12" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-12"></a>    at Module._load (node:internal/modules/cjs/loader:841:27)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-13" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-13" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-13"></a>    at Module.require (node:internal/modules/cjs/loader:1061:19)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-14" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-14" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-14"></a>    at require (node:internal/modules/cjs/helpers:103:18)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-15" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-15" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-15"></a>    at Object.&lt;anonymous&gt; (/tmp/mynpmproject/node_modules/is-even/index.js:10:13)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-16" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-16" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-16"></a>    at Module._compile (node:internal/modules/cjs/loader:1159:14)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-17" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-17" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-17"></a>    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-18" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-18" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-18"></a>    at Module.load (node:internal/modules/cjs/loader:1037:32)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-19" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-19" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-19"></a>    at Module._load (node:internal/modules/cjs/loader:878:12)
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-20" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-20" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-20"></a>    at Module.require (node:internal/modules/cjs/loader:1061:19) {
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-21" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-21" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-21"></a>  code: &#39;MODULE_NOT_FOUND&#39;,
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-22" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-22" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-22"></a>  requireStack: [
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-23" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-23" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-23"></a>    &#39;/tmp/mynpmproject/node_modules/is-even/index.js&#39;,
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-24" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-24" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-24"></a>    &#39;/tmp/mynpmproject/index.js&#39;
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-25" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-25" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-25"></a>  ]
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-26" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-26" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-26"></a>}
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-27" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-27" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-27"></a>
<a id="rest_code_1d4b90082f5249fe9d590a962f9584c8-28" name="rest_code_1d4b90082f5249fe9d590a962f9584c8-28" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_1d4b90082f5249fe9d590a962f9584c8-28"></a>Node.js v18.12.1
</pre></div>
<section id="how-is-node-better-than-python">
<h3>How is Node better than Python?</h3>
<p>Badly designed packages aside, we can see an important difference from Python in that there is <strong>no virtual environment</strong>, and all the packages live in the project directory. If we fix the <code class="docutils literal">node_modules</code> directory by running <code class="docutils literal">npm install</code>, we can see that I can run the script from somewhere else on the file system:</p>
<div class="code"><pre class="code text"><a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-1" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-1"></a>$ pwd
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-2" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-2"></a>/tmp/mynpmproject
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-3" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-3"></a>$ npm install
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-4" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-4"></a>
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-5" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-5"></a>added 1 package, and audited 6 packages in 436ms
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-6" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-6"></a>
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-7" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-7"></a>found 0 vulnerabilities
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-8" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-8"></a>$ node /tmp/mynpmproject/index.js
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-9" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-9"></a>true
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-10" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-10"></a>$ cd ~
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-11" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-11"></a>$ node /tmp/mynpmproject/index.js
<a id="rest_code_bc27cf0914284bc4be97070d5b0880e8-12" name="rest_code_bc27cf0914284bc4be97070d5b0880e8-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_bc27cf0914284bc4be97070d5b0880e8-12"></a>true
</pre></div>
<p><strong>If you try to do that with a Python tool…</strong></p>
<ul class="simple">
<li><p>If you’re using a manually managed venv, you need to remember to activate it, or to use the appropriate Python.</p></li>
<li><p>If you’re using something fancier, it might be tied to the current working directory, and it may expect you to change into that directory, or to pass an argument pointing at that directory.</p></li>
</ul>
<p>I can also run my code as <code class="docutils literal">root</code>, and as an unprivileged <code class="docutils literal">nginx</code> user, without any special preparation (like telling pipenv/poetry to put their venv in the project directory, or running them as the other users):</p>
<div class="code"><pre class="code text"><a id="rest_code_eca6b7c0c3f0414097f504b2b256ead7-1" name="rest_code_eca6b7c0c3f0414097f504b2b256ead7-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_eca6b7c0c3f0414097f504b2b256ead7-1"></a>$ su -
<a id="rest_code_eca6b7c0c3f0414097f504b2b256ead7-2" name="rest_code_eca6b7c0c3f0414097f504b2b256ead7-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_eca6b7c0c3f0414097f504b2b256ead7-2"></a># node /tmp/mynpmproject/index.js
<a id="rest_code_eca6b7c0c3f0414097f504b2b256ead7-3" name="rest_code_eca6b7c0c3f0414097f504b2b256ead7-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_eca6b7c0c3f0414097f504b2b256ead7-3"></a>true
<a id="rest_code_eca6b7c0c3f0414097f504b2b256ead7-4" name="rest_code_eca6b7c0c3f0414097f504b2b256ead7-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_eca6b7c0c3f0414097f504b2b256ead7-4"></a># sudo -u nginx node /tmp/mynpmproject/index.js
<a id="rest_code_eca6b7c0c3f0414097f504b2b256ead7-5" name="rest_code_eca6b7c0c3f0414097f504b2b256ead7-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_eca6b7c0c3f0414097f504b2b256ead7-5"></a>true
</pre></div>
<p><strong>If you try to do that with a Python tool…</strong></p>
<ul class="simple">
<li><p>If you’re using a manually managed venv, you can use its Python as another user (assuming it has the right permissions).</p></li>
<li><p>If your tool puts the venv in the project directory, this will work too.</p></li>
<li><p>If your tool puts the venv in some weird place in your home folder, the other users will get their own venvs. The <code class="docutils literal">uwsgi</code> user on Fedora uses <code class="docutils literal">/run/uwsgi</code> as its home directory, and <code class="docutils literal">/run</code> is ephemeral (tmpfs), so a reboot forces you to reinstall things.</p></li>
</ul>
<p>We can even try to change the name of our project:</p>
<div class="code"><pre class="code text"><a id="rest_code_336d5b5f844a4ec6a1fac2fd04133907-1" name="rest_code_336d5b5f844a4ec6a1fac2fd04133907-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_336d5b5f844a4ec6a1fac2fd04133907-1"></a>$ cd /tmp
<a id="rest_code_336d5b5f844a4ec6a1fac2fd04133907-2" name="rest_code_336d5b5f844a4ec6a1fac2fd04133907-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_336d5b5f844a4ec6a1fac2fd04133907-2"></a>$ mv mynpmproject mynodeproject
<a id="rest_code_336d5b5f844a4ec6a1fac2fd04133907-3" name="rest_code_336d5b5f844a4ec6a1fac2fd04133907-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_336d5b5f844a4ec6a1fac2fd04133907-3"></a>$ node /tmp/mynodeproject/index.js
<a id="rest_code_336d5b5f844a4ec6a1fac2fd04133907-4" name="rest_code_336d5b5f844a4ec6a1fac2fd04133907-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_336d5b5f844a4ec6a1fac2fd04133907-4"></a>true
</pre></div>
<p><strong>If you try to do that with a Python tool…</strong></p>
<ul class="simple">
<li><p>If you’re using a manually managed venv, and it lives in a central directory, all is well.</p></li>
<li><p>If you or your tool places the venv in the project directory, the venv is now broken, and you need to recreate it (hope you have a recent <code class="docutils literal">requirements.txt</code>!)</p></li>
<li><p>If your tool puts the venv in some weird place in your home folder, it may decide that this is a different project, which means it will recreate it, and you’ll have an unused virtual environment somewhere on your filesystem.</p></li>
</ul>
</section>
<section id="other-packaging-topics">
<h3>Other packaging topics</h3>
<p>Some packages may expose executable scripts (with the <code class="docutils literal">bin</code> property). Those can be run in three ways:</p>
<ol class="arabic simple">
<li><p>Installed globally using <code class="docutils literal">npm install <span class="pre">-g</span></code>, which would put the script in a global location that’s likely in <code class="docutils literal">$PATH</code> (e.g. <code class="docutils literal">/usr/local/bin</code>).</p></li>
<li><p>Installed locally using <code class="docutils literal">npm install</code>, and executed with the <code class="docutils literal">npx</code> tool or manually by running the script in <code class="docutils literal"><span class="pre">node_packages/.bin</span></code>.</p></li>
<li><p>Not installed at all, but executed using the <code class="docutils literal">npx</code> tool, which will install it into a cache and run it.</p></li>
</ol>
<p>Also, if we wanted to publish our thing, we can just run <code class="docutils literal">npm publish</code> (after logging in with <code class="docutils literal">npm login</code>).</p>
</section>
</section>
<section id="c-net-with-dotnet-cli-msbuild">
<h2>C#/.NET (with dotnet CLI/MSBuild)</h2>
<p>In modern .NET, the One True Tool is the dotnet CLI, which uses MSBuild for most of the heavy lifting. (In the classic .NET Framework, the duties were split between MSBuild and NuGet.exe, but let’s focus on the modern workflow.)</p>
<p>Let’s create a project:</p>
<div class="code"><pre class="code text"><a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-1" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-1"></a>$ mkdir mydotnetproject
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-2" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-2"></a>$ cd mydotnetproject
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-3" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-3"></a>$ dotnet new console
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-4" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-4"></a>The template &quot;Console App&quot; was created successfully.
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-5" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-5"></a>
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-6" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-6"></a>Processing post-creation actions...
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-7" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-7"></a>Running &#39;dotnet restore&#39; on /tmp/mydotnetproject/mydotnetproject.csproj...
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-8" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-8"></a>  Determining projects to restore...
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-9" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-9"></a>  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 92 ms).
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-10" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-10"></a>Restore succeeded.
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-11" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-11"></a>$ ls
<a id="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-12" name="rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_5b2a5a2f4beb43b1a1391c52a8baf0e3-12"></a>mydotnetproject.csproj  obj/  Program.cs
</pre></div>
<p>We get three things: a <code class="docutils literal">mydotnetproject.csproj</code> file, which defines a few properties of our project; <code class="docutils literal">Program.cs</code>, which is a hello world program, and <code class="docutils literal">obj/</code>, which contains a few files you don’t need to care about.</p>
<p>Let’s try adding a dependency. For a pointless example, but slightly more reasonable than the JS one, we’ll use <code class="docutils literal">AutoFixture</code>, which brings in a dependency on <code class="docutils literal">Fare</code>. If we run <code class="docutils literal">dotnet add package AutoFixture</code>, we get some console output, and our <code class="docutils literal">mydotnetproject.csproj</code> now looks like this:</p>
<div class="code"><pre class="code xml"><a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-1" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-1"></a><span class="nt">&lt;Project</span><span class="w"> </span><span class="na">Sdk=</span><span class="s">&quot;Microsoft.NET.Sdk&quot;</span><span class="nt">&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-2" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-2"></a>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-3" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-3"></a><span class="w">  </span><span class="nt">&lt;PropertyGroup&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-4" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-4"></a><span class="w">    </span><span class="nt">&lt;OutputType&gt;</span>Exe<span class="nt">&lt;/OutputType&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-5" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-5"></a><span class="w">    </span><span class="nt">&lt;TargetFramework&gt;</span>net6.0<span class="nt">&lt;/TargetFramework&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-6" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-6"></a><span class="w">    </span><span class="nt">&lt;ImplicitUsings&gt;</span>enable<span class="nt">&lt;/ImplicitUsings&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-7" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-7"></a><span class="w">    </span><span class="nt">&lt;Nullable&gt;</span>enable<span class="nt">&lt;/Nullable&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-8" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-8"></a><span class="w">  </span><span class="nt">&lt;/PropertyGroup&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-9" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-9"></a>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-10" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-10"></a><span class="w">  </span><span class="nt">&lt;ItemGroup&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-11" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-11"></a><span class="w">    </span><span class="nt">&lt;PackageReference</span><span class="w"> </span><span class="na">Include=</span><span class="s">&quot;AutoFixture&quot;</span><span class="w"> </span><span class="na">Version=</span><span class="s">&quot;4.17.0&quot;</span><span class="w"> </span><span class="nt">/&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-12" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-12"></a><span class="w">  </span><span class="nt">&lt;/ItemGroup&gt;</span>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-13" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-13" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-13"></a>
<a id="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-14" name="rest_code_f08cc48954da4007ab1cd6ab092bfb4f-14" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f08cc48954da4007ab1cd6ab092bfb4f-14"></a><span class="nt">&lt;/Project&gt;</span>
</pre></div>
<p>The first <code class="docutils literal">&lt;PropertyGroup&gt;</code> specifies what our project is (Exe = something you can run), specifies the target framework (.NET 6.0 <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-3" id="footnote-reference-3" role="doc-noteref"><span class="fn-bracket">[</span>3<span class="fn-bracket">]</span></a>), and enables a few opt-in features of C#. The second <code class="docutils literal">&lt;ItemGroup&gt;</code> was inserted when we installed AutoFixture.</p>
<p>We can now write a pointless program in C#. Here’s our new <code class="docutils literal">Program.cs</code>:</p>
<div class="code"><pre class="code csharp"><a id="rest_code_3d97641a75b0423c987f8f6fb0c78955-1" name="rest_code_3d97641a75b0423c987f8f6fb0c78955-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_3d97641a75b0423c987f8f6fb0c78955-1"></a><span class="k">using</span><span class="w"> </span><span class="nn">AutoFixture</span><span class="p">;</span>
<a id="rest_code_3d97641a75b0423c987f8f6fb0c78955-2" name="rest_code_3d97641a75b0423c987f8f6fb0c78955-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_3d97641a75b0423c987f8f6fb0c78955-2"></a><span class="kt">var</span><span class="w"> </span><span class="n">fixture</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">Fixture</span><span class="p">();</span>
<a id="rest_code_3d97641a75b0423c987f8f6fb0c78955-3" name="rest_code_3d97641a75b0423c987f8f6fb0c78955-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_3d97641a75b0423c987f8f6fb0c78955-3"></a><span class="kt">var</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fixture</span><span class="p">.</span><span class="n">Create</span><span class="o">&lt;</span><span class="kt">int</span><span class="o">&gt;</span><span class="p">();</span>
<a id="rest_code_3d97641a75b0423c987f8f6fb0c78955-4" name="rest_code_3d97641a75b0423c987f8f6fb0c78955-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_3d97641a75b0423c987f8f6fb0c78955-4"></a><span class="kt">var</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fixture</span><span class="p">.</span><span class="n">Create</span><span class="o">&lt;</span><span class="kt">int</span><span class="o">&gt;</span><span class="p">();</span>
<a id="rest_code_3d97641a75b0423c987f8f6fb0c78955-5" name="rest_code_3d97641a75b0423c987f8f6fb0c78955-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_3d97641a75b0423c987f8f6fb0c78955-5"></a><span class="kt">var</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">a</span><span class="p">;</span>
<a id="rest_code_3d97641a75b0423c987f8f6fb0c78955-6" name="rest_code_3d97641a75b0423c987f8f6fb0c78955-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_3d97641a75b0423c987f8f6fb0c78955-6"></a><span class="n">Console</span><span class="p">.</span><span class="n">WriteLine</span><span class="p">(</span><span class="n">result</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="s">&quot;Math is working&quot;</span><span class="p">:</span><span class="w"> </span><span class="s">&quot;Math is broken&quot;</span><span class="p">);</span>
</pre></div>
<p>(We could just use C#’s/.NET’s built-in random number generator, AutoFixture is complete overkill here—it’s meant for auto-generating test data, with support for arbitrary classes and other data structures, and we’re just getting two random ints here. I’m using AutoFixture for this example, because it’s simple to use and demonstrate, and because it gets us a transitive dependency.)</p>
<p>And now, we can run it:</p>
<div class="code"><pre class="code text"><a id="rest_code_d9dee65276204d4cbb9b65e089915241-1" name="rest_code_d9dee65276204d4cbb9b65e089915241-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_d9dee65276204d4cbb9b65e089915241-1"></a>$ dotnet run
<a id="rest_code_d9dee65276204d4cbb9b65e089915241-2" name="rest_code_d9dee65276204d4cbb9b65e089915241-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_d9dee65276204d4cbb9b65e089915241-2"></a>Math is working
</pre></div>
<p>If we want something that can be run outside of the project, and possibly without .NET installed on the system, we can use dotnet publish. The most basic scenario:</p>
<div class="code"><pre class="code text"><a id="rest_code_0ac8438553dd4f7baedc301fd45a9d61-1" name="rest_code_0ac8438553dd4f7baedc301fd45a9d61-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0ac8438553dd4f7baedc301fd45a9d61-1"></a>$ dotnet publish
<a id="rest_code_0ac8438553dd4f7baedc301fd45a9d61-2" name="rest_code_0ac8438553dd4f7baedc301fd45a9d61-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0ac8438553dd4f7baedc301fd45a9d61-2"></a>$ ls bin/Debug/net6.0/publish
<a id="rest_code_0ac8438553dd4f7baedc301fd45a9d61-3" name="rest_code_0ac8438553dd4f7baedc301fd45a9d61-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0ac8438553dd4f7baedc301fd45a9d61-3"></a>AutoFixture.dll*  Fare.dll*  mydotnetproject*  mydotnetproject.deps.json  mydotnetproject.dll  mydotnetproject.pdb  mydotnetproject.runtimeconfig.json
<a id="rest_code_0ac8438553dd4f7baedc301fd45a9d61-4" name="rest_code_0ac8438553dd4f7baedc301fd45a9d61-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0ac8438553dd4f7baedc301fd45a9d61-4"></a>$ du -h bin/Debug/net6.0/publish
<a id="rest_code_0ac8438553dd4f7baedc301fd45a9d61-5" name="rest_code_0ac8438553dd4f7baedc301fd45a9d61-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0ac8438553dd4f7baedc301fd45a9d61-5"></a>424K    bin/Debug/net6.0/publish
<a id="rest_code_0ac8438553dd4f7baedc301fd45a9d61-6" name="rest_code_0ac8438553dd4f7baedc301fd45a9d61-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0ac8438553dd4f7baedc301fd45a9d61-6"></a>$ bin/Debug/net6.0/publish/mydotnetproject
<a id="rest_code_0ac8438553dd4f7baedc301fd45a9d61-7" name="rest_code_0ac8438553dd4f7baedc301fd45a9d61-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0ac8438553dd4f7baedc301fd45a9d61-7"></a>Math is working
</pre></div>
<p>You can see that we’ve got a few files related to our project, as well as <code class="docutils literal">AutoFixture.dll</code> and <code class="docutils literal">Fare.dll</code>, which are our dependencies (<code class="docutils literal">Fare.dll</code> is a dependency of <code class="docutils literal">AutoFixture.dll</code>). Now, let’s try to remove <code class="docutils literal">AutoFixture.dll</code> from the published distribution:</p>
<div class="code"><pre class="code text"><a id="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-1" name="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-1"></a>$ rm bin/Debug/net6.0/publish/AutoFixture.dll
<a id="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-2" name="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-2"></a>$ bin/Debug/net6.0/publish/mydotnetproject
<a id="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-3" name="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-3"></a>Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly &#39;AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f&#39;. The system cannot find the file specified.
<a id="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-4" name="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-4"></a>
<a id="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-5" name="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-5"></a>File name: &#39;AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f&#39;
<a id="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-6" name="rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_0a8ff1b606084dffa7e66c2b7f31fb5a-6"></a>[1]    45060 IOT instruction (core dumped)  bin/Debug/net6.0/publish/mydotnetproject
</pre></div>
<p>We can also try a more advanced scenario:</p>
<div class="code"><pre class="code text"><a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-1" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-1"></a>$ rm -rf bin obj  # clean up, just in case
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-2" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-2"></a>$ dotnet publish --sc -r linux-x64 -p:PublishSingleFile=true -o myoutput
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-3" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-3"></a>Microsoft (R) Build Engine version 17.0.1+b177f8fa7 for .NET
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-4" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-4"></a>Copyright (C) Microsoft Corporation. All rights reserved.
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-5" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-5"></a>
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-6" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-6"></a>  Determining projects to restore...
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-7" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-7"></a>  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 4.09 sec).
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-8" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-8"></a>  mydotnetproject -&gt; /tmp/mydotnetproject/bin/Debug/net6.0/linux-x64/mydotnetproject.dll
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-9" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-9"></a>  mydotnetproject -&gt; /tmp/mydotnetproject/myoutput/
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-10" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-10"></a>$ ls myoutput
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-11" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-11"></a>mydotnetproject*  mydotnetproject.pdb
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-12" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-12"></a>$ myoutput/mydotnetproject
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-13" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-13" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-13"></a>Math is working
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-14" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-14" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-14"></a>$ du -h myoutput/*
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-15" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-15" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-15"></a>62M     myoutput/mydotnetproject
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-16" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-16" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-16"></a>12K     myoutput/mydotnetproject.pdb
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-17" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-17" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-17"></a>$ file -k myoutput/mydotnetproject
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-18" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-18" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-18"></a>myoutput/mydotnetproject: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=47637c667797007d777f4322729d89e7fa53a870, for GNU/Linux 2.6.32, stripped, too many notes (256)\012- data
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-19" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-19" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-19"></a>$ file -k myoutput/mydotnetproject.pdb
<a id="rest_code_f3c7156ebb3d43e2afd22d946c03183c-20" name="rest_code_f3c7156ebb3d43e2afd22d946c03183c-20" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_f3c7156ebb3d43e2afd22d946c03183c-20"></a>myoutput/mydotnetproject.pdb: Microsoft Roslyn C# debugging symbols version 1.0\012- data
</pre></div>
<p>We have a single output file that contains our program, its dependencies, and parts of the .NET runtime. We also get debugging symbols if we want to run our binary with a .NET debugger and see the associated source code. (There are ways to make the binary file smaller, and we can move most arguments of <code class="docutils literal">dotnet publish</code> to the .csproj file, but this post is about Python, not .NET, so I’m not going to focus on them too much.)</p>
<section id="how-is-net-better-than-python">
<h3>How is .NET better than Python?</h3>
<p>I’m not going to bore you with the same demonstrations I’ve already shown when discussing <a class="reference internal" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#how-is-node-better-than-python">How is Node better than Python?</a>, but:</p>
<ul class="simple">
<li><p>You can run built .NET projects as any user, from anywhere in the filesystem.</p></li>
<li><p>All you need to run your code is the output directory (publishing is optional, but useful to have a cleaner output, to simplify deployment, and to possibly enable compilation to native code).</p></li>
<li><p>If you do publish in single-executable mode, you can just distribute the single executable, and your users don’t even need to have .NET installed.</p></li>
<li><p>You do not need to manage environments, you do not need special tools to run your code, you do not need to think about the current working directory when running code.</p></li>
</ul>
</section>
<section id="other-packaging-topics-1">
<h3>Other packaging topics</h3>
<p>Locking dependencies is disabled by default, but if you add <code class="docutils literal"><span class="pre">&lt;RestorePackagesWithLockFile&gt;true&lt;/RestorePackagesWithLockFile&gt;</span></code> to the <code class="docutils literal">&lt;PropertyGroup&gt;</code> in your <code class="docutils literal">.csproj</code> file, you can enable it (and get a <code class="docutils literal">packages.lock.json</code> file in output).</p>
<p>Regarding <a class="reference external" href="https://learn.microsoft.com/en-us/dotnet/core/tools/global-tools">command line tools</a>, .NET has support for those as well. They can be installed globally or locally, and may be accessed via $PATH or via the <code class="docutils literal">dotnet</code> command.</p>
<p>As for publishing your package to NuGet.org or to another repository, you might want to look at the <a class="reference external" href="https://learn.microsoft.com/en-us/nuget/quickstart/create-and-publish-a-package-using-the-dotnet-cli">full docs</a> for more details, but the short version is:</p>
<ol class="arabic simple">
<li><p>Add some metadata to the <code class="docutils literal">.csproj</code> file (e.g. <code class="docutils literal">PackageId</code> and <code class="docutils literal">Version</code>)</p></li>
<li><p>Run <code class="docutils literal">dotnet pack</code> to get a <code class="docutils literal">.nupkg</code> file</p></li>
<li><p>Run <code class="docutils literal">dotnet nuget push</code> to upload the <code class="docutils literal">.nupkg</code> file (passing the file name and an API key)</p></li>
</ol>
<p>Once again, everything is done with a single <code class="docutils literal">dotnet</code> tool. The .NET IDEs (in particular, Visual Studio and Rider) do offer friendly GUI versions of many features. Some of those GUIs might be doings things slightly differently behind the scenes, but this is transparent to the user (and the backend is still MSBuild or a close derivative of it). I can take a CLI-created project, add a dependency from Rider, and publish an executable from VS, and everything will work the same. And perhaps XML files aren’t as cool as TOML, but they’re still easy to work with in this case.</p>
</section>
</section>
<section id="other-languages-and-ecosystems">
<h2>Other languages and ecosystems</h2>
<p>While we have explored two tools for two languages in depth, there are also other languages that deserve at least a mention. In the <strong>Java</strong> world, the two most commonly used tools are Maven and Gradle. Both tools can be used to manage dependencies and build artifacts that can be executed or distributed further (things like JAR files). Other tools with support for building Java projects exist, but most people just pick one of the two. The community of <strong>Scala</strong>, which is another JVM-based language, prefers sbt (which can be used for plain Java as well), but there are also Maven or Gradle users in that community. Finally, two new-ish languages which are quite popular in the recent times, <strong>Go</strong> and <strong>Rust</strong>, have first-party tooling integrated with the rest of the toolchain. The <code class="docutils literal">go</code> command-line tool can accomplish many build/dependency/packaging tasks. Rust’s <code class="docutils literal">cargo</code>, which ships with the standard distribution of Rust, handles dependencies, builds, running code and tests, as well as publishing your stuff to a registry.</p>
</section>
<section id="are-those-ecosystems-tools-perfect">
<h2>Are those ecosystems’ tools perfect?</h2>
<p>Not always, they have their deficiencies as well. In the Node ecosystem, packages may execute arbitrary code on install, which can be a security risk (there are some known examples, like a npm package <a class="reference external" href="https://arstechnica.com/information-technology/2022/03/sabotage-code-added-to-popular-npm-package-wiped-files-in-russia-and-belarus/">wiping hard drives in Russia and Belarus</a>, or another one <a href="https://arstechnica.com/information-technology/2018/11/hacker-backdoors-widely-used-open-source-software-to-steal-bitcoin/">stealing <s>imaginary Internet money</s> Bitcoin</a>). Binary packages are not distributed on the npm registry directly, they’re either built with <code class="docutils literal"><span class="pre">node-gyp</span></code>, or have prebuilt packages downloaded via <code class="docutils literal"><span class="pre">node-pre-gyp</span></code> (which is a third-party tool).</p>
<p>In the .NET ecosystem, the tools also create an <code class="docutils literal">obj</code> directory with temporary files. Those temporary files are tied to the environment they’re running in, and while the tooling will usually re-create them if something changes, it can sometimes fail and leave you with confusing errors (which can generally be solved by removing the <code class="docutils literal">bin</code> and <code class="docutils literal">obj</code> directories). If a package depends on native code (which is not already available on the target OS as part of a shared library), it must include binary builds in the NuGet package for all the platforms it supports, as there is <a class="reference external" href="https://github.com/NuGet/Home/issues/9631">no standard way</a> to allow building something from source.</p>
<p>You can also find deficiencies in the tools for the other languages mentioned. Some people think Maven is terrible because it uses XML and Gradle is the way to go, and others think Gradle’s use of a Groovy-based DSL makes things much harder than they need to be and prefer Maven instead.</p>
</section>
</section>
<section id="pep-582-the-future-of-python-packaging">
<h1>PEP 582: the future of Python packaging?</h1>
<p>Recall that when introducing PDM, I mentioned <a class="reference external" href="https://peps.python.org/pep-0582/">PEP 582</a>. This PEP defines a <code class="docutils literal">__pypackages__</code> directory. This directory would be taken into consideration by Python when looking for imports. It would behave similarly to <code class="docutils literal">node_modules</code>. Since there will be no symlinks to the system Python, it will resolve the issues with moving the virtual environment. Because the packages live in the project, there is no problem with sharing a project directory between multiple system users. It might even be possible for different computers (but with the same Python version and OS) to share the <code class="docutils literal">__pypackages__</code> directory (in some specific cases). The proposed <code class="docutils literal">__pypackages__</code> directory structure has <code class="docutils literal"><span class="pre">lib/python3.10/site-packages/</span></code> subfolders, which still makes the “reinstall on Python upgrade” step mandatory, but it doesn’t apply to minor version upgrades, and if you’re dealing with a pure-Python dependency tree, <code class="docutils literal">mv __pypackages__/lib/python3.10 __pypackages__/lib/python3.11</code> might just work. This structure does make sense for binary dependencies, or for dependencies necessary only on older Python versions, as it allows you to use multiple Python versions with the same project directory. The PEP does not say anything about sharing <code class="docutils literal">__pypackages__</code> between projects, but you could probably solve that problem with symlinks (assuming the tooling doesn’t care if the directory is a symlink, and it shouldn’t care IMO).</p>
<p>While PEP 582 is a great vision, and it would simplify many package-related workflows, it hasn’t seen much care from the powers-that-be. The PEP was proposed in May 2018, and there’s even <a class="reference external" href="https://github.com/kushaldas/pep582/blob/main/pep582.py">a usable implementation</a> that’s less than 50 lines of code, there <a class="reference external" href="https://discuss.python.org/t/pep-582-python-local-packages-directory/963/">hasn’t been much progress</a> on having it accepted and implemented in Python proper. However, PDM does not care, and it allows you to enable the future on your own machine.</p>
<section id="enabling-the-future-on-your-own-machine">
<h2>Enabling the future on your own machine</h2>
<p>Let’s enable the future on my own machine. That will require one simple command:</p>
<div class="code"><pre class="code text"><a id="rest_code_57dd2fcf65b6410d89b4d31b34f42435-1" name="rest_code_57dd2fcf65b6410d89b4d31b34f42435-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_57dd2fcf65b6410d89b4d31b34f42435-1"></a>$ eval &quot;$(pdm --pep582)&quot;
</pre></div>
<p>After that, we can initialize our project and install requests into it. Let’s try:</p>
<div class="code"><pre class="code text"><a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-1" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-1"></a>$ mkdir mypdmproject
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-2" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-2"></a>$ cd mypdmproject
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-3" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-3"></a>$ pdm init
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-4" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-4"></a>Creating a pyproject.toml for PDM...
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-5" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-5"></a>Please enter the Python interpreter to use
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-6" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-6"></a>0. /usr/bin/python (3.11)
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-7" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-7"></a>1. /usr/bin/python3.11 (3.11)
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-8" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-8"></a>2. /usr/bin/python2.7 (2.7)
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-9" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-9"></a>Please select (0): 1
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-10" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-10"></a>Using Python interpreter: /usr/bin/python3.11 (3.11)
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-11" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-11"></a>Would you like to create a virtualenv with /usr/bin/python3.11? [y/n] (y): n
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-12" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-12"></a>You are using the PEP 582 mode, no virtualenv is created.
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-13" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-13" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-13"></a>For more info, please visit https://peps.python.org/pep-0582/
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-14" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-14" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-14"></a>Is the project a library that will be uploaded to PyPI [y/n] (n): n
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-15" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-15" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-15"></a>License(SPDX name) (MIT):
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-16" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-16" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-16"></a>Author name (Chris Warrick):
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-17" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-17" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-17"></a>Author email (…):
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-18" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-18" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-18"></a>Python requires(&#39;*&#39; to allow any) (&gt;=3.11):
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-19" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-19" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-19"></a>Changes are written to pyproject.toml.
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-20" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-20" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-20"></a>$ ls
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-21" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-21" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-21"></a>pyproject.toml
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-22" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-22" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-22"></a>$ pdm add requests
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-23" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-23" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-23"></a>Adding packages to default dependencies: requests
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-24" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-24" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-24"></a>🔒 Lock successful
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-25" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-25" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-25"></a>Changes are written to pdm.lock.
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-26" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-26" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-26"></a>Changes are written to pyproject.toml.
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-27" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-27" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-27"></a>Synchronizing working set with lock file: 5 to add, 0 to update, 0 to remove
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-28" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-28" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-28"></a>
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-29" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-29" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-29"></a>  ✔ Install charset-normalizer 2.1.1 successful
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-30" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-30" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-30"></a>  ✔ Install certifi 2022.12.7 successful
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-31" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-31" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-31"></a>  ✔ Install idna 3.4 successful
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-32" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-32" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-32"></a>  ✔ Install requests 2.28.1 successful
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-33" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-33" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-33"></a>  ✔ Install urllib3 1.26.13 successful
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-34" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-34" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-34"></a>
<a id="rest_code_8bf4eb95599e4a33a6173a9a786fd022-35" name="rest_code_8bf4eb95599e4a33a6173a9a786fd022-35" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_8bf4eb95599e4a33a6173a9a786fd022-35"></a>🎉 All complete!
</pre></div>
<p>So far, so good (I’m not a fan of emoji in terminals, but that’s my only real complaint here.) Our <code class="docutils literal">pyproject.toml</code> looks like this:</p>
<div class="code"><pre class="code toml"><a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-1" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-1"></a><span class="k">[tool.pdm]</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-2" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-2"></a>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-3" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-3"></a><span class="k">[project]</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-4" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-4"></a><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;&quot;</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-5" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-5"></a><span class="n">version</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;&quot;</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-6" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-6"></a><span class="n">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;&quot;</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-7" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-7"></a><span class="n">authors</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-8" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-8"></a><span class="w">    </span><span class="p">{</span><span class="n">name</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s2">&quot;Chris Warrick&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">email</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s2">&quot;…&quot;</span><span class="p">},</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-9" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-9"></a><span class="p">]</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-10" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-10"></a><span class="n">dependencies</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-11" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-11"></a><span class="w">    </span><span class="s2">&quot;requests&gt;=2.28.1&quot;</span><span class="p">,</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-12" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-12"></a><span class="p">]</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-13" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-13" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-13"></a><span class="n">requires-python</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;&gt;=3.11&quot;</span>
<a id="rest_code_05fafc9392584b41a4c7d09ebcf57aee-14" name="rest_code_05fafc9392584b41a4c7d09ebcf57aee-14" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_05fafc9392584b41a4c7d09ebcf57aee-14"></a><span class="n">license</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="n">text</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s2">&quot;MIT&quot;</span><span class="p">}</span>
</pre></div>
<p>If we try to look into our file structure, we have this:</p>
<div class="code"><pre class="code text"><a id="rest_code_dfba376b40f84648b2865a0745760a19-1" name="rest_code_dfba376b40f84648b2865a0745760a19-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-1"></a>$ ls
<a id="rest_code_dfba376b40f84648b2865a0745760a19-2" name="rest_code_dfba376b40f84648b2865a0745760a19-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-2"></a>pdm.lock  __pypackages__/  pyproject.toml
<a id="rest_code_dfba376b40f84648b2865a0745760a19-3" name="rest_code_dfba376b40f84648b2865a0745760a19-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-3"></a>$ ls __pypackages__
<a id="rest_code_dfba376b40f84648b2865a0745760a19-4" name="rest_code_dfba376b40f84648b2865a0745760a19-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-4"></a>3.11/
<a id="rest_code_dfba376b40f84648b2865a0745760a19-5" name="rest_code_dfba376b40f84648b2865a0745760a19-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-5"></a>$ ls __pypackages__/3.11
<a id="rest_code_dfba376b40f84648b2865a0745760a19-6" name="rest_code_dfba376b40f84648b2865a0745760a19-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-6"></a>bin/  include/  lib/
<a id="rest_code_dfba376b40f84648b2865a0745760a19-7" name="rest_code_dfba376b40f84648b2865a0745760a19-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-7"></a>$ ls __pypackages__/3.11/lib
<a id="rest_code_dfba376b40f84648b2865a0745760a19-8" name="rest_code_dfba376b40f84648b2865a0745760a19-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-8"></a>certifi/             certifi-2022.12.7.dist-info/
<a id="rest_code_dfba376b40f84648b2865a0745760a19-9" name="rest_code_dfba376b40f84648b2865a0745760a19-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-9"></a>idna/                idna-3.4.dist-info/
<a id="rest_code_dfba376b40f84648b2865a0745760a19-10" name="rest_code_dfba376b40f84648b2865a0745760a19-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-10"></a>charset_normalizer/  charset_normalizer-2.1.1.dist-info/
<a id="rest_code_dfba376b40f84648b2865a0745760a19-11" name="rest_code_dfba376b40f84648b2865a0745760a19-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-11"></a>requests/            requests-2.28.1.dist-info/
<a id="rest_code_dfba376b40f84648b2865a0745760a19-12" name="rest_code_dfba376b40f84648b2865a0745760a19-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_dfba376b40f84648b2865a0745760a19-12"></a>urllib3/             urllib3-1.26.13.dist-info/
</pre></div>
<p>We’ll write a simple Python program (let’s call it <code class="docutils literal">mypdmproject.py</code>) that makes a HTTP request using <code class="docutils literal">requests</code>. It will also print <code class="docutils literal">requests.__file__</code> so we’re sure it isn’t using some random system copy: <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-4" id="footnote-reference-4" role="doc-noteref"><span class="fn-bracket">[</span>4<span class="fn-bracket">]</span></a></p>
<div class="code"><pre class="code python"><a id="rest_code_88da4613f9c142128800d8f4917c7e96-1" name="rest_code_88da4613f9c142128800d8f4917c7e96-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_88da4613f9c142128800d8f4917c7e96-1"></a><span class="kn">import</span><span class="w"> </span><span class="nn">requests</span>
<a id="rest_code_88da4613f9c142128800d8f4917c7e96-2" name="rest_code_88da4613f9c142128800d8f4917c7e96-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_88da4613f9c142128800d8f4917c7e96-2"></a><span class="nb">print</span><span class="p">(</span><span class="n">requests</span><span class="o">.</span><span class="vm">__file__</span><span class="p">)</span>
<a id="rest_code_88da4613f9c142128800d8f4917c7e96-3" name="rest_code_88da4613f9c142128800d8f4917c7e96-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_88da4613f9c142128800d8f4917c7e96-3"></a><span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;https://chriswarrick.com/&quot;</span><span class="p">)</span>
<a id="rest_code_88da4613f9c142128800d8f4917c7e96-4" name="rest_code_88da4613f9c142128800d8f4917c7e96-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_88da4613f9c142128800d8f4917c7e96-4"></a><span class="nb">print</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">[:</span><span class="mi">15</span><span class="p">])</span>
</pre></div>
<div class="code"><pre class="code text"><a id="rest_code_d287b0c578984663a2928bb41455147e-1" name="rest_code_d287b0c578984663a2928bb41455147e-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_d287b0c578984663a2928bb41455147e-1"></a>$ python mypdmproject.py
<a id="rest_code_d287b0c578984663a2928bb41455147e-2" name="rest_code_d287b0c578984663a2928bb41455147e-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_d287b0c578984663a2928bb41455147e-2"></a>/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<a id="rest_code_d287b0c578984663a2928bb41455147e-3" name="rest_code_d287b0c578984663a2928bb41455147e-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_d287b0c578984663a2928bb41455147e-3"></a>&lt;!DOCTYPE html&gt;
</pre></div>
<p>Let’s finally try the tests we’ve done in the other languages. Requests is useless without urllib3, so let’s remove it <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-5" id="footnote-reference-5" role="doc-noteref"><span class="fn-bracket">[</span>5<span class="fn-bracket">]</span></a> and see how well it works.</p>
<div class="code"><pre class="code text"><a id="rest_code_025901aac58247a4a73a4e28c999ae5d-1" name="rest_code_025901aac58247a4a73a4e28c999ae5d-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-1"></a>$ rm -rf __pypackages__/3.11/lib/urllib3*
<a id="rest_code_025901aac58247a4a73a4e28c999ae5d-2" name="rest_code_025901aac58247a4a73a4e28c999ae5d-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-2"></a>$ python mypdmproject.py
<a id="rest_code_025901aac58247a4a73a4e28c999ae5d-3" name="rest_code_025901aac58247a4a73a4e28c999ae5d-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-3"></a>Traceback (most recent call last):
<a id="rest_code_025901aac58247a4a73a4e28c999ae5d-4" name="rest_code_025901aac58247a4a73a4e28c999ae5d-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-4"></a>  File &quot;/tmp/mypdmproject/mypdmproject.py&quot;, line 1, in &lt;module&gt;
<a id="rest_code_025901aac58247a4a73a4e28c999ae5d-5" name="rest_code_025901aac58247a4a73a4e28c999ae5d-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-5"></a>    import requests
<a id="rest_code_025901aac58247a4a73a4e28c999ae5d-6" name="rest_code_025901aac58247a4a73a4e28c999ae5d-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-6"></a>  File &quot;/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py&quot;, line 43, in &lt;module&gt;
<a id="rest_code_025901aac58247a4a73a4e28c999ae5d-7" name="rest_code_025901aac58247a4a73a4e28c999ae5d-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-7"></a>    import urllib3
<a id="rest_code_025901aac58247a4a73a4e28c999ae5d-8" name="rest_code_025901aac58247a4a73a4e28c999ae5d-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_025901aac58247a4a73a4e28c999ae5d-8"></a>ModuleNotFoundError: No module named &#39;urllib3&#39;
</pre></div>
<p>Finally, can we try with a different directory? How about a different user?</p>
<div class="code"><pre class="code text"><a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-1" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-1" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-1"></a>$ pdm install
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-2" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-2" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-2"></a>Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-3" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-3" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-3"></a>
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-4" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-4" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-4"></a>  ✔ Install urllib3 1.26.13 successful
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-5" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-5" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-5"></a>
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-6" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-6" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-6"></a>🎉 All complete!
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-7" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-7" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-7"></a>$ pwd
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-8" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-8" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-8"></a>/tmp/mypdmproject
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-9" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-9" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-9"></a>$ cd ~
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-10" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-10" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-10"></a>$ python /tmp/mypdmproject/mypdmproject.py
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-11" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-11" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-11"></a>/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-12" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-12" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-12"></a>&lt;!DOCTYPE html&gt;
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-13" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-13" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-13"></a># su -s /bin/bash -c &#39;eval &quot;$(/tmp/pdmvenv/bin/pdm --pep582 bash)&quot;; python /tmp/mypdmproject/mypdmproject.py&#39; - nobody
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-14" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-14" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-14"></a>su: warning: cannot change directory to /nonexistent: No such file or directory
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-15" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-15" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-15"></a>/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<a id="rest_code_9544e824a8e84543b9a705b203b8f3f0-16" name="rest_code_9544e824a8e84543b9a705b203b8f3f0-16" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#rest_code_9544e824a8e84543b9a705b203b8f3f0-16"></a>&lt;!DOCTYPE html&gt;
</pre></div>
<p>This is looking pretty good. An independent project manages to do what the big Authority failed to do over so many years.</p>
</section>
<section id="is-this-the-perfect-thing">
<h2>Is this the perfect thing?</h2>
<p>Well, almost. There are two things that I have complaints about. The first one is the <code class="docutils literal">pdm <span class="pre">--pep582</span></code> hack, but hopefully, the PyPA gets its act together and gets it into Python core soon. However, another important problem is the lack of separation from system site-packages. Avid readers of footnotes <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-6" id="footnote-reference-6" role="doc-noteref"><span class="fn-bracket">[</span>6<span class="fn-bracket">]</span></a> might have noticed I had to use a Docker container in my PDM experiments, because requests is very commonly found in system <code class="docutils literal"><span class="pre">site-packages</span></code> (especially when using system Pythons, which have requests because of some random package, or because it was unbundled from pip). <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-7" id="footnote-reference-7" role="doc-noteref"><span class="fn-bracket">[</span>7<span class="fn-bracket">]</span></a> This can break things in ways you don’t expect, because you might end up importing and depending on system-wide things, or mixing system-wide and local packages (if you don’t install an extra requirement, but those packages are present system-wide, then you might end up using an extra you haven’t asked for). This is an important problem—a good solution would be to disable system site-packages if a <code class="docutils literal">__pypackages__</code> directory is in use.</p>
</section>
<section id="the-part-where-the-steering-council-kills-it">
<h2>The part where the Steering Council kills it</h2>
<p>In late March 2023, the Python Steering Council has announced <a class="reference external" href="https://discuss.python.org/t/pep-582-python-local-packages-directory/963/430">the rejection of PEP 582</a>. The reasons cited in the SC decision cited the limitations of the PEP (the <code class="docutils literal">__pypackages__</code> directory not always being enough, and the lack of specification on how it would behave in edge cases). Another argument is that it is possible to get <code class="docutils literal">__pypackages__</code> via “one of the many existing customization mechanisms for <code class="docutils literal">sys.path</code>, like <code class="docutils literal">.pth</code> files or a <code class="docutils literal">sitecustomize</code> module” — things commonly considered hacks, not real solutions. While users certainly can do anything they want to the <code class="docutils literal">sys.path</code> (often with tragic consequences), the point of having a common standard is to encourage tools to add support for it — if you use the aforementioned hacks, your IDE might end up not noticing the packages or considering them part of your code (trying to index them and search for things in them). Another reason cited for the rejection is the disagreement among the packaging community, which should not be surprising, especially in light of the next section.</p>
<p>The PEP 582/<code class="docutils literal">__pypackages__</code> mechanism may become official one day, and finally make Python packaging approachable. That would probably require someone to step up and write a new PEP that would make more people happy. Or Python might be stuck with all these incompatible tools, and invent 10 more in the next few years. (PDM is still there, and it still supports <code class="docutils literal">__pypackages__</code>, even though its implementation isn’t exactly the same as suggested by the now-rejected PEP.) Python’s current trajectory, as demonstrated by this decision, and by many people still being forced to struggle with the needlessly complicated virtual environments, sounds an awful lot like <a class="reference external" href="https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_Says_Only_Nation_Where_This_Regularly_Happens">the classic Onion headline</a>: ‘No Way to Prevent This,’ Says Only Programming Community Where This Regularly Happens.</p>
</section>
</section>
<section id="pypa-versus-reality-packaging-survey-results-and-pypa-reaction">
<h1>PyPA versus reality: packaging survey results and PyPA reaction</h1>
<p>Some time ago, the PSF ran a survey on packaging. Over 8000 people responded. <a class="reference external" href="https://drive.google.com/file/d/1U5d5SiXLVkzDpS0i1dJIA4Hu5Qg704T9/view">The users have spoken:</a></p>
<ul class="simple">
<li><p>Most people think packaging is too complex.</p></li>
<li><p>An overwhelming majority prefers using just a single tool.</p></li>
<li><p>Most people also think the existence of multiple tools is not beneficial for the Python packaging ecosystem.</p></li>
<li><p>Virtually everyone would prefer a clearly defined official workflow.</p></li>
<li><p>Over 50% of responses think tools for other ecosystems are better at managing dependencies and installing packages.</p></li>
</ul>
<p>The next step after this survey was for the packaging community to <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420">discuss its results</a> and try to come up with a new packaging strategy. The first post from Shamika Mohanan (the Packaging Project Manager at PSF) that triggered the discussion also focused heavily on the users’ vision to unify packaging tools and to have One True Tool. This discussion was open to people involved with the packaging world; many participants of the discussion are involved with PyPA, and I don’t think I’ve seen a single comment from the people behind Poetry or PDM.</p>
<p>Most of the thread ended up being discussion of binary extensions, including discussions of how to help tool proliferation by making it possible for tools that aren’t setuptools to build binary extensions. There was also a lot of focus on the scientific community’s issues with <a class="reference external" href="https://pypackaging-native.github.io/">libraries with native code</a>, heavily rooted in C/C++, and with attempts to replace Conda with new PyPA-approved tools. The “unified tool” for everyone else was mentioned in some posts, but they were certainly the minority.</p>
<p>Some PyPA members talked about a UX analysis, and that they expect the unified tool to be re-exporting functionality from existing tools—which immediately raises the question: which tools should it export functionality from and why? Is <code class="docutils literal">pip install <span class="pre">unified-packaging-tool</span></code> going to bring in all fourteen? Is the fact that users are unhappy with what they have, and many of them would be happy with something lke npm/dotnet/cargo, not enough to determine the UX direction of the unified tool?</p>
<p>Some of them are also against breaking existing workflows. Is a unified packaging tool going to work for every single user? Definitely not. But are there that many distinct basic workflows? If we ignore things that border on bikeshedding, such as src vs no-src, or venv locations, are there that many workflows to consider? Someone making a library and someone making an application do have different needs (e.g. with regard to publishing the package or acceptable dependency versions). Someone working with C extensions (or extensions using something like Cython) may have different needs, but their needs would usually be a superset of the needs of someone working on a pure-Python project. The scientific community might have more specialized needs, related to complex non-Python parts, but I am positive many of their points could be solved by the unified tool as well, even if it’s not by the time this tool reaches v1.0. It is also possible that the scientific community might prefer to stay with Conda, or with some evolution of it that brings it closer in line with the Unified Packaging Tool but also solves the scientists’ needs better than a tool also solving the non-scientists’ needs can.</p>
<p>Then there’s a discussion about the existing tools and which one is the tool for the future. The maintainer of Hatch (Ofek Lev) says that <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/4">Hatch can provide the “unified UX”</a>. But do the maintainers of Poetry or PDM agree? Poetry seems to be far more active than Hatch, going by GitHub issues, and it’s also worth noting that Hatch’s bus factor is 1 (with Ofek Lev responsible for 542 out of 576 commits to the master branch). <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/46">Russell Keith-Magee from BeeWare</a> has highlighted the fact that tooling aside, the PyPA does a bad job at communicating things. Russell mentioned that one of PyPA tutorials now uses Hatch, but there is no way to know if the PyPA considers Hatch to be the future, are people supposed to migrate onto Hatch, and is Flit, another recent PyPA tool, now useless? Russell also makes good points about focusing efforts: should people focus on helping Hatch support extension modules (which, according to the Hatch maintainer, is the last scenario requiring setuptools; other participants note that you can already build native code without setuptools), or should people focus on improving setuptools compatibility with PEP 517?</p>
<p>There were also some people stating their opinions on unifying things in various ways—and many of them are <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/136">against</a> <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/137">unifying</a> things. There were some voices of reason, like that of Russell Keith-Magee, or of <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/140">Simon Notley</a>, who correctly noticed the thread fails to resolve problems of developers, who are confused about packaging, and don’t understand the different choices available and how they interoperate. Simon does agree that native dependencies are important and happen often in Python projects (and so do I), but the users who responded to the survey had something else in mind — as exemplified by the discussion opening post, mentioning the user expecting the simplicity of Rust’s cargo, and by the survey results. 70% of the survey respondents also use <code class="docutils literal">npm</code>, so many Python users have already seen the simpler workflows. The survey respondents were also asked to rank a few focus areas based on importance. “Making Python packaging better serve common use cases and workflows” was ranked first out of the provided options <a class="brackets" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-8" id="footnote-reference-8" role="doc-noteref"><span class="fn-bracket">[</span>8<span class="fn-bracket">]</span></a> by 3248 participants. “Supporting a wider range of use cases (e.g. edge cases, etc.)” was ranked first by 379 people, and it was the least important in the minds of 2989 people.</p>
<p>One more point that highlights the detachment of packaging folk from reality was mentioned by Anderson Bravalheri. To Anderson, a new unified tool would be <a class="reference external" href="https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/133">disrespectful of the work</a> the maintainers of the existing tools put into maintaining them, and disrespectful of users who had to adapt to the packaging mess. This point is completely absurd. Was the replacement of MS-DOS/Windows 9x and Classic Mac OS with Windows NT and <s>Mac OS X</s> <s>OS X</s> macOS disrespectful to their respective designers, and the users who had to adapt to manually configuring minutiae, figuring out how to get all your software and hardware to run with weird limitations that were necessary in the 1980s, and the system crashing every once in a while? Was the replacement of horses with cars disrespectful to horses, and the people who were removing horse manure from the streets? Was the replacement of the Ford Model T with faster, safer, more efficient, and easier to use cars disrespectful to Henry Ford? Technology comes and goes, and sometimes, getting an improvement means we need to get rid of the old stuff. This applies outside of technology, too—you could come up with many examples of change in the world, which might have put some people out of power, but has greatly improved the lives of millions of people (the fall of communism in Europe, for example). Also, going back to the technology world of today, this sentiment suggests Anderson is far too attached to the software they write—is this a healthy approach?</p>
<p>Nobody raised PEP 582 or the complexity of virtual environments. It might not be visible from the ivory towers of packaging tool maintainers, who have years of experience dealing with them, but it certainly does exist for regular people, for people who think the Python provided by their Linux distro is good enough, and especially for people for whom Python is their introduction to programming.</p>
<p>I would like to once again highlight: that’s not just the opinion of one random rambling Chris. The opinion that Python packaging needs to be simplified and unified is held by about half of the 8774 people who took the survey.</p>
<p>But here’s one more interesting thing: Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be always accurate, but if we assume it is, the link to the results summary was clicked only 14 times (as of 2023-01-14 21:20 UTC). The discussion has 28 participants and 2.2k views. If we believe the link click counter, <strong>half of the discussion participants did not even bother reading what the people think</strong>.</p>
<img alt="/images/python-packaging/discourse-link-clicks.png" class="align-center" src="https://chriswarrick.com/images/python-packaging/discourse-link-clicks.png">
</section>
<section id="summary">
<h1>Summary</h1>
<p>Python packaging is a mess, and it always has been. There are tons of tools, mostly incompatible with each other, and no tool can solve <em>all</em> problems (especially no tool from the PyPA). PDM is really close to the ideal, since it can do away with the overhead of managing virtual environments—which is hopefully the future of Python packaging, or the 2010s of Node.js packaging (although it is not going to be the 2023 of Python packaging, considering the Steering Council rejection). Perhaps in a few years, Python developers (and more importantly, Python learners!) will be able to just <code class="docutils literal">pip install</code> (or <code class="docutils literal">pdm install</code>?) what they need, without worrying about some “virtual environment” thing, that is separate but not quite from a system Python, and that is not a virtual machine. Python needs less tools, not more.</p>
<p><a class="reference external" href="https://en.wikipedia.org/wiki/Carthago_delenda_est">Furthermore, I consider that the PyPA must be destroyed.</a> The strategy discussion highlights the fact that they are unable to make Python packaging work the way the users expect. The PyPA should focus on producing one good tool, and on getting PEP 582 into Python. A good way to achieve this would be to put its resources behind PDM. The issues with native code and binary wheels are important, but plain-Python workflows, or workflows with straightforward binary dependencies, are much more common, and need to be improved. This improvement needs to happen now.</p>
<p>Discuss in the comments below, on <a class="reference external" href="https://news.ycombinator.com/item?id=34390585">Hacker News</a>, or on <a class="reference external" href="https://www.reddit.com/r/Python/comments/10cnx5i/how_to_improve_python_packaging_or_why_fourteen/">Reddit</a>.</p>
</section>
<section id="footnotes">
<h1>Footnotes</h1>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="footnote-1" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-1">1</a><span class="fn-bracket">]</span></span>
<p>Funnily enough, the aphorism itself fails at “one obvious way to do it”. It is with dashes set in two different ways (with spaces after but not before, and with spaces before but not after), and none of them is the correct one (most English style guides prefer no spaces, but some allow spaces on both sides).</p>
</aside>
<aside class="footnote brackets" id="footnote-2" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-2">2</a><span class="fn-bracket">]</span></span>
<p>Apologies for the slight Linux focus of this post; all the points I make apply on Windows as well, but perhaps with some slightly different names and commands.</p>
</aside>
<aside class="footnote brackets" id="footnote-3" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-3">3</a><span class="fn-bracket">]</span></span>
<p>There’s a new major version of .NET every year, with the even-numbered versions being LTS. Those are far less revolutionary than the Python 2 → 3 transition, and after you jump on the modern .NET train, upgrading a project to the new major version is fairly simple (possibly even just bumping the version number).</p>
</aside>
<aside class="footnote brackets" id="footnote-4" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-4">4</a><span class="fn-bracket">]</span></span>
<p>And to be extra sure, I used a clean <code class="docutils literal">python:latest</code> Docker container, since requests is so commonly found in system site packages.</p>
</aside>
<aside class="footnote brackets" id="footnote-5" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-5">5</a><span class="fn-bracket">]</span></span>
<p>A little caveat here, I also had to remove the <code class="docutils literal"><span class="pre">dist-info</span></code> folder, so that PDM would know it needs to be reinstalled.</p>
</aside>
<aside class="footnote brackets" id="footnote-6" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-6">6</a><span class="fn-bracket">]</span></span>
<p>Yes, that’s you!</p>
</aside>
<aside class="footnote brackets" id="footnote-7" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-7">7</a><span class="fn-bracket">]</span></span>
<p>Also, why is there no good HTTP client library in Python’s standard library? Is the “standard library is where packages go to die” argument still relevant, if requests had four releases in 2022, and urllib3 had six, and most of the changes were minor?</p>
</aside>
<aside class="footnote brackets" id="footnote-8" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#footnote-reference-8">8</a><span class="fn-bracket">]</span></span>
<p>I have removed the “Other” option, and shifted all options ranked below it by one place, since we don’t know what the other thing was and how it related to the options presented (the free-form responses were removed from the public results spreadsheet to preserve the users’ anonymity). In the event a respondent left some of the options without a number, the blank options were not considered neither first nor last.</p>
</aside>
</aside>
</section>
<section id="revision-history">
<h1>Revision History</h1>
<p>This post got amended in April 2023 with an update about the SC rejection of PEP 582 (in a new subsection and in the Summary section).</p>
</section>
]]></content:encoded><category>Python</category><category>.NET</category><category>C#</category><category>JavaScript</category><category>Node.js</category><category>npm</category><category>packaging</category><category>PDM</category><category>pip</category><category>PyPA</category><category>Python</category><category>virtual environments</category></item><item><title>Python Hackery: merging signatures of two Python functions</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/</link><pubDate>Thu, 20 Sep 2018 13:52:20 GMT</pubDate><guid>https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/</guid><description>
Today’s blog post is going to contain fairly advanced Python hackery. We’ll
take two functions — one is a wrapper for the other, but also adds some
positional arguments.  And we’ll change the signature displayed everywhere from
the uninformative f(new_arg, *args, **kwargs) to something more
appropriate.
</description><content:encoded><![CDATA[
<p>Today’s blog post is going to contain fairly advanced Python hackery. We’ll
take two functions — one is a wrapper for the other, but also adds some
positional arguments.  And we’ll change the signature displayed everywhere from
the uninformative <code class="docutils literal">f(new_arg, *args, **kwargs)</code> to something more
appropriate.</p>



<p>This blog post was inspired by F4D3C0D3 on #python (freenode IRC). I also took
some inspiration from
Gynvael Coldwind’s classic <a class="reference external" href="https://www.youtube.com/watch?v=7VJaprmuHcw">Python 101</a> (April Fools) video. (Audio and some comments are in Polish, but even if you don’t speak the language, it’s still worth it to click through the time bar and see some (fairly unusual) magic happen.)</p>
<section id="starting-point">
<h1>Starting point</h1>
<div class="code"><pre class="code python"><a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-1" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-1"></a><span class="k">def</span><span class="w"> </span><span class="nf">old</span><span class="p">(</span><span class="n">foo</span><span class="p">,</span> <span class="n">bar</span><span class="p">):</span>
<a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-2" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-2"></a><span class="w">    </span><span class="sd">&quot;&quot;&quot;This is old&#39;s docstring.&quot;&quot;&quot;</span>
<a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-3" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-3" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-3"></a>    <span class="nb">print</span><span class="p">(</span><span class="n">foo</span><span class="p">,</span> <span class="n">bar</span><span class="p">)</span>
<a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-4" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-4" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-4"></a>    <span class="k">return</span> <span class="n">foo</span> <span class="o">+</span> <span class="n">bar</span>
<a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-5" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-5" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-5"></a>
<a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-6" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-6" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-6"></a>
<a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-7" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-7" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-7"></a><span class="k">def</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">prefix</span><span class="p">,</span> <span class="n">foo</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<a id="rest_code_4d7b7ce96f28484bb5c554955ef085fa-8" name="rest_code_4d7b7ce96f28484bb5c554955ef085fa-8" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_4d7b7ce96f28484bb5c554955ef085fa-8"></a>    <span class="k">return</span> <span class="n">old</span><span class="p">(</span><span class="n">prefix</span> <span class="o">+</span> <span class="n">foo</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</pre></div>
<p>Let’s test it.</p>
<div class="code"><pre class="code pycon"><a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-1" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-1"></a><span class="gp">&gt;&gt;&gt; </span><span class="n">o</span> <span class="o">=</span> <span class="n">old</span><span class="p">(</span><span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">)</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-2" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-2"></a><span class="go">a b</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-3" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-3" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-3"></a><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span> <span class="o">=</span> <span class="n">new</span><span class="p">(</span><span class="s1">&#39;!&#39;</span><span class="p">,</span> <span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">)</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-4" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-4" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-4"></a><span class="go">!a b</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-5" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-5" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-5"></a><span class="gp">&gt;&gt;&gt; </span><span class="nb">print</span><span class="p">(</span><span class="n">o</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s1">&#39; - &#39;</span><span class="p">)</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-6" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-6" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-6"></a><span class="go">ab - !ab</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-7" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-7" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-7"></a><span class="gp">&gt;&gt;&gt; </span><span class="n">help</span><span class="p">(</span><span class="n">old</span><span class="p">)</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-8" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-8" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-8"></a><span class="go">Help on function old in module __main__:</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-9" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-9" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-9"></a>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-10" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-10" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-10"></a><span class="go">old(foo, bar)</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-11" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-11" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-11"></a><span class="go">    This is old&#39;s docstring.</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-12" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-12" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-12"></a>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-13" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-13" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-13"></a><span class="gp">&gt;&gt;&gt; </span><span class="n">help</span><span class="p">(</span><span class="n">new</span><span class="p">)</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-14" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-14" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-14"></a><span class="go">Help on function new in module __main__:</span>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-15" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-15" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-15"></a>
<a id="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-16" name="rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-16" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_bed2964a8ddd429a8b6c770e6bd1cbd7-16"></a><span class="go">new(prefix, foo, *args, **kwargs)</span>
</pre></div>
<p>The last line is not exactly informative — it doesn’t tell us that we need to
pass <code class="docutils literal">bar</code> as an argument.  Sure, you could define <code class="docutils literal">new</code> as just <code class="docutils literal">(prefix, foo,
bar)</code> — but that means every change to <code class="docutils literal">old</code> requires editing <code class="docutils literal">new</code> as
well. So, not ideal. Let’s try to fix this.</p>
</section>
<section id="the-existing-infrastructure-functools-wraps">
<h1>The existing infrastructure: functools.wraps</h1>
<p>First, let’s start with the basic facility Python already has.  The standard
library already comes with <code class="docutils literal">functools.wraps</code> and
<code class="docutils literal">functools.update_wrapper</code>.</p>
<p>If you’ve never heard of those two functions, here’s a crash course:</p>
<div class="code"><pre class="code python"><a id="rest_code_36bca7dffa6042ad98ceab813c492019-1" name="rest_code_36bca7dffa6042ad98ceab813c492019-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-1"></a><span class="k">def</span><span class="w"> </span><span class="nf">decorator</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-2" name="rest_code_36bca7dffa6042ad98ceab813c492019-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-2"></a>    <span class="nd">@functools</span><span class="o">.</span><span class="n">wraps</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-3" name="rest_code_36bca7dffa6042ad98ceab813c492019-3" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-3"></a>    <span class="k">def</span><span class="w"> </span><span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-4" name="rest_code_36bca7dffa6042ad98ceab813c492019-4" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-4"></a>        <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Inside wrapper&quot;</span><span class="p">)</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-5" name="rest_code_36bca7dffa6042ad98ceab813c492019-5" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-5"></a>        <span class="n">f</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-6" name="rest_code_36bca7dffa6042ad98ceab813c492019-6" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-6"></a>    <span class="k">return</span> <span class="n">wrapper</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-7" name="rest_code_36bca7dffa6042ad98ceab813c492019-7" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-7"></a>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-8" name="rest_code_36bca7dffa6042ad98ceab813c492019-8" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-8"></a><span class="nd">@decorator</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-9" name="rest_code_36bca7dffa6042ad98ceab813c492019-9" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-9"></a><span class="k">def</span><span class="w"> </span><span class="nf">square</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">float</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-10" name="rest_code_36bca7dffa6042ad98ceab813c492019-10" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-10"></a><span class="w">    </span><span class="sd">&quot;&quot;&quot;Square a number.&quot;&quot;&quot;</span>
<a id="rest_code_36bca7dffa6042ad98ceab813c492019-11" name="rest_code_36bca7dffa6042ad98ceab813c492019-11" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_36bca7dffa6042ad98ceab813c492019-11"></a>    <span class="k">return</span> <span class="n">n</span> <span class="o">*</span> <span class="n">n</span>
</pre></div>
<p>If we try to inspect the <code class="docutils literal">square</code> function, we’ll see the original name, arguments,
annotations, and the docstring.  If we ran this code again, but with the
<code class="docutils literal">&#64;functools.wraps(f)</code> line commented out, we would only see <code class="docutils literal"><span class="pre">wrapper(*args,</span>
**kwargs)</code>.</p>
<p>This approach gives us a hint of what we need to do.  However, if we apply
<code class="docutils literal">wraps</code> (or <code class="docutils literal">update_wrapper</code>, which is what <code class="docutils literal">wraps</code> ends up calling)
to our function, it will only have <code class="docutils literal">foo</code> and <code class="docutils literal">bar</code> as arguments, and its
name will be displayed as <code class="docutils literal">old</code>.</p>
<p>So, let’s take a look at <a class="reference external" href="https://github.com/python/cpython/blob/4fe8dc68577f9e22aaf24db08fb6647277c42d4c/Lib/functools.py#L27-L79">functools.update_wrapper</a>. What does it do? Two things:</p>
<ul class="simple">
<li><p>copy some attributes from the old function to the new one
(<code class="docutils literal">__module__</code>, <code class="docutils literal">__name__</code>, <code class="docutils literal">__qualname__</code>, <code class="docutils literal">__doc__</code>, <code class="docutils literal">__annotations__</code>)</p></li>
<li><p>update <code class="docutils literal">__dict__</code> of the new function</p></li>
<li><p>set <code class="docutils literal">wrapper.__wrapped__</code></p></li>
</ul>
<p>If we try to experiment with it — by changing the list of things to copy, for
example — we’ll find out that the annotations, the docstring, and the displayed name come from
the copied attributes, but the signature itself is apparently taken from <code class="docutils literal">__wrapped__</code>.</p>
<p>Further investigation reveals this fact about <code class="docutils literal">inspect.signature</code>:</p>
<blockquote>
<p><code class="docutils literal">inspect.signature(callable, *, follow_wrapped=True)</code></p>
<p><em>New in version 3.5:</em> <code class="docutils literal">follow_wrapped</code> parameter. Pass <code class="docutils literal">False</code> to get a signature of callable specifically (<code class="docutils literal">callable.__wrapped__</code> will not be used to unwrap decorated callables.)</p>
</blockquote>
<p>And so, this is our <strong>end goal:</strong></p>
<p class="lead">Craft a function with a specific signature (that merges <code class="docutils literal">old</code> and <code class="docutils literal">new</code>) and set it as <code class="docutils literal">new.__wrapped__</code>.</p>
<p>But first, we need to talk about parallel universes.</p>
<p>Or actually, code objects.</p>
</section>
<section id="defining-a-function-programmatically">
<h1>Defining a function programmatically</h1>
<p>Let’s try an experiment.</p>
<div class="code"><pre class="code pycon"><a id="rest_code_a62b013b29454ad78fe76dca6cddcf0f-1" name="rest_code_a62b013b29454ad78fe76dca6cddcf0f-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_a62b013b29454ad78fe76dca6cddcf0f-1"></a><span class="gp">&gt;&gt;&gt; </span><span class="k">def</span><span class="w"> </span><span class="nf">foo</span><span class="p">(</span><span class="n">bar</span><span class="p">):</span> <span class="k">pass</span>
<a id="rest_code_a62b013b29454ad78fe76dca6cddcf0f-2" name="rest_code_a62b013b29454ad78fe76dca6cddcf0f-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_a62b013b29454ad78fe76dca6cddcf0f-2"></a><span class="gp">&gt;&gt;&gt; </span><span class="n">foo</span><span class="o">.</span><span class="n">__wrapped__</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="kc">None</span>
<a id="rest_code_a62b013b29454ad78fe76dca6cddcf0f-3" name="rest_code_a62b013b29454ad78fe76dca6cddcf0f-3" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_a62b013b29454ad78fe76dca6cddcf0f-3"></a><span class="gp">&gt;&gt;&gt; </span><span class="n">help</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<a id="rest_code_a62b013b29454ad78fe76dca6cddcf0f-4" name="rest_code_a62b013b29454ad78fe76dca6cddcf0f-4" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_a62b013b29454ad78fe76dca6cddcf0f-4"></a><span class="go">foo(x, y)</span>
</pre></div>
<p>So, there are two ways to do this.  The first one would be to generate a string
with the signature and just use <code class="docutils literal">eval</code> to get a <code class="docutils literal">__wrapped__</code> function. But
that would be cheating, and honestly, quite boring. (The inspect module could
help us with preparing the string.)  The second one? Create code objects
manually.</p>
<section id="code-objects">
<h2>Code objects</h2>
<p>To create a function, we’ll need the <code class="docutils literal">types</code> module. <code class="docutils literal">types.FunctionType</code>
gives us a function, but it asks us for a code object. As the <a class="reference external" href="https://docs.python.org/3/reference/datamodel.html">docs</a> state,
<em>Code objects represent byte-compiled executable Python code, or bytecode.</em></p>
<p>To create one by
hand, we’ll need <code class="docutils literal">types.CodeType</code>. Well, not exactly by hand — we’ll end up doing a three-way merge between
<code class="docutils literal">source</code> (<code class="docutils literal">old</code>), <code class="docutils literal">dest</code> (<code class="docutils literal">new</code>) and <code class="docutils literal">def <span class="pre">_blank():</span> pass</code> (a function
that does nothing).</p>
<p>Let’s look at the docstring for <code class="docutils literal">CodeType</code>:</p>
<div class="code"><pre class="code text"><a id="rest_code_6208107589054526ac90219ce627e4dc-1" name="rest_code_6208107589054526ac90219ce627e4dc-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_6208107589054526ac90219ce627e4dc-1"></a>code(argcount, kwonlyargcount, nlocals, stacksize, flags, codestring,
<a id="rest_code_6208107589054526ac90219ce627e4dc-2" name="rest_code_6208107589054526ac90219ce627e4dc-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_6208107589054526ac90219ce627e4dc-2"></a>    constants, names, varnames, filename, name, firstlineno,
<a id="rest_code_6208107589054526ac90219ce627e4dc-3" name="rest_code_6208107589054526ac90219ce627e4dc-3" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_6208107589054526ac90219ce627e4dc-3"></a>    lnotab[, freevars[, cellvars]])
<a id="rest_code_6208107589054526ac90219ce627e4dc-4" name="rest_code_6208107589054526ac90219ce627e4dc-4" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_6208107589054526ac90219ce627e4dc-4"></a>
<a id="rest_code_6208107589054526ac90219ce627e4dc-5" name="rest_code_6208107589054526ac90219ce627e4dc-5" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_6208107589054526ac90219ce627e4dc-5"></a>Create a code object.  Not for the faint of heart.
</pre></div>
<p>All of the arguments end up being fields of a code objects (name starts with
<code class="docutils literal">co_</code>).  For each
function <code class="docutils literal">f</code>, its code object is <code class="docutils literal">f.__code__</code>. You can find the filename in
<code class="docutils literal">f.__code__.co_filename</code>, for example. The meaning of all fields can be
found in docs for the <a class="reference external" href="https://docs.python.org/3/library/inspect.html#types-and-members">inspect module</a>. We’ll be
interested in the following three fields:</p>
<ul class="simple">
<li><p><code class="docutils literal">argcount</code> — number of arguments (not including keyword only arguments, * or ** args)</p></li>
<li><p><code class="docutils literal">kwonlyargcount</code> — number of keyword only arguments (not including ** arg)</p></li>
<li><p><code class="docutils literal">varnames</code> — tuple of names of arguments and local variables</p></li>
</ul>
<p>For all the other fields, we’ll copy them from the appropriate function (one of
the three).  We don’t expect anyone to call the wrapped function directly; as
long as <code class="docutils literal">help</code> and <code class="docutils literal">inspect</code> members don’t crash when they look into it,
we’re fine.</p>
</section>
<section id="everything-you-need-to-know-about-function-arguments">
<h2>Everything you need to know about function arguments</h2>
<div class="code"><pre class="code pycon"><a id="rest_code_f4b49082358149cf861286d903952272-1" name="rest_code_f4b49082358149cf861286d903952272-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_f4b49082358149cf861286d903952272-1"></a><span class="gp">&gt;&gt;&gt; </span><span class="k">def</span><span class="w"> </span><span class="nf">f</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">d</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span> <span class="k">pass</span>
<a id="rest_code_f4b49082358149cf861286d903952272-2" name="rest_code_f4b49082358149cf861286d903952272-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_f4b49082358149cf861286d903952272-2"></a><span class="gp">&gt;&gt;&gt; </span><span class="n">inspect</span><span class="o">.</span><span class="n">getfullargspec</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<a id="rest_code_f4b49082358149cf861286d903952272-3" name="rest_code_f4b49082358149cf861286d903952272-3" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_f4b49082358149cf861286d903952272-3"></a><span class="go">FullArgSpec(args=[&#39;a&#39;, &#39;b&#39;, &#39;c&#39;], varargs=None, varkw=None, defaults=(1, 2), kwonlyargs=[&#39;d&#39;], kwonlydefaults={&#39;d&#39;: 3}, annotations={})</span>
</pre></div>
<p>A function signature has the following syntax:</p>
<ol class="arabic simple">
<li><p>Any positional (non-optional) arguments</p></li>
<li><p>Variable positional arguments (<code class="docutils literal">*x</code>, name stored in <code class="docutils literal">varargs</code>)</p></li>
<li><p>Arguments with defaults (keyword-maybe arguments); their value is stored in <code class="docutils literal">__defaults__</code> left-to-right</p></li>
<li><p>Keyword-only arguments (after an asterisk); their values are stored in a dictionary.  Cannot be used if <code class="docutils literal">varargs</code> are defined.</p></li>
<li><p>Variable keyword arguments (<code class="docutils literal">**y</code>, name stored in <code class="docutils literal">varkw</code>)</p></li>
</ol>
<p>We’re going to make one assumption: we aren’t going to support a <code class="docutils literal">source</code>
function that uses variable arguments of any kind.  So, our final signature
will be composed like this:</p>
<ol class="arabic simple">
<li><p><code class="docutils literal">dest</code> positional arguments</p></li>
<li><p><code class="docutils literal">source</code> positional arguments</p></li>
<li><p><code class="docutils literal">dest</code> keyword-maybe arguments</p></li>
<li><p><code class="docutils literal">source</code> keyword-maybe arguments</p></li>
<li><p><code class="docutils literal">dest</code> keyword-only arguments</p></li>
<li><p><code class="docutils literal">source</code> keyword-only arguments</p></li>
</ol>
<p>That will be saved into <code class="docutils literal">co_names</code>.  The first two arguments are counts —
the first one is <code class="docutils literal">len(1+2+3+4)</code> and the other is <code class="docutils literal">len(5+6)</code>. The remaining
arguments to <code class="docutils literal">CodeType</code> will be either safe minimal defaults, or things taken from
one of the three functions.</p>
<p>We’ll also need to do one more thing: we must ensure <code class="docutils literal">__defaults__</code>,
<code class="docutils literal">__kwdefaults__</code>, and <code class="docutils literal">__annotations__</code> are all in the right places.
That’s also a fairly simple thing to do (it requires more tuple/dict merging).
And with that, we’re done.</p>
</section>
</section>
<section id="final-results">
<h1>Final results</h1>
<p>Before I show you the code, let’s test it out:</p>
<div class="code"><pre class="code python"><a id="rest_code_0f4fd7ac3e33455b8abe64839a219429-1" name="rest_code_0f4fd7ac3e33455b8abe64839a219429-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_0f4fd7ac3e33455b8abe64839a219429-1"></a><span class="c1"># old defined as before</span>
<a id="rest_code_0f4fd7ac3e33455b8abe64839a219429-2" name="rest_code_0f4fd7ac3e33455b8abe64839a219429-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_0f4fd7ac3e33455b8abe64839a219429-2"></a><span class="nd">@merge_args</span><span class="p">(</span><span class="n">old</span><span class="p">)</span>
<a id="rest_code_0f4fd7ac3e33455b8abe64839a219429-3" name="rest_code_0f4fd7ac3e33455b8abe64839a219429-3" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_0f4fd7ac3e33455b8abe64839a219429-3"></a><span class="k">def</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">prefix</span><span class="p">,</span> <span class="n">foo</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<a id="rest_code_0f4fd7ac3e33455b8abe64839a219429-4" name="rest_code_0f4fd7ac3e33455b8abe64839a219429-4" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_0f4fd7ac3e33455b8abe64839a219429-4"></a>    <span class="k">return</span> <span class="n">old</span><span class="p">(</span><span class="n">prefix</span> <span class="o">+</span> <span class="n">foo</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</pre></div>
<p>And the end result — <code class="docutils literal">help(new)</code> says:</p>
<div class="code"><pre class="code text"><a id="rest_code_1088cc0d6c3d483fb32f8946fd791655-1" name="rest_code_1088cc0d6c3d483fb32f8946fd791655-1" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_1088cc0d6c3d483fb32f8946fd791655-1"></a>new(prefix, foo, bar)
<a id="rest_code_1088cc0d6c3d483fb32f8946fd791655-2" name="rest_code_1088cc0d6c3d483fb32f8946fd791655-2" href="https://chriswarrick.com/blog/2018/09/20/python-hackery-merging-signatures-of-two-python-functions/#rest_code_1088cc0d6c3d483fb32f8946fd791655-2"></a>    This is old&#39;s docstring.
</pre></div>
<p>We did it!</p>
<p class="lead">The code is available on <a class="reference external" href="https://github.com/Kwpolska/merge_args">GitHub</a> and on <a class="reference external" href="https://pypi.org/project/merge-args/">PyPI</a> (<code class="docutils literal">pip install merge_args</code>).
There’s also an extensive test suite.</p>
<p>PS. you might be interested in another related post of mine, in which I
reverse-engineer the compilation of a function: <a class="reference external" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/">Gynvael’s Mission 11 (en): Python bytecode reverse-engineering</a></p>
</section>
]]></content:encoded><category>Python</category><category>hacking</category><category>Python</category><category>Python hackery</category><category>Python internals</category></item><item><title>Python Virtual Environments in Five Minutes</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/</link><pubDate>Tue, 04 Sep 2018 18:15:00 GMT</pubDate><guid>https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/</guid><description>
In Python, virtual environments are used to isolate projects from each other
(if they require different versions of the same library, for example). They let
you install and manage packages without administrative privileges, and without
conflicting with the system package manager.  They also allow to quickly create
an environment somewhere else with the same dependencies.
Virtual environments are a crucial tool for any Python developer. And at that,
a very simple tool to work with.
</description><content:encoded><![CDATA[
<p>In Python, virtual environments are used to isolate projects from each other
(if they require different versions of the same library, for example). They let
you install and manage packages without administrative privileges, and without
conflicting with the system package manager.  They also allow to quickly create
an environment somewhere else with the same dependencies.</p>
<p>Virtual environments are a crucial tool for any Python developer. And at that,
a very simple tool to work with.</p>



<p>Let’s get started!</p>
<section id="install">
<h1>Install</h1>
<p>The best tool that can be used to create virtual environments is the
<a class="reference external" href="https://docs.python.org/3/library/venv.html">venv</a> module, which is part of
the standard library since Python 3.3.</p>
<p><code class="docutils literal">venv</code> is built into Python, and most users don’t need to install anything.
However, Debian/Ubuntu users will need to run <code class="docutils literal">sudo <span class="pre">apt-get</span> install <span class="pre">python3-venv</span></code> to make it work (due to Debian not installing some components
that <code class="docutils literal">venv</code> needs by default). <a class="brackets" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-1" id="footnote-reference-1" role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a></p>
<p>The alternative (and original, and previously standard) virtual environment tool is <a class="reference external" href="https://virtualenv.pypa.io/">virtualenv</a>. It works with Python 2.7, and has a couple
extra fetures (that you generally won’t need). virtualenv can be installed with your system package manager, or <code class="docutils literal">pip install <span class="pre">--user</span> virtualenv</code>.</p>
<p>Which one to use? Probably <code class="docutils literal">venv</code>. Both tools achieve the same goal in similar
ways. And if one of them does not work, you can try the other and it might just
work better.</p>
<p><em>(Terminology note: most of the time, the names of both tools are used
interchargeably, “venv” was often used as an abbreviation for “virtualenv”
before the stdlib tool was created)</em></p>
</section>
<section id="create">
<h1>Create</h1>
<p>To create a virtual environment named <code class="docutils literal">env</code>, you need to run the <code class="docutils literal">venv</code>
tool with the Python you want to use in that environment.</p>
<div class="code"><pre class="code text"><a id="rest_code_1fb767790a68420a97f0697df45d2e92-1" name="rest_code_1fb767790a68420a97f0697df45d2e92-1" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_1fb767790a68420a97f0697df45d2e92-1"></a>Linux:   $ python3 -m venv env
<a id="rest_code_1fb767790a68420a97f0697df45d2e92-2" name="rest_code_1fb767790a68420a97f0697df45d2e92-2" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_1fb767790a68420a97f0697df45d2e92-2"></a>Windows: &gt; py -m venv env
</pre></div>
<p>or, if you’re using <code class="docutils literal">virtualenv</code>:</p>
<div class="code"><pre class="code text"><a id="rest_code_d2d40ad9883b450e9264cf578d6f68af-1" name="rest_code_d2d40ad9883b450e9264cf578d6f68af-1" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_d2d40ad9883b450e9264cf578d6f68af-1"></a>$ python3 -m virtualenv env
<a id="rest_code_d2d40ad9883b450e9264cf578d6f68af-2" name="rest_code_d2d40ad9883b450e9264cf578d6f68af-2" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_d2d40ad9883b450e9264cf578d6f68af-2"></a>&gt; py -m virtualenv env
</pre></div>
<p>Afterwards, you will end up with a folder named <code class="docutils literal">env</code> that contains folders
named <code class="docutils literal">bin</code> (<code class="docutils literal">Scripts</code> on Windows — contains executables and scripts
installed by packages, including
<code class="docutils literal">python</code>), <code class="docutils literal">lib</code> (contains code), and <code class="docutils literal">include</code> (contains C headers).</p>
<p>Both tools install <code class="docutils literal">pip</code> and <code class="docutils literal">setuptools</code>, but <code class="docutils literal">venv</code> does not ship with
<code class="docutils literal">wheel</code>. In addition, the default versions tend to be more-or-less outdated.
Let’s upgrade them real quick: <a class="brackets" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-2" id="footnote-reference-2" role="doc-noteref"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></a></p>
<div class="code"><pre class="code text"><a id="rest_code_39b9e0a93e0746fbac8d7b7f11734578-1" name="rest_code_39b9e0a93e0746fbac8d7b7f11734578-1" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_39b9e0a93e0746fbac8d7b7f11734578-1"></a>$ env/bin/python -m pip install --upgrade pip setuptools wheel
<a id="rest_code_39b9e0a93e0746fbac8d7b7f11734578-2" name="rest_code_39b9e0a93e0746fbac8d7b7f11734578-2" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_39b9e0a93e0746fbac8d7b7f11734578-2"></a>&gt; env\Scripts\python -m pip install --upgrade pip setuptools wheel
</pre></div>
<section id="where-to-store-virtual-environments">
<h2>Where to store virtual environments?</h2>
<p>While the tools allow you to put your virtual environments anywhere in the
system, it is not a desirable thing to do. There are two options:</p>
<ol class="arabic simple">
<li><p>Have one global place for them, like <code class="docutils literal">~/virtualenvs</code>.</p></li>
<li><p>Store them in each project’s directory, like <code class="docutils literal"><span class="pre">~/git/foobar/.venv</span></code>.</p></li>
</ol>
<p>The first option can be easier to manage, there are tools that can help manage
those (eg. <code class="docutils literal">virtualenvwrapper</code>, shell auto-activation scripts, or the
<code class="docutils literal">workon</code> functions described below).  The second option is equally easy to
work with, but comes with one caveat — you must add the venv directory to your
<code class="docutils literal">.gitignore</code> file (or <code class="docutils literal">.git/info/exclude</code> if you don’t want to commit
changes to <code class="docutils literal">.gitignore</code>), since you don’t want it in your repository (it’s
binary bloat, and works only on your machine).</p>
<p>If you pick the global virtual environment store option, you can use the following short
function (put it in <code class="docutils literal">.bashrc</code> / <code class="docutils literal">.zshrc</code> / your shell configuration file)
to get a simple way to activate an environment (by running <code class="docutils literal">workon foo</code>).
<code class="docutils literal">virtualenvwrapper</code> also has a <code class="docutils literal">workon</code> feature, although I don’t think
<code class="docutils literal">virtualenvwrapper</code> is really necessary and too helpful — the <code class="docutils literal">workon</code>
feature is handy though, and so here’s a way to do it without
<code class="docutils literal">virtualenvwrapper</code>:</p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-1"><code data-line-number="1"></code></a></td><td class="code"><code><a id="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-1" name="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-1"></a><span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">WORKON_HOME</span><span class="o">=</span>~/virtualenvs
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-2"><code data-line-number="2"></code></a></td><td class="code"><code><a id="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-2" name="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-2"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-3"><code data-line-number="3"></code></a></td><td class="code"><code><a id="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-3" name="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-3"></a><span class="w"> </span><span class="k">function</span><span class="w"> </span>workon<span class="w"> </span><span class="o">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-4"><code data-line-number="4"></code></a></td><td class="code"><code><a id="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-4" name="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-4"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="nb">source</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$WORKON_HOME</span><span class="s2">/</span><span class="nv">$1</span><span class="s2">/bin/activate&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-5"><code data-line-number="5"></code></a></td><td class="code"><code><a id="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-5" name="rest_code_d4c94a5f476e4e5ab3fe0a7682a1bdb7-5"></a><span class="w"> </span><span class="o">}</span>
</code></td></tr></table></div><p>And for PowerShell fans, here’s a <code class="docutils literal">workon.ps1</code> script:</p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_33d18c86da274b70953d97d327e4e0e7-1"><code data-line-number="1"></code></a></td><td class="code"><code><a id="rest_code_33d18c86da274b70953d97d327e4e0e7-1" name="rest_code_33d18c86da274b70953d97d327e4e0e7-1"></a> <span class="nv">$WORKON_HOME</span> <span class="p">=</span> <span class="s2">&quot;$home\virtualenvs&quot;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_33d18c86da274b70953d97d327e4e0e7-2"><code data-line-number="2"></code></a></td><td class="code"><code><a id="rest_code_33d18c86da274b70953d97d327e4e0e7-2" name="rest_code_33d18c86da274b70953d97d327e4e0e7-2"></a> <span class="p">&amp;</span> <span class="s2">&quot;$WORKON_HOME\</span><span class="p">$(</span><span class="nv">$args</span><span class="p">[</span><span class="n">0</span><span class="p">])</span><span class="s2">\Scripts\activate.ps1&quot;</span>
</code></td></tr></table></div><p>And for cmd.exe fans… you should switch to PowerShell, it’s a very nice and
friendly shell (though perhaps requiring some effort to learn how to be
productive with it).</p>
</section>
</section>
<section id="use">
<h1>Use</h1>
<p>There are three ways of working with virtual environments interactively (in a
shell):</p>
<ul class="simple">
<li><p>activation (run <code class="docutils literal">source env/bin/activate</code> on *nix;
<code class="docutils literal">env\Scripts\activate</code> on Windows) — it simplifies work and requires less
typing, although it can sometimes fail to work properly. (After installing
scripts, <code class="docutils literal">hash <span class="pre">-r</span></code> may be necessary on *nix to use them.)</p></li>
<li><p>executing <code class="docutils literal">env/bin/python</code> (<code class="docutils literal">env\Scripts\python</code>) and other scripts directly, as
activation only changes <code class="docutils literal">$PATH</code> and some helper variables — those variables
are not mandatory for operation, running the correct <code class="docutils literal">python</code> is, and that
method is failsafe.</p></li>
<li><p><a class="reference external" href="https://gist.github.com/datagrok/2199506">in subshells</a> (IMO, it’s bad UX)</p></li>
</ul>
<p>Whichever method you use, you must remember that without doing any of these
things, you will still be working with the system Python.</p>
<p>For non-interactive work (eg. crontab entries, system services, etc.),
activation and subshells are not viable solutions. In these cases, you must
always use the full path to Python.</p>
<p>Here are some usage examples (paths can be relative, of course):</p>
<div class="code"><pre class="code text"><a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-1" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-1" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-1"></a>## *nix, activation ##
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-2" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-2" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-2"></a>$ source /path/to/env/bin/activate
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-3" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-3" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-3"></a>(env)$ pip install Django
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-4" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-4" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-4"></a>(env)$ deactivate
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-5" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-5" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-5"></a>
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-6" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-6" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-6"></a>## *nix, manual execution ##
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-7" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-7" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-7"></a>$ /path/to/env/bin/pip install Django
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-8" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-8" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-8"></a>
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-9" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-9" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-9"></a>## Windows, activation ##
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-10" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-10" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-10"></a>&gt; C:\path\to\env\Scripts\activate
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-11" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-11" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-11"></a>(venv)&gt; pip install Django
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-12" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-12" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-12"></a>(venv)&gt; deactivate
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-13" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-13" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-13"></a>
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-14" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-14" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-14"></a>## Windows, manual execution ##
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-15" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-15" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-15"></a>&gt; C:\path\to\env\Scripts\pip install Django
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-16" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-16" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-16"></a>
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-17" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-17" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-17"></a>## Windows, updating pip/setuptools/wheel ##
<a id="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-18" name="rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-18" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_99f32ee1312e43b19d00feb1c5e6ae7f-18"></a>&gt; C:\path\to\env\Scripts\python -m pip install -U pip setuptools wheel
</pre></div>
<p>The same principle applies to running Python itself, or any other script
installed by a package. (With Django’s <code class="docutils literal">manage.py</code>, calling it as
<code class="docutils literal">./manage.py</code> requires activation, or you can run
<code class="docutils literal">venv/bin/python manage.py</code>.)</p>
<section id="moving-renaming-copying-environments">
<h2>Moving/renaming/copying environments?</h2>
<p>If you try to copy or rename a virtual environment, you will discover that the
copied environment does not work. This is because a virtual environment is
closely tied to both the Python it was created with, and the location it was
created in. (The “relocatable” option of <code class="docutils literal">virtualenv</code> does not work and is deprecated.) <a class="brackets" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-3" id="footnote-reference-3" role="doc-noteref"><span class="fn-bracket">[</span>3<span class="fn-bracket">]</span></a></p>
<p>However, this is very easy to fix. Instead of moving/copying, just create a new
environment in the new location. Then, run <code class="docutils literal">pip freeze &gt; requirements.txt</code> in
the old environment to create a list of packages installed in it. With that,
you can just run <code class="docutils literal">pip install <span class="pre">-r</span> requirements.txt</code> in the new environment to
install packages from the saved list. (Of course, you can copy <code class="docutils literal">requirements.txt</code>
between machines. In many cases, it will just work; sometimes, you might need a few
modifications to <code class="docutils literal">requirements.txt</code> to remove OS-specific stuff.)</p>
<div class="code"><pre class="code text"><a id="rest_code_118b2d54954242548dd7d3e590824a4c-1" name="rest_code_118b2d54954242548dd7d3e590824a4c-1" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_118b2d54954242548dd7d3e590824a4c-1"></a>$ oldenv/bin/pip freeze &gt; requirements.txt
<a id="rest_code_118b2d54954242548dd7d3e590824a4c-2" name="rest_code_118b2d54954242548dd7d3e590824a4c-2" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_118b2d54954242548dd7d3e590824a4c-2"></a>$ python3 -m venv newenv
<a id="rest_code_118b2d54954242548dd7d3e590824a4c-3" name="rest_code_118b2d54954242548dd7d3e590824a4c-3" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_118b2d54954242548dd7d3e590824a4c-3"></a>$ newenv/bin/pip install -r requirements.txt
<a id="rest_code_118b2d54954242548dd7d3e590824a4c-4" name="rest_code_118b2d54954242548dd7d3e590824a4c-4" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#rest_code_118b2d54954242548dd7d3e590824a4c-4"></a>(You may rm -rf oldenv now if you desire)
</pre></div>
<p>Note that it might also be necessary to re-create your virtual environment
after a Python upgrade, <a class="brackets" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-4" id="footnote-reference-4" role="doc-noteref"><span class="fn-bracket">[</span>4<span class="fn-bracket">]</span></a> so it might be handy to keep an up-to-date
<code class="docutils literal">requirements.txt</code> for your virtual environments (for many projects, it makes
sense to put that in the repository).</p>
<p>To manage those <code class="docutils literal">requirements.txt</code> files in a more orgnized yet still simple
way, you might be interested in <a class="reference external" href="https://github.com/jazzband/pip-tools">pip-tools</a>.</p>
</section>
</section>
<section id="frequently-asked-questions">
<h1>Frequently Asked Questions</h1>
<section id="im-using-virtualenv-do-i-need-to-install-it-for-each-python-i-want-to-use-it-with">
<h2>I’m using virtualenv. Do I need to install it for each Python I want to use it with?</h2>
<p>In most cases, you can use <code class="docutils literal">virtualenv <span class="pre">-p</span> pythonX env</code> to specify a different
Python version, but with some Python version combinations, that approach might
be unsuccessful. (The <code class="docutils literal">venv</code> module is tied to the Python version it’s
installed in.)</p>
</section>
<section id="im-the-only-user-on-my-system-do-i-still-need-virtual-environments">
<h2>I’m the only user on my system. Do I still need virtual environments?</h2>
<p>Yes, you do. First, you will still need separation between projects, sooner or
later.  Moreover, if you were to install packages system-wide with pip, you
might end up causing conflicts between packages installed by the system package
manager and by pip. Running <code class="docutils literal">sudo pip</code> is never a good idea because of this.</p>
</section>
<section id="im-using-docker-do-i-still-need-virtual-environments">
<h2>I’m using Docker. Do I still need virtual environments?</h2>
<p>They are still a good idea in that case. They protect you against any bad
system-wide Python packages your OS image might have (and one popular base OS
is famous for those). They don’t introduce any extra overhead, while allowing
to have a clean environment and the ability to re-create it outside of Docker
(eg. for local development without Docker)</p>
</section>
<section id="what-about-pipenv">
<h2>What about Pipenv?</h2>
<p>Pipenv is a dependency management tool. It isn’t compatible with most workflows, and comes with many issues. In my opinion, it’s not worth using (Also, that thing about it being an officially recommended tool? Turns out it’s not true.)</p>
<p>I also wrote a blog post detailing concerns with that tool, titled <a class="reference external" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/">Pipenv: promises a lot, delivers very little</a>.</p>
<p>Consider using <a class="reference external" href="https://github.com/jazzband/pip-tools">pip-tools</a> instead.</p>
</section>
</section>
<section id="footnotes">
<h1>Footnotes</h1>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="footnote-1" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-reference-1">1</a><span class="fn-bracket">]</span></span>
<p>The thing you’re actually installing is <code class="docutils literal">ensurepip</code>. In general, Debian isn’t exactly friendly with Python packaging.</p>
</aside>
<aside class="footnote brackets" id="footnote-2" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-reference-2">2</a><span class="fn-bracket">]</span></span>
<p>On Windows, you <em>must</em> run <code class="docutils literal">python <span class="pre">-m</span> pip</code> instead of <code class="docutils literal">pip</code> if you want to upgrade the package manager itself.</p>
</aside>
<aside class="footnote brackets" id="footnote-3" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-reference-3">3</a><span class="fn-bracket">]</span></span>
<p>All script shebangs contain the direct path to the environment’s Python executable.  Many things in the virtual environment are symlinks that point to the original Python.</p>
</aside>
<aside class="footnote brackets" id="footnote-4" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/#footnote-reference-4">4</a><span class="fn-bracket">]</span></span>
<p>Definitely after a minor version (3.x → 3.y) upgrade, sometimes (I’m looking at you Homebrew) after a patch version upgrade (3.x.y → 3.x.z) as well.</p>
</aside>
</aside>
</section>
]]></content:encoded><category>Python</category><category>best practices</category><category>devel</category><category>guide</category><category>Python</category><category>venv</category><category>virtual environments</category><category>virtualenv</category></item><item><title>Pipenv: promises a lot, delivers very little</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/</link><pubDate>Tue, 17 Jul 2018 17:40:00 GMT</pubDate><guid>https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/</guid><description>
Pipenv is a Python packaging tool that does one thing reasonably well — application dependency management. However, it is also plagued by issues, limitations and a break-neck development process. In the past, Pipenv’s promotional material was highly misleading as to its purpose and backers.
In this post, I will explore the problems with Pipenv. Was it really
recommended by Python.org? Can everyone — or at least, the vast majority
of people — benefit from it?
(This post has been updated in February 2020 and May 2020 to reflect the
current state of Pipenv.)
</description><content:encoded><![CDATA[
<p>Pipenv is a Python packaging tool that does one thing reasonably well — application dependency management. However, it is also plagued by issues, limitations and a break-neck development process. In the past, Pipenv’s promotional material was highly misleading as to its purpose and backers.</p>
<p>In this post, I will explore the problems with Pipenv. Was it really
recommended by Python.org? Can everyone — or at least, the vast majority
of people — benefit from it?</p>
<p>(This post has been updated in February 2020 and May 2020 to reflect the
current state of Pipenv.)</p>



<section id="a-2020-update-updated">
<h1>A 2020 update (updated)</h1>
<p>This blog post was written in 2018, and it’s still pretty accurate when it
comes to the criticisms of Pipenv, but something else happened since then.</p>
<p class="lead">No release was made between November 2018 and May 2020. Pipenv was effectively
dead for 1.5 years, and the state of Pipenv maintenance is alarming.</p>
<p>A release of Pipenv was made in late 2018 (aptly named v2018.11.26). But then,
there was silence. New commits were made (on the order of 600-700 by the end of
the year). People asked for new releases, in more or less strong words, <a class="reference external" href="https://github.com/pypa/pipenv/issues/3742">in May
2019</a>, then <a class="reference external" href="https://github.com/pypa/pipenv/issues/3978">in October</a>, and again <a class="reference external" href="https://github.com/pypa/pipenv/issues/4058">in December</a>. Many people — including yours
truly, in this post — considered Pipenv dead. On 13th December 2019, the current
maintainer claimed <a class="reference external" href="https://github.com/pypa/pipenv/issues/4058#issuecomment-565550646">a new release is almost finished</a>.</p>
<p>Pipenv 2019/2020 was vaporware for five months. Not much progress was made since the
December post until March 2020, when an issue from 2018 was renamed the <a class="reference external" href="https://github.com/pypa/pipenv/issues/3369">March 2020
Release Tracking Issue</a>. Some
progress was happening, and many release dates were given, but delays stacked
up. March became April. The first beta release was promised by 21st April, it
was delayed until the 29th. The final release was scheduled for a week from
that, but it didn’t happen. Finally, Beta 2 came out on 20th May 2020, and the
final release landed as v2020.5.28.</p>
<p>If you read further into the post, you’ll encounter a chapter titled <a class="reference internal" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#the-break-neck-pace-of-pipenv">The
break-neck pace of Pipenv</a>. Am I being a hypocrite right now? No, not at all.
Projects that are being depended on, such as a package manager, should have
clear policies about how they’re maintained. A new release when the maintainer
feels like adding a feature is too often. A new release every 1.5 years is not
often enough. And silence from maintainers, when faced with questions about
releases, is simply unacceptable.  Pip, for example, has updates every few
months in a fairly stable pace (with the exception of emergency bugfix
releases), and pip has years of development behind it, unlike the fairly new
Pipenv.</p>
<p>And even if the May release was successful, you can’t be sure about any future
releases, and what will happen with Pipenv. At the same time, Pipenv isn’t a
good tool, as this post tries to explain — those criticisms are still valid,
since they are at the core of what Pipenv is. Instead, perhaps consider using
<strong>pip-tools</strong> for locking dependencies? It does one thing, and one thing well.
It doesn’t enforce any specific structures on users, and supports any workflow
you have. (If you don’t need to lock dependencies, <strong>pip + venv</strong> will
suffice.)</p>
</section>
<section id="officially-recommended-tool-or-how-we-got-here">
<h1>“Officially recommended tool”, or how we got here</h1>
<blockquote>
<p>“Pipenv — the officially recommended Python packaging tool from Python.org, free (as in freedom).”</p>
</blockquote>
<p>Pipenv’s README used to have a version of the above line in their README for
many months: it was added on  <a class="reference external" href="https://github.com/pypa/pipenv/commit/6e06fc451767a57e6fccb828c74a1412f6cef687">2017-08-31</a> and eventually disappeared on <a class="reference external" href="https://github.com/pypa/pipenv/commit/47debed9a1c2a3649bef4d59a3f1cf01bf059522">2018-05-19</a>. For a short while (2018-05-16), it was clarified (<em>managing application dependencies</em>, and <em>PyPA</em> instead of <em>Python.org</em>), and for about 15 minutes, the tagline called Pipenv <em>the world’s worst</em> or <a class="reference external" href="https://github.com/pypa/pipenv/commit/6d77e4a0551528d5d72d81e8a15da4722ad82f26">something</a> <a class="reference external" href="https://github.com/pypa/pipenv/commit/1c956d37e6ad20babdb5021610b2ed2c9c4203f2">along</a> <a class="reference external" href="https://github.com/pypa/pipenv/commit/e3c72e167d21b921bd3bd89d4217b04628919bb2">these</a> <a class="reference external" href="https://github.com/pypa/pipenv/commit/fe78628903948013e8687d1a3be9fd4da2b6bd3d">lines</a> (this coming from the maintainer).</p>
<p>The README tagline claimed that Pipenv is the be-all, end-all of Python
packaging. The problem is: it isn’t that. There are some use cases that benefit
from Pipenv, but for many others, trying to use that tool will only lead to
frustration. We will explore this issue later.</p>
<p>Another issue with this tagline was the <em>Python.org</em> and <em>official</em> parts. The
thing that made it “official” was a <a class="reference external" href="https://packaging.python.org/tutorials/managing-dependencies/">short tutorial</a> <a class="brackets" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-1" id="footnote-reference-1" role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a> on packaging.python.org,
which is the PyPA’s packaging user guide. Also of note is the <em>Python.org</em>
domain used. It makes it sound as if Pipenv was endorsed by the Python core
team. PyPA (Python Packaging Authority) is a separate organization — they are
responsible for the packaging parts (including pypi.org, setuptools, pip,
wheel, virtualenv, etc.) of Python. This made the endorsement misleading. Of
course, PyPA is a valued part of the Python world; an endorsement by the core
team — say, <a class="reference external" href="https://docs.python.org/3/library/ensurepip.html">inclusion in official Python distributions</a> — is something far more
important.</p>
<p>This tagline has led to many discussions and flamewars, perhaps with <a class="reference external" href="https://np.reddit.com/r/Python/comments/8jd6aq/why_is_pipenv_the_recommended_packaging_tool_by/">this
Reddit thread from May</a> being the most heated and most important. The change
was the direct result of this Reddit thread. I recommend reading this thread in
full.</p>
</section>
<section id="what-pipenv-does">
<h1>What pipenv does</h1>
<p>We’ve already learned that Pipenv is used to <em>manage application dependencies</em>.
Let’s learn what that term really means.</p>
<section id="application-dependencies">
<h2>Application dependencies</h2>
<p>Here is an example use case for Pipenv:
I’m working on a website based on Django.  I create <code class="docutils literal">~/git/website</code> and run
<code class="docutils literal">pipenv install Django</code> in that directory.  Pipenv:</p>
<ul class="simple">
<li><p>automatically creates a virtualenv somewhere in my home directory</p></li>
<li><p>writes a Pipfile, which lists Django as my dependency</p></li>
<li><p>installs Django using pip</p></li>
<li><p>proceeds to write <code class="docutils literal">Pipfile.lock</code>, which stores the exact version and source file hash <a class="brackets" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-2" id="footnote-reference-2" role="doc-noteref"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></a> of each package installed (including <code class="docutils literal">pytz</code>, Django’s dependency).</p></li>
</ul>
<p>The last part of the process was the most time consuming. At one point, while
locking the dependency versions, Pipenv hangs for 46 seconds. This is one of
Pipenv’s notable issues: <strong>it’s slow.</strong> Of course, this isn’t the only one,
but it defintely doesn’t help. Losing 46 seconds isn’t much, but when we get to
the longer waits in the timing test section later, we’ll see something that
could easily discourage users from using a package.</p>
</section>
<section id="running-scripts-badly">
<h2>Running scripts (badly)</h2>
<p>But let’s continue with our workflow. <code class="docutils literal">pipenv run <span class="pre">django-admin</span> startproject
foobanizer</code> is what I must use now, which is rather unwieldy to type, and
requires running pipenv even for the smallest things. (The <code class="docutils literal">manage.py</code> script
has <code class="docutils literal">/usr/bin/env python</code> in its shebang.) I can run <code class="docutils literal">pipenv shell</code> to get
a new shell which runs the <code class="docutils literal">activate</code> script by default, giving you the worst
of both worlds when it comes to virtualenv activation: the unwieldiness of a
new shell, and the activate script, which the proponents of the shell spawning
dislike.</p>
<p>Using <code class="docutils literal">pipenv shell</code> means spawning a new subshell, executing the shell
startup scripts (eg. <code class="docutils literal">.bashrc</code>), and requiring you to exit with <code class="docutils literal">exit</code> or
^D. If you type <code class="docutils literal">deactivate</code>, you are working with an extra shell, but now
outside of the virtualenv. Or you can use the <code class="docutils literal"><span class="pre">--fancy</span></code> mode that manipulates
<code class="docutils literal">$PATH</code> before launching the subshell, but it requires a specific shell
configuration, in which <code class="docutils literal">$PATH</code> is not overridden in non-login shells — and
also often changing the config of your terminal emulator to run a login shell,
as many of the Linux terminals don’t do it.</p>
<p>Now, why does all this happen? Because a command cannot manipulate the
environment of the shell it spawns. This means that Pipenv must pretend what it
does is a reasonable thing instead of a workaround. This can be solved with
manual activation using <code class="docutils literal">source $(pipenv <span class="pre">--venv)/bin/activate</span></code> (can be made
into a neat alias), or shell wrappers (similar to what virtualenvwrapper does).</p>
</section>
<section id="finishing-it-all-up">
<h2>Finishing it all up</h2>
<p>Anyway, I want a blog on my site. I want to write them in Markdown syntax, so I
run <code class="docutils literal">pipenv install Markdown</code>, and a few long seconds later, it’s added to
both Pipfiles.  Another thing I can do is <code class="docutils literal">pipenv install <span class="pre">--dev</span> ipython</code> and
get a handy shell for tinkering, but it will be marked as a development
dependency — so, not installed in production. That last part is an important
advantage of using Pipenv.</p>
<p>When I’m done working on my website, I commit both Pipfiles to my git
repository, and push it to the remote server. Then I can clone it to, say,
<code class="docutils literal">/srv/website</code>. Now I can just <code class="docutils literal">pipenv install</code> to get all the production
packages installed (but not the development ones — Django, pytz, Markdown will
be installed, but IPython and all its million dependencies won’t). There’s just
one caveat: by default, the virtualenv will still be created in the current
user’s home directory. This is a problem in this case, since it needs to be
accessible by <a class="reference external" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/">nginx and uWSGI</a>, which do not have access to my (or root’s)
home directory, and don’t have a home directory of their own.  This can be
solved with <code class="docutils literal">export PIPENV_VENV_IN_PROJECT=1</code>. But note that I will now need
to export this environment variable every time I work with the app in <code class="docutils literal">/srv</code>
via Pipenv. The tool supports loading <code class="docutils literal">.env</code> files, <strong>but</strong> only when
running <code class="docutils literal">pipenv shell</code> and <code class="docutils literal">pipenv run</code>. You can’t use it to configure
Pipenv. And to run my app with nginx/uWSGI, I will need to know the exact virtualenv
path anyway, since I can’t use <code class="docutils literal">pipenv run</code> as part of uWSGI configuration.</p>
</section>
</section>
<section id="what-pipenv-doesnt-do">
<h1>What pipenv doesn’t do</h1>
<p>The workflow I mentioned above looks pretty reasonable, right? There are some
deficiencies, but other than that, it seems to work well. The main issue with
Pipenv is: <strong>it works with one workflow, and one workflow only.</strong> Try to do
anything else, and you end up facing multiple obstacles.</p>
<section id="setup-py-source-distributions-and-wheels">
<h2>Setup.py, source distributions, and wheels</h2>
<p>Pipenv only concerns itself with managing dependencies. <strong>It isn’t a packaging
tool.</strong> If you want your thing up on PyPI, Pipenv won’t help you with anything.
You still need to write a <code class="docutils literal">setup.py</code> with <code class="docutils literal">install_requires</code>, because the
Pipfile format only specifies the dependencies and runtime requirements (Python
version), there is no place in it for the package name, and Pipenv does not
mandate/expect you to install your project. It can come in handy to manage the
development environment (as a <code class="docutils literal">requirements.txt</code> replacement, or something
used to write said file), but if your project has a <code class="docutils literal">setup.py</code>, you still
need to manually manage <code class="docutils literal">install_requires</code>. Pipenv can’t create wheels on its
own either. And <code class="docutils literal">pip freeze</code> is going to be a lot faster than Pipenv ever
will be.</p>
</section>
<section id="working-outside-of-the-project-root">
<h2>Working outside of the project root</h2>
<p>Another issue with Pipenv is the use of the working directory to select
the virtual environment. <a class="brackets" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-3" id="footnote-reference-3" role="doc-noteref"><span class="fn-bracket">[</span>3<span class="fn-bracket">]</span></a> Let’s say I’m a library author.  A user of my <code class="docutils literal">foobar</code>
library has just reported a bug and attached a <code class="docutils literal">repro.py</code> file that lets me
reproduce the issue. I download that file to <code class="docutils literal">~/Downloads</code> on my filesystem.
With plain old virtualenv, I can easily confirm the reproduction in a spare
shell with:</p>
<div class="code"><pre class="code shell"><a id="rest_code_83381fdf79414bf799a8ce2777ae054f-1" name="rest_code_83381fdf79414bf799a8ce2777ae054f-1" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_83381fdf79414bf799a8ce2777ae054f-1"></a>$<span class="w"> </span>~/virtualenvs/foobar/bin/python<span class="w"> </span>~/Downloads/repro.py
</pre></div>
<p>And then I can launch my fancy IDE to fix the bug.  I don’t have to <code class="docutils literal">cd</code> into
the project. But with Pipenv, I can’t really do that.  If I put the virtualenv
in <code class="docutils literal">.venv</code> with the command line option, I can type
<code class="docutils literal"><span class="pre">~/git/foobar/.venv/bin/python</span> ~/Downloads/repro.py</code>. If I use the
centralized directory + hashes thing, Tab completion becomes mandatory, if I
haven’t memorized the hash.</p>
<div class="code"><pre class="code shell"><a id="rest_code_227b6f3431ed43459b23802ee8423842-1" name="rest_code_227b6f3431ed43459b23802ee8423842-1" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_227b6f3431ed43459b23802ee8423842-1"></a>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>~/git/foobar
<a id="rest_code_227b6f3431ed43459b23802ee8423842-2" name="rest_code_227b6f3431ed43459b23802ee8423842-2" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_227b6f3431ed43459b23802ee8423842-2"></a>$<span class="w"> </span>pipenv<span class="w"> </span>run<span class="w"> </span>python<span class="w"> </span>~/Downloads/repro.py
</pre></div>
<p>What if I had two <code class="docutils literal">.py</code> files, or <code class="docutils literal">repro.py</code> otherwise depended on being in
the current working directory?</p>
<div class="code"><pre class="code shell"><a id="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-1" name="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-1" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-1"></a>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>~/git/foobar
<a id="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-2" name="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-2" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-2"></a>$<span class="w"> </span>pipenv<span class="w"> </span>shell
<a id="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-3" name="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-3" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-3"></a><span class="o">(</span>foobar-Mwd1l2m9<span class="o">)</span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>~/Downloads
<a id="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-4" name="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-4" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-4"></a><span class="o">(</span>foobar-Mwd1l2m9<span class="o">)</span>$<span class="w"> </span>python<span class="w"> </span>repro.py
<a id="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-5" name="rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-5" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_6d3e90e2fbf547908eaaa3ef2a0e9646-5"></a><span class="o">(</span>foobar-Mwd1l2m9<span class="o">)</span>$<span class="w"> </span><span class="nb">exit</span><span class="w">  </span><span class="c1"># (not deactivate!)</span>
</pre></div>
<p><strong>This is becoming ugly fairly quickly.</strong> Also, with virtualenvwrapper, I can
do this:</p>
<div class="code"><pre class="code shell"><a id="rest_code_1303a58c0d4e43319a2fc0e5e958e914-1" name="rest_code_1303a58c0d4e43319a2fc0e5e958e914-1" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_1303a58c0d4e43319a2fc0e5e958e914-1"></a>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>~/Downloads
<a id="rest_code_1303a58c0d4e43319a2fc0e5e958e914-2" name="rest_code_1303a58c0d4e43319a2fc0e5e958e914-2" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_1303a58c0d4e43319a2fc0e5e958e914-2"></a>$<span class="w"> </span>workon<span class="w"> </span>foobar
<a id="rest_code_1303a58c0d4e43319a2fc0e5e958e914-3" name="rest_code_1303a58c0d4e43319a2fc0e5e958e914-3" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_1303a58c0d4e43319a2fc0e5e958e914-3"></a><span class="o">(</span>foobar<span class="o">)</span>$<span class="w"> </span>python<span class="w"> </span>repro.py
<a id="rest_code_1303a58c0d4e43319a2fc0e5e958e914-4" name="rest_code_1303a58c0d4e43319a2fc0e5e958e914-4" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#rest_code_1303a58c0d4e43319a2fc0e5e958e914-4"></a><span class="o">(</span>foobar<span class="o">)</span>$<span class="w"> </span>deactivate
</pre></div>
<p>And let’s not forget that Pipenv doesn’t help me to write a <code class="docutils literal">setup.py</code>,
distribute code, or manage releases.  It just manages dependencies.  And it
does it pretty badly.</p>
</section>
<section id="nikola">
<h2>Nikola</h2>
<p>I’m a co-maintainer of a static site generator, <a class="reference external" href="https://getnikola.com/">Nikola</a>.  As part of this, I have the following places where
I need to run Nikola:</p>
<ul class="simple">
<li><p><code class="docutils literal">~/git/nikola</code></p></li>
<li><p><code class="docutils literal"><span class="pre">~/git/nikola-site</span></code></p></li>
<li><p><code class="docutils literal"><span class="pre">~/git/nikola-plugins</span></code></p></li>
<li><p><code class="docutils literal"><span class="pre">~/git/nikola-themes</span></code></p></li>
<li><p><code class="docutils literal">~/website</code> (this blog)</p></li>
<li><p><code class="docutils literal">/Volumes/RAMDisk/n</code> (demo site, used for testing and created when needed, on a <a class="reference external" href="https://en.wikipedia.org/wiki/RAM_drive">RAM disk</a>)</p></li>
</ul>
<p>That list is long.  End users of Nikola probably don’t have a list that long,
but they might just have more than one Nikola site.  For me, and for the
aforementioned users, Pipenv does not work.  To use Pipenv, all those
repositories would need to live in one directory. I would also need to have a
<em>separate</em> Pipenv environment for <code class="docutils literal"><span class="pre">nikola-users</span></code>, because that needs Django.
Moreover, the Pipfile would have to be symlinked from <code class="docutils literal">~/git/nikola</code> if we
were to make use of those in the project.  So, I would have a <code class="docutils literal">~/nikola</code>
directory just to make Pipenv happy, do testing/bug reproduction on a SSD (and
wear it out faster), and so on… Well, I could also use the virtualenv directly.
But in that case, Pipenv loses its usefulness, and makes my workflow more
complicated. I can’t use <code class="docutils literal">virtualenvwrapper</code>, because I would need to hack a
fuzzy matching system onto it, or memorize the random string appended to my
virtualenv name.  All because Pipenv relies on the current directory too much.</p>
<p>Nikola end users who want to use Pipenv will also have a specific directory
structure forced on them. What if the site serves as docs for a project, and
lives inside another project’s repo? Two virtualenvs, 100 megabytes wasted.
Or worse, Nikola ends up in the other project’s Pipfile, which is technically
good for our download stats, but not really good for the other project’s
contributors.</p>
</section>
</section>
<section id="the-part-where-i-try-to-measure-times">
<h1>The part where I try to measure times</h1>
<p>Pipenv is famous for being slow.  But how slow is it really?
I put it to the test.  I used two test environments:</p>
<ul class="simple">
<li><p>Remote: a DigitalOcean VPS, the cheapest option (1 vCPU), Python 3.6/Fedora
28, in Frankfurt</p></li>
<li><p>Local: my 2015 13” MacBook Pro (base model), Python 3.7, on a rather slow
Internet connection (10 Mbps on a good day, and the test was not performed on
one of them)</p></li>
</ul>
<p>Both were runninng Pipenv 2018.7.1, installed from pip.</p>
<p>And with the following cache setups:</p>
<ul class="simple">
<li><p>Removed: <code class="docutils literal"><span class="pre">~/.cache/pipenv</span></code> removed</p></li>
<li><p>Partial: <code class="docutils literal">rm <span class="pre">-rf</span> <span class="pre">~/.cache/pipenv/depcache-py*.json</span> <span class="pre">~/.cache/pipenv/hash-cache/</span></code></p></li>
<li><p>Kept: no changes done from previous run</p></li>
</ul>
<p>Well, turns out Pipenv likes doing strange things with caching and locking.  A
look at the Activity Monitor hinted that there is network activity going on
when Pipenv displays its <em>Locking [packages] dependencies...</em> line and
hangs. Now, the docs don’t tell you that. The most atrocious example was a
local Nikola install that was done in two runs: the first <code class="docutils literal">pipenv install
Nikola</code> run was interrupted <a class="brackets" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-4" id="footnote-reference-4" role="doc-noteref"><span class="fn-bracket">[</span>4<span class="fn-bracket">]</span></a> right after it was done installing packages,
so the cache had all the necessary wheels in it. The install took 10 minutes
and 7 seconds, 9:50 of which were taken by locking dependencies and installing
the locked dependencies — so, roughly nine and a half minutes were spent
staring at a static screen, with the tool doing <em>something</em> in the background —
and Pipenv doesn’t tell you what happens in this phase.</p>
<p>(Updated 2018-07-22: In the pipenv measurements: the first entry is the total
time of pipenv executon. The second is the long wait for pipenv to do its
“main” job: locking dependencies and installing them. The timing starts when
pipenv starts locking dependencies and ends when the prompt appears. The third
item is pipenv’s reported installation time.  So, pipenv install ⊇ locking/installing ⊇ Pipfile.lock install.)</p>
<table class="table table-striped table-bordered">
<thead>
<tr><th class="head" rowspan="2"><p>Task</p></th>
<th class="head" rowspan="2"><p>Action</p></th>
<th class="head" rowspan="2"><p>Measurement
method</p></th>
<th class="head" rowspan="2"><p>Environment</p></th>
<th class="head" rowspan="2"><p>Cache</p></th>
<th class="head" colspan="4"><p>Times in seconds</p></th>
</tr>
<tr><th class="head"><p>Attempt 1</p></th>
<th class="head"><p>Attempt 2</p></th>
<th class="head"><p>Attempt 3</p></th>
<th class="head"><p>Average</p></th>
</tr>
</thead>
<tbody>
<tr><td><p>1</p></td>
<td><p>virtualenv</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td><p>Remote</p></td>
<td><p>(not applicable)</p></td>
<td><p>3.911</p></td>
<td><p>4.052</p></td>
<td><p>3.914</p></td>
<td><p>3.959</p></td>
</tr>
<tr><td><p>2</p></td>
<td><p>pip install Nikola</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td><p>Remote</p></td>
<td><p>Removed</p></td>
<td><p>11.562</p></td>
<td><p>11.943</p></td>
<td><p>11.773</p></td>
<td><p>11.759</p></td>
</tr>
<tr><td><p>3</p></td>
<td><p>pip install Nikola</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td><p>Remote</p></td>
<td><p>Kept</p></td>
<td><p>7.404</p></td>
<td><p>7.681</p></td>
<td><p>7.569</p></td>
<td><p>7.551</p></td>
</tr>
<tr><td rowspan="3"><p>4</p></td>
<td><p>pipenv install Nikola</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Remote</p></td>
<td rowspan="3"><p>Removed</p></td>
<td><p>67.536</p></td>
<td><p>62.973</p></td>
<td><p>71.305</p></td>
<td><p>67.271</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>42.6</p></td>
<td><p>40.5</p></td>
<td><p>39.6</p></td>
<td><p>40.9</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>14</p></td>
<td><p>14</p></td>
<td><p>13</p></td>
<td><p>13.667</p></td>
</tr>
<tr><td rowspan="3"><p>5</p></td>
<td><p>adding Django to an environment</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Remote</p></td>
<td rowspan="3"><p>Kept (only Nikola in cache)</p></td>
<td><p>39.576</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>39.576</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>32</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>32</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>14</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>14</p></td>
</tr>
<tr><td rowspan="3"><p>6</p></td>
<td><p>adding Django to another environment</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Remote</p></td>
<td rowspan="3"><p>Kept (both in cache)</p></td>
<td><p>37.978</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>37.978</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>30.2</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>30.2</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>14</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>14</p></td>
</tr>
<tr><td rowspan="3"><p>7</p></td>
<td><p>pipenv install Django</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Remote</p></td>
<td rowspan="3"><p>Removed</p></td>
<td><p>20.612</p></td>
<td><p>20.666</p></td>
<td><p>20.665</p></td>
<td><p>20.648</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>6.6</p></td>
<td><p>6.4</p></td>
<td><p>6</p></td>
<td><p>6.333</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>1</p></td>
<td><p>1</p></td>
<td><p>1</p></td>
<td><p>1</p></td>
</tr>
<tr><td rowspan="3"><p>8</p></td>
<td><p>pipenv install Django (new env)</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Remote</p></td>
<td rowspan="3"><p>Kept</p></td>
<td><p>17.615</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>17.615</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>3.5</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>3.5</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>1</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>1</p></td>
</tr>
<tr><td rowspan="3"><p>9</p></td>
<td><p>pipenv install Nikola</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Remote</p></td>
<td rowspan="3"><p>Partial</p></td>
<td><p>61.507</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>61.507</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>38.40</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>38.40</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>14</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>14</p></td>
</tr>
<tr><td rowspan="3"><p>10</p></td>
<td><p>pipenv install Django</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Local</p></td>
<td rowspan="3"><p>Removed</p></td>
<td><p>73.933</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>73.933</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>46</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>46</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>0</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>0</p></td>
</tr>
<tr><td><p>11</p></td>
<td><p>virtualenv</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td><p>Local</p></td>
<td><p>(not applicable)</p></td>
<td><p>5.864</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>5.864</p></td>
</tr>
<tr><td><p>12</p></td>
<td><p>pip install Nikola (cached)</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td><p>Local</p></td>
<td><p>Kept</p></td>
<td><p>10.951</p></td>
<td><p>—</p></td>
<td><p>—</p></td>
<td><p>10.951</p></td>
</tr>
<tr><td rowspan="3"><p>13</p></td>
<td><p>pipenv install Nikola</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td rowspan="3"><p>Local</p></td>
<td rowspan="3"><p>Partial, after interruption</p></td>
<td><p>607.647</p></td>
<td colspan="2"><p>(10m 7s)</p></td>
<td><p>607.647</p></td>
</tr>
<tr><td><p>├─ locking/installing from lockfile</p></td>
<td><p>stopwatch</p></td>
<td><p>590.85</p></td>
<td colspan="2"><p>(9m 50s)</p></td>
<td><p>590.85</p></td>
</tr>
<tr><td><p>└─ Pipfile.lock install</p></td>
<td><p>pipenv</p></td>
<td><p>6</p></td>
<td colspan="2"></td>
<td><p>6</p></td>
</tr>
<tr><td><p>14</p></td>
<td><p>pipenv install</p></td>
<td><p><code class="docutils literal">time</code></p></td>
<td><p>Local</p></td>
<td><p>Kept</p></td>
<td><p>31.399</p></td>
<td colspan="2"><p>(L/I: 10.51 s)</p></td>
<td><p>31.399</p></td>
</tr>
</tbody>
</table>
</section>
<section id="alternative-tools">
<h1>Alternative tools</h1>
<p>Python packaging is something with the state of which nobody seems to be
satisfied. As such, there are many new contenders for the role of “best new
packaging tool”.</p>
<p>Two popular alternatives packaging tools are <a class="reference external" href="https://github.com/jazzband/pip-tools">pip-tools</a> (by Vincent Driessen
and Jazzband) and <a class="reference external" href="https://github.com/sdispater/poetry">Poetry</a> (by Sébastien Eustace).</p>
<section id="pip-tools-locking-and-hashing-and-that-is-all">
<h2>Pip-tools: locking and hashing, and that is all</h2>
<p><strong>Pip-tools</strong> contains two tools. The first one is <code class="docutils literal"><span class="pre">pip-compile</span></code>. It locks
dependencies in <code class="docutils literal">requirements.txt</code> files, and that’s all it does. It allows
updating dependencies in the file based on what’s on PyPI. You can optionally
add hashes to that file. The second tool is <code class="docutils literal"><span class="pre">pip-sync</span></code>. It will synchronize
your virtualenv and the requirements file: it will delete packages not in that
file, so that you don’t work with stuff not declared in requirements, and will
ensure versions match the requirements file.  <code class="docutils literal"><span class="pre">pip-compile</span></code> takes roughly
10-20 seconds to run in the Nikola repo with a clean pip-tools cache (but with
the pip cache intact).</p>
<p>Its speed is fairly reasonable, and it does not try to be the be-all-end-all
tool for development. It handles a specific task, does it well, and does not
try to handle tasks it should not. Pip-tools lets you work with venvs in any
way you like, and it does not require anything specific. Unlike Pipenv and
Poetry, you can install it into the virtualenvs that need it, and not
system-wide.</p>
</section>
<section id="poetry-better-but-still-not-convincing">
<h2>Poetry: better, but still not convincing</h2>
<p><strong>Poetry</strong> is somewhere in between. Its main aim is close to Pipenv, but it
also makes it possible to distribute things to PyPI. It tries really hard to
hide that it uses Pip behind the scenes. Its README comes with an extensive
<a class="reference external" href="https://github.com/sdispater/poetry#what-about-pipenv">“What about Pipenv?”</a>
section, which I recommend reading — it has a few more examples of bad Pipenv
features.  Poetry claims to use the standardized (PEP 518) <code class="docutils literal">pyproject.toml</code>
file to replace the usual lot of files. Unfortunately, the only thing that is
standardized is the file name and syntax. Poetry uses custom <code class="docutils literal">[tool.poetry]</code>
sections, which means that one needs Poetry to fully use the packages created
with it, leading to vendor lock-in. There is a <code class="docutils literal">build</code> feature to produce a
sdist with setup.py and friends.</p>
<p>In February 2020, in a simple <code class="docutils literal">poetry add Nikola</code> test, it took <strong>about a
minute</strong> (55.1/50.8/53.6 s) to resolve dependencies (according to Poetry’s own
count, Local environment, Poetry cache removed), complete with reassuring output and
no quiet lockups.  Not as good as pip, but it’s more reasonable than Pipenv.
Also, the codebase and its layout are rather convoluted, and the docs are very
sparse and lacking. Poetry produces packages instead of just managing
dependencies, so it’s generally more useful than Pipenv. That said, I am not
convinced by that tool either.</p>
</section>
</section>
<section id="pip-is-here-to-stay">
<h1>Pip is here to stay!</h1>
<p>But in all the talk about new tools, we’re forgetting about the old ones, and
they do their job well — so well in fact, that the new tools still need them
under the covers.</p>
<p>Pip is fast. It does its job well enough. It lacks support for splitting
packages between production and development (as Pipenv and Poetry do). This
means that <code class="docutils literal">pip freeze</code> and <code class="docutils literal">pip install</code> are instant, at the cost of (a)
needing two separate environments, or (b) installing development dependencies
in production (which <em>should</em> only be a waste of HDD space and nothing more in
a well-architected system). But at the same time, pip-tools can help keep the
environments separate, as long as you take some time to write separate
<code class="docutils literal">requirements.in</code> files.</p>
<p>The virtualenv management features can be provided by virtualenvwrapper. That
tool’s main advantage is the shell script implementation, which means that
<code class="docutils literal">workon foo</code> activates the <code class="docutils literal">foo</code> virtualenv without spawning a new
subshell (an issue with Pipenv and Poetry, that I already covered when
describing Pipenv’s operation in the <a class="reference internal" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#running-scripts-badly">Running scripts (badly)</a> chapter.) An
argument often raised by Pipenv proponents is that one does not need to concern
itself with creating the virtualenv, and doesn’t need to care where it is.
Unfortuntately, many tools require this knowledge from their user, or force a
specific location, or require it to be different to the home directory.</p>
<p>And for a reasonable project template with release automation — well, I have my
own entry in that category, called (rather unoriginally) the <a class="reference external" href="https://github.com/Kwpolska/python-project-template">Python Project
Template (PyPT)</a>.</p>
<p>Yes, setup.py files are not ideal, since they use <code class="docutils literal">.py</code> code and a function
execution, making access to meta information hard (<code class="docutils literal">./setup.py egg_info</code>
creates tool-accessible text files). Their main advantage is that they are the
<em>only</em> format that is widely supported — pip is the de-facto default
Python package manager (which is pre-installed on Windows and Mac), and other
tools would require installation/bootstrapping first.</p>
</section>
<section id="the-break-neck-pace-of-pipenv">
<h1>The break-neck pace of Pipenv</h1>
<p>A good packaging tool is stable. In other words, it doesn’t change often, and
it strives to support existing environments. It wouldn’t be fun to re-download
everything on your system, because someone decided that <code class="docutils literal">/usr</code> is now called
<code class="docutils literal">/stuff</code>, and all the files in <code class="docutils literal">/usr</code> would become forgotten and not
removed. Well, this is what Pipenv did:</p>
<table class="table table-striped table-bordered">
<thead>
<tr><th class="head"><p>Date/Time (UTC)</p></th>
<th class="head"><p>Event</p></th>
</tr>
</thead>
<tbody>
<tr><td><p>2017-01-31 22:01</p></td>
<td><p>v3.2.14 released. <code class="docutils literal">pipenv <span class="pre">--three</span></code> creates <code class="docutils literal"><span class="pre">./.venv</span></code> (eg. <code class="docutils literal"><span class="pre">~/git/foo/.venv</span></code>). Last version with the original behavior of pipenv.</p></td>
</tr>
<tr><td><p>2017-02-01 05:36</p></td>
<td><p>v3.3.0 released. <code class="docutils literal">pipenv <span class="pre">--three</span></code> creates <code class="docutils literal"><span class="pre">~/.local/share/virtualenvs/foo</span></code> (to be precise, <code class="docutils literal">$WORKON_HOME/foo</code>).</p></td>
</tr>
<tr><td><p>2017-02-01 06:10</p></td>
<td><p><a class="reference external" href="https://github.com/pypa/pipenv/issues/178">Issue #178</a> is reported regarding the behavior change.</p></td>
</tr>
<tr><td><p>2017-02-01 06:18</p></td>
<td><p>Kenneth Reitz responds: “no plans for making it configurable.” and closes the issue.</p></td>
</tr>
<tr><td><p>2017-02-02 03:05</p></td>
<td><p>Kenneth Reitz responds: “added <code class="docutils literal">PIPENV_VENV_IN_PROJECT</code> mode for classic operation. Not released yet.”</p></td>
</tr>
<tr><td><p>2017-02-02 04:29</p></td>
<td><p>v3.3.3 released. The default is still uses a “remote” location, but <code class="docutils literal">.venv</code> can now be used.</p></td>
</tr>
<tr><td><p>2017-03-02 13:48</p></td>
<td><p>v3.5.0 released. The new default path is <code class="docutils literal"><span class="pre">$WORKON_HOME/foo-HASH</span></code>, eg. <code class="docutils literal"><span class="pre">~/.local/share/virtualenvs/foo-7pl2iuUI</span></code>.</p></td>
</tr>
</tbody>
</table>
<p>Over the course of a month, the location of the virtualenv changed twice. If
the user didn’t read the changelog and didn’t manually intervene (also of note,
the option name was mentioned in the issue and in v3.3.4’s changelog), they
would have a stale <code class="docutils literal">.venv</code> directory, since the new scheme was adopted for
them. And then, after switching to v3.5.0, they would have a stale virtualenv
hidden somewhere in their home directory, because pipenv decided to add hashes.</p>
<p>Also, this is not configurable. One cannot disable the hashes in paths, even
though <a class="reference external" href="https://github.com/pypa/pipenv/issues/589">users</a> <a class="reference external" href="https://github.com/pypa/pipenv/issues/1049">wanted</a> to. It would also help people
who want to mix Pipenv and virtualenvwrapper.</p>
<p>Pipenv is a very <strong>opinionated</strong> tool, and if the dev team changes their mind,
the old way is not supported.</p>
<p>Pipenv moves fast and doesn’t care if anything breaks. As an example, between
2018-03-13 13:21 and 2018-03-14 13:44 (a little over 24 hours), Pipenv had 10
releases, ranging from v11.6.2 to v11.7.3. The <a class="reference external" href="https://github.com/pypa/pipenv/blob/25df09c171a548fd71d4df735767bf763a653b83/HISTORY.txt">changelog</a> is rather unhelpful
when it comes to informing users what happened in each of the releases.</p>
<p>Extra reading:</p>
<ul class="simple">
<li><p><a class="reference external" href="http://web.archive.org/web/20180717140106/https://journal.kennethreitz.org/entry/r-python">Kenneth Reitz, A Letter to /r/python (with some notes about bipolar disorder)</a> (replaced with Wayback Machine link on 2020-02-07)</p></li>
<li><p>Reddit comment threads for the letter: <a class="reference external" href="https://np.reddit.com/r/Python/comments/8kdfd6/kenneth_reitz_a_letter_to_rpython_with_some_notes/">first</a> and <a class="reference external" href="https://np.reddit.com/r/Python/comments/8kjv8x/a_letter_to_rpython_kenneth_reitzs_journal/">second</a></p></li>
</ul>
</section>
<section id="conclusion">
<h1>Conclusion</h1>
<ul class="simple">
<li><p>Pipenv, contrary to popular belief and (now removed) propaganda, is not an
officially recommended tool of Python.org. It merely has a tutorial written
about it on packaging.python.org (page run by the PyPA).</p></li>
<li><p>Pipenv solves one use case reasonably well, but fails at many others, because
it forces a particular workflow on its users.</p></li>
<li><p>Pipenv does not handle any parts of packaging (cannot produce sdists and
wheels).  Users who want to upload to PyPI need to manage a <code class="docutils literal">setup.py</code> file
manually, alongside and independently of Pipenv.</p></li>
<li><p>Pipenv produces lockfiles, which are useful for reproducibility, at the cost
of installation speed. The speed is a noticeable issue with the tool. <code class="docutils literal">pip
freeze</code> is good enough for this, even if there are no dependency classes
(production vs development) and no hashes (which
have minor benefits) <a class="brackets" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-2" id="footnote-reference-5" role="doc-noteref"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></a></p></li>
<li><p>Poetry supports the same niche Pipenv does, while also adding the ability to
create packages and improving over many gripes of Pipenv. A notable issue is
the use of a custom all-encompassing file format, which makes switching tools
more difficult (vendor lock-in).</p></li>
<li><p>Pip, setup.py, and virtualenv — the traditional, tried-and-true tools — are
still available, undergoing constant development. Using them can lead to a
simpler, better experience.  Also of note, tools like virtualenvwrapper
can manage virtualenvs better than the aforementioned new Python tools,
because it is based on shell scripts (which can modify the enivironment).</p></li>
<li><p>Since 2018, the packaging scene deteriorated even more. See <a class="reference external" href="https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/">How to improve
Python packaging, or why fourteen tools are at least twelve too many</a> (from
January 2023) and <a class="reference external" href="https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/">Python Packaging, One Year Later: A Look Back at 2023 in
Python Packaging</a> (from January 2024).</p></li>
</ul>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="footnote-1" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-reference-1">1</a><span class="fn-bracket">]</span></span>
<p>On a side note, the tutorial explains nothing. A prospective user only learns it’s similar to npm or bundler (what does that mean?), installs one package, and runs a <code class="docutils literal">.py</code> file through <code class="docutils literal">pipenv run</code>.</p>
</aside>
<aside class="footnote brackets" id="footnote-2" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></span>
<span class="backrefs">(<a role="doc-backlink" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-reference-2">1</a>,<a role="doc-backlink" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-reference-5">2</a>)</span>
<p>Note that one can’t change the file on PyPI after uploading it, so this would only be protection against rogue PyPI admins or a MitM attack (in which case you’ve got bigger problems anyways). <a class="reference external" href="https://github.com/nedbat/coveragepy/issues/679#issuecomment-406396761">Also, the feature is fairly broken.</a></p>
</aside>
<aside class="footnote brackets" id="footnote-3" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-reference-3">3</a><span class="fn-bracket">]</span></span>
<p>Fortunately, it looks in the parent directories for Pipfiles as well. Otherwise, you might end up with one environment for <code class="docutils literal">foo</code> and another for <code class="docutils literal">foo/foo</code> and yet another for <code class="docutils literal">foo/docs</code> and so on…</p>
</aside>
<aside class="footnote brackets" id="footnote-4" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#footnote-reference-4">4</a><span class="fn-bracket">]</span></span>
<p>The interruption happened by mistake due to the RAM disk running out of space, but it was actually a good thing to have happened.</p>
</aside>
</aside>
<hr class="docutils">
<p class="alert alert-info"><strong>Other discussion threads:</strong> <a class="reference external" href="https://www.reddit.com/r/Python/comments/a3h81m/pipenv_promises_a_lot_delivers_very_little/">r/Python</a>, <a class="reference external" href="https://news.ycombinator.com/item?id=18612590">Hacker News</a>.</p>
</section>
]]></content:encoded><category>Python</category><category>packaging</category><category>Pipenv</category><category>Python</category></item><item><title>Gynvael’s Mission 11 (en): Python bytecode reverse-engineering</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/</link><pubDate>Thu, 03 Aug 2017 10:45:40 GMT</pubDate><guid>https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/</guid><description>
Gynvael Coldwind is a security researcher at Google, who hosts weekly livestreams about security and programming in Polish and English). As part of the streams, he gives out missions — basically, CTF-style reverse engineering tasks. Yesterday’s mission was about Elvish — I mean Paint — I mean Python programming and bytecode.
</description><content:encoded><![CDATA[
<p>Gynvael Coldwind is a security researcher at Google, who hosts weekly livestreams about security and programming in <a class="reference external" href="https://gaming.youtube.com/user/GynvaelColdwind/live">Polish</a> and <a class="reference external" href="https://gaming.youtube.com/user/GynvaelEN/live">English</a>). As part of the streams, he gives out missions — basically, CTF-style reverse engineering tasks. Yesterday’s mission was about Elvish — I mean Paint — I mean Python programming and bytecode.</p>



<div class="code"><pre class="code text"><a id="rest_code_a8243f6403b44a13a671955d2b908777-1" name="rest_code_a8243f6403b44a13a671955d2b908777-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-1"></a>MISSION 011               goo.gl/13Bia9             DIFFICULTY: ██████░░░░ [6╱10]
<a id="rest_code_a8243f6403b44a13a671955d2b908777-2" name="rest_code_a8243f6403b44a13a671955d2b908777-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-2"></a>┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
<a id="rest_code_a8243f6403b44a13a671955d2b908777-3" name="rest_code_a8243f6403b44a13a671955d2b908777-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-3"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-4" name="rest_code_a8243f6403b44a13a671955d2b908777-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-4"></a>Finally some real work!
<a id="rest_code_a8243f6403b44a13a671955d2b908777-5" name="rest_code_a8243f6403b44a13a671955d2b908777-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-5"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-6" name="rest_code_a8243f6403b44a13a671955d2b908777-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-6"></a>One of our field agents managed to infiltrate suspects hideout and steal a
<a id="rest_code_a8243f6403b44a13a671955d2b908777-7" name="rest_code_a8243f6403b44a13a671955d2b908777-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-7"></a>pendrive possibly containing important information. However, the pendrive
<a id="rest_code_a8243f6403b44a13a671955d2b908777-8" name="rest_code_a8243f6403b44a13a671955d2b908777-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-8"></a>actually requires one to authenticate themselves before accessing the stored
<a id="rest_code_a8243f6403b44a13a671955d2b908777-9" name="rest_code_a8243f6403b44a13a671955d2b908777-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-9"></a>files.
<a id="rest_code_a8243f6403b44a13a671955d2b908777-10" name="rest_code_a8243f6403b44a13a671955d2b908777-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-10"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-11" name="rest_code_a8243f6403b44a13a671955d2b908777-11" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-11"></a>We gave the pendrive to our laboratory and they managed to dump the firmware. We
<a id="rest_code_a8243f6403b44a13a671955d2b908777-12" name="rest_code_a8243f6403b44a13a671955d2b908777-12" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-12"></a>looked at the deadlisting they sent and for our best knowledge it&#39;s some form of
<a id="rest_code_a8243f6403b44a13a671955d2b908777-13" name="rest_code_a8243f6403b44a13a671955d2b908777-13" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-13"></a>Elvish. We can&#39;t read it.
<a id="rest_code_a8243f6403b44a13a671955d2b908777-14" name="rest_code_a8243f6403b44a13a671955d2b908777-14" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-14"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-15" name="rest_code_a8243f6403b44a13a671955d2b908777-15" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-15"></a>Here is the firmware: goo.gl/axsAHt
<a id="rest_code_a8243f6403b44a13a671955d2b908777-16" name="rest_code_a8243f6403b44a13a671955d2b908777-16" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-16"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-17" name="rest_code_a8243f6403b44a13a671955d2b908777-17" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-17"></a>And off you go. Bring us back the password.
<a id="rest_code_a8243f6403b44a13a671955d2b908777-18" name="rest_code_a8243f6403b44a13a671955d2b908777-18" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-18"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-19" name="rest_code_a8243f6403b44a13a671955d2b908777-19" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-19"></a>Good luck!
<a id="rest_code_a8243f6403b44a13a671955d2b908777-20" name="rest_code_a8243f6403b44a13a671955d2b908777-20" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-20"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-21" name="rest_code_a8243f6403b44a13a671955d2b908777-21" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-21"></a>---------------------------------------------------------------------------------
<a id="rest_code_a8243f6403b44a13a671955d2b908777-22" name="rest_code_a8243f6403b44a13a671955d2b908777-22" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-22"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-23" name="rest_code_a8243f6403b44a13a671955d2b908777-23" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-23"></a>If you decode the answer, put it in the comments under this video! If you write
<a id="rest_code_a8243f6403b44a13a671955d2b908777-24" name="rest_code_a8243f6403b44a13a671955d2b908777-24" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-24"></a>a blogpost / post your solution online, please add a link in the comments too!
<a id="rest_code_a8243f6403b44a13a671955d2b908777-25" name="rest_code_a8243f6403b44a13a671955d2b908777-25" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-25"></a>
<a id="rest_code_a8243f6403b44a13a671955d2b908777-26" name="rest_code_a8243f6403b44a13a671955d2b908777-26" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-26"></a>P.S. I&#39;ll show/explain the solution on the stream in ~two weeks.
<a id="rest_code_a8243f6403b44a13a671955d2b908777-27" name="rest_code_a8243f6403b44a13a671955d2b908777-27" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_a8243f6403b44a13a671955d2b908777-27"></a>P.S.2. Bonus points for recreating the original high-level code.
</pre></div>
<p>Here’s the firmware:</p>
<div class="code"><pre class="code text"><a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-1" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-1"></a>co_argcount 1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-2" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-2"></a>co_consts (None, &#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;, &#39;hex&#39;, 89, 255, 115, 50)
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-3" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-3"></a>co_flags 67
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-4" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-4"></a>co_name check_password
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-5" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-5"></a>co_names (&#39;decode&#39;, &#39;len&#39;, &#39;False&#39;, &#39;all&#39;, &#39;zip&#39;, &#39;ord&#39;)
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-6" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-6"></a>co_nlocals 4
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-7" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-7"></a>co_stacksize 6
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-8" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-8"></a>co_varnames (&#39;s&#39;, &#39;good&#39;, &#39;cs&#39;, &#39;cg&#39;)
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-9" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-9"></a>              0 LOAD_CONST               1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-10" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-10"></a>              3 LOAD_ATTR                0
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-11" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-11" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-11"></a>              6 LOAD_CONST               2
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-12" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-12" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-12"></a>              9 CALL_FUNCTION            1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-13" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-13" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-13"></a>             12 STORE_FAST               1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-14" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-14" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-14"></a>             15 LOAD_GLOBAL              1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-15" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-15" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-15"></a>             18 LOAD_FAST                0
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-16" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-16" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-16"></a>             21 CALL_FUNCTION            1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-17" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-17" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-17"></a>             24 LOAD_GLOBAL              1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-18" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-18" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-18"></a>             27 LOAD_FAST                1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-19" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-19" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-19"></a>             30 CALL_FUNCTION            1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-20" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-20" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-20"></a>             33 COMPARE_OP               3 (!=)
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-21" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-21" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-21"></a>             36 POP_JUMP_IF_FALSE       43
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-22" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-22" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-22"></a>             39 LOAD_GLOBAL              2
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-23" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-23" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-23"></a>             42 RETURN_VALUE
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-24" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-24" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-24"></a>        &gt;&gt;   43 LOAD_GLOBAL              3
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-25" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-25" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-25"></a>             46 BUILD_LIST               0
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-26" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-26" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-26"></a>             49 LOAD_GLOBAL              4
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-27" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-27" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-27"></a>             52 LOAD_FAST                0
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-28" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-28" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-28"></a>             55 LOAD_FAST                1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-29" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-29" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-29"></a>             58 CALL_FUNCTION            2
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-30" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-30" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-30"></a>             61 GET_ITER
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-31" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-31" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-31"></a>        &gt;&gt;   62 FOR_ITER                52 (to 117)
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-32" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-32" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-32"></a>             65 UNPACK_SEQUENCE          2
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-33" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-33" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-33"></a>             68 STORE_FAST               2
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-34" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-34" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-34"></a>             71 STORE_FAST               3
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-35" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-35" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-35"></a>             74 LOAD_GLOBAL              5
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-36" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-36" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-36"></a>             77 LOAD_FAST                2
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-37" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-37" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-37"></a>             80 CALL_FUNCTION            1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-38" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-38" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-38"></a>             83 LOAD_CONST               3
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-39" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-39" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-39"></a>             86 BINARY_SUBTRACT
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-40" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-40" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-40"></a>             87 LOAD_CONST               4
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-41" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-41" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-41"></a>             90 BINARY_AND
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-42" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-42" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-42"></a>             91 LOAD_CONST               5
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-43" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-43" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-43"></a>             94 BINARY_XOR
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-44" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-44" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-44"></a>             95 LOAD_CONST               6
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-45" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-45" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-45"></a>             98 BINARY_XOR
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-46" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-46" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-46"></a>             99 LOAD_GLOBAL              5
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-47" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-47" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-47"></a>            102 LOAD_FAST                3
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-48" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-48" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-48"></a>            105 CALL_FUNCTION            1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-49" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-49" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-49"></a>            108 COMPARE_OP               2 (==)
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-50" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-50" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-50"></a>            111 LIST_APPEND              2
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-51" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-51" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-51"></a>            114 JUMP_ABSOLUTE           62
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-52" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-52" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-52"></a>        &gt;&gt;  117 CALL_FUNCTION            1
<a id="rest_code_ee28059f26f346f09e4a2ab239ddd619-53" name="rest_code_ee28059f26f346f09e4a2ab239ddd619-53" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ee28059f26f346f09e4a2ab239ddd619-53"></a>            120 RETURN_VALUE
</pre></div>
<p>To the uninitiated, this might look like <em>Elvish</em>. In reality, this is Python bytecode — the instruction set understood by Python’s (CPython 2.7) virtual machine. Python, like many other languages, uses a compiler to translate human-readable source code into something more appropriate for computers. Python code compiles to bytecode, which is then executed by CPython’s virtual machine. CPython bytecode can be ported between different hardware, while machine code cannot. However, machine code can often be faster than languages based on virtual machines and bytecode. (Java and C# work the same way as Python, C compiles directly to machine code)</p>
<p>This is the internal representation of a Python function. The first few lines are the member variables of the <code class="docutils literal">f.__code__</code> object of our function. We know that:</p>
<ul class="simple">
<li><p>it takes 1 argument</p></li>
<li><p>it has 7 constants: None, a long string of hex digits, the string <code class="docutils literal">'hex'</code>, and numbers: 89, 255, 115, 50.</p></li>
<li><p>its <a class="reference external" href="https://docs.python.org/2.7/library/inspect.html#code-objects-bit-flags">flags</a> are set to 67 (CO_NOFREE, CO_NEWLOCALS, CO_OPTIMIZED). This is the “standard” value that most uncomplicated functions take.</p></li>
<li><p>its name is <code class="docutils literal">check_password</code></p></li>
<li><p>it uses the following globals or attribute names: <code class="docutils literal">decode</code>, <code class="docutils literal">len</code>, <code class="docutils literal">False</code>, <code class="docutils literal">all</code>, <code class="docutils literal">zip</code>, <code class="docutils literal">ord</code></p></li>
<li><p>it has 4 local variables</p></li>
<li><p>it uses a stack of size 6</p></li>
<li><p>its variables are named <code class="docutils literal">s</code>, <code class="docutils literal">good</code>, <code class="docutils literal">cs</code>, <code class="docutils literal">cg</code></p></li>
</ul>
<p>There are two ways to solve this task: you can re-assemble the <code class="docutils literal">dis</code> output with the help of the <code class="docutils literal">opcode</code> module, or try to re-create the function by hand, using the bytecode. I chose the latter method.</p>
<section id="reverse-engineering-python-bytecode-re-creating-the-function-by-hand">
<h1>Reverse-engineering Python bytecode: re-creating the function by hand</h1>
<p>I started by recreating the original firmware file. I created an empty function and wrote some code to print out <code class="docutils literal">__code__</code> contents and <code class="docutils literal">dis.dis</code> output. I also added color-coding to help me read it:</p>
<div class="code"><pre class="code python"><a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-1" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-1"></a><span class="ch">#!/usr/bin/env python2</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-2" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-2"></a><span class="kn">import</span><span class="w"> </span><span class="nn">dis</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-3" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-3"></a><span class="kn">import</span><span class="w"> </span><span class="nn">sys</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-4" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-4"></a>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-5" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-5"></a><span class="c1"># Write code here</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-6" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-6"></a><span class="k">def</span><span class="w"> </span><span class="nf">check_password</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-7" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-7"></a>    <span class="k">pass</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-8" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-8"></a>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-9" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-9"></a><span class="c1"># Reverse engineering the code</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-10" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-10"></a><span class="n">cnames</span> <span class="o">=</span> <span class="p">(</span><span class="s1">&#39;co_argcount&#39;</span><span class="p">,</span> <span class="s1">&#39;co_consts&#39;</span><span class="p">,</span> <span class="s1">&#39;co_flags&#39;</span><span class="p">,</span> <span class="s1">&#39;co_name&#39;</span><span class="p">,</span> <span class="s1">&#39;co_names&#39;</span><span class="p">,</span> <span class="s1">&#39;co_nlocals&#39;</span><span class="p">,</span> <span class="s1">&#39;co_stacksize&#39;</span><span class="p">,</span> <span class="s1">&#39;co_varnames&#39;</span><span class="p">)</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-11" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-11" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-11"></a><span class="n">cvalues</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="s1">&#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;</span><span class="p">,</span> <span class="s1">&#39;hex&#39;</span><span class="p">,</span> <span class="mi">89</span><span class="p">,</span> <span class="mi">255</span><span class="p">,</span> <span class="mi">115</span><span class="p">,</span> <span class="mi">50</span><span class="p">),</span> <span class="mi">67</span><span class="p">,</span> <span class="s1">&#39;check_password&#39;</span><span class="p">,</span> <span class="p">(</span><span class="s1">&#39;decode&#39;</span><span class="p">,</span> <span class="s1">&#39;len&#39;</span><span class="p">,</span> <span class="s1">&#39;False&#39;</span><span class="p">,</span> <span class="s1">&#39;all&#39;</span><span class="p">,</span> <span class="s1">&#39;zip&#39;</span><span class="p">,</span> <span class="s1">&#39;ord&#39;</span><span class="p">),</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="p">(</span><span class="s1">&#39;s&#39;</span><span class="p">,</span> <span class="s1">&#39;good&#39;</span><span class="p">,</span> <span class="s1">&#39;cs&#39;</span><span class="p">,</span> <span class="s1">&#39;cg&#39;</span><span class="p">))</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-12" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-12" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-12"></a>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-13" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-13" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-13"></a><span class="k">for</span> <span class="n">n</span><span class="p">,</span> <span class="n">ov</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">cnames</span><span class="p">,</span> <span class="n">cvalues</span><span class="p">):</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-14" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-14" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-14"></a>    <span class="n">v</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">check_password</span><span class="o">.</span><span class="vm">__code__</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-15" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-15" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-15"></a>    <span class="k">if</span> <span class="n">v</span> <span class="o">==</span> <span class="n">ov</span><span class="p">:</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-16" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-16" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-16"></a>        <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\033</span><span class="s1">[1;32m&#39;</span><span class="p">)</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-17" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-17" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-17"></a>    <span class="k">else</span><span class="p">:</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-18" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-18" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-18"></a>        <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\033</span><span class="s1">[1;31m&#39;</span><span class="p">)</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-19" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-19" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-19"></a>    <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-20" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-20" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-20"></a>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-21" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-21" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-21"></a>    <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot; &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;</span><span class="se">\n</span><span class="s2">&quot;</span><span class="p">)</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-22" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-22" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-22"></a>    <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-23" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-23" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-23"></a>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-24" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-24" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-24"></a>    <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\033</span><span class="s1">[0m&#39;</span><span class="p">)</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-25" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-25" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-25"></a>    <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-26" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-26" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-26"></a>
<a id="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-27" name="rest_code_db3df9519cfe4fcebd1d69fff256cc2e-27" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_db3df9519cfe4fcebd1d69fff256cc2e-27"></a><span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="n">check_password</span><span class="p">)</span>
</pre></div>
<p>If we run this solver, we get the following output (text in brackets added by me):</p>
<div class="code"><pre class="code text"><a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-1" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-1"></a>co_argcount 1            [OK]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-2" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-2"></a>co_consts (None,)        [1/7 match]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-3" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-3"></a>co_flags 67              [OK]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-4" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-4"></a>co_name check_password   [OK]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-5" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-5"></a>co_names ()              [0/6 match]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-6" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-6"></a>co_nlocals 1             [should be 4]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-7" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-7"></a>co_stacksize 1           [should be 6]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-8" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-8"></a>co_varnames (&#39;s&#39;,)       [1/4 match]
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-9" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-9"></a>  7           0 LOAD_CONST               0 (None)
<a id="rest_code_27f66539bcdd4afa8277aa02dff98f11-10" name="rest_code_27f66539bcdd4afa8277aa02dff98f11-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_27f66539bcdd4afa8277aa02dff98f11-10"></a>              3 RETURN_VALUE
</pre></div>
<p>We can see (with the help of colors, not reproduced here), that we’ve got <code class="docutils literal">co_argcount</code>, <code class="docutils literal">co_flags</code>, <code class="docutils literal">co_name</code> correctly. We also have one constant (<code class="docutils literal">None</code>, in every function) and one variable name (<code class="docutils literal">s</code>, the argument name). We can also see <code class="docutils literal">dis.dis()</code> output. While it looks similar to the assignment, there are a few noticeable differences: there is no <code class="docutils literal">7</code> (line number) at the start, and <code class="docutils literal">LOAD_CONST</code> instructions in the original code did not have anything in parentheses (only comparisions and loops did).  This makes reading bytecode harder, but still possible. (I originally thought about using <code class="docutils literal">diff</code> for help, but it’s not hard to do it by hand. I did use <code class="docutils literal">diff</code> for the final checking after a manual “conversion”)</p>
<p>Let’s stop to look at the constants and names for a second. The long string is followed by <code class="docutils literal">hex</code>, and one of the constants is <code class="docutils literal">decode</code>. This means that we need to use <code class="docutils literal"><span class="pre">str.decode('hex')</span></code> to create a (byte)string of some information. Puzzle answers tend to be human-readable, and this string isn’t — so we need to do some more work.</p>
<p>So, let’s try reproducing the start of the original mission code using what we’ve just discussed. Python’s VM is based on a stack. In the bytecode above, you can see that instructions take 0 or 1 arguments. Some of them put things on the stack, others do actions and remove them. Most instruction names are self-explanatory, but the full list can be found in the <a class="reference external" href="https://docs.python.org/2/library/dis.html#python-bytecode-instructions">dis module documentation</a>.</p>
<p>Instructions like <code class="docutils literal">LOAD</code> and <code class="docutils literal">STORE</code> refer to indices in the constants/names/varnames tuples. To make it easier, here’s a “table” of them:</p>
<div class="code"><pre class="code text"><a id="rest_code_8c089b4cab264b4f851b62c686149db7-1" name="rest_code_8c089b4cab264b4f851b62c686149db7-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-1"></a>constants
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-2" name="rest_code_8c089b4cab264b4f851b62c686149db7-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-2"></a> 0     1                                                       2      3   4    5    6
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-3" name="rest_code_8c089b4cab264b4f851b62c686149db7-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-3"></a>(None, &#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;, &#39;hex&#39;, 89, 255, 115, 50)
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-4" name="rest_code_8c089b4cab264b4f851b62c686149db7-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-4"></a>
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-5" name="rest_code_8c089b4cab264b4f851b62c686149db7-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-5"></a>names (globals, attributes)
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-6" name="rest_code_8c089b4cab264b4f851b62c686149db7-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-6"></a> 0         1      2        3      4      5
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-7" name="rest_code_8c089b4cab264b4f851b62c686149db7-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-7"></a>(&#39;decode&#39;, &#39;len&#39;, &#39;False&#39;, &#39;all&#39;, &#39;zip&#39;, &#39;ord&#39;)
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-8" name="rest_code_8c089b4cab264b4f851b62c686149db7-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-8"></a>
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-9" name="rest_code_8c089b4cab264b4f851b62c686149db7-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-9"></a>varnames (locals, _fast)
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-10" name="rest_code_8c089b4cab264b4f851b62c686149db7-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-10"></a> 0    1       2     3
<a id="rest_code_8c089b4cab264b4f851b62c686149db7-11" name="rest_code_8c089b4cab264b4f851b62c686149db7-11" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_8c089b4cab264b4f851b62c686149db7-11"></a>(&#39;s&#39;, &#39;good&#39;, &#39;cs&#39;, &#39;cg&#39;)
</pre></div>
<p>In order to improve readability, I will use “new” <code class="docutils literal">dis</code> output with names in parentheses below:</p>
<div class="code"><pre class="code text"><a id="rest_code_fb6ce8efa1624395b2049e1148156b1c-1" name="rest_code_fb6ce8efa1624395b2049e1148156b1c-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_fb6ce8efa1624395b2049e1148156b1c-1"></a> 0 LOAD_CONST               1 (&#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;)
<a id="rest_code_fb6ce8efa1624395b2049e1148156b1c-2" name="rest_code_fb6ce8efa1624395b2049e1148156b1c-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_fb6ce8efa1624395b2049e1148156b1c-2"></a> 3 LOAD_ATTR                0 (decode)
<a id="rest_code_fb6ce8efa1624395b2049e1148156b1c-3" name="rest_code_fb6ce8efa1624395b2049e1148156b1c-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_fb6ce8efa1624395b2049e1148156b1c-3"></a> 6 LOAD_CONST               2 (&#39;hex&#39;)
<a id="rest_code_fb6ce8efa1624395b2049e1148156b1c-4" name="rest_code_fb6ce8efa1624395b2049e1148156b1c-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_fb6ce8efa1624395b2049e1148156b1c-4"></a> 9 CALL_FUNCTION            1 # function takes 1 argument from stack
<a id="rest_code_fb6ce8efa1624395b2049e1148156b1c-5" name="rest_code_fb6ce8efa1624395b2049e1148156b1c-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_fb6ce8efa1624395b2049e1148156b1c-5"></a>12 STORE_FAST               1 (good)
</pre></div>
<p>As I guessed before, the first line of our function is as follows:</p>
<div class="code"><pre class="code python"><a id="rest_code_85beff924755471e8039dad55a327578-1" name="rest_code_85beff924755471e8039dad55a327578-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_85beff924755471e8039dad55a327578-1"></a><span class="k">def</span><span class="w"> </span><span class="nf">check_password</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<a id="rest_code_85beff924755471e8039dad55a327578-2" name="rest_code_85beff924755471e8039dad55a327578-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_85beff924755471e8039dad55a327578-2"></a>    <span class="n">good</span> <span class="o">=</span> <span class="s1">&#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;hex&#39;</span><span class="p">)</span>  <span class="c1"># new</span>
</pre></div>
<p>If we run the solver again, we’ll see that the first 12 bytes of our bytecode match the mission text. We can also see that <code class="docutils literal">varnames</code> is filled in half, we’ve added two constants, and one name.  The next few lines are as follows:</p>
<div class="code"><pre class="code text"><a id="rest_code_506359919e4b49e6b2cd2712f3301286-1" name="rest_code_506359919e4b49e6b2cd2712f3301286-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-1"></a>15 LOAD_GLOBAL              1
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-2" name="rest_code_506359919e4b49e6b2cd2712f3301286-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-2"></a>18 LOAD_FAST                0
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-3" name="rest_code_506359919e4b49e6b2cd2712f3301286-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-3"></a>21 CALL_FUNCTION            1
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-4" name="rest_code_506359919e4b49e6b2cd2712f3301286-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-4"></a>24 LOAD_GLOBAL              1
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-5" name="rest_code_506359919e4b49e6b2cd2712f3301286-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-5"></a>27 LOAD_FAST                1
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-6" name="rest_code_506359919e4b49e6b2cd2712f3301286-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-6"></a>30 CALL_FUNCTION            1
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-7" name="rest_code_506359919e4b49e6b2cd2712f3301286-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-7"></a>33 COMPARE_OP               3 (!=)
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-8" name="rest_code_506359919e4b49e6b2cd2712f3301286-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-8"></a>36 POP_JUMP_IF_FALSE       43
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-9" name="rest_code_506359919e4b49e6b2cd2712f3301286-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-9"></a>39 LOAD_GLOBAL              2
<a id="rest_code_506359919e4b49e6b2cd2712f3301286-10" name="rest_code_506359919e4b49e6b2cd2712f3301286-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_506359919e4b49e6b2cd2712f3301286-10"></a>42 RETURN_VALUE
</pre></div>
<p>We can see that we’re putting a global object on stack and calling it with one argument. In both cases, the global has the index 1, that’s <code class="docutils literal">len</code>. The two arguments are <code class="docutils literal">s</code> and <code class="docutils literal">good</code>. We put both lengths on stack, then compare them. If the comparison fails (they’re equal), we jump to the instruction starting at byte 43, otherwise we continue execution to load the second global (False) and return it.  This wall of text translates to the following simple code:</p>
<div class="code"><pre class="code python"><a id="rest_code_ff50cca92832460f9863e211d84e2c63-1" name="rest_code_ff50cca92832460f9863e211d84e2c63-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ff50cca92832460f9863e211d84e2c63-1"></a><span class="k">def</span><span class="w"> </span><span class="nf">check_password</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<a id="rest_code_ff50cca92832460f9863e211d84e2c63-2" name="rest_code_ff50cca92832460f9863e211d84e2c63-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ff50cca92832460f9863e211d84e2c63-2"></a>    <span class="n">good</span> <span class="o">=</span> <span class="s1">&#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;hex&#39;</span><span class="p">)</span>
<a id="rest_code_ff50cca92832460f9863e211d84e2c63-3" name="rest_code_ff50cca92832460f9863e211d84e2c63-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ff50cca92832460f9863e211d84e2c63-3"></a>    <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="n">good</span><span class="p">):</span>  <span class="c1"># new</span>
<a id="rest_code_ff50cca92832460f9863e211d84e2c63-4" name="rest_code_ff50cca92832460f9863e211d84e2c63-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_ff50cca92832460f9863e211d84e2c63-4"></a>        <span class="k">return</span> <span class="kc">False</span>         <span class="c1"># new</span>
</pre></div>
<p>Let’s take another look at our names. We can see we’re missing <code class="docutils literal">all</code>, <code class="docutils literal">zip</code>, <code class="docutils literal">ord</code>. You can already see a common pattern here: we will iterate over both strings at once (using <code class="docutils literal">zip</code>), do some math based on the character’s codes (<code class="docutils literal">ord</code>), and then check if <code class="docutils literal">all</code> results (of a comparison, usually) are truthy.</p>
<p>Here’s the bytecode with value annotations and comments, which explain what happens where:</p>
<div class="code"><pre class="code text"><a id="rest_code_600c535191904a66bf231f1cfe9efc77-1" name="rest_code_600c535191904a66bf231f1cfe9efc77-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-1"></a>&gt;&gt;   43 LOAD_GLOBAL              3 (all)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-2" name="rest_code_600c535191904a66bf231f1cfe9efc77-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-2"></a>     46 BUILD_LIST               0
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-3" name="rest_code_600c535191904a66bf231f1cfe9efc77-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-3"></a>     49 LOAD_GLOBAL              4 (zip)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-4" name="rest_code_600c535191904a66bf231f1cfe9efc77-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-4"></a>     52 LOAD_FAST                0 (s)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-5" name="rest_code_600c535191904a66bf231f1cfe9efc77-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-5"></a>     55 LOAD_FAST                1 (good)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-6" name="rest_code_600c535191904a66bf231f1cfe9efc77-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-6"></a>     58 CALL_FUNCTION            2           # zip(s, good)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-7" name="rest_code_600c535191904a66bf231f1cfe9efc77-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-7"></a>     61 GET_ITER                             # Start iterating: iter()
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-8" name="rest_code_600c535191904a66bf231f1cfe9efc77-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-8"></a>&gt;&gt;   62 FOR_ITER                52 (to 117)  # for loop iteration start (if iterator exhausted, jump +52 bytes to position 117)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-9" name="rest_code_600c535191904a66bf231f1cfe9efc77-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-9"></a>     65 UNPACK_SEQUENCE          2           # unpack a sequence (a, b = sequence)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-10" name="rest_code_600c535191904a66bf231f1cfe9efc77-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-10"></a>     68 STORE_FAST               2 (cs)      # cs = item from s
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-11" name="rest_code_600c535191904a66bf231f1cfe9efc77-11" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-11"></a>     71 STORE_FAST               3 (cg)      # cg = item from good
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-12" name="rest_code_600c535191904a66bf231f1cfe9efc77-12" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-12"></a>     74 LOAD_GLOBAL              5 (ord)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-13" name="rest_code_600c535191904a66bf231f1cfe9efc77-13" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-13"></a>     77 LOAD_FAST                2 (cs)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-14" name="rest_code_600c535191904a66bf231f1cfe9efc77-14" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-14"></a>     80 CALL_FUNCTION            1           # put ord(cs) on stack
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-15" name="rest_code_600c535191904a66bf231f1cfe9efc77-15" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-15"></a>     83 LOAD_CONST               3 (89)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-16" name="rest_code_600c535191904a66bf231f1cfe9efc77-16" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-16"></a>     86 BINARY_SUBTRACT                      # - 89   [subtract 89 from topmost value]
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-17" name="rest_code_600c535191904a66bf231f1cfe9efc77-17" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-17"></a>     87 LOAD_CONST               4 (255)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-18" name="rest_code_600c535191904a66bf231f1cfe9efc77-18" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-18"></a>     90 BINARY_AND                           # &amp; 255  [bitwise AND with topmost value]
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-19" name="rest_code_600c535191904a66bf231f1cfe9efc77-19" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-19"></a>     91 LOAD_CONST               5 (115)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-20" name="rest_code_600c535191904a66bf231f1cfe9efc77-20" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-20"></a>     94 BINARY_XOR                           # ^ 115  [bitwise XOR with topmost value]
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-21" name="rest_code_600c535191904a66bf231f1cfe9efc77-21" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-21"></a>     95 LOAD_CONST               6 (50)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-22" name="rest_code_600c535191904a66bf231f1cfe9efc77-22" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-22"></a>     98 BINARY_XOR                           # ^ 50   [bitwise XOR with topmost value]
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-23" name="rest_code_600c535191904a66bf231f1cfe9efc77-23" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-23"></a>     99 LOAD_GLOBAL              5 (ord)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-24" name="rest_code_600c535191904a66bf231f1cfe9efc77-24" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-24"></a>    102 LOAD_FAST                3 (cg)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-25" name="rest_code_600c535191904a66bf231f1cfe9efc77-25" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-25"></a>    105 CALL_FUNCTION            1           # put ord(cs) on stack
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-26" name="rest_code_600c535191904a66bf231f1cfe9efc77-26" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-26"></a>    108 COMPARE_OP               2 (==)      # compare the two values on stack
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-27" name="rest_code_600c535191904a66bf231f1cfe9efc77-27" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-27"></a>    111 LIST_APPEND              2           # append topmost value to the list in topmost-1; pop topmost (append to list created in comprehension)
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-28" name="rest_code_600c535191904a66bf231f1cfe9efc77-28" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-28"></a>    114 JUMP_ABSOLUTE           62           # jump back to start of loop
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-29" name="rest_code_600c535191904a66bf231f1cfe9efc77-29" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-29"></a>&gt;&gt;  117 CALL_FUNCTION            1           # after loop: call all([list comprehension result])
<a id="rest_code_600c535191904a66bf231f1cfe9efc77-30" name="rest_code_600c535191904a66bf231f1cfe9efc77-30" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_600c535191904a66bf231f1cfe9efc77-30"></a>    120 RETURN_VALUE                         # return value returned by all()
</pre></div>
<p>We can now write the full answer.</p>
<p><a class="reference external" href="link://listing/listings/gynvaels-mission-11-en/mission11.py">listings/gynvaels-mission-11-en/mission11.py</a>  <a class="reference external" href="link://listing_source/listings/gynvaels-mission-11-en/mission11.py">(Source)</a></p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_937a75fde167457d96c306bb9cde4520-1"><code data-line-number="1"></code></a></td><td class="code"><code><a id="rest_code_937a75fde167457d96c306bb9cde4520-1" name="rest_code_937a75fde167457d96c306bb9cde4520-1"></a><span class="k">def</span><span class="w"> </span><span class="nf">check_password</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_937a75fde167457d96c306bb9cde4520-2"><code data-line-number="2"></code></a></td><td class="code"><code><a id="rest_code_937a75fde167457d96c306bb9cde4520-2" name="rest_code_937a75fde167457d96c306bb9cde4520-2"></a>&nbsp;&nbsp;&nbsp;&nbsp;<span class="n">good</span> <span class="o">=</span> <span class="s1">&#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;hex&#39;</span><span class="p">)</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_937a75fde167457d96c306bb9cde4520-3"><code data-line-number="3"></code></a></td><td class="code"><code><a id="rest_code_937a75fde167457d96c306bb9cde4520-3" name="rest_code_937a75fde167457d96c306bb9cde4520-3"></a>&nbsp;&nbsp;&nbsp;&nbsp;<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="n">good</span><span class="p">):</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_937a75fde167457d96c306bb9cde4520-4"><code data-line-number="4"></code></a></td><td class="code"><code><a id="rest_code_937a75fde167457d96c306bb9cde4520-4" name="rest_code_937a75fde167457d96c306bb9cde4520-4"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="k">return</span> <span class="kc">False</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_937a75fde167457d96c306bb9cde4520-5"><code data-line-number="5"></code></a></td><td class="code"><code><a id="rest_code_937a75fde167457d96c306bb9cde4520-5" name="rest_code_937a75fde167457d96c306bb9cde4520-5"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_937a75fde167457d96c306bb9cde4520-6"><code data-line-number="6"></code></a></td><td class="code"><code><a id="rest_code_937a75fde167457d96c306bb9cde4520-6" name="rest_code_937a75fde167457d96c306bb9cde4520-6"></a>&nbsp;&nbsp;&nbsp;&nbsp;<span class="k">return</span> <span class="nb">all</span><span class="p">([</span><span class="nb">ord</span><span class="p">(</span><span class="n">cs</span><span class="p">)</span> <span class="o">-</span> <span class="mi">89</span> <span class="o">&amp;</span> <span class="mi">255</span> <span class="o">^</span> <span class="mi">115</span> <span class="o">^</span> <span class="mi">50</span> <span class="o">==</span> <span class="nb">ord</span><span class="p">(</span><span class="n">cg</span><span class="p">)</span> <span class="k">for</span> <span class="n">cs</span><span class="p">,</span> <span class="n">cg</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">good</span><span class="p">)])</span>
</code></td></tr></table></div><p>In the end, our <code class="docutils literal">dis.dis()</code> output matches the mission text (except the removed values, but their IDs do match), our <code class="docutils literal">co_*</code> variables are all green, and we can get to work on solving the puzzle itself!</p>
<p><strong>Side note:</strong> this task uses a list comprehension. You might want to optimize it, remove the brackets, and end up with a generator expression. This would make the task harder, since would require working with the internal generator code object as well:</p>
<div class="code"><pre class="code text"><a id="rest_code_2486792360754ae283652134d687e43c-1" name="rest_code_2486792360754ae283652134d687e43c-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2486792360754ae283652134d687e43c-1"></a>co_consts (None, &#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;, &#39;hex&#39;, &lt;code object &lt;genexpr&gt; at 0x104a86c30, file &quot;mission11-genexpr.py&quot;, line 11&gt;)
<a id="rest_code_2486792360754ae283652134d687e43c-2" name="rest_code_2486792360754ae283652134d687e43c-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2486792360754ae283652134d687e43c-2"></a>
<a id="rest_code_2486792360754ae283652134d687e43c-3" name="rest_code_2486792360754ae283652134d687e43c-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2486792360754ae283652134d687e43c-3"></a>46 LOAD_CONST               3 (&lt;code object &lt;genexpr&gt; at 0x104a86c30, file &quot;mission11-genexpr.py&quot;, line 11&gt;)
</pre></div>
<p><code class="docutils literal">BINARY_*</code> and <code class="docutils literal">ord</code> disappeared from the new listing. You can see the <a class="reference external" href="https://chriswarrick.com/listings/gynvaels-mission-11-en/mission11-genexpr.py.html">modified code</a> (which differs by two bytes) and <a class="reference external" href="https://chriswarrick.com/listings/gynvaels-mission-11-en/mission11-genexpr.txt.html">solver output</a>.</p>
</section>
<section id="solving-the-real-puzzle">
<h1>Solving the real puzzle</h1>
<p>I solved the extra credit part of the puzzle. The <em>real</em> aim of the puzzle was to recover the password — the text for which <code class="docutils literal">check_password()</code> will return True.</p>
<p>This part is pretty boring. I built a dictionary, where I mapped every byte (0…255) to the result of the calculation done in the <code class="docutils literal">check_password()</code> function’s loop. Then I used that to recover the original text.</p>
<div class="code"><pre class="code python"><a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-1" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-1" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-1"></a><span class="n">pass_values</span> <span class="o">=</span> <span class="p">{}</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-2" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-2" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-2"></a><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">256</span><span class="p">):</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-3" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-3" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-3"></a>    <span class="n">result</span> <span class="o">=</span> <span class="n">i</span> <span class="o">-</span> <span class="mi">89</span> <span class="o">&amp;</span> <span class="mi">255</span> <span class="o">^</span> <span class="mi">115</span> <span class="o">^</span> <span class="mi">50</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-4" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-4" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-4"></a>    <span class="n">pass_values</span><span class="p">[</span><span class="n">result</span><span class="p">]</span> <span class="o">=</span> <span class="n">i</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-5" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-5" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-5"></a>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-6" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-6" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-6"></a><span class="n">good</span> <span class="o">=</span> <span class="s1">&#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;hex&#39;</span><span class="p">)</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-7" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-7" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-7"></a><span class="n">password</span> <span class="o">=</span> <span class="s1">&#39;&#39;</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-8" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-8" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-8"></a><span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">good</span><span class="p">:</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-9" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-9" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-9"></a>    <span class="n">password</span> <span class="o">+=</span> <span class="nb">chr</span><span class="p">(</span><span class="n">pass_values</span><span class="p">[</span><span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">)])</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-10" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-10" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-10"></a>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-11" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-11" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-11"></a><span class="nb">print</span><span class="p">(</span><span class="n">password</span><span class="p">)</span>
<a id="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-12" name="rest_code_0f5c1acbabcc4365a8eccce0dc13f132-12" href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_0f5c1acbabcc4365a8eccce0dc13f132-12"></a><span class="nb">print</span><span class="p">(</span><span class="n">check_password</span><span class="p">(</span><span class="n">password</span><span class="p">))</span>
</pre></div>
<p><strong>The password is:</strong> <code class="docutils literal">huh, that actually worked!</code>.</p>
</section>
<section id="what-was-that-paint-thing-about">
<h1>What was that Paint thing about?</h1>
<blockquote>Yesterday’s mission was about Elvish — <strong>I mean Paint</strong> — I mean Python programming.<footer>yours truly in this post’s teaser</footer></blockquote><p>Most of my readers were probably puzzled by the mention of Paint. Long-time viewers of Gynvael’s streams in Polish remember the Python 101 video he posted on April Fools last year. See <a class="reference external" href="https://www.youtube.com/watch?v=7VJaprmuHcw">original video</a>, <a class="reference external" href="http://gynvael.coldwind.pl/?id=599">explanation</a>, <a class="reference external" href="https://github.com/gynvael/stream/tree/master/007-python-101">code</a> (video and explanation are both Polish; you can get the gist of the video without hearing the audio commentary though.) <strong>Spoilers ahead.</strong></p>
<p>In that prank, Gynvael taught Python basics. The first part concerned itself with writing bytecode by hand. The second part (starts around 12:00) was about drawing custom Python modules. In Paint. Yes, Paint, the simple graphics program included with Microsoft Windows. He drew a custom Python module in Paint, and saved it using the BMP format. It looked like this (zoomed PNG below; <a class="reference external" href="https://chriswarrick.com/pub/gynvaels-mission-11-en/gynmod.bmp">download gynmod.bmp</a>):</p>
<img alt="/images/gynvaels-mission-11-en/gynmod-zoom.png" class="align-center" src="https://chriswarrick.com/images/gynvaels-mission-11-en/gynmod-zoom.png">
<p>How was this done? There are three things that come into play:</p>
<ul class="simple">
<li><p>Python can import modules from a ZIP file (if it’s appended to sys.path). Some tools that produce <code class="docutils literal">.exe</code> files of Python code use this technique; the old <code class="docutils literal">.egg</code> file format also used ZIPs this way.</p></li>
<li><p>BMP files have their header at the start of a file.</p></li>
<li><p>ZIP files have their header at the end of a file.</p></li>
<li><p>Thus, one file can be a valid BMP and ZIP at the same time.</p></li>
</ul>
<p>I took the code of <code class="docutils literal">check_password</code> and put it in <code class="docutils literal">mission11.py</code> (which I already cited above). Then I compiled to <code class="docutils literal">.pyc</code> and created a <code class="docutils literal">.zip</code> out of it.</p>
<p><a class="reference external" href="link://listing/listings/gynvaels-mission-11-en/mission11.py">listings/gynvaels-mission-11-en/mission11.py</a>  <a class="reference external" href="link://listing_source/listings/gynvaels-mission-11-en/mission11.py">(Source)</a></p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_bec3e104bb384b61922373724d0100a2-1"><code data-line-number="1"></code></a></td><td class="code"><code><a id="rest_code_bec3e104bb384b61922373724d0100a2-1" name="rest_code_bec3e104bb384b61922373724d0100a2-1"></a><span class="k">def</span><span class="w"> </span><span class="nf">check_password</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_bec3e104bb384b61922373724d0100a2-2"><code data-line-number="2"></code></a></td><td class="code"><code><a id="rest_code_bec3e104bb384b61922373724d0100a2-2" name="rest_code_bec3e104bb384b61922373724d0100a2-2"></a>&nbsp;&nbsp;&nbsp;&nbsp;<span class="n">good</span> <span class="o">=</span> <span class="s1">&#39;4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;hex&#39;</span><span class="p">)</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_bec3e104bb384b61922373724d0100a2-3"><code data-line-number="3"></code></a></td><td class="code"><code><a id="rest_code_bec3e104bb384b61922373724d0100a2-3" name="rest_code_bec3e104bb384b61922373724d0100a2-3"></a>&nbsp;&nbsp;&nbsp;&nbsp;<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="n">good</span><span class="p">):</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_bec3e104bb384b61922373724d0100a2-4"><code data-line-number="4"></code></a></td><td class="code"><code><a id="rest_code_bec3e104bb384b61922373724d0100a2-4" name="rest_code_bec3e104bb384b61922373724d0100a2-4"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="k">return</span> <span class="kc">False</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_bec3e104bb384b61922373724d0100a2-5"><code data-line-number="5"></code></a></td><td class="code"><code><a id="rest_code_bec3e104bb384b61922373724d0100a2-5" name="rest_code_bec3e104bb384b61922373724d0100a2-5"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_bec3e104bb384b61922373724d0100a2-6"><code data-line-number="6"></code></a></td><td class="code"><code><a id="rest_code_bec3e104bb384b61922373724d0100a2-6" name="rest_code_bec3e104bb384b61922373724d0100a2-6"></a>&nbsp;&nbsp;&nbsp;&nbsp;<span class="k">return</span> <span class="nb">all</span><span class="p">([</span><span class="nb">ord</span><span class="p">(</span><span class="n">cs</span><span class="p">)</span> <span class="o">-</span> <span class="mi">89</span> <span class="o">&amp;</span> <span class="mi">255</span> <span class="o">^</span> <span class="mi">115</span> <span class="o">^</span> <span class="mi">50</span> <span class="o">==</span> <span class="nb">ord</span><span class="p">(</span><span class="n">cg</span><span class="p">)</span> <span class="k">for</span> <span class="n">cs</span><span class="p">,</span> <span class="n">cg</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">good</span><span class="p">)])</span>
</code></td></tr></table></div><p>Since I’m not an expert in any of the formats, I booted my Windows virtual machine and blindly copied the <a class="reference external" href="http://gynvael.coldwind.pl/img/secapr16_3.png">parameters used by Gynvael</a> to open the ZIP file (renamed <code class="docutils literal">.raw</code>) in IrfanView and saved as <code class="docutils literal">.bmp</code>. I changed the size to 83×2, because my ZIP file was 498 bytes long (3 BPP * 83 px * 2 px = 498 bytes) — by doing that, and through sheer luck with the size, I could avoid adding comments and editing the ZIP archive. I ended up with this (PNG again; <a class="reference external" href="https://chriswarrick.com/pub/gynvaels-mission-11-en/mission11.bmp">download mission11.bmp</a>):</p>
<img alt="/images/gynvaels-mission-11-en/mission11-zoom.png" class="align-center" src="https://chriswarrick.com/images/gynvaels-mission-11-en/mission11-zoom.png">
<p>The <code class="docutils literal">.bmp</code> file is runnable! We can use this code:</p>
<p><a class="reference external" href="link://listing/listings/gynvaels-mission-11-en/ziprunner.py">listings/gynvaels-mission-11-en/ziprunner.py</a>  <a class="reference external" href="link://listing_source/listings/gynvaels-mission-11-en/ziprunner.py">(Source)</a></p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2c416583ffef4a72a08094db61f84464-1"><code data-line-number="1"></code></a></td><td class="code"><code><a id="rest_code_2c416583ffef4a72a08094db61f84464-1" name="rest_code_2c416583ffef4a72a08094db61f84464-1"></a><span class="ch">#!/usr/bin/env python2</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2c416583ffef4a72a08094db61f84464-2"><code data-line-number="2"></code></a></td><td class="code"><code><a id="rest_code_2c416583ffef4a72a08094db61f84464-2" name="rest_code_2c416583ffef4a72a08094db61f84464-2"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2c416583ffef4a72a08094db61f84464-3"><code data-line-number="3"></code></a></td><td class="code"><code><a id="rest_code_2c416583ffef4a72a08094db61f84464-3" name="rest_code_2c416583ffef4a72a08094db61f84464-3"></a><span class="kn">import</span><span class="w"> </span><span class="nn">sys</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2c416583ffef4a72a08094db61f84464-4"><code data-line-number="4"></code></a></td><td class="code"><code><a id="rest_code_2c416583ffef4a72a08094db61f84464-4" name="rest_code_2c416583ffef4a72a08094db61f84464-4"></a><span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s2">&quot;mission11.bmp&quot;</span><span class="p">)</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2c416583ffef4a72a08094db61f84464-5"><code data-line-number="5"></code></a></td><td class="code"><code><a id="rest_code_2c416583ffef4a72a08094db61f84464-5" name="rest_code_2c416583ffef4a72a08094db61f84464-5"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2c416583ffef4a72a08094db61f84464-6"><code data-line-number="6"></code></a></td><td class="code"><code><a id="rest_code_2c416583ffef4a72a08094db61f84464-6" name="rest_code_2c416583ffef4a72a08094db61f84464-6"></a><span class="kn">import</span><span class="w"> </span><span class="nn">mission11</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2017/08/03/gynvaels-mission-11-en-python-bytecode-reverse-engineering/#rest_code_2c416583ffef4a72a08094db61f84464-7"><code data-line-number="7"></code></a></td><td class="code"><code><a id="rest_code_2c416583ffef4a72a08094db61f84464-7" name="rest_code_2c416583ffef4a72a08094db61f84464-7"></a><span class="nb">print</span> <span class="s2">&quot;Result:&quot;</span><span class="p">,</span> <span class="n">mission11</span><span class="o">.</span><span class="n">check_password</span><span class="p">(</span><span class="s1">&#39;huh, that actually worked!&#39;</span><span class="p">)</span>
</code></td></tr></table></div><p>And we get this:</p>
<img alt="/images/gynvaels-mission-11-en/running-bmp.png" class="align-center" src="https://chriswarrick.com/images/gynvaels-mission-11-en/running-bmp.png">
</section>
<section id="resources">
<h1>Resources</h1>
<ul class="simple">
<li><p><a class="reference external" href="https://chriswarrick.com/listings/gynvaels-mission-11-en/mission11-solver.py.html">mission11-solver.py (full solver code)</a></p></li>
<li><p><a class="reference external" href="https://chriswarrick.com/listings/gynvaels-mission-11-en/mission11-genexpr.py.html">mission11-genexpr.py</a>, <a class="reference external" href="https://chriswarrick.com/listings/gynvaels-mission-11-en/mission11-genexpr.txt.html">mission11-genexpr.txt</a> (used for side note regarding generator expressions vs. list comprehensions)</p></li>
<li><p><a class="reference external" href="https://chriswarrick.com/listings/gynvaels-mission-11-en/mission11.py.html">mission11.py code, used in BMP file</a></p></li>
<li><p><a class="reference external" href="https://chriswarrick.com/listings/gynvaels-mission-11-en/ziprunner.py.html">ziprunner.py, file that runs the BMP/ZIP module</a> (adapted from Gynvael’s)</p></li>
<li><p><a class="reference external" href="https://chriswarrick.com/pub/gynvaels-mission-11-en/gynmod.bmp">gynmod.bmp</a></p></li>
<li><p><a class="reference external" href="https://chriswarrick.com/pub/gynvaels-mission-11-en/mission11.bmp">mission11.bmp</a></p></li>
<li><p><a class="reference external" href="https://docs.python.org/2/library/dis.html#python-bytecode-instructions">dis module documentation</a>.</p></li>
</ul>
<p>Thanks for the mission (and BMP idea), Gynvael!</p>
</section>
]]></content:encoded><category>Python</category><category>BMP</category><category>Gynvael Coldwind</category><category>hacking</category><category>Paint</category><category>Python</category><category>Python hackery</category><category>Python internals</category><category>reverse engineering</category><category>writeup</category></item><item><title>Setting up a Python development environment</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2017/07/03/setting-up-a-python-development-environment/</link><pubDate>Mon, 03 Jul 2017 10:40:00 GMT</pubDate><guid>https://chriswarrick.com/blog/2017/07/03/setting-up-a-python-development-environment/</guid><description>
Setting up Python is usually simple, but there are some places where newcomers
(and experienced users) need to be careful. What versions are there? What’s the
difference between Python, CPython, Anaconda, PyPy? Those and many other
questions may stump new developers, or people wanting to use Python.
</description><content:encoded><![CDATA[
<p>Setting up Python is usually simple, but there are some places where newcomers
(and experienced users) need to be careful. What versions are there? What’s the
difference between Python, CPython, Anaconda, PyPy? Those and many other
questions may stump new developers, or people wanting to use Python.</p>



<p>Note: this guide is opinionated.</p>
<section id="glossary-and-questions">
<h1>Glossary and questions</h1>
<section id="python-versions-2-vs-3">
<h2>Python versions: 2 vs 3</h2>
<p>The Python community has undergone sort of a <em>schism</em> in recent years. Python
3, released in 2008, broke backwards compatibility: deprecated some bad
constructs and libraries (eg. <code class="docutils literal">raw_input()</code> became <code class="docutils literal">input()</code> and the
original Python 2 function that ran code input by users is gone; <code class="docutils literal">print()</code>
became a function; many things that returned lists now are iterators — <code class="docutils literal">zip</code>,
<code class="docutils literal">range</code>), and completely remodelled strings (which are now Unicode by
default, and the interpreter behavior is stricter when the wrong type is used)</p>
<p>For new code, you should use Python 3. <a class="reference external" href="https://python3wos.appspot.com/">Most popular packages support Python 3</a>, and many of them support both Pythons at
the same time. The early bugs were ironed out in the first few point releases,
some features that made porting easier were added (back).</p>
<p>But what if you end up needing Python 2 later? No problem: you can learn the
differences in a short time, and with the help of a few libraries (eg. <code class="docutils literal">six</code>)
you can easily write code that is compatible with Python 2 and 3 at the same
time, using the same codebase (most libraries out there do that).</p>
<p>Python 2 will go EOL and lose official support and updates in 2020.</p>
<p>Read more: <a class="reference external" href="https://wiki.python.org/moin/Python2orPython3">Python 2 or Python 3 on Python Wiki</a></p>
</section>
<section id="can-i-run-multiple-pythons-on-the-same-machine">
<h2>Can I run multiple Pythons on the same machine?</h2>
<p>Yes. Note that multiple Python interpreters are completely separate: they have
their own pip and packages, and you can’t run Python 2 code in a Python 3
interpreter. You need to specify which interpreter to use when installing
packages and running some scripts (eg. <code class="docutils literal">pip2</code>, <code class="docutils literal">pip3</code> or <code class="docutils literal">python3 <span class="pre">-m</span> pip</code>).</p>
<p>It’s best to limit yourself to the latest Python 2 and 3 versions. Python is
backwards-compatible within the major release, so Python 2.7 runs code
written with older 2.x versions in mind.</p>
</section>
<section id="implementations">
<h2>Implementations</h2>
<p>A programming language is an abstract construct. To run code written in that
language, an interpreter or compiler needs to be written. In Python’s case,
there’s a plethora of implementations. Some of them are:</p>
<ul class="simple">
<li><p><strong>CPython</strong> is the reference implementation. This is the implementation
distributed on <a class="reference external" href="https://python.org/">https://python.org/</a> and as part of many operating systems.
Most Python features are first implemented in CPython, and then they are
ported to other implementations.  If you don’t know what to choose, use
CPython.</p></li>
<li><p><strong>PyPy</strong> is a fast implementation, written in a subset of Python. It’s compatible with
Python 2.7 and 3.5 (beta support). It can run all pure Python code, and many
extension libraries that use CFFI.</p></li>
<li><p><strong>IronPython</strong> is a .NET CLR implementation. It can integrate with .NET code.</p></li>
<li><p><strong>Jython</strong> is a Java JVM implementation. It can integrate with Java code, as
well as other JVM languages.</p></li>
</ul>
<p>Read more: <a class="reference external" href="https://wiki.python.org/moin/PythonImplementations">Python Implementations on Python Wiki</a></p>
</section>
<section id="distributions">
<h2>Distributions</h2>
<p>There are also Python (CPython) distributions. They ship the CPython
interpreter and add some extra packages/features.  They are maintained by other
communities or corporate entities.</p>
<p>The most popular third-party distribution is <a class="reference external" href="https://www.continuum.io/downloads">Anaconda</a> from Continuum Analytics. It’s popular
for data scientists, and includes over 100 packages, with extra pre-built
binaries available from the <code class="docutils literal">conda</code> package manager.</p>
<p>I personally recommend to avoid Anaconda:</p>
<ul class="simple">
<li><p>Most packages have binary wheels for Windows, macOS and Linux (yes, Linux!)
making the installation as simple as <code class="docutils literal">pip install numpy</code>.</p></li>
<li><p>You waste disk space for packages Anaconda installs that you won’t ever need.</p></li>
<li><p>It’s provided by some random for-profit company.</p></li>
<li><p>I’ve seen bugs that were not reproducible outside of Anaconda.</p></li>
<li><p>You can still do data science using the official distribution. There’s
nothing special about Anaconda.</p></li>
</ul>
<p>Read more: <a class="reference external" href="https://wiki.python.org/moin/PythonDistributions">Python distributions on Python Wiki</a></p>
</section>
<section id="can-i-make-exe-files-from-python-programs">
<h2>Can I make .exe files from Python programs?</h2>
<p>Yes, you can. There are tools for this — <a class="reference external" href="http://www.pyinstaller.org/">PyInstaller</a> is the best one. Note that you usually need to
run it on the destination operating system. And remember that “compiling” to
exe files like that <strong>is not</strong> a security measure — your source code is still
easily recoverable. (It’s not a security measure in other languages either,
even if getting source code back might be more expensive/tricky in those.)</p>
</section>
<section id="where-to-learn-python-where-to-get-help">
<h2>Where to learn Python? Where to get help?</h2>
<p>The choice of learning material is important. If you get a bad book, it might
discourage you from learning (because it’s boring), or may teach you
bad/outdated practices.</p>
<p>If you can already program in another language, I recommend the <a class="reference external" href="https://docs.python.org/3/tutorial/">official
Python tutorial</a>. For newcomers to
programming, I recommend <a class="reference external" href="http://greenteapress.com/wp/think-python-2e/">Think Python</a> or <a class="reference external" href="https://automatetheboringstuff.com/">Automate the Boring Stuff
with Python</a>.  They teach Python 3, and
(mostly) best practices.</p>
<p>If you need help, try <code class="docutils literal">#python</code> on freenode IRC, the <a class="reference external" href="https://mail.python.org/mailman/listinfo/tutor">Tutor</a> or <a class="reference external" href="https://mail.python.org/mailman/listinfo/python-list">Python-list</a> mailing lists, or a bunch of other communities. (I’m a regular on <code class="docutils literal">#python</code>)</p>
</section>
</section>
<section id="installing-python">
<h1>Installing Python</h1>
<p>This guide will focus on installing CPython 2.7 and 3.x (latest), using the standard
distribution. This choice is satisfactory for most people. Third-party
distributions, while handy in some cases, are not needed for most. (See
<a class="reference internal" href="https://chriswarrick.com/blog/2017/07/03/setting-up-a-python-development-environment/#distributions">Distributions</a> for arguments)</p>
<p>Throughout this guide, I’ll refer to the Python interpreter executable as
<code class="docutils literal">python</code>. The exact name depends on your system and desired version. On most
OSes, <code class="docutils literal">python</code> is Python 2 and <code class="docutils literal">python3</code> is 3; <code class="docutils literal">python2</code> should also
exist.  On Arch Linux, <code class="docutils literal">python</code> is Python 3. On Windows, use the <code class="docutils literal">py</code>
launcher.</p>
<section id="windows">
<h2>Windows</h2>
<p>Download the installer(s): <a class="reference external" href="https://www.python.org/downloads/">https://www.python.org/downloads/</a></p>
<p>Those installers come with <code class="docutils literal">pip</code>, and modern Python 3.x versions come with
the <code class="docutils literal">py</code> launcher.  You can use that launcher to pick a specific Python
version, eg.:</p>
<ul class="simple">
<li><p><code class="docutils literal">py <span class="pre">-3</span> <span class="pre">-m</span> pip install &lt;package&gt;</code></p></li>
<li><p><code class="docutils literal">py <span class="pre">-2</span> somefile.py</code></p></li>
<li><p><code class="docutils literal">py <span class="pre">-2.7</span></code></p></li>
<li><p><code class="docutils literal">py</code> (default system version)</p></li>
</ul>
<p>It’s recommended for most use, and mandatory for upgrading pip.</p>
<p>The 32-bit versions are more versatile. Most packages support both (the only
exception I’m aware of is Tensorflow, which only allows 64-bit Python 3.5 as of
now).</p>
</section>
<section id="macos">
<h2>macOS</h2>
<p>macOS ships with Python 2.7.10 (as of macOS Sierra). It’s not the latest
version; it’s good enough for most people, but I still recommend installing
your own (the system Python doesn’t include <code class="docutils literal">pip</code>, for example). You can
install the latest 2.7 version, as well as Python 3, using a package manager. I
recommend Homebrew — it’s the most popular solution, and lets you install many
other packages.</p>
<p><strong>DO NOT</strong> use the python.org installers: they do not have uninstallers, so you
will have outdated versions lying around after some time. There is no
auto-update as well.  <strong>DO NOT</strong> attempt to remove the system-installed Python,
this will only damage your system and you’ll need to reinstall.</p>
<p>If you already have a package manager installed (MacPorts, Fink), don’t install
a new one and just use the existing one.</p>
<ol class="arabic simple">
<li><p>Install <a class="reference external" href="https://brew.sh/">Homebrew</a>.</p></li>
<li><p>Run <code class="docutils literal">brew install python python3</code>.</p></li>
<li><p>You should now have <code class="docutils literal">python</code>, <code class="docutils literal">python3</code>, <code class="docutils literal">pip</code> and <code class="docutils literal">pip3</code>.</p></li>
</ol>
<p>To update Homebrew and Python, run <code class="docutils literal">brew update</code>.</p>
</section>
<section id="linux-and-other-unix-like-oses">
<h2>Linux (and other Unix-like OSes)</h2>
<p>On Linux, there usually are good enough packages in your OS repositories. You
should be able to install the appropriate package for Python (2 and/or 3).
Most (if not all) distributions require Python — <strong>do not</strong> remove the
pre-installed packages, and be careful not to overwrite them with something
newer.</p>
<p>If the version that ships with your distribution is too old, there are some
options. There might be some repositories with better versions, eg. the
<a class="reference external" href="https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes">deadsnakes PPA</a>
for Ubuntu. Then there’s the other option of compiling Python. There
are some tools to help with this, like <code class="docutils literal">pyenv</code> or <code class="docutils literal">pythonz</code> (they can also
manage multiple Python versions), or you can do it manually.
The instructions depend on your exact requirements, but here’s a summary:</p>
<ol class="arabic simple">
<li><p>Download the <a class="reference external" href="https://www.python.org/downloads/source/">source distribution from Python.org</a> and unpack it. Go into the unpacked source directory.</p></li>
<li><p>Ensure you’ve got a functional C compiler and Python’s dependencies. You can
usually use your system’s package manager to install the build dependencies
of your system Python. Some dependencies are optional (eg. <code class="docutils literal">sqlite3</code>
requires SQLite headers).</p></li>
<li><p>Run <code class="docutils literal">./configure <span class="pre">--prefix=/opt/python3.6</span></code> and then <code class="docutils literal">make</code>. (You may add other options to both. It will
take a while.)</p></li>
<li><p>Run <code class="docutils literal">make altinstall</code> as root. Avoid <code class="docutils literal">make install</code>, as it can override
<code class="docutils literal">python</code> executables.</p></li>
</ol>
<p>Remember: compiling Python should be considered a <strong>last resort</strong>, unless you
have very specific Python version requirements.</p>
</section>
</section>
<section id="installing-packages">
<h1>Installing packages</h1>
<p>To install third-party packages, you should use pip, the Python package
manager. If you’re using Windows or macOS (from Homebrew), pip is included with
your copy of Python.  If you’re on Linux and installed Python from a system
repository, install the correct system package (<code class="docutils literal"><span class="pre">python-pip</span></code>,
<code class="docutils literal"><span class="pre">python3-pip</span></code>). If you compiled your own Python, pip is also included.</p>
<p>To run pip, use <code class="docutils literal">py <span class="pre">-m</span> pip</code> (Windows), <code class="docutils literal">python <span class="pre">-m</span> pip</code> (other platforms),
or the short <code class="docutils literal">pip</code>/<code class="docutils literal">pip3</code> commands.</p>
<p><strong>NEVER use sudo pip.</strong> This can cause numerous problems:</p>
<ul class="simple">
<li><p>conflicts between packages installed by pip and your system package
manager</p></li>
<li><p>pip modifying system packages, leading to issues when updating them, or
breaking dependencies</p></li>
<li><p>no isolation between package versions, which is sometimes needed to satisfy
dependencies</p></li>
</ul>
<p>Note that a package install is specific to the Python interpreter used to run
<code class="docutils literal">pip</code>. Packages installed to a virtualenv are separate from system packages;
packages installed for “global” Python 2.7 are separate from 3.6 packages.
Virtual environments generally don’t use the system packages, unless
specifically enabled during creation.</p>
<p>Some distros have popular packages in their repositories. Sometimes they’re
good; in other cases they’re terribly outdated or they lack important
components, making package managers angry and sick of supporting a 2-year-old
version. (Especially since most bugs are closed with “we’ve fixed that long
ago”)</p>
<section id="user-installs">
<h2>User installs</h2>
<p>At a small scale, you can install packages with pip for a single user.  Use
<code class="docutils literal">pip install <span class="pre">--user</span> PACKAGE</code> to do this. If your package installs <a class="reference external" href="https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/">scripts</a>,
they will be installed to <code class="docutils literal"><span class="pre">~/.local/bin</span></code> on Linux, and
<code class="docutils literal">~/Library/Python/X.Y/bin</code> on macOS (X.Y is Python version), or you can use
<code class="docutils literal">python <span class="pre">-m</span></code> if the package supports it.</p>
<p>For most people and projects, virtual environments are better. There are,
however, use cases for putting some packages user-wide — if you don’t work on
projects, but instead are doing one-off research projects, those are better
suited by user-wide installs.</p>
</section>
<section id="virtual-environments">
<h2>Virtual environments</h2>
<p class="lead">I wrote a newer, more detailed post about virtualenvs: <a class="reference external" href="https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/">Python Virtual
Environments in Five Minutes</a></p>
<p>Virtual environments are the best way to install and manage Python packages.
Advantages include:</p>
<ul class="simple">
<li><p>Isolation of projects and their requirements: if one app/package requires
library version X, but another requires version Y, they can live in separate
virtual environments</p></li>
<li><p>Independent from system-wide packages</p></li>
<li><p>Lightweight (an empty virtualenv is about 10 MB)</p></li>
<li><p>Simple to re-create in any place (<code class="docutils literal">pip freeze &gt; requirements.txt</code> → <code class="docutils literal">pip install <span class="pre">-r</span> requirements.txt</code>)</p></li>
</ul>
<section id="tools-and-management">
<h3>Tools and management</h3>
<p>There are two tools to facilitate creation of virtual environments: the older
<a class="reference external" href="https://virtualenv.pypa.io/en/stable/">virtualenv</a> project, and the newer
<code class="docutils literal">venv</code> module. The <code class="docutils literal">venv</code> module is shipped with Python 3.x; some
distributions may put it in a separate package or remove it altogether. Use
whichever works for you.  Virtualenv is compatible with more Python versions
and cannot be broken by incompetent OS package maintainers (<code class="docutils literal">venv</code> requires
an extra package on Debian).</p>
<p>There are multiple schools of thought regarding virtualenv placement and
content. Myself, I use <a class="reference external" href="https://virtualenvwrapper.readthedocs.io/en/latest/">virtualenvwrapper</a> to manage virtualenvs
and put them in <code class="docutils literal">~/virtualenvs</code>. Other people put virtualenvs inside their
git repositories (but they <em>must</em> be in <code class="docutils literal">.gitignore</code>) Virtualenvs should only contain packages
installed with <code class="docutils literal">pip</code> so they can be recreated quickly.</p>
<p>I also use the <code class="docutils literal">virtualenvwrapper</code> plugin for Oh My Zsh, which also
activates virtualenvs with the same name as a git repo, or the environment
named by a <code class="docutils literal">.venv</code> file.</p>
</section>
<section id="installation-and-usage">
<h3>Installation and usage</h3>
<p>To install virtualenv user-wide, use <code class="docutils literal">pip install <span class="pre">--user</span> virtualenv</code>. You can
then use it with <code class="docutils literal">python <span class="pre">-m</span> virtualenv DIRECTORY</code>. You may pass extra
options, eg. interpreter to use (<code class="docutils literal"><span class="pre">-p</span> python3</code>). Sometimes you need to install
virtualenv for every Python version; usually, one copy is enough.</p>
<p>How to use them? This is a subject of heated debate in the Python community.</p>
<ul class="simple">
<li><p>Some people believe that activating (<code class="docutils literal">source bin/activate</code> on *nix;
<code class="docutils literal">Scripts\activate</code> on Windows) is the right thing to do and simplifies work.</p></li>
<li><p>Others think that you should use <code class="docutils literal">bin/python</code> (or other scripts in that
directory) directly, as activation only changes <code class="docutils literal">$PATH</code> and some helper
variables — those variables are not mandatory for operation, running
the correct <code class="docutils literal">python</code> is.</p></li>
<li><p>Others still think <a class="reference external" href="https://gist.github.com/datagrok/2199506">virtualenvs should be used in subshells</a>.</p></li>
</ul>
<p>In my opinion, if activating virtualenvs works in your environment, you should
do it — it’s the most convenient option. There are, however, cases when
activation fails, or is otherwise impossible — calling <code class="docutils literal">bin/python</code> directly
is your best bet in that case. If you are working inside shell scripts, do not
activate virtualenvs.  I’m not a fan of the subshell option, because it
complicates stuff if you work on multiple projects, and requires tracking usage
manually.</p>
</section>
<section id="upgrading-and-moving">
<h3>Upgrading and moving</h3>
<p>Upgrading the system Python may make your virtualenvs unusable.
For patch version upgrades, you can just update symlinks (see <a class="reference external" href="https://github.com/Kwpolska/scripts/blob/master/fix-venvs.sh">fix-venvs.sh</a>).
However, if the minor version changes, it’s best to re-create the virtualenv
(you need to create <code class="docutils literal">requirements.txt</code> ahead of time).</p>
<p>You cannot move a virtualenv between directories/machines or rename
virtualenvs. You need to use <code class="docutils literal">pip freeze &gt; requirements.txt</code>, create a new
virtualenv, and run <code class="docutils literal">pip install <span class="pre">-r</span> requirements.txt</code> (you can then delete
the old environment with a simple <code class="docutils literal">rm <span class="pre">-rf</span></code>)</p>
</section>
</section>
<section id="packages-with-c-extensions-binary">
<h2>Packages with C extensions (binary)</h2>
<p>The situation improved drastically in the past year or so. Nowadays, almost
all packages have a pre-compiled package available in PyPI. Those packages work
for Windows, macOS, and Linux. There are packages for some of the most
common <em>offenders</em>, including Pillow, lxml, PyQt5, numpy… However, there might
still be packages without wheels on PyPI.</p>
<p>If there is no wheel for a package and you are on Windows, check out <a class="reference external" href="http://www.lfd.uci.edu/~gohlke/pythonlibs/">Christoph
Gohlke’s unofficial binaries</a>.
If you can’t find any wheels online, you would have to resort to compiling it
manually — this requires installing Visual Studio (Visual C++) in a version
that matches your Python, and it’s kind of a pain to do.</p>
<p>If you are not on Windows, you must install a C compiler and toolchain.
If you get a warning about missing <code class="docutils literal">Python.h</code>, install the appropriate development
package — for example, <code class="docutils literal"><span class="pre">python-dev</span></code> or <code class="docutils literal"><span class="pre">python3-dev</span></code>) on Debian/Ubuntu,
<code class="docutils literal"><span class="pre">python-devel</span></code> or <code class="docutils literal"><span class="pre">python3-devel</span></code> on RHEL/Fedora. The package you’re trying
to install might have other dependencies that you need to install (the
<code class="docutils literal"><span class="pre">-dev(el)</span></code> part is important, too)</p>
</section>
<section id="other-stuff">
<h2>Other stuff</h2>
<p>If you’re working on a project, use <code class="docutils literal">pip install <span class="pre">-e</span> .</code> inside the project
directory to install the package in your environment in development (editable)
mode. This loads code directly from your repository — you don’t need to
re-install on every change; you might need to re-install when your version
number changes.</p>
</section>
</section>
<section id="editors-and-ides">
<h1>Editors and IDEs</h1>
<p>Another important thing a developer should take care of is the choice of an
editor. This is an important decision, and is the reason for many holy wars in
the programmer community.</p>
<p>A good editor should have syntax highlighting for all languages you need to
work with. It should also have features like visual block/multiple selections,
sophisticated find-and-replace, file finding, code completion, and many more minor
but helpful features.</p>
<p>Then there’s the difference between IDEs and text editors. Text editors are
simpler, whereas IDEs try to include many extra things not necessarily related
to writing code. IDEs often use more resources, but you won’t notice it with a
modern computer (especially with a SSD).</p>
<p>The best IDE out there is <a class="reference external" href="https://www.jetbrains.com/pycharm/">PyCharm</a> from
JetBrains. It has both a free Community and paid Professional edition. The
JetBrains folks are experts at IDEs — they have fully-fledged tools for many
languages. Their Python solution offers a plethora of options that aid
programmers in their work.  Also, if you work with Java, or otherwise more than
one IDEA-supported language, then install IntelliJ IDEA and the Python plugin
(which has the same features as PyCharm).  Students can get <a class="reference external" href="https://www.jetbrains.com/student/">free
Professional/Ultimate licenses for JetBrains products</a>.</p>
<p>I also spend a lot of time in <a class="reference external" href="http://www.vim.org/">Vim</a> (<a class="reference external" href="https://neovim.io/">neovim</a>/<a class="reference external" href="http://vimr.org/">VimR</a> to be precise). Vim is the
most powerful text editor out there, and with the right set of plugins it can
beat IDEs at speed and productivity. Vim has a steep learning curve, but it’s
worth it — you can do large changes with just a few keystrokes. Vim is
considered so good that many IDEs (Visual Studio, IntelliJ IDEA/PyCharm) have
Vim emulation plugins.</p>
<p>Another option is <a class="reference external" href="https://code.visualstudio.com/">Visual Studio Code</a> — it’s
a text editor, but can offer many IDE-like features with the right set of
plugins. It’s Electron-based architecture, or effectively being based on top of
Google’s Chromium, is unfortunate and can lead to terrible performance on
lower-end machines, and on higher-end ones in some cases. (In my experience,
it’s better than Atom.) You can also try <a class="reference external" href="https://www.sublimetext.com/">Sublime Text</a> ($80).</p>
<p>But really, almost any editor will do. But please <strong>avoid</strong> IDLE, the editor
included with Python. It lacks some of the most basic things — it doesn’t even
have an option to show line numbers. Not to mention its ugliness. Also, don’t
use Notepad and TextEdit. Those are too simple, and Notepad has encoding
issues.</p>
</section>
<section id="update-history">
<h1>Update history</h1>
<dl class="simple">
<dt>2018-09-21</dt>
<dd><p>Link to python-virtual-environments post.</p>
</dd>
<dt>2017-07-19</dt>
<dd><p>Better description of problems caused by using sudo pip.</p>
</dd>
<dt>2017-07-10</dt>
<dd><p>Added notes about not removing built-in Pythons.</p>
</dd>
<dt>2017-07-07</dt>
<dd><p>Spelling fixes and updates to the virtualenv usage section.</p>
</dd>
</dl>
</section>
]]></content:encoded><category>Python</category><category>best practices</category><category>devel</category><category>guide</category><category>guide</category><category>Python</category></item><item><title>Deploying Python Web Applications with nginx and uWSGI Emperor</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/</link><pubDate>Wed, 10 Feb 2016 14:00:00 GMT</pubDate><guid>https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/</guid><description>
You’ve just written a great Python web application. Now, you want to share it with the world. In order to do that, you need a server, and some software to do that for you.
The following is a comprehensive guide on how to accomplish that, on multiple Linux-based operating systems, using nginx and uWSGI Emperor. It doesn’t force you to use any specific web framework — Flask, Django, Pyramid, Bottle will all work. Written for Ubuntu, Debian, Fedora, CentOS 7, Alma Linux, Rocky Linux and Arch Linux (should be helpful for other systems, too). Now with an Ansible Playbook.
Revision 8 (2022-02-20): works with Fedora 35, AlmaLinux 8, RockyLinux 8
While this guide is still valid, in 2026 I switched to Docker and Gunicorn for my Python web app deployments. Check out Deploying Python Web Applications with Docker for more details.
</description><content:encoded><![CDATA[
<p>You’ve just written a great Python web application. Now, you want to share it with the world. In order to do that, you need a server, and some software to do that for you.</p>
<p>The following is a comprehensive guide on how to accomplish that, on multiple Linux-based operating systems, using nginx and uWSGI Emperor. It doesn’t force you to use any specific web framework — Flask, Django, Pyramid, Bottle will all work. Written for Ubuntu, Debian, Fedora, CentOS 7, Alma Linux, Rocky Linux and Arch Linux (should be helpful for other systems, too). Now with an Ansible Playbook.</p>
<p><em>Revision 8 (2022-02-20): works with Fedora 35, AlmaLinux 8, RockyLinux 8</em></p>
<p><em>While this guide is still valid, in 2026 I switched to Docker and Gunicorn for my Python web app deployments. Check out <a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/">Deploying Python Web Applications with Docker</a> for more details.</em></p>



<p>CI status for the associated Ansible Playbook: <img alt="ci-status" src="https://github.com/Kwpolska/ansible-nginx-uwsgi/workflows/CI in Docker for ansible-nginx-uwsgi %28pyweb%29/badge.svg"></p>
<p>For easy linking, I set up some aliases: <a class="reference external" href="https://go.chriswarrick.com/pyweb">https://go.chriswarrick.com/pyweb</a> and <a class="reference external" href="https://go.chriswarrick.com/uwsgi-tut">https://go.chriswarrick.com/uwsgi-tut</a>.</p>
<section id="prerequisites">
<h1>Prerequisites</h1>
<p>In order to deploy your web application, you need a server that gives you root and ssh access — in other words, a VPS (or a dedicated server, or a datacenter lease…). If you’re looking for a great VPS service for a low price, I recommend <a class="reference external" href="https://hetzner.cloud/?ref=Qy1lehF8PwzP">Hetzner Cloud</a> (reflink <a class="brackets" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#footnote-1" id="footnote-reference-1" role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a>), which offers a pretty good entry-level VPS for €3.49 + VAT / month (with higher plans available for equally good prices). If you want to play along at home, without buying a VPS, you can create a virtual machine on your own, or use Vagrant with a Vagrant box for Fedora 35 (<code class="docutils literal"><span class="pre">fedora/35-cloud-base</span></code>).</p>
<p>Your server should also run a modern Linux-based operating system. This guide was written and tested on:</p>
<ul class="simple">
<li><p>Ubuntu 18.04 LTS, 20.04 LTS or newer</p></li>
<li><p>Debian 10 (buster), 11 (bullseye) or newer</p></li>
<li><p>Fedora 33 or newer (with SELinux enabled and disabled)</p></li>
<li><p>CentOS 7 (with SELinux enabled and disabled) — manual guide should also work on RHEL 7.</p></li>
<li><p>AlmaLinux 8 (with SELinux enabled and disabled) — manual guide should also work on RHEL 8. Referred to as “EL8” collectively with Rocky Linux.</p></li>
<li><p>Rocky Linux 8 (with SELinux enabled and disabled) — manual guide should also work on RHEL 8. Referred to as “EL8” collectively with AlmaLinux.</p></li>
<li><p>Arch Linux</p></li>
</ul>
<p>Debian 8 (jessie) 9 (stretch), Ubuntu 16.04 LTS, and Fedora 24 through 32 are not officially supported, even though they still probably work.</p>
<p>What if you’re using <strong>Docker</strong>? This guide does not apply, you may want to read <a class="reference external" href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/">Deploying Python Web Applications with Docker</a> instead.</p>
<p>Users of other Linux distributions (and perhaps other Unix flavors) can also follow this tutorial. This guide assumes <code class="docutils literal">systemd</code> as your init system; if you are not using systemd, you will have to get your own daemon files somewhere else. In places where the instructions are split three-way, try coming up with your own, reading documentation and config files; the Arch Linux instructions are probably the closest to upstream (but not always).  Unfortunately, all Linux distributions have their own ideas when it comes to running and managing nginx and uWSGI.</p>
<p>nginx and uWSGI are considered best practices by most people. nginx is a fast, modern web server, with uWSGI support built in (without resorting to reverse proxying).  uWSGI is similarly aimed at speed.  The Emperor mode of uWSGI is recommended for init system integration by the uWSGI team, and it’s especially useful for multi-app deployments. (This guide is opinionated.)</p>
</section>
<section id="automate-everything-ansible-playbook">
<h1>Automate everything: Ansible Playbook</h1>
<p class="lead">A <a class="reference external" href="https://github.com/Kwpolska/ansible-nginx-uwsgi">Playbook</a> that automates everything in this tutorial is available. <img alt="ci-status" src="https://github.com/Kwpolska/ansible-nginx-uwsgi/workflows/CI in Docker for ansible-nginx-uwsgi %28pyweb%29/badge.svg"></p>
<section id="how-to-use">
<h2>How to use</h2>
<ol class="arabic simple">
<li><p>Install <a class="reference external" href="https://docs.ansible.com/ansible/intro_installation.html">Ansible</a> on your control computer (not necessarily the destination server).</p></li>
<li><p>Clone the <a class="reference external" href="https://github.com/Kwpolska/ansible-nginx-uwsgi">Playbook</a> from GitHub.</p></li>
<li><p>Read <code class="docutils literal">README.md</code>. You should also understand how Ansible works.</p></li>
<li><p>Configure (change three files: <code class="docutils literal">hosts</code>, <code class="docutils literal">group_vars/all</code>, and <code class="docutils literal">group_vars/os_&lt;destination OS&gt;</code></p></li>
<li><p>Make sure all the dependencies are installed on your destination server</p></li>
<li><p>Run <code class="docutils literal"><span class="pre">ansible-playbook</span> <span class="pre">-v</span> <span class="pre">nginx-uwsgi.yml</span> <span class="pre">-i</span> hosts</code> and watch magic happen.</p></li>
<li><p>Skip over to <a class="reference internal" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#end-result">End result</a> and test your site.</p></li>
</ol>
</section>
</section>
<section id="the-manual-guide">
<h1>The manual guide</h1>
<p>Even though I personally recommend the Playbook as a much less error-prone way to set up your app, it might not be compatible with everyone’s system, or otherwise be the wrong solution. The original manual configuration guide is still maintained.</p>
<p>Even if you are using the Playbook, you should still read this to find out what happens under the hood, and to find out about other caveats/required configuration changes.</p>
<aside class="admonition note">
<p class="admonition-title">Note</p>
<p>All the commands in this tutorial are meant to be run <strong>as root</strong> — run <code class="docutils literal">su</code> or <code class="docutils literal">sudo su</code> first to get an administrative shell. This tutorial assumes familiarity with basic Linux administration and command-line usage.</p>
</aside>
<section id="getting-started">
<h2>Getting started</h2>
<p>Start by installing Python 3 (with venv), nginx and uWSGI. I recommend using your operating system’s packages. Make sure you are downloading the latest versions available for your OS (update the package cache first). For uWSGI, we need the <code class="docutils literal">logfile</code> and <code class="docutils literal">python3</code> plugins. (Arch Linux names the <code class="docutils literal">python3</code> plugin <code class="docutils literal">python</code>; the <code class="docutils literal">logfile</code> plugin may be built-in — check with your system repositories!). I’ll also install Git to clone the tutorial app, but it’s optional if your workflow does not involve Git.</p>
<p><strong>Ubuntu, Debian:</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_dc96945fddd249e3b571e60cd43c2440-1" name="rest_code_dc96945fddd249e3b571e60cd43c2440-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_dc96945fddd249e3b571e60cd43c2440-1"></a>apt<span class="w"> </span>install<span class="w"> </span>python3<span class="w"> </span>python3-venv<span class="w"> </span>uwsgi<span class="w"> </span>uwsgi-emperor<span class="w"> </span>uwsgi-plugin-python3<span class="w"> </span>nginx-full<span class="w"> </span>git
</pre></div>
<p><strong>Fedora:</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_684497d7a94a42268b1cbf5bb326c87c-1" name="rest_code_684497d7a94a42268b1cbf5bb326c87c-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_684497d7a94a42268b1cbf5bb326c87c-1"></a>dnf<span class="w"> </span>install<span class="w"> </span>python3<span class="w"> </span>uwsgi<span class="w"> </span>uwsgi-plugin-python3<span class="w"> </span>uwsgi-logger-file<span class="w"> </span>nginx<span class="w"> </span>git
</pre></div>
<p><strong>CentOS 7:</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_20fd4da87897484280bca847882fd041-1" name="rest_code_20fd4da87897484280bca847882fd041-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_20fd4da87897484280bca847882fd041-1"></a>yum<span class="w"> </span>install<span class="w"> </span>epel-release
<a id="rest_code_20fd4da87897484280bca847882fd041-2" name="rest_code_20fd4da87897484280bca847882fd041-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_20fd4da87897484280bca847882fd041-2"></a>yum<span class="w"> </span>install<span class="w"> </span>python36<span class="w"> </span>uwsgi<span class="w"> </span>uwsgi-plugin-python36<span class="w"> </span>uwsgi-logger-file<span class="w"> </span>nginx<span class="w"> </span>git<span class="w"> </span>wget
</pre></div>
<p><strong>EL8 (AlmaLinux 8, Rocky Linux 8):</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_146ba186238e4081a6ecc5df6adebac5-1" name="rest_code_146ba186238e4081a6ecc5df6adebac5-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_146ba186238e4081a6ecc5df6adebac5-1"></a>dnf<span class="w"> </span>install<span class="w"> </span>epel-release
<a id="rest_code_146ba186238e4081a6ecc5df6adebac5-2" name="rest_code_146ba186238e4081a6ecc5df6adebac5-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_146ba186238e4081a6ecc5df6adebac5-2"></a>dnf<span class="w"> </span>install<span class="w"> </span>python36<span class="w"> </span>uwsgi<span class="w"> </span>uwsgi-plugin-python3<span class="w"> </span>uwsgi-logger-file<span class="w"> </span>nginx<span class="w"> </span>git<span class="w"> </span>wget
</pre></div>
<p><strong>Arch Linux:</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_d74d4895ec02488689a7fe3ff44dbb98-1" name="rest_code_d74d4895ec02488689a7fe3ff44dbb98-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_d74d4895ec02488689a7fe3ff44dbb98-1"></a>pacman<span class="w"> </span>-S<span class="w"> </span>python<span class="w"> </span>uwsgi<span class="w"> </span>uwsgi-plugin-python<span class="w"> </span>nginx<span class="w"> </span>git
</pre></div>
</section>
<section id="preparing-your-application">
<h2>Preparing your application</h2>
<p>This tutorial will work for any web framework. I will use <a class="reference external" href="https://github.com/Kwpolska/flask-demo-app">a really basic Flask app</a> that has just one route (<code class="docutils literal">/</code>), a static <code class="docutils literal">hello.png</code> file and a <code class="docutils literal">favicon.ico</code> for demonstration purposes. The app is pretty basic, but all the usual advanced features (templates, user logins, database access, etc.) would work without any other web server-related config. Note that the app does not use <code class="docutils literal">app.run()</code>. While you could add it, it would be used for local development and debugging only, and would have to be prepended by <code class="docutils literal">if __name__ == '__main__':</code> (if it wasn’t, that server would run instead of uWSGI, which is bad)</p>
<p>The app will be installed somewhere under the <code class="docutils literal">/srv</code> directory, which is a great place to store things like this. I’ll choose <code class="docutils literal">/srv/myapp</code> for this tutorial, but for real deployments, you should use something more distinguishable — the domain name is a great idea.</p>
<p>If you don’t use Flask, this tutorial also has instructions for other web frameworks (Django, Pyramid, Bottle) in the configuration files; it should be adjustable to any other WSGI-compliant framework/script nevertheless.</p>
<aside class="sidebar">
<p class="sidebar-title">Paths and locations</p>
<p>This guide used to recommend creating the venv in <code class="docutils literal">/srv/myapp</code>. This was changed to improve in-place Python upgrades. Virtual environments should be ephemeral, so that <code class="docutils literal">rm <span class="pre">-rf</span> $VIRTUAL_ENV</code> is recoverable in less than 10 minutes and 2 commands. The old structure made the venv hard to delete without deleting <code class="docutils literal">appdata</code>. The current structure has <code class="docutils literal">/srv/myapp/venv</code> and <code class="docutils literal">/srv/myapp/appdata</code> separate. An alternative structure would put the app in <code class="docutils literal">/srv/myapp</code>, but that requires including <code class="docutils literal">venv</code>, sockets and other deployment-specific files in <code class="docutils literal">.gitignore</code> (or having dirty working directories).</p>
</aside>
<p>We’ll start by creating a virtual environment, which is very easy with Python 3:</p>
<div class="code"><pre class="code sh"><a id="rest_code_1d7f4017262f40ca9004a3fe734e356d-1" name="rest_code_1d7f4017262f40ca9004a3fe734e356d-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_1d7f4017262f40ca9004a3fe734e356d-1"></a>mkdir<span class="w"> </span>/srv/myapp
<a id="rest_code_1d7f4017262f40ca9004a3fe734e356d-2" name="rest_code_1d7f4017262f40ca9004a3fe734e356d-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_1d7f4017262f40ca9004a3fe734e356d-2"></a>python3<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span>--prompt<span class="w"> </span>myapp<span class="w"> </span>/srv/myapp/venv
</pre></div>
<p>(The <code class="docutils literal"><span class="pre">--prompt</span></code> option is not supported on some old versions of Python, but you can just skip it if that’s the case, it’s just to make the prompt after <code class="docutils literal">source bin/activate</code> more informative.)</p>
<p>Now, we need to put our app there and install requirements. An example for the tutorial demo app:</p>
<div class="code"><pre class="code sh"><a id="rest_code_f4e2ef824ed045478948be971f94b599-1" name="rest_code_f4e2ef824ed045478948be971f94b599-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_f4e2ef824ed045478948be971f94b599-1"></a><span class="nb">cd</span><span class="w"> </span>/srv/myapp
<a id="rest_code_f4e2ef824ed045478948be971f94b599-2" name="rest_code_f4e2ef824ed045478948be971f94b599-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_f4e2ef824ed045478948be971f94b599-2"></a>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/Kwpolska/flask-demo-app<span class="w"> </span>appdata
<a id="rest_code_f4e2ef824ed045478948be971f94b599-3" name="rest_code_f4e2ef824ed045478948be971f94b599-3" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_f4e2ef824ed045478948be971f94b599-3"></a>venv/bin/pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>appdata/requirements.txt
</pre></div>
<p>I’m storing my application data in the <code class="docutils literal">appdata</code> subdirectory so that it doesn’t clutter the virtual environment (or vice versa).  You may also install the <code class="docutils literal">uwsgi</code> package in the virtual environment, but it’s optional.</p>
<p>What this directory should be depends on your web framework.  For example, for a Django app, you should have an <code class="docutils literal">appdata/manage.py</code> file (in other words, <code class="docutils literal">appdata</code> is where your app structure starts).  I also assumed that the <code class="docutils literal">appdata</code> folder should have a <code class="docutils literal">static</code> subdirectory with all static files, including <code class="docutils literal">favicon.ico</code> if you have one (we will add support for both in nginx).</p>
<p>At this point, you should chown this directory to the user and group your server is going to run as.  This is especially important if uwsgi and nginx run as different users (as they do on Fedora). Run one of the following commands:</p>
<p><strong>Ubuntu, Debian:</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_68d4b156f0bb491d92e8f5a85b8a211f-1" name="rest_code_68d4b156f0bb491d92e8f5a85b8a211f-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_68d4b156f0bb491d92e8f5a85b8a211f-1"></a>chown<span class="w"> </span>-R<span class="w"> </span>www-data:www-data<span class="w"> </span>/srv/myapp
</pre></div>
<p><strong>Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux):</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_e0bdfa35e6be477ea4b37661d6b9562f-1" name="rest_code_e0bdfa35e6be477ea4b37661d6b9562f-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_e0bdfa35e6be477ea4b37661d6b9562f-1"></a>chown<span class="w"> </span>-R<span class="w"> </span>uwsgi:nginx<span class="w"> </span>/srv/myapp
</pre></div>
<p><strong>Arch Linux:</strong></p>
<div class="code"><pre class="code sh"><a id="rest_code_c7fe9bf6369947f18fd80bd5a01204f2-1" name="rest_code_c7fe9bf6369947f18fd80bd5a01204f2-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_c7fe9bf6369947f18fd80bd5a01204f2-1"></a>chown<span class="w"> </span>-R<span class="w"> </span>http:http<span class="w"> </span>/srv/myapp
</pre></div>
</section>
<section id="configuring-uwsgi-and-nginx">
<h2>Configuring uWSGI and nginx</h2>
<aside class="admonition note">
<p class="admonition-title">Note</p>
<p>Parts of the configuration depend on your operating system. I tried to provide advice for Ubuntu, Debian, Fedora, CentOS, EL8, and Arch Linux. If you experience any issues, in particular with plugins, please consult the documentation.</p>
</aside>
<p>We need to write a configuration file for uWSGI and nginx.</p>
<section id="uwsgi-configuration">
<h3>uWSGI configuration</h3>
<p>Start with this, but read the notes below and change the values accordingly:</p>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-1"><code data-line-number=" 1"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-1" name="rest_code_3dc7358e7e1040868406e431b3fa0248-1"></a><span class="k">[uwsgi]</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-2"><code data-line-number=" 2"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-2" name="rest_code_3dc7358e7e1040868406e431b3fa0248-2"></a><span class="na">socket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/srv/myapp/uwsgi.sock</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-3"><code data-line-number=" 3"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-3" name="rest_code_3dc7358e7e1040868406e431b3fa0248-3"></a><span class="na">chmod-socket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">775</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-4"><code data-line-number=" 4"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-4" name="rest_code_3dc7358e7e1040868406e431b3fa0248-4"></a><span class="na">chdir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/srv/myapp/appdata</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-5"><code data-line-number=" 5"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-5" name="rest_code_3dc7358e7e1040868406e431b3fa0248-5"></a><span class="na">master</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">true</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-6"><code data-line-number=" 6"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-6" name="rest_code_3dc7358e7e1040868406e431b3fa0248-6"></a><span class="na">binary-path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/srv/myapp/venv/bin/uwsgi</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-7"><code data-line-number=" 7"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-7" name="rest_code_3dc7358e7e1040868406e431b3fa0248-7"></a><span class="na">virtualenv</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/srv/myapp/venv</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-8"><code data-line-number=" 8"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-8" name="rest_code_3dc7358e7e1040868406e431b3fa0248-8"></a><span class="na">module</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">flaskapp:app</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-9"><code data-line-number=" 9"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-9" name="rest_code_3dc7358e7e1040868406e431b3fa0248-9"></a><span class="na">uid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">www-data</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-10"><code data-line-number="10"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-10" name="rest_code_3dc7358e7e1040868406e431b3fa0248-10"></a><span class="na">gid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">www-data</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-11"><code data-line-number="11"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-11" name="rest_code_3dc7358e7e1040868406e431b3fa0248-11"></a><span class="na">processes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">1</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-12"><code data-line-number="12"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-12" name="rest_code_3dc7358e7e1040868406e431b3fa0248-12"></a><span class="na">threads</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">1</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-13"><code data-line-number="13"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-13" name="rest_code_3dc7358e7e1040868406e431b3fa0248-13"></a><span class="na">plugins</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">python3,logfile</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3dc7358e7e1040868406e431b3fa0248-14"><code data-line-number="14"></code></a></td><td class="code"><code><a id="rest_code_3dc7358e7e1040868406e431b3fa0248-14" name="rest_code_3dc7358e7e1040868406e431b3fa0248-14"></a><span class="na">logger</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">file:/srv/myapp/uwsgi.log</span>
</code></td></tr></table></div><p>Save this file as:</p>
<ul class="simple">
<li><p>Ubuntu, Debian: <code class="docutils literal"><span class="pre">/etc/uwsgi-emperor/vassals/myapp.ini</span></code></p></li>
<li><p>Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux): <code class="docutils literal">/etc/uwsgi.d/myapp.ini</code></p></li>
<li><p>Arch Linux: <code class="docutils literal">/etc/uwsgi/vassals/myapp.ini</code> (create the directory first and <strong>chown</strong> it to http: <code class="docutils literal">mkdir <span class="pre">-p</span> /etc/uwsgi/vassals; chown <span class="pre">-R</span> http:http /etc/uwsgi/vassals</code>)</p></li>
</ul>
<p>The options are:</p>
<ul>
<li><p><code class="docutils literal">socket</code> — the socket file that will be used by your application. It’s usually a file path (Unix domain socket). You could use a local TCP socket, but it’s not recommended.</p></li>
<li><p><code class="docutils literal">chdir</code> — the app directory.</p></li>
<li><p><code class="docutils literal"><span class="pre">binary-path</span></code> — the uWSGI executable to use. Remove if you didn’t install the (optional) <code class="docutils literal">uwsgi</code> package in your virtual environment.</p></li>
<li><p><code class="docutils literal">virtualenv</code> — the virtual environment for your application.</p></li>
<li><p><code class="docutils literal">module</code> — the name of the module that houses your application, and the object that speaks the WSGI interface, separated by colons. This depends on your web framework:</p>
<div class="table-responsive-lg">
<table class="table table-bordered">
<thead><tr>
<th style="width: 10%">Framework</th>
<th style="width: 30%">Flask, Bottle</th>
<th style="width: 30%">Django</th>
<th style="width: 30%">Pyramid</th>
</tr></thead>
<tbody>
<tr>
<th>Package</th>
<td>module where <code>app</code> is defined</td>
<td><code><em>project</em>.wsgi</code><br><span style="font-size: 0.9rem">(<code style="font-size: 0.9rem"><em>project</em></code> is the package with <code style="font-size: 0.9rem">settings.py</code>)</span></td>
<td>module where <code>app</code> is defined</td>
</tr>
<tr>
<th>Callable</th>
<td>Flask: <code>app</code> instance<br>Bottle: <code>app = bottle.default_app()</code></td>
<td><code>application</code></td>
<td><code>app = config.make_wsgi_app()</code></td>
</tr>
<tr class="table-active">
<th>Module</th>
<td><code style="font-size: 1.2rem"><em>package</em>:app</code></td>
<td><code style="font-size: 1.2rem"><em>project</em>.wsgi:application</code></td>
<td><code style="font-size: 1.2rem"><em>package</em>:app</code></td>
</tr>
<tr>
<th>Caveats</th>
<td>Make sure <code>app</code> is <strong>not</strong> in an <code style="font-size: 0.85rem">if __name__ == '__main__':</code> block</td>
<td>Add environment variable for settings:<br><code style="font-size: 0.7rem">env = DJANGO_SETTINGS_MODULE=<em>project</em>.settings</code></td>
<td>Make sure <code>app</code> is <strong>not</strong> in an <code style="font-size: 0.85rem">if __name__ == '__main__':</code> block (the demo quickstart does that!)</td>
</tr>
</tbody>
</table>
</div></li>
<li><p><code class="docutils literal">uid</code> and <code class="docutils literal">gid</code> — the names of the user account to use for your server.  Use the same values as in the <code class="docutils literal">chown</code> command above.</p></li>
<li><p><code class="docutils literal">processes</code> and <code class="docutils literal">threads</code> — control the resources devoted to this application. Because this is a simple hello app, I used one process with one thread, but for a real app, you will probably need more (you need to see what works the best; there is no algorithm to decide). Also, remember that if you use multiple processes, they don’t share memory (you need a database to share data between them).</p></li>
<li><p><code class="docutils literal">plugins</code> — the list of uWSGI plugins to use. For Arch Linux, use <code class="docutils literal">plugins = python</code> (the <code class="docutils literal">logfile</code> plugin is always active).  For CentOS 7 only (i.e. <strong>not</strong> for EL8), use <code class="docutils literal">plugins = python36</code>.</p></li>
<li><p><code class="docutils literal">logger</code> — the path to your app-specific logfile. (Other logging facilities are available, but this one is the easiest, especially for multiple applications on the same server)</p></li>
<li><p><code class="docutils literal">env</code> — environment variables to pass to your app. Useful for configuration, may be specified multiple times. Example for Django: <code class="docutils literal">env = DJANGO_SETTINGS_MODULE=project.settings</code></p></li>
</ul>
<p>You can test your configuration by running <code class="docutils literal">uwsgi <span class="pre">--ini</span> /path/to/myapp.ini</code> (disable the logger for stderr output or run <code class="docutils literal">tail <span class="pre">-f</span> /srv/myapp/uwsgi.log</code> in another window).</p>
<p>If you’re using <strong>Fedora</strong>, <strong>CentOS</strong>, or <strong>EL8</strong>, there are two configuration changes you need to make globally: in <code class="docutils literal">/etc/uwsgi.ini</code>, disable the <code class="docutils literal"><span class="pre">emperor-tyrant</span></code> option (which we don’t need, as it sets uid/gid for every process based on the owner of the related <code class="docutils literal">.ini</code> config file — we use one global setup) and set <code class="docutils literal">gid = nginx</code>.  We’ll need this so that nginx can talk to your socket.</p>
</section>
<section id="nginx-configuration">
<h3>nginx configuration</h3>
<p>We need to configure our web server. Here’s a basic configuration that will get us started:</p>
<p>Save this file as:</p>
<ul class="simple">
<li><p>Ubuntu, Debian: <code class="docutils literal"><span class="pre">/etc/nginx/sites-enabled/myapp.conf</span></code></p></li>
<li><p>Fedora, CentOS, EL8 (AlmaLinux, Rocky Linux): <code class="docutils literal">/etc/nginx/conf.d/myapp.conf</code></p></li>
<li><p>Arch Linux: add <code class="docutils literal">include <span class="pre">/etc/nginx/conf.d/*.conf;</span></code> to your <code class="docutils literal">http</code> directive in <code class="docutils literal">/etc/nginx/nginx.conf</code> and use <code class="docutils literal">/etc/nginx/conf.d/myapp.conf</code></p></li>
</ul>
<div class="code"><table class="codetable"><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-1"><code data-line-number=" 1"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-1" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-1"></a><span class="k">server</span><span class="w"> </span><span class="p">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-2"><code data-line-number=" 2"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-2" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-2"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1"># for a public HTTP server:</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-3"><code data-line-number=" 3"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-3" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-3"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-4"><code data-line-number=" 4"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-4" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-4"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1"># for a public HTTPS server:</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-5"><code data-line-number=" 5"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-5" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-5"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c1"># listen 443 ssl;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-6"><code data-line-number=" 6"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-6" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-6"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">server_name</span><span class="w"> </span><span class="s">localhost</span><span class="w"> </span><span class="s">myapp.local</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-7"><code data-line-number=" 7"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-7" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-7"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-8"><code data-line-number=" 8"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-8" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-8"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-9"><code data-line-number=" 9"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-9" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-9"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">include</span><span class="w"> </span><span class="s">uwsgi_params</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-10"><code data-line-number="10"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-10" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-10"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">uwsgi_pass</span><span class="w"> </span><span class="s">unix:/srv/myapp/uwsgi.sock</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-11"><code data-line-number="11"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-11" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-11"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="p">}</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-12"><code data-line-number="12"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-12" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-12"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-13"><code data-line-number="13"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-13" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-13"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">location</span><span class="w"> </span><span class="s">/static</span><span class="w"> </span><span class="p">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-14"><code data-line-number="14"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-14" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-14"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">alias</span><span class="w"> </span><span class="s">/srv/myapp/appdata/static</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-15"><code data-line-number="15"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-15" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-15"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="p">}</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-16"><code data-line-number="16"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-16" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-16"></a>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-17"><code data-line-number="17"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-17" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-17"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">location</span><span class="w"> </span><span class="s">/favicon.ico</span><span class="w"> </span><span class="p">{</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-18"><code data-line-number="18"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-18" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-18"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="kn">alias</span><span class="w"> </span><span class="s">/srv/myapp/appdata/static/favicon.ico</span><span class="p">;</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-19"><code data-line-number="19"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-19" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-19"></a><span class="w">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="p">}</span>
</code></td></tr><tr><td class="linenos linenodiv"><a href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_87fa085cf558477a847e9e0988b1b3e2-20"><code data-line-number="20"></code></a></td><td class="code"><code><a id="rest_code_87fa085cf558477a847e9e0988b1b3e2-20" name="rest_code_87fa085cf558477a847e9e0988b1b3e2-20"></a><span class="p">}</span>
</code></td></tr></table></div><p>Note that this file is a very basic and rudimentary configuration. This configuration is fine for local testing, but for a real deployment, you will need to adjust it:</p>
<ul class="simple">
<li><p>set <code class="docutils literal">listen</code> to <code class="docutils literal">443 ssl</code> and create a http→https redirect on port 80 (you can get a free SSL certificate from <a class="reference external" href="https://letsencrypt.org/">Let’s Encrypt</a>; make sure to <a class="reference external" href="https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html">configure SSL properly</a>).</p></li>
<li><p>set <code class="docutils literal">server_name</code> to your real domain name</p></li>
<li><p>you might also want to add custom error pages, log files, or change anything else that relates to your web server — consult other nginx guides for details</p></li>
<li><p>nginx usually has some server already enabled by default — edit <code class="docutils literal">/etc/nginx/nginx.conf</code> or remove their configuration files from your sites directory to disable it</p></li>
</ul>
</section>
</section>
<section id="service-setup">
<h2>Service setup</h2>
<p>After you’ve configured uWSGI and nginx, you need to enable and start the system services.</p>
<section id="for-arch-linux">
<h3>For Arch Linux</h3>
<p>All you need is:</p>
<div class="code"><pre class="code sh"><a id="rest_code_f4fee615da8f41d4811b829f880176d6-1" name="rest_code_f4fee615da8f41d4811b829f880176d6-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_f4fee615da8f41d4811b829f880176d6-1"></a>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>nginx<span class="w"> </span>emperor.uwsgi
<a id="rest_code_f4fee615da8f41d4811b829f880176d6-2" name="rest_code_f4fee615da8f41d4811b829f880176d6-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_f4fee615da8f41d4811b829f880176d6-2"></a>systemctl<span class="w"> </span>start<span class="w"> </span>nginx<span class="w"> </span>emperor.uwsgi
</pre></div>
<p>Verify the service is running with <code class="docutils literal">systemctl status emperor.uwsgi</code></p>
</section>
<section id="for-fedora-centos-el8">
<h3>For Fedora, CentOS, EL8</h3>
<p>Make sure you followed the extra note about editing <code class="docutils literal">/etc/uwsgi.ini</code> earlier and run:</p>
<div class="code"><pre class="code sh"><a id="rest_code_4a91378499014435877a7e9c80fb5673-1" name="rest_code_4a91378499014435877a7e9c80fb5673-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_4a91378499014435877a7e9c80fb5673-1"></a>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>nginx<span class="w"> </span>uwsgi
<a id="rest_code_4a91378499014435877a7e9c80fb5673-2" name="rest_code_4a91378499014435877a7e9c80fb5673-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_4a91378499014435877a7e9c80fb5673-2"></a>systemctl<span class="w"> </span>start<span class="w"> </span>nginx<span class="w"> </span>uwsgi
</pre></div>
<p>Verify the service is running with <code class="docutils literal">systemctl status uwsgi</code></p>
<p>If you disabled SELinux, this is enough to get an app working and you can skip over to the next section.</p>
<p>If you want to use SELinux, you need to do the following to allow nginx to read static files:</p>
<div class="code"><pre class="code sh"><a id="rest_code_41a1d2f1251346b2af452af27fd6ea47-1" name="rest_code_41a1d2f1251346b2af452af27fd6ea47-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_41a1d2f1251346b2af452af27fd6ea47-1"></a>setenforce<span class="w"> </span><span class="m">0</span>
<a id="rest_code_41a1d2f1251346b2af452af27fd6ea47-2" name="rest_code_41a1d2f1251346b2af452af27fd6ea47-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_41a1d2f1251346b2af452af27fd6ea47-2"></a>chcon<span class="w"> </span>-R<span class="w"> </span>system_u:system_r:httpd_t:s0<span class="w"> </span>/srv/myapp/appdata/static
<a id="rest_code_41a1d2f1251346b2af452af27fd6ea47-3" name="rest_code_41a1d2f1251346b2af452af27fd6ea47-3" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_41a1d2f1251346b2af452af27fd6ea47-3"></a>setenforce<span class="w"> </span><span class="m">1</span>
</pre></div>
<p>We now need to install a <a class="reference external" href="https://chriswarrick.com/pub/nginx-uwsgi.pp">SELinux policy</a> (that I created for this project; updated 2020-05-02) to allow nginx and uWSGI to communicate.
Download <a class="reference external" href="https://chriswarrick.com/pub/nginx-uwsgi.pp">nginx-uwsgi.pp</a> and run:</p>
<div class="code"><pre class="code sh"><a id="rest_code_3cf167a88c294332abedc17e9c7f3735-1" name="rest_code_3cf167a88c294332abedc17e9c7f3735-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_3cf167a88c294332abedc17e9c7f3735-1"></a>semodule<span class="w"> </span>-i<span class="w"> </span>nginx-uwsgi.pp
</pre></div>
<p>Hopefully, this is enough (you can delete the file). In case it isn’t, please read SELinux documentation, check audit logs, and look into <code class="docutils literal">audit2allow</code>.</p>
</section>
<section id="for-ubuntu-and-debian">
<h3>For Ubuntu and Debian</h3>
<p>Ubuntu and Debian (still!) use LSB services for uWSGI. Because LSB services are awful, we’re going to set up our own systemd-based (native) service.</p>
<p>Start by disabling the LSB service that comes with Ubuntu and Debian:</p>
<div class="code"><pre class="code sh"><a id="rest_code_e3ea69a4dabd430fb3155f4d1d8ed983-1" name="rest_code_e3ea69a4dabd430fb3155f4d1d8ed983-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_e3ea69a4dabd430fb3155f4d1d8ed983-1"></a>systemctl<span class="w"> </span>stop<span class="w"> </span>uwsgi-emperor
<a id="rest_code_e3ea69a4dabd430fb3155f4d1d8ed983-2" name="rest_code_e3ea69a4dabd430fb3155f4d1d8ed983-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_e3ea69a4dabd430fb3155f4d1d8ed983-2"></a>systemctl<span class="w"> </span>disable<span class="w"> </span>uwsgi-emperor
</pre></div>
<p>Copy the <code class="docutils literal">.service</code> file from the <a class="reference external" href="https://uwsgi-docs.readthedocs.org/en/latest/Systemd.html#adding-the-emperor-to-systemd">uWSGI systemd documentation</a> to <code class="docutils literal">/etc/systemd/system/emperor.uwsgi.service</code>.  Change the ExecStart line to:</p>
<div class="code"><pre class="code ini"><a id="rest_code_0af5cc3b6ac14570b52f79551d4b8985-1" name="rest_code_0af5cc3b6ac14570b52f79551d4b8985-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_0af5cc3b6ac14570b52f79551d4b8985-1"></a><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/bin/uwsgi --ini /etc/uwsgi-emperor/emperor.ini</span>
</pre></div>
<p>You can now reload systemd daemons and enable the services:</p>
<div class="code"><pre class="code sh"><a id="rest_code_cf61a6a5205149b7989dbb4550f60700-1" name="rest_code_cf61a6a5205149b7989dbb4550f60700-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_cf61a6a5205149b7989dbb4550f60700-1"></a>systemctl<span class="w"> </span>daemon-reload
<a id="rest_code_cf61a6a5205149b7989dbb4550f60700-2" name="rest_code_cf61a6a5205149b7989dbb4550f60700-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_cf61a6a5205149b7989dbb4550f60700-2"></a>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>nginx<span class="w"> </span>emperor.uwsgi
<a id="rest_code_cf61a6a5205149b7989dbb4550f60700-3" name="rest_code_cf61a6a5205149b7989dbb4550f60700-3" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_cf61a6a5205149b7989dbb4550f60700-3"></a>systemctl<span class="w"> </span>reload<span class="w"> </span>nginx
<a id="rest_code_cf61a6a5205149b7989dbb4550f60700-4" name="rest_code_cf61a6a5205149b7989dbb4550f60700-4" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_cf61a6a5205149b7989dbb4550f60700-4"></a>systemctl<span class="w"> </span>start<span class="w"> </span>emperor.uwsgi
</pre></div>
<p>Verify the service is running with <code class="docutils literal">systemctl status emperor.uwsgi</code>.  (Ignore
the warning about no request plugin)</p>
</section>
</section>
</section>
<section id="end-result">
<h1>End result</h1>
<p>Your web service should now be running at <a class="reference external" href="http://localhost/">http://localhost/</a> (or wherever you set up server to listen).</p>
<p>If you used the demo application, you should see something like this (complete with the favicon and image greeting):</p>
<img alt="/images/nginx-uwsgi-demo.png" class="centered" src="https://chriswarrick.com/images/nginx-uwsgi-demo.png">
<p>If you want to test with cURL:</p>
<div class="code"><pre class="code sh"><a id="rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-1" name="rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-1"></a>curl<span class="w"> </span>-v<span class="w"> </span>http://localhost/
<a id="rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-2" name="rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-2"></a>curl<span class="w"> </span>-I<span class="w"> </span>http://localhost/favicon.ico
<a id="rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-3" name="rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-3" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_a6f6d19b27a4438eaffbdd0c3b8add49-3"></a>curl<span class="w"> </span>-I<span class="w"> </span>http://localhost/static/hello.png
</pre></div>
<section id="troubleshooting">
<h2>Troubleshooting</h2>
<p>Hopefully, everything works. If it doesn’t:</p>
<ul class="simple">
<li><p>Check your nginx, system (<code class="docutils literal">journalctl</code>, <code class="docutils literal">systemctl status SERVICE</code>) and uwsgi (<code class="docutils literal">/srv/myapp/uwsgi.log</code>) logs.</p></li>
<li><p>Make sure you followed all instructions.</p></li>
<li><p>If you get a default site, disable that site in nginx config (<code class="docutils literal">/etc/nginx/nginx.conf</code> or your sites directory).</p></li>
<li><p>If you have a firewall installed, make sure to open the ports your web server runs on (typically 80/443). For <code class="docutils literal">firewalld</code> (Fedora, CentOS, EL8):</p></li>
</ul>
<div class="code"><pre class="code sh"><a id="rest_code_523bca1d77ee4cedb025cba2a35e403d-1" name="rest_code_523bca1d77ee4cedb025cba2a35e403d-1" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_523bca1d77ee4cedb025cba2a35e403d-1"></a>firewall-cmd<span class="w"> </span>--add-service<span class="w"> </span>http
<a id="rest_code_523bca1d77ee4cedb025cba2a35e403d-2" name="rest_code_523bca1d77ee4cedb025cba2a35e403d-2" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#rest_code_523bca1d77ee4cedb025cba2a35e403d-2"></a>firewall-cmd<span class="w"> </span>--add-service<span class="w"> </span>https
</pre></div>
<ul class="simple">
<li><p>If it still does not work, feel free to ask in the comments, mentioning your distribution, installation method, and what doesn’t work.</p></li>
</ul>
</section>
</section>
<section id="can-i-use-docker">
<h1>Can I use Docker?</h1>
<p>This blog post is written for systems running standalone. But Docker is a bit special, in that it offers a limited subset of OS features this workflow expects. The main issue is with user accounts, which generally work weird in Docker, and I had issues with <code class="docutils literal">setuid</code>/<code class="docutils literal">setgid</code> as used by uWSGI. Another issue is the lack of systemd, which means that another part of the tutorial fails to apply.</p>
<p>This tutorial uses uWSGI Emperor, which can run multiple sites at once, and offers other management features (such as seamless code restarts with <code class="docutils literal">touch /etc/uwsgi/vassals/myapp.ini</code>) that may not be useful or easy to use in a Docker environment. You’d probably also run uWSGI and nginx in separate containers in a typical Docker deployment.</p>
<p>Regardless, many parts of this tutorial can be used with Docker, although with the aforementioned adjustments. I have done some work on this topic. This tutorial has an Ansible Playbook attached, and the tutorial/playbook are compatible with five Linux distros in multiple versions. How do I know that there were no unexpected bugs in an older version? I could grab a Vagrant image or set up a VM. I do that when I need specific testing, but doing it for each of the distros on each update would take at least half an hour, probably even more. Yeah, that needs automating. I decided to use GitHub Actions for the CI, which can run anything, as long as you provide a Dockerfile.</p>
<p>The Docker images were designed to support running the Playbook and testing it. But the changes, setups and patches could be a good starting point if you wanted to make your own Docker containers that could run in production. You can take a look at <a class="reference external" href="https://github.com/Kwpolska/ansible-nginx-uwsgi/tree/master/ci">the Docker files for CI</a> The images support all 5 distros using their base images, but you could probably use Alpine images, or the <code class="docutils literal">python</code> docker images; be careful not to mix Python versions in the latter case.</p>
<p>In 2026, I gave up on setting up Python applications with uWSGI using the system Python and virtual environments. I’ve switched all my deployments to Docker containers with Gunicorn as the application server. Nowadays, Docker is a much better choice, as not having to deal with system Python and system virtual environments makes deployments much easier. I wrote up my experiences and shared the configuration files in <a class="reference external" href="https://chriswarrick.com/blog/2026/02/06/deploying-python-web-applications-with-docker/">Deploying Python Web Applications with Docker</a>.</p>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="footnote-1" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/#footnote-reference-1">1</a><span class="fn-bracket">]</span></span>
<p>This reflink gives you €20 in credit (expires the next month). I earn €10 after you spend €10 of your own.</p>
</aside>
</aside>
</section>
]]></content:encoded><category>Python</category><category>Ansible</category><category>Arch Linux</category><category>Django</category><category>Flask</category><category>guide</category><category>Internet</category><category>Linux</category><category>nginx</category><category>Python</category><category>systemd</category><category>uWSGI</category></item><item><title>Rewriting a Flask app in Django</title><dc:creator>Chris Warrick</dc:creator><link>https://chriswarrick.com/blog/2015/10/11/rewriting-a-flask-app-in-django/</link><pubDate>Sun, 11 Oct 2015 15:24:43 GMT</pubDate><guid>https://chriswarrick.com/blog/2015/10/11/rewriting-a-flask-app-in-django/</guid><description>
I spent Saturday on rewriting a Flask app in Django.  The app in question was
Nikola Users, which is a very simple CRUD
app.  And yet, the Flask code was a mess, full of bugs and vulnerabilities.
Eight hours later, I had a fully functional Django app that did more and fixed
all problems.
</description><content:encoded><![CDATA[
<p>I spent Saturday on rewriting a Flask app in Django.  The app in question was
<a class="reference external" href="https://users.getnikola.com/">Nikola Users</a>, which is a very simple CRUD
app.  And yet, the Flask code was a mess, full of bugs and vulnerabilities.
Eight hours later, I had a fully functional Django app that did more and fixed
all problems.</p>



<section id="original-flask-app">
<h1>Original Flask app</h1>
<p>The original Flask app had a ton of problems.  In order to make it anywhere
near useful, I would need to spend hours.  Here’s just a few of
them:</p>
<ul class="simple">
<li><p>357 lines of spaghetti code (295 SLOC), all in one file</p></li>
<li><p>No form data validation, no CSRF <a class="brackets" href="https://chriswarrick.com/blog/2015/10/11/rewriting-a-flask-app-in-django/#footnote-1" id="footnote-reference-1" role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a> protection (it did have XSS protection
though)</p></li>
<li><p>Login using Mozilla Persona, which requries JavaScript, is a bit kludgey, and
feels desolate (and also had me store the admin e-mail list in code)</p></li>
<li><p>Geopolitics issues: using country flags for languages</p></li>
<li><p>A lot of things were implemented by hand</p></li>
<li><p>SQLAlchemy is very verbose</p></li>
<li><p>no DB migrations (makes enhancements harder)</p></li>
<li><p>Languages implemented as a PostgreSQL integer array</p></li>
<li><p>Adding a language required running a command-line script and <strong>restarting the
app</strong> (languages were cached in Python dicts with no way to reload them from
the database; that would require talking through uWSGI anyway because there
were multiple processes involved)</p></li>
<li><p>The templates were slightly hacky (the page title was set in each individual
template and not in the view code); menus hacked together in HTML with no
highlighting</p></li>
<li><p>Python 2.7</p></li>
</ul>
</section>
<section id="the-rewrite">
<h1>The rewrite</h1>
<p>I started the process by opening <a class="reference external" href="https://docs.djangoproject.com/en/">Django documentation</a>, with its wonderful
<a class="reference external" href="https://docs.djangoproject.com/en/1.8/intro/tutorial01/">tutorial</a>.  Now, I have written a couple basic Django apps before, but
the majority of them didn’t do much.  In other words, I didn’t have a lot of experience.  Especially with taking user input and relationships.  It took me about 8 hours to get feature parity, and more.</p>
<p>Getting all the features was really simple.  For example, to get a many-to-many
relationship for languages, I had to write just one line.</p>
<div class="code"><pre class="code python"><a id="rest_code_ea0160f8c13f453b8c5b1d4887725fb8-1" name="rest_code_ea0160f8c13f453b8c5b1d4887725fb8-1" href="https://chriswarrick.com/blog/2015/10/11/rewriting-a-flask-app-in-django/#rest_code_ea0160f8c13f453b8c5b1d4887725fb8-1"></a><span class="n">languages</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ManyToManyField</span><span class="p">(</span><span class="n">Language</span><span class="p">)</span>
</pre></div>
<p>That’s it.  I didn’t have to run through complicated SQLAlchemy documentation,
which provides a <a class="reference external" href="http://docs.sqlalchemy.org/en/rel_1_0/orm/basic_relationships.html#many-to-many">13-line solution</a> to the same problem.</p>
<p>Django also simplified New Relic integration, as the browser JS can be implemented
using Django template tags.</p>
<p>Django is not without its problems, though.  I got a very cryptic traceback
when I did this:</p>
<div class="code"><pre class="code python"><a id="rest_code_c47a1c8a76894a0fb5173c04c7d6277b-1" name="rest_code_c47a1c8a76894a0fb5173c04c7d6277b-1" href="https://chriswarrick.com/blog/2015/10/11/rewriting-a-flask-app-in-django/#rest_code_c47a1c8a76894a0fb5173c04c7d6277b-1"></a><span class="n">publish_email</span> <span class="o">=</span> <span class="n">forms</span><span class="o">.</span><span class="n">BooleanField</span><span class="p">(</span><span class="s2">&quot;Publish e-mail&quot;</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<a id="rest_code_c47a1c8a76894a0fb5173c04c7d6277b-2" name="rest_code_c47a1c8a76894a0fb5173c04c7d6277b-2" href="https://chriswarrick.com/blog/2015/10/11/rewriting-a-flask-app-in-django/#rest_code_c47a1c8a76894a0fb5173c04c7d6277b-2"></a><span class="ne">TypeError</span><span class="p">:</span> <span class="s2">&quot;BooleanField() got multiple values for argument &#39;required&#39;&quot;</span>
</pre></div>
<p>The real problem with this code?  I forgot the <code class="docutils literal">label=</code> keyword.  The
problem is, the model API accepts this syntax — <code class="docutils literal">verbose_name</code> is the first
argument.  (I am not actually using the labels though, I write my own form
HTML)</p>
<p>Still, the Django version is much cleaner.  And the best part of all?  There
are no magic global objects (<code class="docutils literal">g</code>, <code class="docutils literal">session</code>, <code class="docutils literal">request</code>) and
decorator-based views (which are a bit of syntax abuse IMO).</p>
<p>In the end, I have:</p>
<ul class="simple">
<li><p>382 lines of code (297 SLOC) over 6 files — much cleaner, and with less long lines</p></li>
<li><p>form data validation (via Django), CSRF and XSS protection</p></li>
<li><p>Login using Django built-in authentication, without JavaScript</p></li>
<li><p>Language codes (granted, I could’ve done that really easily back in Flask)</p></li>
<li><p>Tried-and-true implementations of common patterns</p></li>
<li><p>Django models are much more readable and friendly</p></li>
<li><p>Django-provided DB migrations (generated automatically!)</p></li>
<li><p>Languages implemented using Django many-to-many relationships</p></li>
<li><p>Adding a language is possible from the Django built-in admin panel and is
reflected immediately (no caching)</p></li>
<li><p>Titles and menus in code</p></li>
<li><p>Python 3</p></li>
<li><p>New features: featured sites; show only a specified language — were really easy to add</p></li>
</ul>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="footnote-1" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="https://chriswarrick.com/blog/2015/10/11/rewriting-a-flask-app-in-django/#footnote-reference-1">1</a><span class="fn-bracket">]</span></span>
<p>I had some <code class="docutils literal">CSRF_ENABLED</code> variable, but it did not seem to be actually
used by anything.</p>
</aside>
</aside>
</section>
]]></content:encoded><category>Python</category><category>Django</category><category>Flask</category><category>Internet</category><category>Nikola</category><category>Python</category></item></channel></rss>