How will it work out in Apple land?

For developers who work only on iOS apps, the transition also won’t mean much. Maybe a faster, more accurate Simulator. They’ll need to buy an ARM Mac sooner or later (within the next 5 years), because Apple requires them to use the latest Xcode version for App Store submissions, and Xcode supports at best the previous version of macOS. But that has been Apple’s policy forever, and the Intel Macs will probably be within the usual deprecation range when that happens.

The requirements for macOS-only developers are pretty obvious, they will need to buy an ARM Mac on day one, so they can test their apps on the new platform. They will also need to work on ARM compatibility — although updating your app for the new OS is a yearly ritual in Apple land, so that’s also mostly business-as-usual (unless you do a lot of unportable low-level stuff in your code). There are some pro apps that tend to lag behind new Apple decrees (some might have been hit by Catalina), and users of those apps might prefer to stay with Intel for a little bit longer.

But then, we get to the requirements of developers who use Macs, but don’t work exclusively with the Apple platforms. This is a fairly large group, since many developers like Macs for the good hardware, Unix-based software, and the integration of both. And for some part, non-developers are affected too.

Who needs non-Apple operating systems?

The first group are people tied to Windows, somehow. Some of them might be using Boot Camp to play games. Others might be using Boot Camp or virtualization software (Parallels Desktop, VMware Fusion, Oracle VM VirtualBox) to run Windows and Windows-specific apps — perhaps they need the Windows version of Office, or various Windows-onlypro apps, or they need Windows to file their taxes, because their government does not care about non-Windows OSes. Or perhaps they’re web developers, and they need to test compatibility with the Windows versions of browsers, or the old Microsoft browsers (IE and pre-Chromium Edge).

The second group is software developers who need Linux. While macOS provides a very competent development environment, and many things can be run directly on macOS, some use-cases may require a Linux VM. Perhaps the most notable case is Docker.

Docker is a solution for lightweight app containers, that can offer separation between apps, and that can simplify and standardize deployment. Docker itself is not a virtualization solution (at least in the traditional sense). Docker must run on top of Linux (there’s also Docker-on-Windows, but that’s another story). The Docker Desktop for Mac app runs a lightweight Linux VM, and runs containers in that VM. The virtualization solution Docker for Mac uses is Hypervisor.framework, which is part of macOS itself.

Who else needs virtualization? Android developers. The Android Emulator is also a virtual machine that runs the Android operating system. Android can run on different architectures, and so, a x86 system image is typically used for the Emulator.

Is virtualization possible on ARM?

Yes, definitely. Apple has been testing it much earlier, since the aforementioned Hypervisor.framework was found on iOS in April. And Apple announced virtualization support for ARM Macs during the keynote, and showed an example of a Linux VM. That VM was, of course, running an ARM64 distribution of Linux.

But what can we use this for? Turns out, it’s complicated. The easiest thing from the few use-cases mentioned before is Android. Google just needs to get the Emulator working on ARM Macs and ship that to the devs.

What about Linux in general? Many mainstream distributions support ARM64, so that’s not a problem in general. The support for a particular distro or software might be worse than on x86_64, but it’s generally not a problem for users.

But for Docker, there’s a problem. One of the many advantages of Docker is dev-prod parity. If you deploy your app with Docker to an x86_64 Linux server, you can also install Docker on an x86_64 Linux developer machine (or a Linux VM on an Intel Mac/Windows PC). Both the server and the dev machine can run the same image, the same code, the same configuration. That won’t happen if they are a different architecture. This means that you can end up with bugs happening because of different environments, and it’s also possible that some images you depend on are not available for both architectures.

And then we get to Windows. Windows also has an ARM version, but it’s currently available only with a new ARM device (you can’t buy it standalone). If Microsoft were to sell this, we’d have an issue with the software. Windows 10 on ARM supports 32/64-bit ARM software, and can run 32-bit Intel (x86) software using emulation. It cannot, however, emulate apps that require 64-bit Intel processors (x86_64). This makes the software situation on that platform a bit better. While many developers don’t care about ARM and might not have builds for ARM available, most Windows software is available in both x86 and x86_64 versions, or is exclusively 32-bit. But certain pro apps are x86_64 only, so if there is no ARM build of it, an ARM Windows PC currently cannot run it. (Update: Microsoft announced x86_64 emulation on ARM, which means more software will work.)

And note that Microsoft knows about the transition, but we haven’t heard anything about Windows during the keynote…

Can we emulate x86(_64) and run x86(_64) Windows 10?

Theoretically? Yes. Practically? No.

The issue with emulation is speed. There are a few x86 emulators available, and those emulators can be run on an ARM device just fine. You can find videos on YouTube (not a very reliable source of information, I know) in which people try to benchmark those, or try to run Windows using an emulator like that. And even with an ancient Windows version, the emulation is painfully slow. Windows 10 would be basically unusable if you tried to emulate all of it.

How does the x86 emulation on Windows 10 for ARM work? You can watch the Channel 9 video about Windows 10 on ARM (around 6:00) for more details. The trick is that system DLLs are using a hybrid x86/ARM64 library format, which means x86 code can call those DLLs at native speeds. This means that many apps run at near-native speed (depending on the ratio of custom code to system DLL calls). This technique cannot work for emulating the entire operating system. If Windows 10 on ARM was made available for ARM Macs, running x86 Windows apps would become feasible.

Rosetta probably uses similar technique. Most apps will be translated at install time, not at run time. But you can’t do that with an entire OS.

What’s next for people who rely on both macOS and Windows?

For a few more years, Intel Macs will still be supported by Apple (with new macOS versions) and by software vendors. But after that? Well, you’re stuck with two machines, at least until Windows on ARM becomes viable and runnable on Macs. Or you can start exploring alternatives to macOS software. If you’re one of the macOS-as-UNIX-with-great-UX developers (hello!), perhaps you’ll have to switch to Linux — or perhaps Windows with Windows Subsystem for Linux? (The latter is becoming more usable with every Windows release, so keep an eye on that… I wrote this post in NeoVim in WSL2, with Windows Terminal supporting many advanced terminal features, and the transparent filesystem integration letting me access Windows files directly).

Post-M1 announcement update (2020-11-14)

Parallels have confirmed support for M1 Macs and are offering a Technical Preview of their M1 virtualization product. This announcement’s mention of Windows 10 ARM supporting x86_64 apps has caused some tech writers to assume Parallels will support Windows 10 ARM on M1 Macs. This is not what the post says. Parallels is not, and cannot announce support for that OS, because Windows 10 ARM is (still) available to ARM OEMs only to install on their devices — making an official announcement about this feature today would be admitting to doing something illegal/not allowed by the EULA. I’m pretty sure they are not working on support for Windows 10 ARM now and in the foreseeable future, until Microsoft opens up Windows 10 ARM to the public — their own legal issues aside, who would they sell the Windows support to?

In other news, Docker is not ready yet.

Install

The best tool that can be used to create virtual environments is the venv module, which is part of the standard library since Python 3.3.

venv is built into Python, and most users don’t need to install anything. However, Debian/Ubuntu users will need to run sudo apt-get install python3-venv to make it work (due to Debian not installing some components that venv needs by default). [1]

The alternative (and original, and previously standard) virtual environment tool is virtualenv. It works with Python 2.7, and has a couple extra fetures (that you generally won’t need). virtualenv can be installed with your system package manager, or pip install --user virtualenv.

Which one to use? Probably venv. Both tools achieve the same goal in similar ways. And if one of them does not work, you can try the other and it might just work better.

(Terminology note: most of the time, the names of both tools are used interchargeably, “venv” was often used as an abbreviation for “virtualenv” before the stdlib tool was created)

Create

To create a virtual environment named env, you need to run the venv tool with the Python you want to use in that environment.

Linux:   $ python3 -m venv env
Windows: > py -m venv env

or, if you’re using virtualenv:

$ python3 -m virtualenv env
> py -m virtualenv env

Afterwards, you will end up with a folder named env that contains folders named bin (Scripts on Windows — contains executables and scripts installed by packages, including python), lib (contains code), and include (contains C headers).

Both tools install pip and setuptools, but venv does not ship with wheel. In addition, the default versions tend to be more-or-less outdated. Let’s upgrade them real quick: [2]

$ env/bin/python -m pip install --upgrade pip setuptools wheel
> env\Scripts\python -m pip install --upgrade pip setuptools wheel

Use

There are three ways of working with virtual environments interactively (in a shell):

• activation (run source env/bin/activate on *nix; env\Scripts\activate on Windows) — it simplifies work and requires less typing, although it can sometimes fail to work properly. (After installing scripts, hash -r may be necessary on *nix to use them.)

• executing env/bin/python (env\Scripts\python) and other scripts directly, as activation only changes $PATH and some helper variables — those variables are not mandatory for operation, running the correct python is, and that method is failsafe.

• in subshells (IMO, it’s bad UX)

Whichever method you use, you must remember that without doing any of these things, you will still be working with the system Python.

For non-interactive work (eg. crontab entries, system services, etc.), activation and subshells are not viable solutions. In these cases, you must always use the full path to Python.

Here are some usage examples (paths can be relative, of course):

## *nix, activation ##
$ source /path/to/env/bin/activate
(env)$ pip install Django
(env)$ deactivate

## *nix, manual execution ##
$ /path/to/env/bin/pip install Django

## Windows, activation ##
> C:\path\to\env\Scripts\activate
(venv)> pip install Django
(venv)> deactivate

## Windows, manual execution ##
> C:\path\to\env\Scripts\pip install Django

## Windows, updating pip/setuptools/wheel ##
> C:\path\to\env\Scripts\python -m pip install -U pip setuptools wheel

The same principle applies to running Python itself, or any other script installed by a package. (With Django’s manage.py, calling it as ./manage.py requires activation, or you can run venv/bin/python manage.py.)

$ oldenv/bin/pip freeze > requirements.txt
$ python3 -m venv newenv
$ newenv/bin/pip install -r requirements.txt
(You may rm -rf oldenv now if you desire)

Frequently Asked Questions

Footnotes

Use the right tool for the job

By calling another process, you introduce a third-party dependency. That dependency isn’t controlled by your code, and your code becomes more fragile. The problems include:

• the program is not installed, or even available, for the user’s OS of choice

• the program is not in the $PATH your process gets

• the hard-coded path is not correct on the end user’s system

• the program is in a different version (eg. GNU vs. BSD, updates/patches), which means different option names or other behaviors

• the program’s output is not what you expected due to user config (including locale)

• error reporting is based on numeric exit codes, and the meaning of those differs between programs (if they have meaning besides 0/1 in the first place)

On the other hand, if your code uses a lot of subprocesses, perhaps you should stay with Bash. You can do the harder parts with Python, Ruby, or some other language by calling them from within your Bash script.

Don’t spawn subprocesses if there’s an alternative

Spawning a subprocess always incurs a (minor) [1] performance hit minor compared to the alternatives. With that in mind, and the resiliency issues listed above, you should always try to find an alternative for the external command.

The simplest ones are the basic Unix utilities. Replace grep, sed and awk with string operations and regular expressions. Filesystem utilities will have equivalents — for Python, in os or shutil. Your language of choice can also handle things like networking (don’t call curl), file compression, working with date/time…

Similarly, you should check if there are packages available that already do what you want — library bindings or re-implementations. And if there isn’t, perhaps you could help the world by writing one of those and sharing it?

One more important thing: if the program uses the same language as your code, then you should try to import the code and run it from the same process instead of spawning a process, if this is feasible.

Security considerations: shells, spaces, and command injection

We come to the most important part of this article: how to spawn subprocesses without compromising your system. When you spawn a subprocess on a typical Unix system, fork() is called, and your process is copied. Many modern Unix systems have a copy-on-write implementation of that syscall, meaning that the operation does not result in copying all the memory of the host process over. Forking is (almost) immediately followed by calling execve() (or a helper function from the exec family) [2] in the child process — that function transforms the calling process into a new process [3]. This technique is called fork-exec and is the typical way to spawn a new process on Unix. [4]

There are two ways to access this API, from the C perspective:

• directly, by calling fork() and exec*() (or posix_spawn()), and providing an array of arguments passed to the process, or

• through the shell (sh), usually by calling system(). As Linux’s manpage for system(3) puts it,

  The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:

  execl("/bin/sh", "sh", "-c", command, (char *) 0);
  

If you go through the shell, you pass one string argument, whereas exec*() demands you to specify arguments separately. Let’s write a sample program to print all the arguments it receives. I’ll do it in Python to get a more readable output.

#!/usr/bin/env python3
import sys
print(sys.argv)

Let’s see what appears:

$ ./argv.py foo bar
['./argv.py', 'foo', 'bar']
$ ./argv.py 'foo bar'
['./argv.py', 'foo bar']
$ ./argv.py foo\ bar baz
['./argv.py', 'foo bar', 'baz']

$ ./argv.py $(date)
['./argv.py', 'Sat', 'Sep', '2', '16:54:52', 'CEST', '2017']
$ ./argv.py "$(date)"
['./argv.py', 'Sat Sep  2 16:54:52 CEST 2017']

$ ./argv.py /usr/*
['./argv.py', '/usr/X11', '/usr/X11R6', '/usr/bin', '/usr/include', '/usr/lib', '/usr/libexec', '/usr/local', '/usr/sbin', '/usr/share', '/usr/standalone']
$ ./argv.py "/usr/*"
['./argv.py', '/usr/*']

$ ./argv.py $EDITOR
['./argv.py', 'nvim']

$ $PWD/argv.py foo bar
['/Users/kwpolska/Desktop/blog/subprocess/argv.py', 'foo', 'bar']
$ ./argv.py a{b,c}d
['./argv.py', 'abd', 'acd']

$ python argv.py foo bar | cat
['argv.py', 'foo', 'bar']
$ python argv.py foo bar > foo.txt
$ cat foo.txt
['argv.py', 'foo', 'bar']

$ ./argv.py foo; ls /usr
['./argv.py', 'foo']
X11@        X11R6@      bin/        include/    lib/        libexec/    local/      sbin/       share/      standalone/

As you can see, the following things are handled by the shell (the process is unaware of this occurring):

• quotes and escapes

• expanding expressions in braces

• expanding variables

• wildcards (glob, *)

• redirections and pipes (> >> |)

• command substitution (backticks or $(…))

• running multiple commands on the same line (; && || &)

The list is full of potential vulnerabilities. If end users are in control of the arguments passed, and you go through the shell, they can execute arbitrary commands or even get full shell access. Even in other cases, you’ll have to depend on the shell’s parsing, which introduces an unnecessary indirection.

TL;DR: How to do this properly in your language of choice

To ensure spawning subprocess is done securely, do not use the shell in between. If you need any of the operations I listed above as part of your command — wildcards, pipes, etc. — you will need to take care of them in your code; most languages have those features built-in.

In C (Unix)

Perform fork-exec by yourself, or use posix_spawn(). This also lets you communicate with the process if you open a pipe and make it stdout of the child process. Never use system().

In Python

Use the subprocess module. Always pass shell=False and give it a list of arguments. With asyncio, use asyncio.create_subprocess_exec (and not _shell), but note it takes *args and not a list. Never use os.system and os.popen.

In Ruby

Pass arrays to IO.popen. Pass multiple arguments to system() (system(["ls", "ls"]) or system("ls", "-l")). Never use %x{command} or backticks.

In Java

Pass arrays to Runtime.exec. Pass multiple arguments or list to ProcessBuilder.

In PHP

All the standard methods go through the shell. Try escapeshellcmd(), escapeshellarg() — or better, switch to Python. Or anything, really.

In Go

os/exec and os.StartProcess are safe.

In Node.js

Use child_process.execFile or child_process.spawn with shell set to false.

Elsewhere

You should be able to specify multiple strings (using variadic arguments, arrays, or otherwise standard data structures of your language of choice) as the command line. Otherwise, you might be running into something shell-related.

The part where I pretend I know something about Windows

On Windows, argument lists are always passed to processes as strings (Python joins them semi-intelligently if it gets a list). Redirections and variables work in shell mode, but globs (asterisks) are always left for the called process to handle.

Some useful functions are implemented as shell built-ins — in that case, you need to call it via the shell.

Internals: There is no fork() on Windows. Instead, CreateProcess(), ShellExecute(), or lower-level spawn*() functions are used. cmd.exe /c is called in shell calls.

Glossary and questions

Installing Python

This guide will focus on installing CPython 2.7 and 3.x (latest), using the standard distribution. This choice is satisfactory for most people. Third-party distributions, while handy in some cases, are not needed for most. (See Distributions for arguments)

Throughout this guide, I’ll refer to the Python interpreter executable as python. The exact name depends on your system and desired version. On most OSes, python is Python 2 and python3 is 3; python2 should also exist. On Arch Linux, python is Python 3. On Windows, use the py launcher.

Installing packages

To install third-party packages, you should use pip, the Python package manager. If you’re using Windows or macOS (from Homebrew), pip is included with your copy of Python. If you’re on Linux and installed Python from a system repository, install the correct system package (python-pip, python3-pip). If you compiled your own Python, pip is also included.

To run pip, use py -m pip (Windows), python -m pip (other platforms), or the short pip/pip3 commands.

NEVER use sudo pip. This can cause numerous problems:

• conflicts between packages installed by pip and your system package manager

• pip modifying system packages, leading to issues when updating them, or breaking dependencies

• no isolation between package versions, which is sometimes needed to satisfy dependencies

Note that a package install is specific to the Python interpreter used to run pip. Packages installed to a virtualenv are separate from system packages; packages installed for “global” Python 2.7 are separate from 3.6 packages. Virtual environments generally don’t use the system packages, unless specifically enabled during creation.

Some distros have popular packages in their repositories. Sometimes they’re good; in other cases they’re terribly outdated or they lack important components, making package managers angry and sick of supporting a 2-year-old version. (Especially since most bugs are closed with “we’ve fixed that long ago”)

Editors and IDEs

Another important thing a developer should take care of is the choice of an editor. This is an important decision, and is the reason for many holy wars in the programmer community.

A good editor should have syntax highlighting for all languages you need to work with. It should also have features like visual block/multiple selections, sophisticated find-and-replace, file finding, code completion, and many more minor but helpful features.

Then there’s the difference between IDEs and text editors. Text editors are simpler, whereas IDEs try to include many extra things not necessarily related to writing code. IDEs often use more resources, but you won’t notice it with a modern computer (especially with a SSD).

The best IDE out there is PyCharm from JetBrains. It has both a free Community and paid Professional edition. The JetBrains folks are experts at IDEs — they have fully-fledged tools for many languages. Their Python solution offers a plethora of options that aid programmers in their work. Also, if you work with Java, or otherwise more than one IDEA-supported language, then install IntelliJ IDEA and the Python plugin (which has the same features as PyCharm). Students can get free Professional/Ultimate licenses for JetBrains products.

I also spend a lot of time in Vim (neovim/VimR to be precise). Vim is the most powerful text editor out there, and with the right set of plugins it can beat IDEs at speed and productivity. Vim has a steep learning curve, but it’s worth it — you can do large changes with just a few keystrokes. Vim is considered so good that many IDEs (Visual Studio, IntelliJ IDEA/PyCharm) have Vim emulation plugins.

Another option is Visual Studio Code — it’s a text editor, but can offer many IDE-like features with the right set of plugins. It’s Electron-based architecture, or effectively being based on top of Google’s Chromium, is unfortunate and can lead to terrible performance on lower-end machines, and on higher-end ones in some cases. (In my experience, it’s better than Atom.) You can also try Sublime Text ($80).

But really, almost any editor will do. But please avoid IDLE, the editor included with Python. It lacks some of the most basic things — it doesn’t even have an option to show line numbers. Not to mention its ugliness. Also, don’t use Notepad and TextEdit. Those are too simple, and Notepad has encoding issues.

Update history

2018-09-21

Link to python-virtual-environments post.

2017-07-19

Better description of problems caused by using sudo pip.

2017-07-10

Added notes about not removing built-in Pythons.

2017-07-07

Spelling fixes and updates to the virtualenv usage section.

Requirements and Desired Results

You will need:

• a Python project

• a setup.py file using setuptools

• the following directory structure:

  • entry_points_project/
  • • my_project/
    • • __init__.py
      • __main__.py
    • setup.py

(entry_points_project is also where the README and other auxiliary files go, while my_project contains all the Python code.)

When you’re done, you will have a project that can be executed by:

• python -m my_project

• my_project

Provided that you have your Python directory and its Scripts\ subdirectory on the %PATH%, this will also work in Windows.

Step 1: create a __main__.py file

In order to implement the first desired result, you need to create a __main__.py file in your package. This file needs to contain a main() function that takes no arguments, and also a special passage to determine code to run:

entry_points_project/my_project/__main__.py (Source)

1. The if __name__ == "__main__": idiom, as documented here, is used to check whether this is executed as the top-level file, or if it has been imported by someone else (in this case, executing the main() function is not always intended).

2. The main() function must not take any arguments, because that’s how entry_points executes things.

Step 2: adjust setup.py accordingly

This is the real deal: create the entry points in your setup.py file.

entry_points_project/setup.py (Source)

1. You must use setuptools, otherwise this won’t work.

2. The most important piece of code is the entry_points declaration (unsurprisingly).

3. The declaration reads

"name_of_executable = module.with:function_to_execute"

4. If you are developing a GUI application (in Tkinter, PyQt/PySide, wxPython, PyGTK, PyGame…), you should change the declaration to gui_scripts. On *nix, this makes no difference, but on Windows, it means that running your script by opening the created .exe files does not show a console window. Note that stdout/stderr do not work in that mode under Windows, which can lead to spurious application crashes. (GUI-only processes cannot use stdout/stderr because they don’t have a console attached)

5. You can create multiple scripts this way. You can also have multiple console_scripts and gui_scripts in one setup file.

All code samples are freely reusable, but if you mention where you got them from, it’d be really nice.