Use the right tool for the job

By calling another process, you introduce a third-party dependency. That dependency isn’t controlled by your code, and your code becomes more fragile. The problems include:

• the program is not installed, or even available, for the user’s OS of choice

• the program is not in the $PATH your process gets

• the hard-coded path is not correct on the end user’s system

• the program is in a different version (eg. GNU vs. BSD, updates/patches), which means different option names or other behaviors

• the program’s output is not what you expected due to user config (including locale)

• error reporting is based on numeric exit codes, and the meaning of those differs between programs (if they have meaning besides 0/1 in the first place)

On the other hand, if your code uses a lot of subprocesses, perhaps you should stay with Bash. You can do the harder parts with Python, Ruby, or some other language by calling them from within your Bash script.

Don’t spawn subprocesses if there’s an alternative

Spawning a subprocess always incurs a (minor) [1] performance hit minor compared to the alternatives. With that in mind, and the resiliency issues listed above, you should always try to find an alternative for the external command.

The simplest ones are the basic Unix utilities. Replace grep, sed and awk with string operations and regular expressions. Filesystem utilities will have equivalents — for Python, in os or shutil. Your language of choice can also handle things like networking (don’t call curl), file compression, working with date/time…

Similarly, you should check if there are packages available that already do what you want — library bindings or re-implementations. And if there isn’t, perhaps you could help the world by writing one of those and sharing it?

One more important thing: if the program uses the same language as your code, then you should try to import the code and run it from the same process instead of spawning a process, if this is feasible.

Security considerations: shells, spaces, and command injection

We come to the most important part of this article: how to spawn subprocesses without compromising your system. When you spawn a subprocess on a typical Unix system, fork() is called, and your process is copied. Many modern Unix systems have a copy-on-write implementation of that syscall, meaning that the operation does not result in copying all the memory of the host process over. Forking is (almost) immediately followed by calling execve() (or a helper function from the exec family) [2] in the child process — that function transforms the calling process into a new process [3]. This technique is called fork-exec and is the typical way to spawn a new process on Unix. [4]

There are two ways to access this API, from the C perspective:

• directly, by calling fork() and exec*() (or posix_spawn()), and providing an array of arguments passed to the process, or

• through the shell (sh), usually by calling system(). As Linux’s manpage for system(3) puts it,

  The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:

  execl("/bin/sh", "sh", "-c", command, (char *) 0);
  

If you go through the shell, you pass one string argument, whereas exec*() demands you to specify arguments separately. Let’s write a sample program to print all the arguments it receives. I’ll do it in Python to get a more readable output.

#!/usr/bin/env python3
import sys
print(sys.argv)

Let’s see what appears:

$ ./argv.py foo bar
['./argv.py', 'foo', 'bar']
$ ./argv.py 'foo bar'
['./argv.py', 'foo bar']
$ ./argv.py foo\ bar baz
['./argv.py', 'foo bar', 'baz']

$ ./argv.py $(date)
['./argv.py', 'Sat', 'Sep', '2', '16:54:52', 'CEST', '2017']
$ ./argv.py "$(date)"
['./argv.py', 'Sat Sep  2 16:54:52 CEST 2017']

$ ./argv.py /usr/*
['./argv.py', '/usr/X11', '/usr/X11R6', '/usr/bin', '/usr/include', '/usr/lib', '/usr/libexec', '/usr/local', '/usr/sbin', '/usr/share', '/usr/standalone']
$ ./argv.py "/usr/*"
['./argv.py', '/usr/*']

$ ./argv.py $EDITOR
['./argv.py', 'nvim']

$ $PWD/argv.py foo bar
['/Users/kwpolska/Desktop/blog/subprocess/argv.py', 'foo', 'bar']
$ ./argv.py a{b,c}d
['./argv.py', 'abd', 'acd']

$ python argv.py foo bar | cat
['argv.py', 'foo', 'bar']
$ python argv.py foo bar > foo.txt
$ cat foo.txt
['argv.py', 'foo', 'bar']

$ ./argv.py foo; ls /usr
['./argv.py', 'foo']
X11@        X11R6@      bin/        include/    lib/        libexec/    local/      sbin/       share/      standalone/

As you can see, the following things are handled by the shell (the process is unaware of this occurring):

• quotes and escapes

• expanding expressions in braces

• expanding variables

• wildcards (glob, *)

• redirections and pipes (> >> |)

• command substitution (backticks or $(…))

• running multiple commands on the same line (; && || &)

The list is full of potential vulnerabilities. If end users are in control of the arguments passed, and you go through the shell, they can execute arbitrary commands or even get full shell access. Even in other cases, you’ll have to depend on the shell’s parsing, which introduces an unnecessary indirection.

TL;DR: How to do this properly in your language of choice

To ensure spawning subprocess is done securely, do not use the shell in between. If you need any of the operations I listed above as part of your command — wildcards, pipes, etc. — you will need to take care of them in your code; most languages have those features built-in.

In C (Unix)

Perform fork-exec by yourself, or use posix_spawn(). This also lets you communicate with the process if you open a pipe and make it stdout of the child process. Never use system().

In Python

Use the subprocess module. Always pass shell=False and give it a list of arguments. With asyncio, use asyncio.create_subprocess_exec (and not _shell), but note it takes *args and not a list. Never use os.system and os.popen.

In Ruby

Pass arrays to IO.popen. Pass multiple arguments to system() (system(["ls", "ls"]) or system("ls", "-l")). Never use %x{command} or backticks.

In Java

Pass arrays to Runtime.exec. Pass multiple arguments or list to ProcessBuilder.

In PHP

All the standard methods go through the shell. Try escapeshellcmd(), escapeshellarg() — or better, switch to Python. Or anything, really.

In Go

os/exec and os.StartProcess are safe.

In Node.js

Use child_process.execFile or child_process.spawn with shell set to false.

Elsewhere

You should be able to specify multiple strings (using variadic arguments, arrays, or otherwise standard data structures of your language of choice) as the command line. Otherwise, you might be running into something shell-related.

The part where I pretend I know something about Windows

On Windows, argument lists are always passed to processes as strings (Python joins them semi-intelligently if it gets a list). Redirections and variables work in shell mode, but globs (asterisks) are always left for the called process to handle.

Some useful functions are implemented as shell built-ins — in that case, you need to call it via the shell.

Internals: There is no fork() on Windows. Instead, CreateProcess(), ShellExecute(), or lower-level spawn*() functions are used. cmd.exe /c is called in shell calls.

Check currently used locale

The locale command (without arguments) should tell you which locales you’re currently using. (The list might be shorter on your end)

$ locale
LANG="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

If any of those is set to C or POSIX, has a different encoding than UTF-8 (sometimes spelled utf8) is empty (with the exception of LC_ALL), or if you see any errors, you need to reconfigure your locale.

Check locale availability and install missing locales

The first thing you need to do is check locale availability. To do this, run locale -a. This will produce a list of all installed locales. You can use grep to get a more reasonable list.

$ locale -a | grep -i utf
<lists all UTF-8 locales>
$ locale -a | grep -i utf | grep -i en_US
en_US.UTF-8

The best locale to use is the one for your language, with the UTF-8 encoding. The locale will be used by some console apps for output. I’m going to use en_US.UTF-8 in this guide.

If you can’t see any UTF-8 locales, or no appropriate locale setting for your language of choice, you might need to generate those. The required actions depend on your distro/OS.

• Debian, Ubuntu, and derivatives: install language-pack-en-base, run sudo dpkg-reconfigure locales

• RHEL, CentOS, Fedora: install glibc-langpack-en

• Arch Linux: uncomment relevant entries in /etc/locale.gen and run sudo locale-gen (wiki)

• For other OSes, refer to the documentation.

You need a UTF-8 locale to ensure compatibility with software. Avoid the C and POSIX locales (it’s ASCII) and locales with other encodings (those aren’t used by ~anyone these days)

Configure system-wide

On some systems, you may be able to configure locale system-wide. Check your system documentation for details. If your system has systemd, run

sudo localectl set-locale LANG=en_US.UTF-8

Configure for a single user

If your environment does not allow system-wide locale configuration (macOS, shared server with generated but unconfigured locales), or if you want to ensure it’s always configured independently of system settings.

To do this, you need to edit the configuration file for your shell. If you’re using bash, it’s .bashrc (or .bash_profile on macOS). For zsh users, .zshrc. Add this line (or equivalent in your shell):

export LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

That should be enough. Note that those settings don’t apply to programs not launched through a shell.

Python/Windows corner: Python 3.7 will fix this on Unix by assuming UTF-8 if it encounters the C locale. On Windows, Python 3.6 is using UTF-8 interactively, but not when using shell redirections to files or pipes.

This post was brought to you by ą — U+0105 LATIN SMALL LETTER A WITH OGONEK.

Episode I: INotifyPropertyChanged, or Python writing C#

I was building a private C# weekend project (that turned into a weeklong project) — and by the way, WPF and C# are quite pleasant (Windows Forms is a trainwreck, though). One of the things I used in that project was a DataGrid bound to a list of custom objects (a DataGrid is a table, basically). And in order to use it, you need to use the INotifyPropertyChanged interface (MSDN). It involves doing something like this:

That’s 12 lines of code (excluding [JsonProperty] which comes from the Json.NET library) for that pattern. Oh: and I need to do that for every field/property of my class, because otherwise any changes to them would not be reflected in the tables (and maybe one or two fields were not in the table).

Doing that by hand is really not feasible: you need to copy-paste this large block 14 times and take care of 5 instances of the name (3 with underscores, 2 without), 2 instances of the type, and the [JsonProperty] attribute (which does not appear on all properties).

So, I used one of those intelligent computer things to do it for me. I wrote a really simple Python script and ran it. And I ended up with all 14 fields built for me.

code-writing-code/write_properties.py (Source)

That script takes a list of properties and spits out a block of code, ready to be pasted into the code. Visual Studio has a nice Insert File as Text feature, so redirecting the output to a file and using that option is enough.

Episode II: Fixing argument order, or Vim (re)writing Java

Another project, Number Namer, written in Java, and it does what it says on the tin: takes a number and writes it out as words, while being multilingual and extensible. I used Eclipse for this project, because it looks good, is really helpful with its code linting, and does not run slowly on my aging system (I’m looking at you, IntelliJ IDEA aka PyCharm aka Android Studio…)

And so, I was building a test suite, using JUnit. It’s pretty straightforward, and I remember the syntax from Python’s unittest (even though I write tests with pytest nowadays). Or so I thought.

// (incorrect)
assertEquals("Basic integers (7) failed", namer.name(7L), "seven");
// (fixed)                              ^ cursor
assertEquals("Basic integers (7) failed", "seven", namer.name(7L));

You see, the typical Python spelling is self.assertEquals(actual, expected). Java adds a String message parameter and it also swaps actual and expected. Which I didn’t notice at first, and I wrote my assertions incorrectly. While it doesn’t really matter (it will still work), the output looked a bit weird.

And I noticed only when I finished writing my tests (and I had a typo in my expected output). I wanted to fix them all — not manually, of course. So, I closed this file, brought up Vim, searched for the motion I need (it’s t{char} — see :help t). And I ended up with this (cursor placed on the comma after the first argument):

What does this do, you may ask? It’s actually pretty self-explanatory:

delete till comma, (go) till closing parenthesis, paste.

This fixes one line. Automatically. Make it a macro (wrap in qq … q, use with @q) and now you can run it on all lines, either by moving manually or by searching for , and pressing n@q until you run out of lines.

Epilogue

Some of you might say “but VS/Eclipse/IDEA has an option for that somewhere” or “[expensive tool] can do that” — and a Google search shows that there is an Eclipse plugin to swap arguments and that I could also write a regex to solve my second issue. Nevertheless, Python is a great tool in a programmer’s toolbox — especially the interactive interpreter. And Vim is an awesome editor that can accomplish magic in a few keystrokes — and there are many more things you can do with it.

Go learn Python and Vim now.

Also: don’t even bother with VsVim or IdeaVim or any other Vim emulation plugins, they work in unusual ways and often don’t give you everything — for example, VsVim has a Vim visual mode (v key) and Visual Studio selection mode (mouse), and only one allows Vim keystrokes (the other will replace selected text).