(Also, Docker Desktop demands payment for enterprise use; if your company fits the enterprise criteria but cannot afford $16/developer/month, run.)

Console command conventions: $ means Linux, > means Windows (PowerShell).
Path conventions: / means Linux, \ means Windows.

Linux setup

You need to have WSL and a distribution installed, and you must be using WSL 2. Read the install docs if you need to do it.

➡️ Start by installing Docker in the usual way for your Linux distribution of choice. I’m using Ubuntu with the official Docker apt repository. Make sure to add yourself to the Docker group.

➡️ Enable systemd support in WSL. Put those two lines into /etc/wsl.conf:

[boot]
systemd=true

You can do it like so (if you don’t have a wsl.conf file already):

$ printf '[boot]\nsystemd=true\n' | sudo tee /etc/wsl.conf

➡️ Next, enable the systemd services:

$ sudo systemctl enable containerd.service
$ sudo systemctl enable docker.service

Exposing Docker to Windows

By default, Docker only listens on a Unix domain socket. To work with Docker from Windows, it needs to listen on a TCP socket. If you’re fine with using Docker only from the WSL terminal, and won’t run any Docker-compatible tools on the Windows side, you can basically stop reading here.

Adding a socket can be done in two ways: by adding command-line arguments, or by modifying /etc/docker/daemon.json. If your Docker systemd service is passing in arguments, you need to edit the service, and can’t do it using the daemon.json file. Let’s edit the service, as this is more likely to be what you need.

➡️ Run sudo systemctl edit docker.service and override the ExecStart command. The first ExecStart= line clears out the existing configuration defined in the system-provided service. The second line is the new command. To avoid breaking things, review the ExecStart command that is shown as the current content of your docker.service, and make sure you have both -H arguments as in my example below.

➡️ We’re done with the WSL setup, so let’s restart WSL and make sure Docker is working:

Windows setup

➡️ Download the latest version of the Docker binaries. Extract docker.exe and put it somewhere on your Windows PATH. I have a ~\Tools\bin folder for things like that.

We need to tell the Docker CLI where to look for Docker. The easiest way to do this is to set the DOCKER_HOST environment variable. This might not be the cleanest way if you want to work with Windows containers or Docker Desktop, but you don’t, so we don’t need to bother with anything nicer.

➡️ Press the Start key, type environment variables (or the equivalent phrase in your Windows language) and open the environment variables editor. Add an environment variable named DOCKER_HOST with the value tcp://127.0.0.1:2375.

➡️ Restart your Terminal. In one tab, run a WSL shell. In another, run PowerShell (or cmd):

Optional: Docker Compose

Docker Compose is a plugin for Docker CLI, and it not included in the Docker binary we’ve installed on Windows. It is quite easy to add it in if you want.

➡️ Download the latest version of the Docker Compose binary (docker-compose-windows-x86_64.exe). Move it to ~\.docker\cli-plugins\docker-compose.exe.

> mkdir ~\.docker\cli-plugins
> mv ~\Downloads\docker-compose-windows-x86_64.exe ~\.docker\cli-plugins\docker-compose.exe

Let’s test it out:

> docker compose version
Docker Compose version v5.1.0

Optional: WSL VM management

WSL does not automatically start the Linux VM. And it kills it if no user process is running inside of it.

If you’re fine with having a WSL terminal open while you’re working with Docker, you don’t need to do anything. But if you don’t want to see a WSL terminal, make Windows start WSL when you log in and set a really large VM Idle Timeout in the Optional Features tab of the WSL Settings app. The maximum value accepted by the GUI¹ is the maximum value a 32-bit integer can represent, i.e. 2147483647 ms (24 days). You can also manually add the option to ~\.wslconfig:

[wsl2]
vmIdleTimeout=2147483647

Getting Started

PowerShell is so nice, Microsoft made it twice.

Specifically, there concurrently exist two products named PowerShell:

• Windows PowerShell (5.1) is a built-in component of Windows. It is proprietary, Windows-only, and is based on the equally proprietary and equally Windows-only .NET Framework 4.x. It has a blue icon.
• PowerShell (7.x), formerly known as PowerShell Core, is a stand-alone application. It is MIT-licensed (developed on GitHub), available for Windows, Linux, and macOS, and is based on the equally MIT-licensed and equally multi-platform .NET (formerly .NET Core). It has a black icon.

Windows PowerShell development stopped when PowerShell (Core) came out. There are some niceties and commands missing in it, but it is still a fine option for trying it out or for when one can’t install PowerShell on a Windows system but need to solve something with code.

All examples in this post should work in either version of PowerShell on any OS (unless explicitly noted otherwise).

Install the modern PowerShell: Windows, Linux, macOS.

Objects? In my shell?

Let’s try getting a directory listing. This is Microsoft land, so let’s try the DOS command for a directory listing — that would be dir:

This looks like a typical (if slightly verbose) file listing.

Now, let’s try to do something useful with this. Let’s get the total size of all .txt files.

In a Unix shell, one option is du -bc *.txt. The arguments: -b (--bytes) gives the real byte size, and -c (--summarize) produces a total. The result is this:

But how to get just the number? This requires text manipulation (getting the first word of the last line). Something like du -bc *.txt | tail -n 1 | cut -f 1 will do. There’s also wc --total=only --bytes *.txt — but this is specific to GNU wc, so it won’t cut it on *BSD or macOS. Another option would be to parse the output of ls -l — but that might not always be easy, and the output may contain something unexpected added by the specific ls version or the user’s specific shell configuration.

Let’s try something in PowerShell. If we do $x = dir, we’ll have the output of the dir command in $x. Let’s try to analyse it further, is the first character a newline?

That’s interesting, we didn’t get the first character or the first line, we got the first file. And if we try $x[1]?

What if we try getting the Length property out of that?

PS C:\tmp\hello> $x[1].Length
23

It turns out that dir returns an array of objects, and PowerShell knows how to format this array (and a single item from the array) into a nice table. What can we do with it? This:

We can iterate over all file objects, get their length (using ForEach-Object and a lambda), and then use Measure-Object to compute the sum (Measure-Object returns an object, we need to get its Sum property). We can replace the ForEach-Object call with the -Property argument in Measure-Object. And if we want to look into subdirectories, we can easily add -Recurse to Get-ChildItem. We get actual integers we can do math on.

You might have noticed I used Get-ChildItem instead of dir in the previous example. Get-ChildItem is the full name of the command (cmdlet). dir is one of its aliases, alongside gci and ls (Windows-only to avoid shadowing /bin/ls). Many common commands have aliases defined for easier typing and ease of use — Copy-Item can be written as cp (for compatibility with Unix), copy (for compatibility with MS-DOS), and ci. In our examples, we could also use measure for Measure-Object and foreach or % for ForEach-Object. Those aliases are a nice thing to have for interactive use, but for scripts, it’s best to use the full names for readability, and to avoid depending on the environment for those aliases.

More filesystem operations

Files per folder

There’s a photo collection in a Photos folder, grouped into folders. The objective is to see how many .jpg files are in each folder. Here’s the PowerShell solution:

In Unix land, StackOverflow has a lot of solutions. The top solution is du -a | cut -d/ -f2 | sort | uniq -c | sort -nr — a lot of tools mashed together, starting with a tool to check disk usage, and a lot of string manipulation. The second solution uses find, read, and shell globbing. The PowerShell solution is quite simple and obvious to anyone who has ever touched SQL.

The above example works for one level of nesting. For more levels, given Photos\one\two\three.jpg, use Get-ChildItem -Filter '*.jpg' -Recurse Photos, and:

• Group by $_.Directory.Name (same as before) to get two
• Group by Split-Path -Parent ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)) to get one/two
• Group by ([System.IO.Path]::GetRelativePath("$PWD/Photos", $_.FullName)).Split([System.IO.Path]::DirectorySeparatorChar)[0] to get one

(All of the above examples work for a single folder as well. The latter two examples don’t work on Windows PowerShell.)

Duplicate finder

Let’s build a simple tool to detect byte-for-byte duplicated files. Get-FileHash is a shell built-in. We can use Group-Object again, and Where-Object to filter only matching objects. Computing the hash of every file is quite inefficient, so we’ll group by the file length first, and then ensure the hashes match. This gives us a nice pipeline of 6 commands:

Serious Scripting: Software Bill of Materials

Software Bills of Materials (SBOMs) and supply chain security are all the rage these days. The boss wants to have something like that, i.e. a CSV file with a list of packages and versions, and only the direct production dependencies. Sure, there exist standards like SPDX, but the boss does not like those pesky “standards”. The backend is written in C#, and the frontend is written in Node.js. Since we care only about the production dependencies, we can look at the .csproj and package.json files. For Node packages, we’ll also try to fetch the license name from the npm API (the API is a bit more complicated for NuGet, so we’ll keep it as a TODO in this example).

Just like every well-written shell script starts with set -euo pipefail, every PowerShell script should start with $ErrorActionPreference = "Stop" so that execution is stopped as soon as something goes wrong. Note that this does not affect native commands, you still need to check $LASTEXITCODE. Another useful early command is Set-StrictMode -Version 3.0 to catch undefined variables.

For .csproj files, which are XML, we look for PackageReference elements using XPath, and then build a PSCustomObject out of a hashmap — extracting the appropriate attributes from the PackageReference nodes.

For package.json, we read the file, parse the JSON, and extract the properties of the dependencies object (it’s a map of package names to versions). To get the license, we use Invoke-RestMethod, which takes care of parsing JSON for us.

In the main body of the script, we look for the appropriate files (skipping things under node_modules) and call our parser functions. After retrieving all data, we concatenate the two arrays, convert to CSV, and use Tee-Object to output to a file and to standard output. We get this:

Could it be done in a different language? Certainly, but PowerShell is really easy to integrate with CI, e.g. GitHub Actions or Azure Pipelines. On Linux, you might be tempted to use Python — and you could get something done equally simply, as long as you don’t mind using the ugly urllib.request library, or alternatively ensuring requests is installed (and then you get into the hell that is Python package management).

Using .NET classes

PowerShell is built on top of .NET. This isn’t just the implementation technology — PowerShell gives access to everything the .NET standard library offers. For example, the alternate ways to group photos in multiple subdirectories we’ve explored above involve a call to a static method of the .NET System.IO.Path class.

Other .NET types are also available. Need a HashSet? Here goes:

It is also possible to load any .NET DLL into PowerShell (as long as it’s compatible with the .NET version PowerShell is built against) and use it as usual from C# (although possibly with slightly ugly syntax).

Sick Windows Tricks

Microsoft supposedly killed off Internet Explorer last year. Attempting to launch iexplore.exe will bring up Microsoft Edge. But you see, Internet Explorer is a crucial part of Windows, and has been so for over two decades. Software vendors have built software that depends on IE being there and being able to show web content. Some of them are using web views, but some of them prefer something else: COM.

COM, or Component Object Model, is Microsoft’s thing for interoperability between different applications and/or components. COM is basically a way for classes offered by different vendors and potentially written in different languages to talk to one another. Under the hood, COM is C++ vtables plus standard reference counting and class loading/discovery mechanisms. The .NET Framework, and its successor .NET, have always included COM interoperability. The modern WinRT platform is COM on steroids.

Coming back to Internet Explorer, it exposes some COM classes. They were not removed with iexplore.exe. This means you can bring up a regular Internet Explorer window in just two lines of PowerShell:

$ie = New-Object -ComObject InternetExplorer.Application
$ie.Visible = $true

Why would you do that? The InternetExplorer.Application object lets you control the browser, e.g. you can use $ie.Navigate("https://example.com/") to go to a page. Why would you want to launch IE in 2024? I don’t know, I guess you can use it to laugh in the faces of the Microsoft developers who removed the user-accessible shortcuts? But there definitely exist some legacy applications that expect a COM-controllable IE.

We have already explored the possibility of using classes from .NET. .NET comes with a GUI framework named Windows Forms, which can be loaded from PowerShell and used to build a GUI. There is no form designer, so it requires manually defining and positioning controls, but it actually works.

PowerShell can also do various Windows management tasks. It can manage boot settings, BitLocker, Hyper-V, networking, storage… For example, to get the percentage of disk space remaining:

$c = Get-Volume C
"$(($c.SizeRemaining / $c.Size) * 100)%"

Getting out of PowerShell land

As a shell, PowerShell can obviously launch subprocesses. Unlike something like Python, running a subprocess is as simple as running anything else. If you need to git pull, you just type that. Or you can make PowerShell interact with non-PowerShell commands, reading output and passing arguments:

I chose to compute untracked files with the help of standard .NET string manipulation methods, but there’s also a regex option. On a related note, there are three content check operators: -match uses regex, -like uses wildcards, and -contains checks collection membership.

Profile script

I use a fairly small profile script that adds some behaviours I’m used to from Unix, and to make Tab completion show a menu. Here are the most basic bits:

Apart from that, I use a few aliases and a pretty prompt with the help of oh-my-posh.

The unusual and sometimes confusing parts

PowerShell can be verbose. Some of its syntax is a little quirky, compared to other languages, e.g. the equality and logic operators (for example, -eq, -le, -and). The aliases usually help with remembering commands, but they can’t always be depended on — ls is defined as an alias only on Windows, and Windows PowerShell aliases wget and curl to Invoke-WebRequest, even though all three have completely different command line arguments and outputs (this was removed in PowerShell).

Moreover, the Unix/DOS aliases do not change the argument handling. rm -rf foo is invalid. rm -r foo is, since argument names can be abbreviated as long as the abbreviation is unambiguous. rm -r -f foo is not valid, because -f can be an abbreviation of -Filter or -Force (so rm -r -fo foo) will do. rm foo bar does not work, an array is needed: rm foo,bar.

C:\Windows\regedit.exe launches the Registry editor. "C:\Program Files\Mozilla Firefox\firefox.exe" is a string. Launching something with spaces in its name requires the call operator: & "C:\Program Files\Mozilla Firefox\firefox.exe". PowerShell’s tab completion will add the & if necessary.

There are two function call syntaxes. Calling a function/cmdlet uses the shell-style syntax with argument names: Some-Function -Arg1 value1 -Arg2 value2, and argument names can be abbreviated, and can sometimes be omitted. Calling a method requires a more traditional syntax: $obj.SomeMethod(value1, value2). Names are case-insensitive in either case.

The escape character is the backtick. The backslash is the path separator in Windows, so making it an escape character would make everything painful on Windows. At least it makes it easy to write regex.

The ugliest part

The ugliest and the least intuitive part of PowerShell is the handling of single-element arrays. PowerShell really wants to unpack them to a scalar. The command (Get-ChildItem).Length will produce the number of files in the current directory — unless there is exactly one file, in which case it will produce the single file’s size in bytes. And if there are zero items, instead of an empty array, PowerShell produces $null. Sometimes, things will work out in the end (since many cmdlets are happy to get either as inputs), but sometimes, PowerShell must be asked to stop this madness and return an array: @(Get-ChildItem).Length.

The previous example with git status leverages its --null argument to get zero-delimited data, so we expect either $null or a single string according to the rules. If we didn’t want to use --null, we would need to use @(git status --porcelain) to always get an array (but we would also need to remove quotes that git adds to paths that contain spaces).

Conclusion

PowerShell is a fine interactive shell and scripting language. While it does have some warts, it is more powerful than your usual Unix shell, and its strongly-typed, object-oriented code beats stringly-typed sh spaghetti any day.

Many software development workflows involve virtualization. WSL, Docker for Windows, and the Android Emulator are some examples of common virtualization-based tools. Then there are general virtualization tools/hypervisors, such as VMware Workstation, Hyper-V or VirtualBox. All these tools require hardware virtualization extensions (Intel VT-x, AMD-V) or at least are very slow without them. Virtualization extensions are not enabled by default in the CPU, they must be enabled by something. On typical PCs, this is often a firmware-level setting (that might be disabled by default), or it might be unconditionally enabled by the firmware. On a Mac, however, enabling VT-x is done by macOS, as part of the boot process. This means that Windows running in Boot Camp will start without virtualization, unless you want to boot into macOS first and then reboot into Windows. That setup isn’t quite ergonomic (and what if macOS refuses to shut down, as it often does for me?).

Instead, we’re going to use rEFInd, a boot manager for EFI-based systems that can boot into various OSes and also handle other parts of the boot process. But first, let’s prepare our system for this.

DISCLAIMER: Those steps may make your Mac fail to boot. I don’t take any responsibility whatsoever if that happens. Prepare for the worst — make backups, perhaps have install media ready, plan some downtime.

sudo chflags hidden /Volumes/rEFInd

./refind-install --ownhfs disk9s9

I upgraded Windows 10 from the RTM version (build 10240, July 2015) to the November Upgrade version (1511/build 10560). It took me a good two hours, multiple reboots, and a BSOD.

I upgraded to Windows 10 (from Windows 10) back in August. The upgrade experience was okay. The main issue I had was that the error messages were a bit uninformative. There was one error, Windows 10 claimed it couldn’t check available disk space, which is a really strange way to say your active partition is a Linux one and you have a Linux bootloader set up. But after switching to the Windows bootloader, it worked fine. I only had to fight a bit with the display drivers, but other than that, it succeeded.

Windows 10 claims to be Windows as a Service: a fast update cycle with new Windows builds released often. Now, Linux users are accustomed to this. Depending on your distribution, you get new versions somewhere between “every day” (rolling release: Arch Linux, Gentoo, Debian sid), “every 6 months” (Ubuntu, Fedora), or some other time scale. (Three months and two weeks passed between the two “general” builds of Windows 10.) And most of the time, the upgrades are fast and relatively painless.

With Windows 10, this is not the case. I booted to Windows yesterday, to set up some new hardware and mess around with the OS. It offered the upgrade in Windows Update. After downloading it (3 GB, according to Microsoft), it started installing, without success. There were no error messages, and “Error” or “Failed to install” were all I could get.

I found the setup log files. The error log ended with “failed to resurrect new system”. Which didn’t make a lot of sense to me, even though I’m quite proficient with this sort of things.

This can be dangerous, so I backed up my drives with Clonezilla, which is really useful — and it ships with the Arch Linux ISO, which helped a lot.

So, I tried my trusty friend, the Media Creation Tool. I needed it to get the original install to work, and it came useful now. After yet another download, I got a USB stick with Windows 10 on it. So, I rebooted to the installer, which told me I can’t use it and I need to start it from within Windows.

I rebooted back into Windows, started the installer. It took it a while to begin the installation process (including waiting forever to get updates, so I restarted the process and disabled updating now, and another long wait to make sure it will work on my machine), but then it went quite fast and was at 70% after less than 10 minutes. Sadly, that’s too good to be true. It rebooted and started counting from zero. 90 minutes passed since I started, and the login screen came up. Username, password, log in.

Hi, we’ve updated your PC. Six more minutes, but at long last, I got my desktop. There’s just one problem though: 1024×768 is not the screen resolution I started the install with (oddly enough, the “Updating Windows” part was running at 1080p for part of the process, and then went back to 768p after a reboot). I tried installing NVIDIA drivers, and it failed — however, Windows managed to install something and wanted a reboot (what is this, Windows 95?). Fine, let’s reboot and get a fully functional Windows environment?

:( Your PC ran into a problem and needs to restart. A blue screen of death, on the second boot, talking about critical structure corruption in CI.dll. Well, shit. I rebooted, but I first took a little detour to the Arch Linux USB stick I prepared before to get a sane bootloader back. I’ve done this before, and usually requires three commands.

# mount /dev/sda2 /mnt
Metadata kept in Windows cache, refused to mount.
Failed to mount '/dev/sda2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

Wait, what?! I remember my hard drive layout, and /dev/sda2 is my Linux ext4 partition! I checked cfdisk, and apparently I now have a Hidden NTFS WinRE, with Linux moved to /dev/sda3 and my extended partition moved to /dev/sda4 (what would happen if I already had four partitions?). This is typical Windows misbehaviour: not caring about other OSes that might be installed.

I accepted this defeat, mounted /dev/sda3, ran arch-chroot /mnt and syslinux-install_update -i -a -m (I’m not a fan of GRUB 2, and I have a MBR drive layout). I should test this out by rebooting into Linux.

Error getting authority: Error initializing authority: Could not connect: No such file or directory (g-io-error-quark, 1)
Welcome to emergency mode!

systemd really values /etc/fstab, and you can’t boot if any entry fails to mount, which was the case with the Windows drive in /dev/sda1 and my other NTFS partition (note that the error message is very unhelpful)

I guess I have to get into Windows first. I tried booting Windows again, this time it worked (in glorious 1080p, which was the case on the previous boot too).

So, Windows re-enabled Fast Boot (a.k.a. hibernation instead of clean shutdown) as part of the upgrade. I hunted down the setting and restored sanity. But when I was doing that, I noticed things were spelt (or spelled) a bit differently than two hours ago. You see, the original Windows 7 system was US English. After the upgrade to Windows 10, I took the opportunity to install the British and Polish language packs (it’s a multi-user machine). Both of which were gone. I had to fix that, too.

And all I got out of it were colourful window borders, which are not even in my desired colour (Windows has a limited colour palette, even though I explicitly set it to #00AADD). What a great way to waste a Saturday!

This leaves me wondering, how does this work for Windows Insiders (Microsoft’s community beta testers)? Is the process better if Windows Update manages to perform the install? Or is it substantially faster if you’re running on a SSD? Perhaps the testers have dedicated machines and don’t run experimental builds on their daily drivers. But I doubt that, because that’s a significant investment without any financial gain from helping out a corporation. So, virtual machines? I have no idea how they cope.

By the way, cmd.exe and winver.exe claims it’s Copyright © 2016. Previously, Windows copyright notices were outdated. Now, they’re in the future.

This is a short guide on how to do this.

cd C:\mpd
mpd data\mpd.conf

Failed to load database: Failed to open database file "<name>": No such file or directory

cd C:\mpd
mpc play

cd C:\mpd
mpc update
mpc add "FILENAME"
mpc play

sc create mpd binPath= "c:\mpd\mpd.exe c:\mpd\data\mpd.conf"