Step 1. Install Windows in Boot Camp the usual way

The first thing you should do is install Windows 10 in Boot Camp, with the help of the Boot Camp Assistant. The Assistant will take some time to partition your drive and do other preparations (and show barely informative progress bars, but I ranted about that Apple design “feature” already). There are no special preparations for this, the standard process will work. If you already have Windows installed, you can go to the next step.

Step 2. Ensure the setup is stable

We’ll be making changes to how the machine boots, and as such, it’s good to have other things working correctly and in line with your expected configuration. Make sure that:

• Both macOS and Windows boot correctly

• You can change the OS you boot into by holding the Option key after pressing Power (requires disabling the firmware password [1])

• Disk encryption (FileVault, BitLocker) is enabled (if you want that, of course) and fully configured (initial encryption is complete)

• Windows setup (including Boot Camp drivers) is complete

• The OSXRESERVED partition that the Boot Camp Assistant created has been deleted (that should have happened when booting into macOS for the first time after installing Windows — complete with a slowly moving progress bar and no other information, as is usual for this OS — but if that didn’t happen, use Disk Utility in macOS or Recovery OS to do that — pick your drive, click Partition and delete the partition, this will grow the macOS partition)

• System Integrity Protection is enabled (the procedure is a bit safer that way)

Step 3. Create a partition for rEFInd

First, back up your data before making changes to your hard drive layout. We’ll need to create a new partition for rEFInd to live on. This is the safest option — you could install it to the EFI System Partition (ESP), but macOS might want to put its own stuff there, and it’s safer not to use it.

The rEFInd partition doesn’t need to be large (50 MB will be enough); it must use the HFS+ (Mac OS Extended) file system. To create it, you have three options:

• From macOS, by shrinking the macOS partition: open Disk Utility, choose your drive, select Partition, add a new partition, set its size and file system (in that order!). This will take a few minutes (10-15, or possibly more), and you won’t be able to use your Mac during the resize.

• From Recovery OS, by shrinking the macOS partition: same steps apply, but it might be a bit safer than doing it from within macOS.

• From Windows, by shrinking the Windows partition: open Disk Management (press the Windows key and type partition, or open Computer Management from Administrative Tools), right click your Windows partition, select Shrink Volume. Enter the desired size and click Shrink. Then, right click the unallocated space and create a New Simple Volume. For now, choose FAT32 or exFAT; you’ll need to reformat it as HFS+ from within macOS later (Erase in Disk Utility). This will take a few seconds — and even if you include the time to reboot, it’s faster.

After you create the new partition and make sure it’s HFS+ (Mac OS Extended), you can proceed with the setup. Also, if you don’t want the partition to be visible in the Finder, run the following command (insert the correct volume path for your system):

sudo chflags hidden /Volumes/rEFInd

Step 4. Configure and install rEFInd

To set ue rEFInd, you’ll need to boot into macOS. Download rEFInd from the author’s website — you want the file named A binary zip file. Extract this archive anywhere on your system (~/Downloads is fine).

First, you’ll need to change the configuration file refind/refind.conf-sample. Locate the setting named enable_and_lock_vmx, uncomment it (remove the # at the start of the line), and set its value to true. You can also make other configuration changes — the default timeout of 20 seconds is likely to be too much for your needs.

When your configuration file is ready, you can install rEFInd. You can use the refind-install tool, or perform a manual install (check out the installation docs for more details).

Before installing, you’ll need to get the device name of your rEFInd partition. Open Disk Utility, select the partition from the left pane, and check the Device field (for example, disk9s9 — it will be different on your system, depending on your partition layout).

Open a Terminal, cd into the directory where rEFInd was extracted, and run the following command (replace disk9s9 with the device name on your system):

./refind-install --ownhfs disk9s9

This command will produce an error if you have SIP enabled — but this error is not important for us, the install will work without the change that SIP prevented. [2]

You can now shut down your Mac and use the Option key while starting up to choose the OS. You should see three options: Macintosh HD, EFI Boot, and Boot Camp. The EFI Boot option is rEFInd — pick that, boot into Windows (Microsoft EFI boot), et voilà — Windows can now run virtualization software.

There are a few more things that you can do now, depending on your OS preferences.

• You can make rEFInd the default boot loader. Hold Control on the Apple boot device selection screen and click the Power icon under the EFI Boot drive (source for the tip).

• You can use rEFInd to boot into macOS, although this might not work with Big Sur according to the author (it seems to work for me, but YMMV). You can use the standard boot method for macOS (by defaulting to Macintosh HD, or by choosing it from the Power+Option picker) and rEFInd exclusively for Windows (and set your timeout to a low value).

• You can modify rEFInd’s configuration — in this scenario, the config file is /Volumes/rEFInd/System/Library/CoreServices/refind.conf. You can set a custom background image, for example (rEFInd’s site can help you figure out what options are available and what you can set them to).

How will it work out in Apple land?

For developers who work only on iOS apps, the transition also won’t mean much. Maybe a faster, more accurate Simulator. They’ll need to buy an ARM Mac sooner or later (within the next 5 years), because Apple requires them to use the latest Xcode version for App Store submissions, and Xcode supports at best the previous version of macOS. But that has been Apple’s policy forever, and the Intel Macs will probably be within the usual deprecation range when that happens.

The requirements for macOS-only developers are pretty obvious, they will need to buy an ARM Mac on day one, so they can test their apps on the new platform. They will also need to work on ARM compatibility — although updating your app for the new OS is a yearly ritual in Apple land, so that’s also mostly business-as-usual (unless you do a lot of unportable low-level stuff in your code). There are some pro apps that tend to lag behind new Apple decrees (some might have been hit by Catalina), and users of those apps might prefer to stay with Intel for a little bit longer.

But then, we get to the requirements of developers who use Macs, but don’t work exclusively with the Apple platforms. This is a fairly large group, since many developers like Macs for the good hardware, Unix-based software, and the integration of both. And for some part, non-developers are affected too.

Who needs non-Apple operating systems?

The first group are people tied to Windows, somehow. Some of them might be using Boot Camp to play games. Others might be using Boot Camp or virtualization software (Parallels Desktop, VMware Fusion, Oracle VM VirtualBox) to run Windows and Windows-specific apps — perhaps they need the Windows version of Office, or various Windows-onlypro apps, or they need Windows to file their taxes, because their government does not care about non-Windows OSes. Or perhaps they’re web developers, and they need to test compatibility with the Windows versions of browsers, or the old Microsoft browsers (IE and pre-Chromium Edge).

The second group is software developers who need Linux. While macOS provides a very competent development environment, and many things can be run directly on macOS, some use-cases may require a Linux VM. Perhaps the most notable case is Docker.

Docker is a solution for lightweight app containers, that can offer separation between apps, and that can simplify and standardize deployment. Docker itself is not a virtualization solution (at least in the traditional sense). Docker must run on top of Linux (there’s also Docker-on-Windows, but that’s another story). The Docker Desktop for Mac app runs a lightweight Linux VM, and runs containers in that VM. The virtualization solution Docker for Mac uses is Hypervisor.framework, which is part of macOS itself.

Who else needs virtualization? Android developers. The Android Emulator is also a virtual machine that runs the Android operating system. Android can run on different architectures, and so, a x86 system image is typically used for the Emulator.

Is virtualization possible on ARM?

Yes, definitely. Apple has been testing it much earlier, since the aforementioned Hypervisor.framework was found on iOS in April. And Apple announced virtualization support for ARM Macs during the keynote, and showed an example of a Linux VM. That VM was, of course, running an ARM64 distribution of Linux.

But what can we use this for? Turns out, it’s complicated. The easiest thing from the few use-cases mentioned before is Android. Google just needs to get the Emulator working on ARM Macs and ship that to the devs.

What about Linux in general? Many mainstream distributions support ARM64, so that’s not a problem in general. The support for a particular distro or software might be worse than on x86_64, but it’s generally not a problem for users.

But for Docker, there’s a problem. One of the many advantages of Docker is dev-prod parity. If you deploy your app with Docker to an x86_64 Linux server, you can also install Docker on an x86_64 Linux developer machine (or a Linux VM on an Intel Mac/Windows PC). Both the server and the dev machine can run the same image, the same code, the same configuration. That won’t happen if they are a different architecture. This means that you can end up with bugs happening because of different environments, and it’s also possible that some images you depend on are not available for both architectures.

And then we get to Windows. Windows also has an ARM version, but it’s currently available only with a new ARM device (you can’t buy it standalone). If Microsoft were to sell this, we’d have an issue with the software. Windows 10 on ARM supports 32/64-bit ARM software, and can run 32-bit Intel (x86) software using emulation. It cannot, however, emulate apps that require 64-bit Intel processors (x86_64). This makes the software situation on that platform a bit better. While many developers don’t care about ARM and might not have builds for ARM available, most Windows software is available in both x86 and x86_64 versions, or is exclusively 32-bit. But certain pro apps are x86_64 only, so if there is no ARM build of it, an ARM Windows PC currently cannot run it. (Update: Microsoft announced x86_64 emulation on ARM, which means more software will work.)

And note that Microsoft knows about the transition, but we haven’t heard anything about Windows during the keynote…

Can we emulate x86(_64) and run x86(_64) Windows 10?

Theoretically? Yes. Practically? No.

The issue with emulation is speed. There are a few x86 emulators available, and those emulators can be run on an ARM device just fine. You can find videos on YouTube (not a very reliable source of information, I know) in which people try to benchmark those, or try to run Windows using an emulator like that. And even with an ancient Windows version, the emulation is painfully slow. Windows 10 would be basically unusable if you tried to emulate all of it.

How does the x86 emulation on Windows 10 for ARM work? You can watch the Channel 9 video about Windows 10 on ARM (around 6:00) for more details. The trick is that system DLLs are using a hybrid x86/ARM64 library format, which means x86 code can call those DLLs at native speeds. This means that many apps run at near-native speed (depending on the ratio of custom code to system DLL calls). This technique cannot work for emulating the entire operating system. If Windows 10 on ARM was made available for ARM Macs, running x86 Windows apps would become feasible.

Rosetta probably uses similar technique. Most apps will be translated at install time, not at run time. But you can’t do that with an entire OS.

What’s next for people who rely on both macOS and Windows?

For a few more years, Intel Macs will still be supported by Apple (with new macOS versions) and by software vendors. But after that? Well, you’re stuck with two machines, at least until Windows on ARM becomes viable and runnable on Macs. Or you can start exploring alternatives to macOS software. If you’re one of the macOS-as-UNIX-with-great-UX developers (hello!), perhaps you’ll have to switch to Linux — or perhaps Windows with Windows Subsystem for Linux? (The latter is becoming more usable with every Windows release, so keep an eye on that… I wrote this post in NeoVim in WSL2, with Windows Terminal supporting many advanced terminal features, and the transparent filesystem integration letting me access Windows files directly).

Post-M1 announcement update (2020-11-14)

Parallels have confirmed support for M1 Macs and are offering a Technical Preview of their M1 virtualization product. This announcement’s mention of Windows 10 ARM supporting x86_64 apps has caused some tech writers to assume Parallels will support Windows 10 ARM on M1 Macs. This is not what the post says. Parallels is not, and cannot announce support for that OS, because Windows 10 ARM is (still) available to ARM OEMs only to install on their devices — making an official announcement about this feature today would be admitting to doing something illegal/not allowed by the EULA. I’m pretty sure they are not working on support for Windows 10 ARM now and in the foreseeable future, until Microsoft opens up Windows 10 ARM to the public — their own legal issues aside, who would they sell the Windows support to?

In other news, Docker is not ready yet.

Making install media with El Capitan (with access only to Recovery)

The best, safest, least error-prone way to do an install is with a USB stick. Unfortunately, making a USB stick with the macOS installer on it is a nuisance. The expected way to produce macOS install media is to download the installer from App Store/Software Update, and run the createinstallmedia command-line program included with that installer app. All is well, as long as macOS works. If it doesn’t, and Recovery can’t install it for you, that can be difficult to solve.

Apple does not make macOS images publicly available. That’s probably to make Hackintoshing this little bit harder, but this also affects legitimate users. The only thing you can easily download from Apple is El Capitan. Apple offers InstallMacOSX.dmg on their website.

There are also some other downloads available:

• there’s a Sierra download, with a broken link, although it can be fixed by changing updates-http to updates: get Sierra installer.

• if you need something older than El Capitan, downloads are available as far back as Lion, although these might not have the createinstallmedia tool required for this procedure.

This post assumes you’re working with El Capitan; a quick look though that file suggests the instructions in this post should work with Sierra as well.

If you take a look at the instructions, you will see that this is not a bootable OS X image. This image has a .pkg package. This package is expected to install /Applications/Install OS X El Capitan.app. Well, we’re in recovery, we can’t install stuff. So, let’s do this the manual way.

Manually installing .pkg files

Turns out the .pkg format is just an archives all the way down, with all archives being different formats (at least three).

The first archive is the .pkg file itself. Those files are in XAR format, which was invented by the OpenDarwin community. You can either extract it with pkgutil --expand foo.pkg foo_files (the last argument is the destination directory, can be anything, will be created by pkgutil) if you have access to that command (it’s available in Recovery OS), or you can try the xar utility as xar -xf foo.pkg. The structure produced by both tools is a bit different, but we can work with both.

The second archive-in-archive is the Payload. It’s a gzipped cpio archive that contains the files installed by this package. If you have BSD tar (default on macOS, easily installable on Linux), you can just do tar -xvf Payload. Otherwise, you can use gunzip -c Payload | cpio -i (or gzcat). That will extract all the files the package has.

Another nested archive is the Scripts archive, although note that pkgutil will extract it automatically. If it’s not extracted, it’s actually .cpio.gz again, with the same way to extract it.

(PS. If you have 7z around (on Windows/Linux as well), you can just point it at all the compressed files mentioned in this paragraph.)

Making install media out of partial macOS installers (“not a valid OS installer application”)

$ (mount the DMG in Disk Utility)
$ cp /Volumes/Install\ OS\ X/InstallMacOSX.pkg /Volumes/Macintosh\ HD/
     (Or copy it to some other volume you can write to; NOT the USB stick)
$ cd /Volumes/Macintosh\ HD/
$ pkgutil --expand InstallMacOSX.pkg elcapitan
$ ls -F elcapitan
Distribution*       InstallMacOSX.pkg/ Resources/
$ cd elcapitan/InstallMacOSX.pkg/
$ tar -xvf Payload
x .
x ./Install OS X El Capitan.app
x ./Install OS X El Capitan.app/Contents
…

# "Install OS X El Capitan.app/Contents/Resources/createinstallmedia" --volume /Volumes/MyBlankUSBDrive --applicationpath "Install OS X El Capitan.app"
Install OS X El Capitan.app does not appear to be a valid OS installer application.

$ mkdir "Install OS X El Capitan.app/Contents/SharedSupport"
$ mv InstallESD.dmg "Install OS X El Capitan.app/Contents/SharedSupport"
# "Install OS X El Capitan.app/Contents/Resources/createinstallmedia" --volume /Volumes/MyBlankUSBDrive --applicationpath "Install OS X El Capitan.app"
Ready to start.
To continue we need to erase the disk at /Volumes/MyBlankUSBDrive.
If you wish to continue type (Y) then press return:

Install macOS Catalina.app does not appear to be a valid OS installer application.

El Capitan installer can’t be verified

If you get this error, it might be because Apple’s signing keys expired, or because of other date/time weirdness. Regardless, you can force an install if you are sure the installer is not damaged with this command (source):

# installer -pkg  /Volumes/Mac\ OS\ X\ Install\ DVD/Packages/OSInstall.mpkg -target /Volumes/"XXX"

Bonus tidbit 1: how the download works

While messing with all the installer stuff, I found out a few interesting/worrying things about the download process.

The first one is that the macOS installer uses plain HTTP without encryption to download files. That opens you to all the standard issues — an attacker can replace files you download, and the protocol doesn’t do anything to detect errors (the installer will verify files, but where do the checksums come from?).

The second one is how the download happens. You might have noticed it to be a bit slower than usual traffic. The download happens in 10 MB chunks, using the Range HTTP header. The installer asks for 10 MB, gets it, saves, asks for another chunk. Repeat that over 800 times, and the overhead of the entire HTTP dance becomes noticeable. (I haven’t checked, but I hope the installer at least uses Keep-Alive. I wouldn’t be particularly surprised if it didn’t, though.)

But this raises another question. The servers clearly support partial downloads. And yet, if your network disconnects during the download, your download progress for that file is reset, and in Catalina, you can go from 8 GB back to 500 MB if you’re particularly unlucky. The question is, why? This infrastructure should make it trivial to continue the download, perhaps discarding the most recent chunk if you’re concerned about that download of it being unsuccessful.

Bonus tidbit 2: using Terminal from Setup Assistant

The first time you boot a Mac after a clean install, it starts the Setup Assistant. This app asks for basic OS settings (locale, date/time, user accounts), and also lets you restore user data from backups.

Sometimes, you might want to access the Terminal or Console from that screen. You can do that with Ctrl + Opt + Cmd + T and Ctrl + Opt + Cmd + C respectively (source).

How could that come in handy? For example, if you want to check if the backup drive still worked and if the process isn’t stuck (I wrote a test file and also checked top).

Bonus tidbit 3: creating an image of the install media might not work

A few months later, in December, I upgraded to Big Sur and then installed Windows 10 alongside it in Boot Camp. I then did some more hacks, which led to two unbootable OSes.

As part of the upgrade, I had prepared install media and used it to install (so it wouldn’t fail, as it did last time), and made a .dmg of it with Disk Utility. (Also, Apple won’t tell you this, but you need to give Disk Utility Full Disk Access for disk imaging to work. Otherwise, you get a cryptic error.) I erased the USB drive after installing, but hey, I could get it back. I booted into Internet Recovery and restored my image. Big Sur failed to boot and showed a 🚫 sign. I tried restoring my Catalina image from the previous reinstall, and that didn’t work due to a size mismatch. I used a different USB drive than these months ago (I didn’t have that one with me at the moment), and apparently the one I used had a different size (both are marketed as 16 GB). The images could be mounted fine, and createinstallmedia should have worked, likely producing a bootable drive.

Bonus tidbit 4: don’t bother restoring a Time Machine backup

Time Machine is Apple’s magical backup solution. Time Machine saves snapshots of your entire disk. It’s supposed to help restore files that were deleted or changed in an unwanted way, or help you restore a full macOS install.

Time Machine is great at file recovery, but none of my 3 system restore attempts were successful. Attempt #1 was a full Time Machine System Restore, from Recovery, back in June. It failed partway through, it couldn’t read everything from the disk. There might have been underlying hardware issues with that failure, so I had another attempt.

Attempt #2 was a Migration Assistant restore, as part of the initial setup. This one succeeded, and things worked… except for one fairly important app. This app requires online activation with the vendor, and it wouldn’t reactivate after the install. Whatever the third-party vendor is doing didn’t like the reinstall. I tried to nuke all the things in ~/Library related to their software, and ran their nuke-everything uninstaller, but that didn’t work. I reinstalled from scratch and copied over my files, settings and apps from the Time Machine drive.

Attempt #3 involved the System Restore again, this time for the December reinstall. The hardware issues were all fixed in the meantime, so I went for a Time Machine System Restore.

Issue #1: Internet Recovery booted into Catalina. There was an issue on Apple’s side, Big Sur was unavailable in Internet Recovery in December. TM Recovery will not restore a backup created with a newer version of macOS than you’re booted into, so I was forced to restore a slightly older Catalina backup. (I spent most of my time in Windows during that weekend, so other than the need to upgrade macOS to Big Sur again, I didn’t really lose any data due to this.)

Issue #2: It wasted time computing an inaccurate size estimate. Before restoring a backup, macOS first checks if it will fit on your drive. When it does that, an indeterminate progress bar is shown. macOS won’t tell you the result of that computation, but you can read the final value from the full Installer Log (Cmd + L). On my Mac, the value was 96.2 GB. I was at the Mac when it was getting close to that value. 94, 95, 96, 96.1, 96.2, 96.3… hold on a second, 96.3 GB? Hopefully that’s just a bunch of extra things that are installed from the system image directly, or something like that, right? Of course, since the progress bar is based on the pre-computed size, it became indeterminate and I couldn’t tell when it would end. 98, 100, 110, 120, 121.2 GB is where it ultimately ended. So, not only did it waste 20+ minutes computing a size, it was off by 25 GB.

Issue #3: The restore didn’t work. The System Restore finished and claimed to have succeeded, but macOS wouldn’t boot. It showed an Unrecoverable error, SecurityAgent was unable to create requested mechanism. Most people who had a similar error had it caused by a botched TeamViewer uninstall; I didn’t have that installed, and it was referring to a different component. So, wipe and fresh reinstall it is.

I copied my stuff from the TM drive, and it was acting weird. Some apps failed to load their settings copied into Library, others started with a “Move to /Applications?” prompt (even though they were in that directory). For some reason, those files had some hidden attribute set on it. I worked around it by putting files in a .zip archive with Keka, and then unzipping them; xattr might also help. (The attribute was likely com.apple.quarantine.)

After I got the Mac to work, I reinstalled Windows and set up rEFInd, and it now works fine. (I only use rEFInd because I want virtualization in Windows, and that doesn’t work unless you’re warm-rebooting from macOS. I don’t need anything more advanced than the Option key boot menu, but Apple made me use a third-party bootloader.)

We now go back to the original post from June.

An Open Letter to Progress Bar Designers

Dear Progress Bar Designers: can you please make your progress bars functional? The macOS progress bar might look sleek at just 7 px (non-Retina)/6 pt = 12 px (Retina) high, but at the same time, you’re looking at individual pixels if you need to know if it works or if it’s stuck. I have had to point my mouse cursor at the end of the filled-in part just to know if it’s working or not. Or sometimes, put a piece of paper in front of my screen, because there is no mouse cursor when macOS installs on the black screen. How to make that progress bar easier to use and more informative? Just add numbers on top of it. For long-running processes, I wouldn’t mind progress bars that said “12.34%”. That specific Setup/Migration Assistant window should be changed (it only has a remaining time estimate and transfer speed, it should also show moved data/total size), but wouldn’t more things benefit from a clear indication of the progress? Yes, perhaps it looks less sleek, perhaps it requires more space for the bar.

Just compare: which is easier to parse? Which is more informative?

I’d honestly be happy enough with option 2, at least it can be read easily and you can remember the number instead of a vague position.

In the end…

After all this, I managed to get macOS Catalina installed. After various failures in built-in El Capitan recovery and Catalina Internet Recovery, I first installed El Capitan with this hack, then jumped to Mojave because I thought the new Software Update would help (it didn’t, same installer, same failed-to-extract-package issue), then made a Catalina USB stick, and it finally clean-installed, but I was worried about the backup disk’s operation, and I used a proxy on my local network to try and speed up Catalina downloads without much improvement… but hey, at least it works. Apple should really make it easier to install their OS and to make boot media even when stuff doesn’t work, even from Windows. The Hackintosh folks can just find someone with a working Mac and ask them to download from App Store and make install media, or find less legitimate sources, they probably don’t care as much. But if my own system crashes, I’d probably want to get working install media immediately, myself, and from Apple. Without all this mess.