In 2020 with macOS Big Sur, Apple announced that they were introducing a new framework to macOS for virtualization, built on their already existing Hypervisor Framework. The implementation details are rather vague; as always with Apple, the documentation is poor.

Legend for AVF blog series

• Part 1: Basic experimentation and intro to AVF (this post)
• Part 2: Analyzing UEFI firmware
• Part 3: Analyzing Windows booting + pure luck
• Finale: Enabling the GPU

First experience

My first expectation was a command line/GUI interface for creating VMs. Boy, was I wrong! As the name Framework suggests, it is actually an API that developers can use to embed virtual machines into their applications. You can use Objective-C or Swift to interact with it, and unlike a lot of other API's for macOS, Apple has not made a system app that interacts with it at all. This one is purely for developers.

Trying it out

As a non-macOS developer I dived into this head first. I know a little bit of Swift from creating small iOS apps so I chose that as my starting point. The basic infrastructure is simple: You first create a configuration and then the framework does the rest. Annoyingly, you cannot actually interact much with any instantiated objects. For example, if you configure a DVD drive, you can not interact/swap the disk anymore after the VM is created.

The basics

So how do you create the configuration? It is actually surprisingly simple

import Virtualization
...
// Basic VM settings
let configuration = VZVirtualMachineConfiguration()
configuration.cpuCount = 4
configuration.memorySize = 8 * 1024 * 1024 * 1024 // 8 GiB

This is pretty much the base virtual machine, all virtual machines need these lines of code. From here, you can add various devices to interact with the VM:

// Keyboard
configuration.keyboards = [VZUSBKeyboardConfiguration()]
// Mouse
configuration.pointingDevices = [VZUSBScreenCoordinatePointingDeviceConfiguration()]

Storage

Unfortunately, from here on it is no longer as easy. For storage devices, the framework abstracts everything into two groups:

Together you combine these into various storage devices. For example, you can back a DVD drive using an ISO and a disk using an actual BSD disk file:

let hddFile = FileHandle(forUpdatingAtPath: "/dev/diskX")!
let hdd = try! VZDiskBlockDeviceStorageDeviceAttachment(fileHandle: hddFile, readOnly: false, synchronizationMode: VZDiskSynchronizationMode.full)
        
let iso = try! VZDiskImageStorageDeviceAttachment(url: URL(filePath: "/path/to/iso.iso")!, readOnly: true)
configuration.storageDevices = [VZVirtioBlockDeviceConfiguration(attachment: iso),
                                VZVirtioBlockDeviceConfiguration(attachment: hdd)]

Note: Virtio block devices with read only attachments are automatically created as a DVD drive. This is not documented, but makes sense.

Boot loader

You can connect a bunch of components together, but without any firmware you won't get too far booting any OS. Virtualization Framework provides 'interesting' solutions to this. The first solution is VZLinuxBootLoader. This is very useful for embedded applications where you already have a kernel, initramfs and a system image. But it is not very versatile.

Instead, you can also use VZEFIBootLoader! This is a fully UEFI compliant bootloa..... Of course not. This is Apple. Why would it be compliant to anything! This is a boot loader which is worse than the one Apple uses for their Intel Macs. The basics are that it either loads the fallback bootloader on one of the disks, or it uses BootNext to boot. This is fine, but the problem arises after booting an EFI application. From what I can tell, the firmware used does not provide a full UEFI environment. GRUB and rEFInd work fine, but I have had problems booting Windows and some Linux kernels with it. At least it is simple to use:

// EFI Firmware
let bootLoader = VZEFIBootLoader()
// NVRAM, automatically created if file does not exist
bootLoader.variableStore = try! VZEFIVariableStore(creatingVariableStoreAt: URL(filePath: "/path/to/nvram.bin")!, options: VZEFIVariableStore.InitializationOptions.allowOverwrite)
configuration.bootLoader = bootLoader

Graphics

Also relatively simple: You create a GPU and you add screens:

let graphicsScreen = VZVirtioGraphicsScanoutConfiguration(widthInPixels: displayWidth, heightInPixels: displayHeight)
let graphicsDevice = VZVirtioGraphicsDeviceConfiguration()
graphicsDevice.scanouts = [graphicsScreen]
configuration.graphicsDevices = [graphicsDevice]

You can then also use VZVirtualMachineView in GUI applications for displaying the VMs main monitor and interacting with it. Do note however that although AVF exposes the same device as programs like QEMU do, the GPU will not use paravirtualization! It instead will be software rendered.

Networking

Networking is really easy but there is one annoying issue: you are not allowed to use bridged networking for free. I am not even kidding, even when signing locally for debugging, you can not create a bridged network without being in the Apple Developer Program. I do not quite understand why this restriction exists, but regardless, here is the code:

// Network (NAT)
let networkAttachment = VZNATNetworkDeviceAttachment()
let networkDevice = VZVirtioNetworkDeviceConfiguration()
networkDevice.attachment = networkAttachment
configuration.networkDevices = [networkDevice]
        
// Network (Bridged)
// Or at least, this is how you would do it if you paid 100 dollars a year for it
let networkAttachment = VZBridgedNetworkDeviceAttachment(interface: VZBridgedNetworkInterface.networkInterfaces.first(where: { a in
    a.identifier.elementsEqual("en0")
})!)

Running Linux

With the EFI firmware and the devices listed above, Arch Linux ran perfectly on my system. Linux has all Virtio drivers built-in so there was no hassle in setting that up. Just fire it up, create partitions, and install using pacstrap.

Running macOS

This was my main goal for creating this overly simplistic VM. I wanted to see how well macOS perfoms in an in-house Apple VM. Except, I can't. Apple decided it was not worth supporting Intel CPUs when virtualizing macOS. I can do Linux, but not macOS. To me it seems like this is just a management decision. It's not worth it in their book spending development hours on a platform that will be deprecated in a couple of years anyway.

Trying something more fun: Windows

After installing Arch Linux and finding out I can't run macOS, I decided to see if I could get Windows working on this very limited VM. I started off simple: just a standard Windows 11 ISO. Unfortunately, I got nothing. Not even "Press any key to boot from CD or DVD…..". I realized maybe Windows 11 with its strict and odd hardware requirements is not ideal for such a setup.

Windows 8.1 did show the prompt to boot from CD, but the VM stopped immediately after I pressed a key. On Windows 10 however, the operating system seems to be booting, with one glaring issue: There is no display output!. While booting you can see the old fallback boot screen (the one that say Loading files) but after that finishes I get a black screen. Judging by CPU activity, Windows did actually boot. My best guess is that the EFI implementation does not fully implement GOP (Graphics Output Protocol). Windows PE uses GOP as its output instead of loading any advanced graphics drivers.

Trying to boot a full Windows installation with Virtio drivers installed does also not work however, indicatinng that there is more problems than just bad firmware. The system restarts, then freezes. Force quiting and loading again tries to launch automatic repair (again, with the old loading bar) but does not succeed.

My opinion

I really like the APIs! They are a bit vague, as with any framework from Apple, but once you get the hang of it you have quite a bit of options. But there is this pattern with features in Apple products. The core concepts, ideas and technologies are quite good, but they never release anything feature-complete.

Take a look at the MacBook Touch Bar for example: I personally love my Touch Bar. I think it has an extreme amount of potential, but the APIs for it are really really bad. This is another case where Apple has really failed to hit the mark. The base concepts and ideas introduced in the API are very nice to use. But there are too many quirks, to the point where this framework becomes unusable. Odd restrictions, half-assed firmware, non-compliant virtual hardware, etc. From what I can tell, the restrictions are not with the Hypervisor framework!. QEMU can use HVF to accelerate virtualization and it works totally fine. So the problem really lies with Apple's interpretation of what a VM is.

I was thinking that I should give Apple some time to get things fixed. But then I remembered that this API is 5 years old at this point. They will never revisit the weird decisions and restrictions they made up on release anymore. Too bad, I guess.

xx Leet