Strong ARMing with MacOS: Adventures in Cross-Platform Emulation

Summary:

In a world where adversaries are becoming more sophisticated by the day, it is important that threat hunters can keep a competitive advantage and remain one step ahead of threat actors. Recent developments in Apple® hardware have made it even more difficult for security researchers to keep up, and the demand for ARM-targeted testing environments is increasing.

BlackBerry recognizes the importance of supporting the cybersecurity community in the fight against cyberthreats, and is therefore following up its release of the PE Tree Tool in 2020 by sharing this methodology report to inform security researchers and pen-testers on how to successfully emulate a MacOS ARM64 kernel under QEMU.

Pen-testers and researchers can use the virtualized environment of a stripped-down MacOS kernel for debugging and vulnerability discovery, and this illustrates the extent to which one can use emulation to manipulate and control the kernel to their desired ends, whether it be to find a critical bug or to patch an area of the kernel.

More importantly, this project was a successful experiment in cross-platform emulation that has the potential for future development.

Introduction

Demand for ARM-targeted testing environments is increasing. The first Apple silicon processors are appearing in the market in conjunction with the growing extent of ARM64 support on the most popular operating systems. This project was inspired by a series of recent developments in emulation software and Apple hardware as well as a race to be the first to coalesce them. iOS® kernel emulation on a MacOS host had already been attempted, accomplished, and published. Cross-platform virtualization like this is nothing new: ARM-based systems have been virtualizable on Intel-based host systems as early as 2009.

QEMU, the versatile and dynamic emulator responsible for bringing this practice into practicality, is popular among developers and pen-testers for cross-platform emulation. Even the Android™ emulator is based on QEMU. It was only a matter of time before XNU, Apple's own Unix-derived kernel, joined the party.

Background

When emulating a kernel image, the first phase of the kernel boot stage is typically referred to as the 'bootstrap' phase. This is normally when the earliest kernel output appears and is the first visible output during an emulation session of the MacOS® ARM64e kernel. The MacOS 11.1 ARM64e kernel bootstrap process is shown below:

Fifty seconds, 5086 lines, and 113 kexts later:

All of this is virtualized in a QEMU session, on a Linux® host, running an Intel® Core™ i5-7500 CPU @ 3.40GHz. You can see the full output on our GitHub page:

https://github.com/cylance/macos-arm64-emulation/blob/main/macos-qemu.log

Getting the Files

In June 2020, Apple announced the first beta releases of MacOS 11 (Big Sur) along with universal binary support for both x86-64 and ARM64. Does that mean we can expect to find both the x86-64 and ARM64 kernels in this release?

Yes!

The OSX-KVM project provides a script to download the Big Sur installer package. From there, it was simply a matter of extracting one nested archive after the other to find the kernel image. This script does not have a good track record when it comes to reading Apple's software update catalogs. Therefore, we've provided a link to the kernelcache, ramdisk, and device tree files below:

https://mega.nz/file/GZwzGYKb#HscZIOg_K5JdUIvbLwwwW7_Ntc1z9c7QPOcEQRKwp8c

Note that the next few steps are only necessary if these files are extracted from the installer package referenced below, instead of from the link above. Skip ahead to the Modifying QEMU section, or continue below if you are extracting the files from the installer package:

An archive inside the SFR software update directory with a hash-style name contains the files we need. We must also extract the Mac® software update archive that contains the APFS file system. This file system contains many ARM64e binaries that are not present on the ramdisk, including bash and ls:

It's important to note that the long, hash-style archive file names will vary from version to version. The ramdisk, device tree and kernel files can be easily extracted the SFR archive:

The ramdisk file functions as the operating system. The device tree file identifies the devices for loading the relevant drivers. The kernel, begetter of all running processes, boots the system.

Decoding and Decompressing

Now we've discovered the kernel image, ramdisk image, and device tree binary and can gather the requisite files into a single directory. Next, we move ahead to decode each of the three ASN1-encoded files with these scripts:

The decoded device tree file and kernel image were LZFSE compressed, unlike the LZSS-compressed iOS kernel. LZFSE features a -decode option for such files:

Getting Bash and Other Binaries

The root file system on the ramdisk was missing many common command line tools, including a shell client binary. Even the ls program was completely absent. This brings us to the mac.zip archive extracted earlier. Below are the contents of the AssetData/Restore directory in this archive:

BaseSystem.dmg is for x86_64 installations only. 022-10310-098.dmg is for ARM64e installations only. After extracting the ARM64e installer and examining the contents:

Discovering a bash file within, we check the file type:

It turns out all of the Mach-O binaries in this directory were purely ARM64e executables, as well as those in the /sbin, /usr/bin, and /usr/sbin directories. To fit these binaries into the original ramdisk file, the ramdisk had to be resized. Hdiutil is the only tool for the job, but no port of hdiutil existed outside of MacOS. This means the ramdisk needed to be resized in a MacOS system:

This was the only time throughout the entire project that access to a MacOS system was required. Fortunately, this can be done in a MacOS virtual machine (VM) that can be created with OSX-KVM. We mounted the ramdisk, cleared out its /System/Library/LaunchDaemons directory and transferred the binaries into the ramdisk file system:

We then created a new file at ramdisk/System/Library/LaunchDaemons/com.apple.bash.plist:

Afterwards we copied the following code into it:

Finally, we unmounted the file system and ramdisk:

Then it was time to begin testing.

Modifying QEMU

Since the kernelcache binary already contained all the necessary kexts, it was not necessary to create a kext collection. Thanks to the folks at Aleph Research for providing a modified version of QEMU that supports Apple's XNU kernel. With access to this source, we managed to add support for MacOS on top of the iOS support already implemented.

Building QEMU

We skimmed through the source files in xnu-qemu-arm64 and found two files that specifically targeted the iOS kernel used by the iPhone® 6s Plus: include/hw/arm/n66_iphone6plus.h and hw/arm/n66_iphone6splus.c. These files target a very specific iOS kernel: N66, build 16B92. The definitions and configurations in these files would likely be incompatible with the kernel we were using (J273, build 20C69), let alone any macOS kernel. To add additional support for the MacOS kernel, we:

• Copied these files;
• Renamed the variables, functions, and preprocessor directives to match the names of the MacOS kernel (J273), kernel version (20C69), and chipset (A21Z); and
  
• Updated the filenames in the QEMU command line:

As in the original xnu-qemu-arm64, we included the generated object files in hw/arm/Makefile.objs:

After this, we began the QEMU build. Unfortunately, and unsurprisingly, the compiler produced an error:

But what if the linker doesn't even need this qemu-pr-helper.c module or any other potentially unbuildable modules? Let's run make again with the -k flag and CFLAGS='-Wno-error':

This appears to have succeeded. However, we are not out of the woods yet.

Switch to QEMU 5.1.0

QEMU 5.1.0 supports the LDAPR instruction. QEMU 4.2.0 does not. The official ARM documentation states the following about this instruction:

This instruction is supported in architectures ARMv8.3-A and later. It is optionally supported in ARMv8.2-A with the RCpc extension.

Xnu-qemu-arm64 is based on 4.2.0. QEMU 4.2.0 has very limited support of ARMv8.3 and no support for the LDAPR instruction. The immediate task ahead was to move all the xnu-related source files over to a freshly downloaded source of QEMU 5.1.0. Below is a Git diff showing the files added to the official QEMU 5.1.0 source from xnu-qemu-arm64:

In addition to the following modified files in the official source:

One particular function proved problematic:

Due to changes in the source from QEMU 4.2.0 to 5.1.0, memory_region_allocate_system_memory had to be changed to memory_region_init_ram. It takes the same arguments in the same order plus one extra (&error_fatal). The full Git diff file can be downloaded below:

https://github.com/cylance/macos-arm64-emulation/blob/main/xnu-qemu-arm64-5.1.0.diff

To apply the diff and build the modified QEMU source:

• Download the source for QEMU 5.1.0 to the same directory as the Git diff file
• Extract it
• Rename it to xnu-qemu-arm64-5.1.0
• Apply the Git diff:

Configure the source and build it per the instructions provided by Aleph Research:

No bypasses or build flags are necessary.

QEMU Dry Run

The next thing to determine is: will it run? The run.sh script below has the updated QEMU command line, where we enabled remote kernel debugging with the -S -s option:

Attaching the remote debugger on the same host:

0x47ac4580 is the entry point to our MacOS 11 Big Sur ARM64e kernel image. On entry, addresses seen in QEMU will be physical addresses. Since this is an initial dry run, we let it loose:

Or, in this case, we let it spin around endlessly on the same three instructions:

Bit 1 (#0x2) is never set in the system coprocessor register s3_4_c15_c0_4, so it never breaks the loop. This is an Apple-specific hardware register. Apple registers are unrecognized by the official QEMU branch, but xnu-qemu-arm64 added several Apple registers to boot the iOS kernel, including s3_4_c15_c0_4. This register is also known as APCTL_EL1/MIGSTS.

Patching the Kernel

That dry run barely got us on our feet. Not easily discouraged, we began skimming the QEMU source files for clues. After looking at the patching function we disabled, we could find nothing directly addressing the elusive 'APCTL_EL1' register. The infinite loop above does show up in the XNU source in xnu-6153.141.1/osfmk/arm64/start.s:

It is polling a flag by the name of MKEYVld.

MKEYVld

What is the MKEYVld flag? Not many clues are in the XNU source. Perhaps a flag indicating some kind of validation status (MKEYVld = MAC key validated?). Most likely the kernel is waiting for it to be set by some other piece of hardware. We can force set this flag ourselves by adding to the following patch already provided by Aleph Research in our copied hw/arm/j273_macos11.c:

We introduced two more instructions that set the MKEYVld flag (bit 1) in APCTL_EL1:

The presence of several fixed offsets in xnu-qemu-arm64/hw/arm/n66_iphone6splus.c shows that there were 11 places in the iOS kernel that required patching:

There was no way these hard-coded offsets would be compatible with the MacOS kernel image. This is when we realized we would need to tear the MacOS kernel apart in a disassembler to find the offsets.

IDA

Completing this project would have been impossible without a disassembler. IDA 7.5 was the primary candidate, chiefly because of its support for ARM64e binaries and the latest A64 instruction set. For instance, earlier versions of IDA (namely 7.0) do not recognize the pointer authentication code for instruction key B (PACIBSP) instruction, which appears at the start of nearly every function in the MacOS ARM64e kernel:

Figure 1.

Moreover, the kernel image used in this project contained no symbols. Functions had to be manually named, one by one, throughout the two-month testing and research period. ASCII strings offered the most reliable clues. The open-source XNU kernel was crucial in the struggle to identify the culprit of a crash or freeze. A total of 122645 functions have been defined in the IDA project so far. On initial analysis, however, IDA failed to define nearly every single function in the kernel binary. A script was needed to rectify this:

The kernel image, kernelcache.release.j273.out, is ~83MB. While the script only took around 10 minutes to execute, the resulting mass of new functions and cross-references took over an hour to finish generating. After weeks of research and testing, all patches were written, and offsets defined:

MSR Instructions

QEMU is not equipped to emulate ARM-based Apple systems. Over 110 of Apple's model-specific hardware registers (MSR), in addition to hundreds of others, are currently unrecognized by QEMU. The Aleph Research team added support for 12 Apple-specific registers required for the iOS kernel to boot. To reach the goal of fully booting the MacOS ARM64 kernel, 24 more hardware registers needed support. But which registers would we need to add?

Finding the Necessary Registers

Getting to that bash prompt after two grueling months of research and testing was anything but a straightforward process. Unsupported MSR registers tended to pop up intermittently as we diagnosed and fixed one crash after another. We typically followed a 'panic, crash and patch' strategy, adding register support for individual MSR's on an ad hoc basis. The list of definitions below from hw/arm/j273_macos11.c are the result:

Unrecognized system registers typically appear as s#_#_c#_c#_# in various debuggers, where # corresponds to the arguments in each of the definitions above. For example, ARM64_REG_EHID1 or APRR_EL0 are parsed as s3_0_c15_c3_1 and s3_4_c15_c2_0, respectively. The last argument PL1_RW means 'exception level 1 read/write', which specifies the privilege level of the given registers. This indicates that the register is accessible in exception level 1 (EL1). Yet the level 2 (EL2) registers are marked with PL1_RW. This is because the following line in the function j273_cpu_setup prohibits EL2 registers:

Setting this to true proved problematic, as QEMU then became unable to switch from physical to virtual addressing early in the boot process. The fix involved forcing QEMU to treat these registers as EL1 registers in define_one_arm_cp_reg_with_opaque (target/arm/helper.c):

We were initially apprehensive of this fix, as modifying any official QEMU source files to bypass errors may prove to be a dangerous endeavor. Fortunately, no calamities arose, and eventually all MSR registers were accounted for.

Device Tree

The device tree (DeviceTree.j273aap.im4p.out) was responsible for over half of the panics during testing. Several properties and devices were either absent from the tree entirely, usually causing a crash, or needed to be manually adjusted to prevent later issues. We have written a program that can apply the necessary changes to a device tree file using a diff-style file:

https://github.com/cylance/macos-arm64-emulation/tree/main/dtetool

To create a compatible device tree, back up the device tree file and apply the changes specified in dtediff_20C69 with the dtetool program:

The following section provides a more detailed description of each modification.

Device Tree Modifications

The following property is changed to 'running' to avoid an infinite loop in pe_identify_machine:

The first element in arm-io/ranges is changed to 0x100000000:

The following node is removed to ignore dockchannel-uart in order to use the default uart0 in serial_init:

The following properties are added to the 'chosen' node to avoid the panics at the end of arm_init:

The following properties are added to the lock-regs node to avoid the panic in subroutine 0xfffffe0007b2af00:

One final device tree property is modified: the nvram.

NVRAM

No external NVRAM file is necessary, as the device tree file can house all NVRAM data in a property called nvram-proxy-data. By default, this property is completely empty, which led to several problems later (i.e. panics), which were related to null pointers. Crafting and configuring the NVRAM to the kernel's liking was a time-consuming task, albeit with a fairly simple outcome. Several issues cropped up in succession, and additional changes had to be made.

Null Pointers

The first of the NVRAM-related issues happened in IODTNVRAM::init, where a null pointer reference to a lock variable caused a panic. This lock variable was supposed to have been initialized in IODTNVRAM::initNVRAMImage, but this function was never called. We discovered that the device-tree/chosen/nvram-total-size property in the device tree file was zero. Another panic occurred due to a null pointer reference in the IODTNVRAM::initOFVariables function. Apparently the nvram partition dictionary was not being set due to missing partition information in the device tree's nvram data.

Nvram-Proxy-Data

The solution to this problem was to tailor the device-tree/chosen/nvram-proxy-data and device-tree/chosen/nvram-total-size properties to the kernel's needs. Clues as to what format this data must be in were given in IODTNVRAM::initNVRAMImage:

When initialized, the NVRAM must be a valid, non-zero size no greater than 65536. This is specified in the nvram-total-size property. In addition to a valid size the nvram must have at least one valid partition with a size of at least 32 bytes. Valid partition names include 'common' (defined as kIODTNVRAMOFPartitionName) or 'system' (not in the most recent XNU source). Below is the updated device tree data for nvram-proxy-data:

Actual data starts at offset 0xf90 in our modified device tree file. The partition's size is calculated by multiplying the 16-bit integer at offset 0xf92 by 16. In this case, 2 * 16 = 32 bytes. This includes the first 16 bytes containing the partition's name and the remaining 16 null bytes. Now that the kernel is satisfied with our empty partition, the mystery of the missing NVRAM is solved.

Forcing JOP

All the required MSR registers had been added to QEMU. The device tree was properly tailored to the kernel's requirements. The following launchd greeting in a GDB session gave a small boost of optimism:

Thu Jan 1 00:02:10 1970 localhost com.apple.xpc.launchd[1] : hello

The holy grail of a shell prompt is still beyond our grasp at this point. Something was spinning in the kernel, impeding progress once again:

In an infinite loop at 0xfffffe00079ebcbc:

Figure 2.

Notice the infinitely looping B.NE instruction. How did it get here? We looked a bit further up in the disassembly:

Figure 3.

XNU kernel threads have a member called TH_DISABLE_USER_JOP. If set to a non-zero value, Ldisable_jop is invoked. SCTLR_EL1 (system control register) is then validated against a constant (0x7454599d) and freezes execution if the values do not match. Where is this thread property being set, and how can we prevent it in the most orthodox manner possible? Setting a write watchpoint in gdb for the address of the blocking thread's TH_DISABLE_USER_JOP property leads to this location in posix_spawn:

Figure 4.

In posix_spawn (xnu-6153.141.1/bsd/kern/kern_exec.c):

In the case of the blocking thread, imgp->ip_flags is set to 0x80000000, which indicates the IMGPF_NOJOP flag is enabled. The enabled status of this flag is written to the thread's TH_DISABLE_USER_JOP property. So where is imgp->ip_flags set?

Load_machfile

IMGPF_NOJOP is enabled in load_machfile near the end of the function:

Figure 5.

Load_machfile compares the Mach-O executable's identifier against several strings and enables IMGPF_NOJOP if any are a match:

The bash identifier is among them. This flag may be a security mechanism to prevent certain executables from loading at boot time, as they may be used to compromise the system. Among them are several shell clients and script interpreters. The kernel blocks the executing thread if the Mach-O file's identifier matches any of the above strings.

Solution

NOP over the ORR W8, W8 #0x80000000 instruction to keep IMGPF_NOJOP disabled. This gave the go-ahead to the kernel to allow launchd to execute bash (via xpcproxy):

A prompt appeared at last. Yet stdin is not working and there is no keyboard input.

Serial Keyboard and FIQ

A fully functioning bash prompt is useless without keyboard input. Keyboard input is read in a separate thread in a function called serial_keyboard_poll. The serial_keyboard_poll:

• Reads all pending characters from the stdin buffer
• Sets the 16-millisecond deadline by calling assert_wait_deadline
• Blocks execution in thread_block until the deadline has passed

serial_keyboard_poll is then re-invoked by thread_invoke, and this sequence of events repeats indefinitely for as long as bash is open.

Figure 6.

Yet this wasn't happening. Thread_block was called only once, the thread stalled, and serial_keyboard_poll was never invoked again.

We thought: maybe the deadline wasn't being reached? Wait deadlines are specified in the assert_wait_deadline function. This function creates a waitq object for the current keyboard-polling thread with an optional deadline, in nanoseconds. When thread_block is called shortly afterwards, the thread hangs until the thread_clear_waitq_state kernel function clears the thread's waitq object:

But this function never got called, and the deadline was never reached. What does the thread do? It switches context to the system idle thread and blocks indefinitely, never invoking a continuation of serial_keyboard_poll to continue polling for stdin. Surely something somewhere is called that may lead to thread_clear_waitq_state? By back tracing the sequence of function calls in the XNU source, we discovered the origin of interrupt timers:

After analyzing this call flow, we were inching ever closer to the source of the problem. Analyzing the interoperability of threads, timers, clocks, and interrupts in the XNU kernel was slowly paying dividends.

Fast Interrupt Requests (FIQ)

Further investigation revealed that not even fleh_fiq (first-level exception handler for Fast Interrupt Request) was being called. Something was seriously wrong, as fleh_fiq is integral to a working interrupt timer. Peripheral devices like keyboards and mice typically communicate with the kernel using Fast Interrupt Requests (FIQ) in the ARM architecture. We confirmed that FIQs were not firing in the emulator after looking at QEMU's source. We discovered that arm_cpu_exec_interrupt, which is called for a fast interrupt request, never got called. No amount of keyboard-spamming would trigger this or any of the kernel functions listed above.

Hardware Timers

Perhaps another possibility for a non-operational FIQ handler was timer-related? Threads often rely on hardware timers to notify the thread of a passed deadline. If no timer exists, no countdowns can be performed and idle threads waiting for a deadline will stall the system. Without a hardware timer to poll for FIQs, MacOS will ignore them. This is significant, as it would have affected not only standard keyboard input, but other areas of the kernel as well. Wait deadlines would never be reached, threads would hang, and services would never start simply because no hardware timer was present.

Enable_timebase_event_stream

We did a comparison between the iOS and MacOS kernel for the enable_timebase_event_stream function. Why this function? Because this function modifies three timer-related system registers and could provide the answer to our timer dilemma. Below is from the MacOS kernel:

Contrast the above with the corresponding iOS kernel disassembly:

We immediately noticed the presence of an extra register in the MacOS kernel: CNTV_CTL_EL0. With the help of a comprehensive list of iOS ARM64 system registers, we began to make the connection between these registers and the timer issues. The iOS kernel appeared to be enabling the physical timer by writing 1 to it, while the MacOS kernel was enabling the virtual timer by writing 1 to it. Then we noticed the line in the xnu-qemu-ARM64 source:

The GTIMER_VIRT constant gave it away almost instantly. QEMU's official source lists five different global timer types in target/arm/cpu.h:

The solution was incredibly simple: MacOS uses a virtual timer. iOS uses a physical timer. Therefore, QEMU must specify a virtual timer with GTIMER_VIRT instead of GTIMER_PHYS when linking the timer to FIQ. Switching to GTIMER_VIRT:

Followed by booting up the MacOS kernel, waiting for the bash prompt, then:

Such a simple answer to a cryptic problem. This final fix marks the end of the final phase of a fully bootable MacOS 11 ARM64e kernel.

Achieving a Functioning Emulator

This chronicling of discoveries, fixes and accomplishments would not be complete without long-term failures and ineffective bypasses. Below are several examples that involved multiple days of research and testing and created quite a bit of frustration throughout.

IORTC

Before we discovered, diagnosed, and mitigated the timer dilemma, something related to the initialization of the RTC (real-time clock) had been blocking the bsd_init thread. IOKitInitializeTime was waiting for a matching IORTC service. The wait was initiated by IOService::waitForMatchingService, which relies on assert_wait or assert_wait_deadline to begin blocking the thread until a condition is met. assert_wait_deadline is non-functional without a working hardware timer. The kernel had no working hardware timer. We tried adding the no-rtc property to the device tree root and a child node (with the name rtc) to the arm-io node. This would invoke the bootstrap to immediately publish the IORTC service and skip the wait. After the real source of the problem was determined and fixed, these device tree nodes were removed.

Task-access Server and SIP

The following message from the launchd output was a bit disconcerting:

It was originally attributed to code-signing, then attributed to SIP, and finally ignored once a functioning bash prompt was operational. This message caused several unneeded headaches. A task-access server is related to communication between tasks over the task-access port (defined as constant TASK_ACCESS_PORT with a value of 9). Assuming this only affected TCP connections, which this emulation project currently does not support, we ignored the error.

iOS Binary Incompatibility

RootlessJB provides common Mach-O ARM64e binaries, including bash, that are runnable in the iOS kernel. These are iOS binaries for an iOS kernel. This explains why one is likely to see the following message when attempting to run said binaries on a MacOS system and never see a bash prompt:

The necessary command line tools are part of the base ARM64 system, archived deep within the MacOS Big Sur installer package. This is simply a warning to those attempting execution of the aforementioned iOS binaries in a MacOS environment: it probably won't work.

Conclusion

This project was a successful experiment in cross-platform emulation that has potential for future development. Hard disk and TCP tunneling (which xnu-qemu-arm64 already supports for iOS) still await implementation. Multi-core and KVM support would dramatically reduce the boot time, perhaps to mere seconds, and eliminate massive overhead. Full graphical support is a mere prospect (even less so in a cross-platform environment). But graphical support is low priority, so long as a functioning shell client is present. If it works, that is enough of a motivation to make it work well.

Example Commands

To complement the article, we have decided to provide examples of command output from an emulated MacOS 11 ARM64e guest. For example, the following shows output from the lsof program:

The df command lists active file systems, such as the root device where launchd, bash, and all other userland programs are located:

View network interfaces with ifconfig:

Finally, the full output of a shutdown command:

The BlackBerry Research and Intelligence team examines emerging and persistent threats, providing intelligence analysis for the benefit of defenders and the organizations they serve.