Low-level · C++

Ring -1: the hypervisor watching over your kernel

On this page
  1. "Virtualization support not detected"
  2. What a hypervisor is
  3. How the hardware does it: VT-x and AMD-V
  4. Memory of memory: EPT and NPT
  5. The hypervisor as the kernel's guardian
  6. Check it yourself with CPUID
  7. Does your CPU support virtualization?
  8. Is there a hypervisor beneath you?
  9. This is what the cloud is
  10. A rule you learn the hard way
  11. And there's still more below

"Virtualization support not detected"#

While preparing this post, Docker Desktop refused to start on my machine with a curt message: Virtualization support not detected. Odd, because the processor is an Intel Core i9-14900KF, which has hardware virtualization to spare, and it's enabled in the BIOS. Windows confirms it:

output
PS> Get-CimInstance Win32_Processor |
>>     Format-List Virtualization*, VMMonitor*, SecondLevel*
VirtualizationFirmwareEnabled           : True
VMMonitorModeExtensions                 : True
SecondLevelAddressTranslationExtensions : True

PS> (Get-CimInstance Win32_ComputerSystem).HypervisorPresent
False

The key is the last line. The hardware is ready, but no hypervisor is loaded: nobody occupies the level below the kernel. Docker Desktop on Windows usually runs on WSL2, and WSL2 is a lightweight virtual machine that needs Windows to boot its own hypervisor. Without it, there are no containers.

In the previous post we went down from Ring 3 to Ring 0 and said the kernel is in charge. Today we need to qualify that: there can be someone else below the kernel. It's informally called Ring -1.

What a hypervisor is#

A hypervisor is the software that creates and manages virtual machines: it shares the CPU, memory and devices among several operating systems and makes each one believe it has the machine to itself. Two types are usually distinguished:

Type 1 (bare metal)Type 2 (hosted)
Where it runsDirectly on the hardwareAs an application inside another operating system
ExamplesHyper-V, VMware ESXi, XenVirtualBox, VMware Workstation
Typical useServers, cloud, securing the OS itselfDesktop, testing, development

The line isn't as sharp as it looks. KVM, for instance, is a kernel module that turns Linux itself into a hypervisor. And when you enable Hyper-V on Windows, something happens that surprises many people: your Windows stops running directly on the hardware and becomes the root partition, a virtual machine with special privileges running on top of Hyper-V.

RING 3 guest applications RING 0 guest kernel (Windows, Linux...) ▲ VMX non-root VM exit ▼ ▲ VM entry VMX root ▼ RING -1 hypervisor · VMCS/VMCB · EPT/NPT tables HARDWARE CPU with VT-x or AMD-V · memory · devices
Fig. 1 — The guest keeps its rings 3 and 0, but all of it lives in VMX non-root. The hypervisor sits below.

How the hardware does it: VT-x and AMD-V#

Virtualizing x86 purely in software was slow and complicated, because the guest expects to run in Ring 0 and freely touch control registers, page tables and interrupts. Since 2005–2006, Intel (VT-x) and AMD (AMD-V, also called SVM) solve it in hardware with a new execution mode, independent of the rings:

  • VMX root: where the hypervisor lives. That's what we call Ring -1, although technically it isn't a new ring but a CPU mode.
  • VMX non-root: where the guest lives, with its Ring 0 and Ring 3 intact. The guest kernel still runs in Ring 0, but under supervision.

The hypervisor describes each virtual machine in a memory structure (the VMCS on Intel, the VMCB on AMD) holding the guest's register state and, above all, which guest operations it wants to intercept. When the guest performs one of them — accessing certain CPU registers, an I/O port, or memory that isn't its own — the CPU stops it and jumps to the hypervisor. That's a VM exit. The hypervisor decides what to do and hands control back with a VM entry.

The cost of a VM exit

Every VM exit saves the guest's state, loads the hypervisor's, and later makes the trip back. Depending on the CPU and the work the hypervisor has to do, that costs on the order of hundreds to a few thousand cycles, compared with the handful of cycles of a normal instruction. Some instructions, such as CPUID under Intel VT-x, always cause a VM exit. That's why much of a modern hypervisor's work is about avoiding exits: the less it intervenes, the faster the guest runs.

Memory of memory: EPT and NPT#

In the previous post we saw that every process uses virtual addresses that the kernel translates to physical ones with its page tables. In a virtual machine that's no longer enough: the "physical memory" the guest sees isn't real either. A second translation is needed:

  1. Guest virtual address → guest physical address. Done by the guest kernel, with its own page tables, as usual.
  2. Guest physical address → real physical address. Done by the hypervisor, with a second table the guest can neither see nor touch.

Intel calls that second table EPT (Extended Page Tables) and AMD calls it NPT (Nested Page Tables); the generic name is SLAT. Before it existed, around 2007–2008, hypervisors maintained "shadow page tables" in software that had to be resynchronized every time the guest changed its own, and that was very expensive. With SLAT, the hardware walks both tables by itself. It isn't free: in the worst case, a TLB miss with four-level paging can take up to 24 memory accesses instead of 4. That's why translation caches matter so much in virtualization.

The interesting part for security is that every EPT entry has its own read, write and execute permissions, independent of whatever the guest sets. The hypervisor can decide a page can't be modified, even if the guest kernel, from its Ring 0, says otherwise.

The hypervisor as the kernel's guardian#

That last idea is the foundation of Windows' Virtualization-Based Security (VBS). With VBS enabled, Hyper-V splits the system into two virtual trust levels: regular Windows, with its kernel, at level 0 (VTL0), and a small secure kernel at level 1 (VTL1), isolated by the hypervisor. Two of the most important defenses in today's Windows are built on top of it:

  • HVCI (Memory Integrity): kernel code signature checks happen in VTL1, and the hypervisor uses SLAT so that no kernel page is ever both writable and executable. Even if an attacker manages to run code in Ring 0, they can't inject new code into the kernel: the page won't allow it.
  • Credential Guard: authentication secrets are kept in an isolated process in VTL1, out of reach even of the Windows kernel itself.

It's the direct answer to the idea we closed the previous post with: if Ring 0 doesn't forgive, put someone below it to watch over Ring 0. On many new Windows 11 machines, VBS and HVCI are enabled out of the box.

Check it yourself with CPUID#

CPUID is the instruction any program uses to ask the CPU what it is and what it can do. It needs no privileges, so we can use it from Ring 3. First, a small portable header for GCC, Clang and MSVC:

cpuid.hpp
// portable CPUID access (GCC/Clang and MSVC)
#pragma once
#include <cstdint>
#include <cstring>
#include <string>
#if defined(_MSC_VER)
#include <intrin.h>
#else
#include <cpuid.h>
#endif

struct CpuidRegs { std::uint32_t eax, ebx, ecx, edx; };

inline CpuidRegs cpuid(std::uint32_t leaf) {
    CpuidRegs r{};
#if defined(_MSC_VER)
    int out[4];
    __cpuid(out, static_cast<int>(leaf));
    std::memcpy(&r, out, sizeof out);
#else
    __cpuid(leaf, r.eax, r.ebx, r.ecx, r.edx);
#endif
    return r;
}

// Joins three registers into the 12-byte ASCII text they encode.
inline std::string cpuid_text(std::uint32_t a, std::uint32_t b,
                              std::uint32_t c) {
    char s[12];
    std::memcpy(s, &a, 4);
    std::memcpy(s + 4, &b, 4);
    std::memcpy(s + 8, &c, 4);
    std::string t(s, sizeof s);
    t.erase(t.find_last_not_of('\0') + 1);   // strip trailing padding
    return t;
}

Does your CPU support virtualization?#

virt.cpp
// g++ -std=c++20 virt.cpp -o virt
#include "cpuid.hpp"
#include <format>
#include <iostream>

int main() {
    const CpuidRegs v = cpuid(0);                         // vendor
    const bool vmx = cpuid(1).ecx & (1u << 5);            // Intel VT-x
    const bool svm = cpuid(0x8000'0001).ecx & (1u << 2);  // AMD-V
    const std::string vendor = cpuid_text(v.ebx, v.edx, v.ecx);
    auto yes = [](bool b) { return b ? "supported" : "not supported"; };

    std::cout << std::format("CPU:         {}\n", vendor)
              << std::format("VT-x (VMX):  {}\n", yes(vmx))
              << std::format("AMD-V (SVM): {}\n", yes(svm));
}

These are the real outputs on my machine and inside a GitHub Actions runner, which is a virtual machine on Azure:

output
# Intel Core i9-14900KF (physical hardware)
CPU:         GenuineIntel
VT-x (VMX):  supported
AMD-V (SVM): not supported

# GitHub Actions runner (Azure VM, AMD EPYC 7763)
CPU:         AuthenticAMD
VT-x (VMX):  not supported
AMD-V (SVM): supported

Two important nuances. First: the CPU supporting virtualization doesn't mean it's enabled. That's decided by the BIOS in a special CPU register (an MSR) that can only be read from Ring 0; that's why Windows checks it from the kernel and shows it to you in Task Manager, under Performance → CPU → Virtualization. Second: inside a virtual machine, CPUID answers whatever the hypervisor chooses to expose. The Azure runner showing AMD-V means that hypervisor allows nested virtualization: creating virtual machines inside the virtual machine.

Is there a hypervisor beneath you?#

hv.cpp
// g++ -std=c++20 hv.cpp -o hv
#include "cpuid.hpp"
#include <format>
#include <iostream>

int main() {
    // Leaf 1, ECX bit 31: reserved for a hypervisor to announce itself.
    if (!(cpuid(1).ecx & (1u << 31))) {
        std::cout << "No hypervisor announced: running on bare metal.\n";
        return 0;
    }
    // Leaf 0x40000000: the hypervisor's 12-character signature.
    const CpuidRegs h = cpuid(0x4000'0000);
    std::cout << std::format("Hypervisor: \"{}\"\n",
                             cpuid_text(h.ebx, h.ecx, h.edx));
}
output
# Intel Core i9-14900KF (physical hardware)
No hypervisor announced: running on bare metal.

# GitHub Actions runner (Azure VM)
Hypervisor: "Microsoft Hv"

"Microsoft Hv" is Hyper-V's signature: Azure runs on a customized version of Hyper-V. On Linux with KVM you'd see KVMKVMKVM, and on VMware, VMwareVMware. And if you run it on your own Windows with VBS enabled, you'll also see "Microsoft Hv": your "real" system is Hyper-V's root partition.

Notice that the program checks bit 31 first: without a hypervisor, leaf 0x40000000 is undefined and whatever it returns means nothing. That bit is also a voluntary declaration. Hypervisors set it so the guest knows it's virtualized and can use mechanisms designed for it (known as paravirtualization), such as clocks and devices that don't need to emulate legacy hardware.

This is what the cloud is#

When you rent a machine in the cloud, your entire operating system lives in VMX non-root. Azure runs on Hyper-V, as we just saw; AWS uses Nitro, a KVM-based hypervisor, and Google Cloud also uses KVM. Your "server" is really a guest, and someone else's Ring -1 is what guarantees that your machine and your neighbor's can't see each other.

A rule you learn the hard way#

Personal note

A while ago I wrote an academic hypervisor on Intel VT-x as a Windows kernel driver: it enables VMX on every core, virtualizes the very system that's already running, and places beneath it an EPT table that, for now, maps every address to itself. It's the approach of research projects like HyperPlatform and, at heart, the same one VBS uses, except Windows does it from boot. The lesson that took me longest to accept is in my own lab guide, in a warning box: only one hypervisor fits in VMX root. If Hyper-V, WSL2 or VBS are already active, yours can't get in; and if you switch Windows' off to make room (bcdedit /set hypervisorlaunchtype off), you lose WSL2 and Docker. It isn't a bug, it's a decision you have to make. That's why testing was done inside a virtual machine with nested virtualization.

If Docker or WSL2 say there's no virtualization
  1. Check in Task Manager (Performance → CPU) that virtualization shows as enabled. If it doesn't, turn it on in the BIOS/UEFI.
  2. Enable Windows' Virtual Machine Platform, for example with wsl --install from an administrator terminal, and reboot.
  3. If you ever turned off Windows' hypervisor, turn it back on with bcdedit /set hypervisorlaunchtype auto and reboot.

And there's still more below#

Ring -1 isn't the bottom either. Below the hypervisor sits System Management Mode (SMM), sometimes called Ring -2: a CPU mode reserved for firmware, entered through special interrupts, that runs code in a memory region not even the hypervisor can see. And outside the main CPU live microcontrollers such as Intel ME or AMD PSP, which some call Ring -3. But that's another story.

Next time Docker tells you it can't find virtualization, you'll know what it means: the hardware is there, but nobody is occupying Ring -1.

References

  1. Intel® 64 and IA-32 Architectures Software Developer's ManualsIntel · intel.com
  2. Hypervisor Top Level Functional Specification: Feature and Interface DiscoveryMicrosoft · learn.microsoft.com
  3. Hypervisor Top Level Functional Specification: Virtual Secure ModeMicrosoft · learn.microsoft.com
  4. KVM CPUID bitsLinux kernel documentation · docs.kernel.org

Share

Keep reading

Low-level

Anatomy of a Windows driver: from DriverEntry to the IRP

What happens inside a Windows driver: DriverEntry, IRQL, an IRP's journey, the dispatch table, IOCTLs and why a misused spinlock brings the system down. With real code.

· 12 min

Low-level · Part 3

Ring -2: SMI, SMRAM and RSM, how firmware takes over the CPU below the hypervisor

What lives below the hypervisor: System Management Mode, the firmware code that runs hidden from the OS, and how to check from C++ which protections your firmware declares.

· 12 min

Does your project need to go below the application layer?

Systems software, performance and security on Windows and Linux: diagnostics, custom tooling and low-level code review.

Let's talk→

Comments

Commenting requires a GitHub account. Comments live in GitHub Discussions (SoftDryzz/blog-comments) and are moderated: no promotional links.