Snapshot restore fails on Intel hosts when intr_shadow is set

[glitzflitz]

VM snapshot restore fails on Intel hosts with:

$ pfexec ./propolis-standalone --restore arch-1767964216637.tar
Jan 09 13:12:41.257 INFO restoring snapshot of VM from arch-1767964216637.tar
Jan 09 13:12:41.258 INFO Creating VM with 4 vCPUs, 3221225472 lowmem, 1073741824 highmem
Jan 09 13:12:42.571 INFO VM created, name: arch
Jan 09 13:12:42.593 INFO Bootorder declared as Some(["block0", "net0"])
Jan 09 13:12:42.593 INFO Generating E820 map for guest address space
Error: Failed to restore device state for bhyve-vcpu-0

Caused by:
    0: IO Error
    1: Invalid argument (os error 22)
    1: Invalid argument (os error 22)

This occurs when the source vCPU was in an interrupt shadow state (intr_shadow: true) at snapshot time.

[glitzflitz]

Running truss to trace the ioctls

$ pfexec truss -t ioctl  ./propolis-standalone --restore arch-1767964216637.tar 
ioctl(3, DTRACEHIOC_ADDDOF, 0xFFFFFC7FFFDF42D8) = 0
ioctl(1, TCGETA, 0xFFFFFC7FFFDF3410)            Err#22 EINVAL
/1:     ioctl(3, DTRACEHIOC_ADDDOF, 0xFFFFFC7FFFDF2EE8) = 1
/1:     ioctl(3, (('V'<<16)|('M'<<8)|4), 0x00000000)    = 18
/1:     ioctl(3, (('V'<<16)|('C'<<8)|3), 0x00000000)    = 4
/1:     ioctl(3, DTRACEHIOC_ADDDOF, 0xFFFFFC7FFFDF40C8) = 2
/1:     ioctl(3, DP_EPOLLCOMPAT, 0x00000000)            = 0
/1:     ioctl(11, VMM_CREATE_VM, 0xFFFFFC7FFFDEFB58)    Err#17 EEXIST
/1:     ioctl(11, VMM_DESTROY_VM, 0xFFFFFC7FFFDEF850)   = 0
/1:     ioctl(11, VMM_CREATE_VM, 0xFFFFFC7FFFDEFB58)    = 0
/3:     ioctl(3, DP_POLL, 0xFFFFFC7FEDFFE0B0) (sleeping...)
/1:     ioctl(10, VM_ALLOC_MEMSEG, 0xFFFFFC7FFFDEFA80)  = 0
...
/1:     ioctl(10, VM_SET_CAPABILITY, 0xFFFFFC7FFFDF07E8) = 0
/1:     ioctl(10, VM_ACTIVATE_CPU, 0xFFFFFC7FFFDF290C)  = 0
/1:     ioctl(10, VM_RESET_CPU, 0xFFFFFC7FFFDF2908)     = 0
...
/10:    ioctl(3, DP_POLL, 0xFFFFFC7FEB5FE0B0) (sleeping...)
/1:     ioctl(10, VM_SET_RUN_STATE, 0xFFFFFC7FFFDF1828) = 0
/1:     ioctl(10, VM_SET_REGISTER, 0xFFFFFC7FFFDF1820)  Err#22 EINVAL
/10:    ioctl(3, DP_POLL, 0xFFFFFC7FEB5FE0B0)           = 1

so VM_SET_REGISTER fails with EINVAL

[glitzflitz]

Looking at code that sets the register we get to
https://github.com/oxidecomputer/propolis/blob/master/lib/propolis/src/vcpu.rs#L974

        fn write(self, vcpu: &Vcpu) -> Result<()> {
            vcpu.set_run_state(self.run_state, Some(self.sipi_vector))?;
            vcpu.set_reg(
                vm_reg_name::VM_REG_GUEST_INTR_SHADOW,
                u64::from(self.intr_shadow),
            )?;

so set_reg fails for VM_REG_GUEST_INTR_SHADOW

Let me check how VM_REG_GUEST_INTR_SHADOW is handled in bhyve

[glitzflitz]

In vmx_setreg() https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/intel/io/vmm/intel/vmx.c#L3149

if (reg == VM_REG_GUEST_INTR_SHADOW) {
    if (val != 0) {
        /*
         * Forcing the vcpu into an interrupt shadow is not
         * presently supported.
         */
        err = EINVAL;
    } else {
        // Clear interrupt shadow (val == 0) - this IS supported
        uint64_t gi;
        gi = vmcs_read(VMCS_GUEST_INTERRUPTIBILITY);
        gi &= ~HWINTR_BLOCKING;
        vmcs_write(VMCS_GUEST_INTERRUPTIBILITY, gi);
        err = 0;
    }
}

So this is how VM_SET_REGISTER returns EINVAL.

In case of AMD though https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/intel/io/vmm/amd/svm.c#L2214

	case VM_REG_GUEST_INTR_SHADOW:
		vmcb->ctrl.intr_shadow = (val != 0) ? 1 : 0;
		break;

looks like it accepts any value.

[glitzflitz]

As per https://cdrdv2-public.intel.com/825750/326019-sdm-vol-3c.pdf in section 27.7.1 Interruptibility State

f the VM entry is not vectoring, the following apply:
— Events are blocked by STI if and only if bit 0 in the interruptibility-state field is 1. This blocking is cleared
after the guest executes one instruction or incurs an exception (including a debug exception made pending
by VM entry; see Section 27.7.3).
— Events are blocked by MOV SS if and only if bit 1 in the interruptibility-state field is 1. This may affect the
treatment of pending debug exceptions; see Section 27.7.3. This blocking is cleared after the guest
executes one instruction or incurs an exception (including a debug exception made pending by VM entry)

In simpler words, event blocking by both STI or MOV SS is cleared after guest executes one instruction or incurs an exception.
Looking at bhyve,
https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/intel/io/vmm/intel/vmx.c#L1211

#define	HWINTR_BLOCKING	(VMCS_INTERRUPTIBILITY_STI_BLOCKING |		\
			VMCS_INTERRUPTIBILITY_MOVSS_BLOCKING)

So I think we can simply ignore EINVAL when intr_shadow=true since the state is transient (one instruction) and in the worst case an interrupt may arrive one instruction early.