tdx: use timer virtualization for lower VTLs #2483
Conversation
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
There was a problem hiding this comment.
Pull request overview
This PR implements hardware timer virtualization for lower VTLs on TDX CVMs using the TDX L2-VM TSC deadline timer capability. This reduces guest exits to the hypervisor by handling timer operations directly in hardware, improving CVM performance.
Key changes:
- Added
HardwareIsolatedGuestTimertrait to abstract lower VTL timer deadline management - Implemented
TdxTscDeadlineServicefor TDX-specific hardware timer virtualization - Refactored existing
VmTimeinterface into the new trait as a fallback implementation
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| vm/x86/x86defs/src/vmx.rs | Added TIMER_EXPIRED exit reason constant for TDX timer expiry |
| tmk/tmk_vmm/src/paravisor_vmm.rs | Disabled lower VTL timer virtualization for TMK VMM |
| openhcl/virt_mshv_vtl/src/processor/tdx/mod.rs | Implemented TDX TSC deadline timer service and integrated it into TDX backing |
| openhcl/virt_mshv_vtl/src/processor/snp/mod.rs | Added timer interface delegation for SNP (using VmTime fallback) |
| openhcl/virt_mshv_vtl/src/processor/mod.rs | Added timer deadline management methods to HardwareIsolatedBacking trait |
| openhcl/virt_mshv_vtl/src/processor/hardware_cvm/mod.rs | Extracted timer management into new trait with VmTime fallback implementation |
| openhcl/virt_mshv_vtl/src/lib.rs | Added configuration plumbing for lower VTL timer virtualization feature |
| openhcl/underhill_core/src/worker.rs | Propagated timer virtualization configuration to partition parameters |
| openhcl/underhill_core/src/options.rs | Added command-line option to disable lower VTL timer virtualization |
| openhcl/underhill_core/src/lib.rs | Passed timer virtualization option to worker configuration |
| openhcl/hcl/src/protocol.rs | Defined TDX L2 TSC deadline state structure in VP context |
| openhcl/hcl/src/ioctl/tdx.rs | Added accessors for TDX L2 TSC deadline state |
| openhcl/hcl/src/ioctl.rs | Added HCL capability check for lower VTL timer virtualization |
chris-oo
left a comment
chris-oo
left a comment
There was a problem hiding this comment.
looks reasonable but i want to also have @smalis-msft review. We also use this timer for VTL1 - have you tested that?
openhcl/hcl/src/protocol.rs
Outdated
| pub struct tdx_l2_tsc_deadline_state { | ||
| /// Timer deadline value in absolute virtual TSC units for this processor. | ||
| pub deadline: u64, | ||
| /// Indicates if the `mshv_vtl` driver should issue a `TDG.VP.WR` call to update the |
There was a problem hiding this comment.
i assume here a value of 1 - can we be specific what value is expected here?
|
|
||
| /// Returns true if `ref_time` is before `ref_time_last`. | ||
| /// | ||
| /// Note that this is a relative comparison in the 64-bit space and is not |
There was a problem hiding this comment.
I'm not sure i understand the implications of this comment? or how this check cannot be transitive?
There was a problem hiding this comment.
I guess it's because we wrap? That should be clarified.
There was a problem hiding this comment.
Done, clarified the comment.
| fn clear_deadline(&self, vp: &mut UhProcessor<'_, T>); | ||
| } | ||
|
|
||
| /// Interface for managing lower VTL timer deadlines via `VmTime`. |
There was a problem hiding this comment.
nit: use doc links with "[VmTime]" since we're referring to an actual type here (or refer to the right type in the doc comment)
openhcl/virt_mshv_vtl/src/lib.rs
Outdated
| private_dma_client: Arc<dyn DmaClient>, | ||
| hide_isolation: bool, | ||
| proxy_interrupt_redirect: bool, | ||
| lower_vtl_timer_virt: bool, |
There was a problem hiding this comment.
Do we need to keep this field around? I think we could get rid of it and use other signals, like if some options are Some, right?
There was a problem hiding this comment.
Like what if this was only in BackingSharedParams instead?
There was a problem hiding this comment.
We could, but it is also needed later in BackingPrivate::new() where it initializes the per-VP context needed for by this feature.
There was a problem hiding this comment.
It could be put in BackingParams too. Or we could take my other idea of unconditionally storing a VmTimeGuestTimer somewhere and having TdxBackedShared just hold an Option<TdxTscDeadlineService> instead of doing a trait object. But I think I'd prefer to keep PartitionState holding only things that are needed at runtime, whereas this is only needed at construction.
There was a problem hiding this comment.
Done. Moved this to BackingSharedParams
Added a method in the trait to signal if hardware virtualized timer is in use; let me know if this approach works.
On the suggestion of unconditionally storing a VmTimeGuestTimer object:
since VmTimeGuestTimer is only for managing lower vtl timers, when lower_vtl_timer_virt is supported, the backing is not going to need the VmTimeGuestTimer object.
So i felt the trait object might be more appropriate here. Also the above changes should hopefully address the concern around keeping only runtime stuff in the partition state.
But let me know if you strongly favor - unconditionally storing a VmTimeGuestTimer object, I can implement that instead.
@chris-oo Yes, we validated with VTL1 enabled on Windows guest and we don't see any issues. Performance improvements are consistent with expectation. |
babayet2
added
the
release-ci-required
smalis-msft
left a comment
smalis-msft
left a comment
There was a problem hiding this comment.
LGTM but waiting for Chris before merging
balajimc55
dismissed stale reviews from chris-oo and smalis-msft
via
29f4f46
Implements microsoft#2028 This PR implements hardware timer virtualization for lower VTLs on TDX CVM usingL2-VM TSC deadline timer, an architectural capability provided by TDX module. This improves the CVM performance by eliminating guest exits to hypervisor for timer arming and expiry notifications for lower VTL's timer requirements. The related changes in OHCL-kernel is implemented by microsoft/OHCL-Linux-Kernel#107 This allows VTL2 to set an execution deadline for lower VTLs. If the lower VTL is running when the deadline time arrives, it exits to VTL2 with exit reason `VmxExitBasic::TIMER_EXPIRED`. If the TSC deadline is in the past during entry into lower VTL (i.e., TSC deadline value is lower than the current virtual TSC value), it will immediately exit back to VTL2 with exit reason `VmxExitBasic::TIMER_EXPIRED`. The TSC deadline is set using `TDG.VP.WR` for `TDVPS.TSC_DEADLINE[L2-VM Index]`. - With these changes, openvmm evaluates earliest deadline across all lower VTLs and sets it in a `tdx_vp_context ->tdx_l2_tsc_deadline_state ` that is shared with `mshv_vtl` driver. - During entry into lower VTL, `mshv_vtl` driver makes the `TDG.VP.WR` call to set the deadline when an update is needed. - Added `HardwareIsolatedGuestTimer` trait as an abstraction for managing lower VTL timer deadlines. - Moved current `VmTime` interface as default/fallback implementation into this trait. - Added `TdxTscDeadlineService` to implement the TDX specific timer virtualization.
|
Backported to release/1.7.2511 in #2570 |
benhillis
added
backported_1.7.2511
|
Backported to release/1.7.2511 in #2570 |
5 participants
Implements #2028
This PR implements hardware timer virtualization for lower VTLs on TDX CVM usingL2-VM TSC deadline timer, an architectural capability provided by TDX module. This improves the CVM performance by eliminating guest exits to hypervisor for timer arming and expiry notifications for lower VTL's timer requirements.
The related changes in OHCL-kernel is implemented by microsoft/OHCL-Linux-Kernel#107
Background - TDX L2-VM TSC Deadline Timer
This allows VTL2 to set an execution deadline for lower VTLs. If the lower VTL is running when the deadline time arrives, it exits to VTL2 with exit reason
VmxExitBasic::TIMER_EXPIRED.If the TSC deadline is in the past during entry into lower VTL (i.e., TSC deadline value is lower than the current virtual TSC value), it will immediately exit back to VTL2 with exit reason
VmxExitBasic::TIMER_EXPIRED.The TSC deadline is set using
TDG.VP.WRforTDVPS.TSC_DEADLINE[L2-VM Index].Implementation
tdx_vp_context ->tdx_l2_tsc_deadline_statethat is shared withmshv_vtldriver.mshv_vtldriver makes theTDG.VP.WRcall to set the deadline when an update is needed.Changes
HardwareIsolatedGuestTimertrait as an abstraction for managing lower VTL timer deadlines.VmTimeinterface as default/fallback implementation into this trait.TdxTscDeadlineServiceto implement the TDX specific timer virtualization.