netvsp: handle eqe 135 and reconfigure vf on release/1.7.2511 #2610

erfrimod · 2026-01-05T21:58:41Z

Clean cherry-pick of #2576

For netvsp to recover from SoC crash and NMC servicing, it must detect EQE 135 from MANA and reconfigure the Virtual Function.

GDMA_EQE_HWC_RECONFIG_VF added to events handled by the GDMA driver
vf_reconfiguration_pending bool added the GDMA driver. Set when the EQE is received, read by the MANA driver when processing all EQ events.
vf_reconfig_sender added to MANA driver to signal Netvsp VF Manager when it sees 'pending' is true
vf_reconfig_receiver added to Netvsp HclNetworkVFManagerWorker to send VFReconfig message when signaled
VfReconfig message added to Netvsp VF Manager, which removes the old VF and then creates a new VF

Smaller changes:

GDMA_GENERATE_TEST_EQE and GDMA_GENERATE_RECONFIG_VF_EVENT added in order to create a unit test, test_gdma_reconfig_vf()
The logic of NextWorkItem::ManaDeviceArrived refactored into startup_vtl2_device() so the logic can be shared with VfReconfig

Testing:

Unit tests pass
Tested on a lab machine with SoC MANA privates which allowed EQE 135 to be generated by command. Netvsp is able to see the EQE and VfReconfig is called. Before and after sending the EQE, ping and ntttcp traffic succeed.

For netvsp to recover from SoC crash and NMC servicing, it must detect EQE 135 from MANA and reconfigure the Virtual Function. * `GDMA_EQE_HWC_RECONFIG_VF` added to events handled by the GDMA driver * `vf_reconfiguration_pending` bool added the GDMA driver. Set when the EQE is received, read by the MANA driver when processing all EQ events. * `vf_reconfig_sender` added to MANA driver to signal Netvsp VF Manager when it sees 'pending' is true * `vf_reconfig_receiver` added to Netvsp HclNetworkVFManagerWorker to send `VFReconfig` message when signaled * `VfReconfig` message added to Netvsp VF Manager, which removes the old VF and then creates a new VF Smaller changes: * `GDMA_GENERATE_TEST_EQE` and `GDMA_GENERATE_RECONFIG_VF_EVENT` added in order to create a unit test, `test_gdma_reconfig_vf()` * The logic of `NextWorkItem::ManaDeviceArrived` refactored into `startup_vtl2_device()` so the logic can be shared with `VfReconfig` Testing: * Unit tests pass * Tested on a lab machine with SoC MANA privates which allowed EQE 135 to be generated by command. Netvsp is able to see the EQE and VfReconfig is called. Before and after sending the EQE, ping and ntttcp traffic succeed.

Copilot

Pull request overview

This PR adds support for handling EQE 135 (VF reconfiguration event) to enable netvsp recovery from SoC crashes and NMC servicing. The implementation spans three layers: GDMA driver (event detection), MANA driver (event propagation), and Netvsp (VF reconfiguration orchestration).

Key changes:

GDMA driver now handles GDMA_EQE_HWC_RECONFIG_VF events and maintains a vf_reconfiguration_pending flag
MANA driver propagates VF reconfig events through a mesh channel subscription mechanism
Netvsp implements a state machine with exponential backoff retry logic to gracefully restart the VTL2 device after reconfiguration
Full save/restore support for the pending flag to handle servicing scenarios

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
vm/devices/net/gdma_defs/src/lib.rs	Adds EQE 135 constant and test request type for VF reconfiguration
vm/devices/net/gdma/src/hwc.rs	Implements hardware control handler to generate VF reconfig events for testing
vm/devices/net/mana_driver/src/gdma_driver.rs	Adds vf_reconfiguration_pending flag, event handling, getter method, and test helper
vm/devices/net/mana_driver/src/mana.rs	Implements subscription mechanism for VF reconfig events with sender/receiver channel
vm/devices/net/mana_driver/src/save_restore.rs	Extends saved state to include vf_reconfiguration_pending flag
vm/devices/net/mana_driver/src/tests.rs	Adds unit test verifying VF reconfiguration event detection and flag behavior
openhcl/underhill_core/src/emuplat/netvsp.rs	Implements VF reconfiguration state machine with shutdown, restart, and exponential backoff retry logic; refactors device startup into shared method

Copilot · 2026-01-05T22:04:46Z

openhcl/underhill_core/src/emuplat/netvsp.rs

+                    let update_vtl2_device_bind_state = true;
+                    if self
+                        .startup_vtl2_device(update_vtl2_device_bind_state)
                        .await
                    {
-                        tracing::error!(
-                            err = err.as_ref() as &dyn std::error::Error,
-                            "Failed to report new binding state to host"
-                        );
-                    }
-                    if device_bound {
-                        vtl2_device_present = true;
-                        if matches!(&self.vtl0_bus_control, Vtl0Bus::Present(_)) {
-                            *self.guest_state.vtl0_vfid.lock().await =
-                                vtl0_vfid_from_bus_control(&self.vtl0_bus_control);
-                            self.notify_vtl0_vf_arrival();
-                        }
+                        vtl2_device_state = Vtl2DeviceState::Present;
                    }


When ManaDeviceArrived occurs during normal startup (vtl2_device_state is Missing), but startup_vtl2_device fails, the vtl2_device_state remains Missing (line 978 is skipped). However, there's no mechanism to retry startup in this failure case unlike VF reconfiguration which has backoff logic. Consider adding similar retry logic or at least logging when startup fails to help diagnose issues.

Copilot · 2026-01-05T22:04:46Z

openhcl/underhill_core/src/emuplat/netvsp.rs

-                        NextWorkItem::Continue
+                    let exists = Path::new(&device_path).exists();
+                    match (vtl2_device_state, exists) {
+                        (Vtl2DeviceState::Missing, true) => NextWorkItem::ManaDeviceArrived,


The device_arrival event is only processed when vtl2_device_state is Missing (line 644), but this excludes the Reconfiguring state. If a uevent for device arrival occurs while in the Reconfiguring state (e.g., the device actually becomes available), it will be ignored until the next uevent. This could cause unnecessary retry delays. Consider also handling device arrival during Reconfiguring state to allow faster recovery.

Suggested change

(Vtl2DeviceState::Missing, true) => NextWorkItem::ManaDeviceArrived,

(Vtl2DeviceState::Missing | Vtl2DeviceState::Reconfiguring, true) => {

NextWorkItem::ManaDeviceArrived

}

Copilot · 2026-01-05T22:04:47Z

vm/devices/net/mana_driver/src/gdma_driver.rs

+                GDMA_EQE_HWC_RECONFIG_VF => {
+                    // No data is supplied for VF reconfiguration events.
+                    tracing::info!("HWC VF reconfiguration event");


The comment says "No data is supplied for VF reconfiguration events." However, the hwc.rs handler (line 311) passes an empty slice &[] as data, which is consistent. Consider adding a similar comment in hwc.rs to maintain consistency with the driver-side expectations about the event format.

Copilot · 2026-01-05T22:04:47Z

openhcl/underhill_core/src/emuplat/netvsp.rs

+
+        const RECONFIG_INITIAL_SLEEP: std::time::Duration = std::time::Duration::from_millis(100);
+        const RECONFIG_MAX_SLEEP: std::time::Duration = std::time::Duration::from_secs(2);
+        const RECONFIG_MAX_ATTEMPTS: u64 = 300; // ~10 minutes of retries at max backoff


The comment states "~10 minutes of retries at max backoff" but the calculation is: 300 attempts * 2 seconds (max backoff) = 600 seconds = 10 minutes. However, the exponential backoff starts at 100ms and doubles each time, so the actual total time would be less than 10 minutes since earlier attempts use shorter sleep times. Consider clarifying the comment to say "up to 300 attempts (approximately 10 minutes of retries)" to better reflect the exponential backoff behavior.

Suggested change

const RECONFIG_MAX_ATTEMPTS: u64 = 300; // ~10 minutes of retries at max backoff

const RECONFIG_MAX_ATTEMPTS: u64 = 300; // up to 300 attempts (~10 minutes of retries)

Copilot · 2026-01-05T22:04:47Z

openhcl/underhill_core/src/emuplat/netvsp.rs

+                    // Don't 'keep alive'. VTL2 is reconfigured when in a bad state.
+                    let keep_vf_alive = false;
+                    self.shutdown_vtl2_device(keep_vf_alive).await;


After shutdown_vtl2_device is called (line 892), the vf_reconfig_receiver retains the old receiver whose sender was dropped during shutdown. If startup_vtl2_device fails to create a new device during retry attempts, subsequent VF reconfiguration events will be lost until the device successfully restarts. Consider either clearing vf_reconfig_receiver in shutdown_vtl2_device and checking for None in the event loop, or documenting this behavior to make the subtle coupling explicit.

Copilot · 2026-01-05T22:04:47Z

openhcl/underhill_core/src/emuplat/netvsp.rs

+                device.start_notification_task(&self.driver_source).await;
+                self.vf_reconfig_receiver = Some(device.subscribe_vf_reconfig().await);


There is a potential race condition between subscribing to VF reconfig events and processing pending events. The subscription happens in startup_vtl2_device (line 555) after start_notification_task (line 554). However, if a VF reconfiguration event was already pending in the GDMA driver before the subscription, it could be processed and sent to a non-existent receiver, causing the event to be lost. Consider subscribing to VF reconfig events before starting the notification task, or ensure the pending flag is checked after subscription.

Suggested change

device.start_notification_task(&self.driver_source).await;

self.vf_reconfig_receiver = Some(device.subscribe_vf_reconfig().await);

self.vf_reconfig_receiver = Some(device.subscribe_vf_reconfig().await);

device.start_notification_task(&self.driver_source).await;

Copilot · 2026-01-05T22:04:48Z

vm/devices/net/mana_driver/src/tests.rs

+    assert!(
+        gdma.get_vf_reconfiguration_pending(),
+        "vf_reconfiguration_pending should be true after reconfig event"
+    );
+}


The test should verify that calling get_vf_reconfiguration_pending() a second time returns false, since the method uses mem::take which should clear the flag. This ensures the flag is properly consumed and won't cause duplicate VF reconfigurations.

Copilot · 2026-01-05T22:04:48Z

openhcl/underhill_core/src/emuplat/netvsp.rs

+                NextWorkItem::VfReconfig => {
+                    if self.is_shutdown_active
+                        || matches!(vtl2_device_state, Vtl2DeviceState::Missing)
+                    {


If multiple VF reconfiguration events occur (e.g., EQE 135 is received while already in Reconfiguring state), the second VfReconfig event will be silently ignored (line 876-878 continue). While this behavior may be intentional, it would be clearer to log this scenario at debug or info level to help operators understand that a reconfiguration event was received but is already being handled.

Suggested change

{

{

tracing::debug!(

"VF reconfiguration event received but ignored (is_shutdown_active: {}, vtl2_device_state: {:?})",

self.is_shutdown_active,

vtl2_device_state

);

github-actions · 2026-01-05T22:58:26Z

At least one Petri test failed.

erfrimod requested a review from a team as a code owner January 5, 2026 21:58

Copilot AI review requested due to automatic review settings January 5, 2026 21:58

github-actions bot added the release_1.7.2511 Targets the release/1.7.2511 branch. label Jan 5, 2026

Copilot started reviewing on behalf of erfrimod January 5, 2026 21:59 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

netvsp: handle eqe 135 and reconfigure vf on release/1.7.2511 #2610

netvsp: handle eqe 135 and reconfigure vf on release/1.7.2511 #2610

erfrimod commented Jan 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

github-actions bot commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	const RECONFIG_MAX_ATTEMPTS: u64 = 300; // ~10 minutes of retries at max backoff
	const RECONFIG_MAX_ATTEMPTS: u64 = 300; // up to 300 attempts (~10 minutes of retries)

		device.start_notification_task(&self.driver_source).await;
		self.vf_reconfig_receiver = Some(device.subscribe_vf_reconfig().await);

-                    {
+                    {
+                        tracing::debug!(
+                            "VF reconfiguration event received but ignored (is_shutdown_active: {}, vtl2_device_state: {:?})",
+                            self.is_shutdown_active,
+                            vtl2_device_state
+                        );

netvsp: handle eqe 135 and reconfigure vf on release/1.7.2511 #2610

Are you sure you want to change the base?

netvsp: handle eqe 135 and reconfigure vf on release/1.7.2511 #2610

Conversation

erfrimod commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

erfrimod commented Jan 5, 2026 •

edited

Loading