Poor TCP RX performance from Windows 10 hosts with default dynamic/adaptive interrupt mode

[logan893]

My journey described in more detail here:
https://forums.servethehome.com/index.php?threads/intel-x550-in-linux-poor-receive-throughput-from-windows-10.52610

I have one Linux (Debian, Proxmox 8.4) system where both X550 and X540 result in terrible out-of-box TCP RX performance, from Windows 10 hosts. I rebuilt the latest ixgbe driver from this source to make sure I am using the most recent code.

Easiest way to reproduce the problematic behavior is using iperf3, with Windows 10 hosts transmitting using default settings. Throughput is terrible, only 300-700 Mbps (average 450) from one host with 2.5Gb networking, and 800-1500 Mbps from another (faster) host with 10Gb networking. The target system interrupt load is merely 1% (4-core CPU without HT) during the test from the slower system.

iperf3 TX from a Linux host on the same network can reach 9+ Gbps with its default settings, albeit with some packet loss, especially at the start, which is not seem as much when reversing the transfer direction. And, the Windows 10 hosts can also achieve good throughput if I limit their iperf3 TX buffer from default 128k down to 32k (--length 32k, with iperf3).

ethtool output shows "rx-usecs" set to 1, and "Adaptive RX: n/a" for both this poorly performing server, and for a good server. The only difference here between the two is "tx-frames-irq = 256" for the server with X540 card, and n/a for the server with X550. I have not tried swapping them around to see if this is system or card specific.

# ethtool -c enp1s0f1
Coalesce parameters for enp1s0f1:
Adaptive RX: n/a  TX: n/a
stats-block-usecs: n/a
sample-interval: n/a
pkt-rate-low: n/a
pkt-rate-high: n/a

rx-usecs: 1
rx-frames: n/a
rx-usecs-irq: n/a
rx-frames-irq: n/a

tx-usecs: 0
tx-frames: n/a
tx-usecs-irq: n/a
tx-frames-irq: 256

rx-usecs-low: n/a
rx-frame-low: n/a
tx-usecs-low: n/a
tx-frame-low: n/a

rx-usecs-high: n/a
rx-frame-high: n/a
tx-usecs-high: n/a
tx-frame-high: n/a

CQE mode RX: n/a  TX: n/a

Even with the X5x0 NIC in 1Gb mode on this Linux server, the transfer speed from Windows 10 is terrible, while the i210 NICs using igb driver in the same system work flawlessly at gigabit speeds. The igb NICs use "rx-usecs: 3", if this is relevant.

Manually changing rx-usecs to a value of e.g. 8 (any multiple of 2 between 2-16 works pretty well with interrupt load being the main difference), makes it possible to achieve good TCP throughput from Win10 to the Linux host using ixgbe driver. Still, even with this, the throughput is not as stable and can drop down into the 500-1500 Mbps range for quite some time during a continuous TCP session (especially from the Win10 host with the 10Gb networking).

These same Windows 10 hosts have no problem reaching their maximum TX TCP throughput via iperf3 to two other Linux servers (also running Debian, Proxmox 8.4 or 9.0) on the same network, one with a Mellanox ConnectX-2 NIC, and another where I tried the same X550 NIC as was giving me poor performance in the original Linux server.

Comparing the good performing hosts with the poor performing one, shows all three use MSI-X interrupts for the network cards.

Could this be an issue with the driver and its interaction with some part of my problematic Linux server's system?
What more data can I provide to help dig into this further?