connection.switch.arp_inspection potential spurious result

[duncangreene]

Describe the bug
Conducted separate 2 tests on the same DUT, from separate Tetsrun machines (not sure whether this makes a difference, as the machine hardware/spec for both machines is identical).

Test 1

Test 2

For test 2 this was shown in the logs.

However at the same timestamp in connection.pcap, the DUT can clearly be seen announcing that 10.10.10.12 is at 00:60:74:f0:29:90.

Is this a race condition?

Additionally, an ARP packet pertaining to 10.10.10.14 doesn't appear in connection.pcap until 7 minutes later at 15:45:07. How can this be reported/inferred at 15:38:11 in the log file above?

Expected behavior
Identical results should occur on identical tests of the same DUT.

Error logs
QSC QIO-L4O v9.13.0-2412.002 (test 1).zip
QSC QIO-L4O v9.13.0-2412.002 (test 2).zip

Environment (please provide the following information about your setup):

• Version 2.1

[duncangreene]

Same "Device is sending false ARP response" observed on this completely different device.

RDL DDS-RN42 v4.1.4.2.zip

[duncangreene]

I have now observed this same issue (sporadic Passing and Failing) separately on a third separate DUT from a completely different manufacturer, on a totally separate machine running Testrun.

arp_fail.zip
arp_pass.zip

[duncangreene]

Just confirming also observed on 2.1.1. Reports below for info.

arpfail-2-1-1.zip
arppass-2-1-1.zip

Below taken from arpfail-2-1-1.zip.

Complete side note BTW @jboddey, the line in orange above, this appears to be being truncated here?

Back to the topic, if I'm following the code correctly, the test is analysing monitor.pcap and startup.pcap. Are the ARP announcements from the DUT in those captures applicable if the _device_ipv4_addr has changed (as presumably shown in green above) since testing formally started?

[jhughesoti]

I would recommend reviewing the installation for file permissions as I could see some errors that could indicate an issue with the installation. I also noticed there are a number of devices on the network not being tested which may be causing issues with proper DHCP leasing for single device testing. Possibly these devices are powered on prior to test causing other DHCP caching issues to cause false failures.

[duncangreene]

I would recommend reviewing the installation for file permissions as I could see some errors that could indicate an issue with the installation.

@jhughesoti Could you let me know which file/directories might warrant a look? I will gather the permissions from them and report back if so.

[duncangreene]

I also noticed there are a number of devices on the network not being tested which may be causing issues with proper DHCP leasing for single device testing. Possibly these devices are powered on prior to test causing other DHCP caching issues to cause false failures.

@jhughesoti Just for the avoidance of doubt, in all cases there is only 1 physical device under test with 1 physical ethernet connection, however having looked back at some of the devices in question (and pre-empting some other devices/protocols that I know exist out there and will likely also exhibit this behaviour in the future), I believe you may be seeing 'extra' source MACs in the Wireshark trace from the exact same device, and the reason(s) for this are entirely legitimate.

Some devices out there, by design and by specification of the protocols that they implement, may well 'have' multiple MAC addresses. There's an excellent chance that the RDL device mentioned in one of the posts above implements Dante for Audio over IP on a completely separate MAC address and IP address to that of its 'management' interface, which may have been the one used/configured to run against Testrun.

Similarly, a Grundfos CIM 500 (an IP-based comms card for a pump) features an inbuilt Ethernet switch (in itself not a problem when managed into an enterprise network properly) but by implication of it supporting PROFINET (a widely established industrial automation protocol), it also needs to implement LLDP. This means that each individual switch port has a MAC address for the LLDP comms (this is also how STP works), but these will be in addition to the MAC address used for the PROFINET comms (the CIM 500 goes one further by having a fourth MAC address used for the management/web interface too).

In a nutshell, it's not at all uncommon/unacceptable for a 'single' device to present/have multiple source MAC addresses. Many widely established networking and communication protocols implicate it by design.

P.S. Happy to jump on a call to go into this if needed, I appreciate it's not a straightforward topic. 😅

[jhughesoti]

Understood, but in this instance, as this is a testing tool to validate devices that will go on a google managed network, this is not an acceptable solution. Each device must have only one unchanging MAC address. This device behavior may indicate expected failures or inconsistent passes within the scope of testrun as it attempts to simulate this production network architecture as best as possible.

[duncangreene]

I would add that the MAC address(es) in these examples are definitely unchanging, but it is the case that multiple MAC addresses can/will present from one device if the systems/protocols in use implicate it. A server (and Testrun obviously! 😅) running VMs/containers is another classic example of this (and we know 'server' in this case doesn't need to mean something in a rack, it could just as easily be an IoT device).

In either case back to the OP and subsequent logs, how would this scenario be regarded/interpreted as an ARP failure per se? Worth also recapping that the above results have been sporadic, that is, one test yields pass, the next could yield fail. The timestamps of the logs in the OP also seem odd. Any ideas?

[jboddey]

Agreed. The name of the test does not match the requirement. @KrystsinaT probably worth checking in with the lab about this one?

[KrystsinaT]

The ticket has been reported 417428949 and will be discussed with the lab.