IX won't TX

[ssps]

Dear IX developers,

I'd like to reproduce experiments from IX and Zygos paper in my own environment, but I've encountered following issues.
Here are some details about the environment and problems.

0. General environment
   CPU: Intel Xeon E5-2690
   RAM: 120GB
   Kernel: 4.4.0-104-generic
   DPDK version: 16.04 (installed following the guides)
   10GbE NIC: Intel 82599ES
   40GbE NIC: Intel XL710

1. 10GbE Multi-port TX

• Only the first NIC listed in the configuration can RX / TX packets.
• Rest of the NICs can RX packets, but it cannot TX packets out to wire.

2. 40GbE TX

• 40GbE NIC can RX packets, but it won't TX packets out to wire

Regards,
Ilwoo

[prekageo]

Regarding DPDK: You say that you installed it following the guides. What exactly do you mean? IX and Zygos automatically compile and use a compatible version of DPDK as part of their Makefile. I suggest that you use the following repository to build and use IX and Zygos: https://github.com/ix-project/zygos-bench

Keep in mind that Zygos doesn't support multiple NICs unlike IX.

Regarding the 40GbE, is it IX or Zygos that doesn't work? Or both? Can you give the ix.conf and the command line used?

[ssps]

Dear @prekageo ,

Thanks for your response.

1. DPDK
   Looks like I've chosen wrong word "install".
   I've used the commands provided in README.md to build and run IX.

2. 40GbE and IX, Zygos
   I've tested both system under 10GbE link.
   For 40GbE NIC, I've tested only with IX and couldn't detect any outgoing packets in the wire.

3. IX configuration file
   Uploaded as .txt as Github won't support file attachment with unknown file type.

ix.conf.txt

[prekageo]

I guess you have 2 different problems:

1. IX with multiple NICs uses only the first NIC for tx. Is that correct? In that case, what kind of packets do you send and receive? ARP, UDP, TCP? How do you observe rx and tx?

2. IX with 40GbE doesn't TX. What kind of packets do you try to transmit? Have you tried doing an arping against IX from another host of your subnet?

[ssps]

Dear @prekageo ,

I've wired two machines directly with cable during the test.
Server (echoserver) runs on top of IX stack and Client runs on top of kernel stack.

Correct.
Stack has been tested with ICMP ping (linux ping command) and TCP client (provided in ix-bench).
Neither of clients received return packets from IX server.
I've setup some hooks in ix/dp/net/ip.c and ix/inc/ix/ethqueue.h to verify whether stack received correct packets and enqueues outgoing packets.
IX successfully receives SYN packet and enqueues SYN/ACK to the target device, but the SYN/ACK packet cannot be observed in the other end.

As in 1., ICMP ping and TCP connection.
I've tested the system with arping as you've recommended and IX fails to reply to ARP request.
mTCP applications replies fine without any problem.

[prekageo]

Regarding point 1. ICMP always uses the first NIC. TCP should use the same NIC for Tx as Rx. How do connect 2 servers together via cable if one of them has multiple NICs? I don't understand your network topology.

[ssps]

Dear @prekageo

I'll explain how I tested TCP connection with multiple NIC.

0. Hardware

Both machines have two Intel 10G NICs. (for 40GbE I've replaced one of NICs to 40GbE).
Two cables are used to connect first port of each NIC to the other machine's NICs.
If I denote cable connection as "<->", it can be drawn as follows.

[machine0] [machine1]
05:00.0 <-> 03:00.0
07:00.0 <-> 05:00.0

1. Configuration

1.1 Server (IX)
[ix.conf]
host_addr="10.0.0.2/24"
port=8000
...
devices=["05:00.0", "07:00.0"]
...

1.2 Client (Linux)
Two netdev entries are available.

p3p1 (directly connected with "05:00.0")
p5p1 (directly connected with "07:00.0")

2. Shell commands

[Server]
Run server with following command.
# ./dp/ix -- apps/echoserver 64

[Client]
Run client with two set of commands.

# ifconfig p5p1 down
# ifconfig p3p1 10.0.0.6/24
$ ./client 10.0.0.2 8000 100 64 100 (works fine)

and

# ifconfig p3p1 down
# ifconfig p5p1 10.0.0.6/24
$ ./client 10.0.0.2 8000 100 64 100 (does not work)

3. Side note

If I modify "devices" line in [ix.conf] to ["07:00.0", "05:00.0"], now p5p1 (which is connected to 07:00.0) can receive packets from IX while p3p1 (which is connected to 05:00.0) won't.

[prekageo]

Unfortunately, this is an invalid network configuration. Unless you modify the MAC addresses of your devices, this setup will not work. Moreover, I am not really sure what you are trying to achieve with this setup. If you want to understand how to properly setup a bond read the following: https://www.kernel.org/doc/Documentation/networking/bonding.txt

[ssps]

Dear @prekageo ,

1. Purpose
   I cannot run 40Gbps experiment (1 server, 4 clients) in different setup due to this problem.
   I'm currently trying to identify the cause and the above is my debug setup.

2. Bonding
   You're suggesting me to use bonding driver setup in client, right?
   I'm using static ARP entry on both sides and re-launching IX server whenever I'm switching client configuration.
   MAC address in client side and server side won't interfere here.
   Bonding driver could reduce the time setting up the client, I'll use that in the future.

P.S. Does IX tries to send packets with the MAC address of first NIC port?

[prekageo]

Use tcpdump on Linux side to verify. Your setup is not valid. Even Linux will not work with this setup. You have to setup a proper bond interface and first make sure that Linux works with it.

[ssps]

Dear @prekageo ,

Connectivity on both NIC is verified with Linux stacks on both sides.
Obviously I've used different subnet to overcome routing issue in kernel.
However this is not an issue in my debug setup as I'll turn off the other NIC when the other one is active.

Could you clarify why bonding NIC is so important?

[prekageo]

Are you using bonding or not with Linux? There is a single way to setup bonding under Linux. You have to follow the document I pointed out earlier. If you don't using bonding under Linux, how do you expect IX to work under your setup when you specifically ask from IX to operate in bond mode? You have to read and understand better what is a bond and how to set it up. Hint: There is a single MAC for all the adapters in a bond.

[ssps]

Dear @prekageo

Okay now I understand.

I was thinking that you asked me to setup bonding in client-side.
I'll try to bond the NICs in IX server-side and see how it works.

BTW, shouldn't the IX use DPDK bonding driver internally, if it's designed around it?

[prekageo]

I assume that you are no longer interested in this issue. If that's not the case, please re-open it.

[kkaffes]

I have the same problem as @ssps. IX using a 40Gb Intel XL710 NIC is able to RX packets but cannot transmit anything on the wire.

Setup:
CPU: Intel Xeon E5-2620
Kernel: 4.4.0-145-generic
DPDK version: 16.04
40GbE NIC: Intel XL710
Manually set ARP entries