LINE is a network emulator designed for realism, accuracy and powerful measurement capabilities. It is capable of emulating networks in real-time, forwarding actual network traffic generated by real Linux network stacks.
LINE is released under the GPLv2 license.
Get LINE:
git clone https://github.com/nal-epfl/line.git
A typical LINE setup consists of three computers, each having a specific role.
This machine performs the emulation of the routers in the network. It should be a system optimized for low latency.
CPU: 64-bit quad-core with good single thread performance. Our current setup uses an Intel Xeon E3-1290 V2 (Ivy Bridge) CPU, but there might be other good options. We found this classification useful: PassMark - CPU Mark Single Thread Performance.
Network: two network cards, one for WAN and another one dedicated for exchanging traffic with the traffic generator. The dedicated card should run at 10 Gbps (1 Gbps will work too, but will limit the amount of traffic that can be sent through the emulated network to < 1 Gbps). The traffic generator and the network emulator should be connected directly through a network cable (no switches/routers, to avoid background traffic and other effects).
OS: Ubuntu 12.04.4 (LTS) 64-bit server.
The machine should not be used for anything else, and should be connected to a secure network, since we are going to disable the firewall.
This machine generates the traffic that will be routed through the emulated network. It should be a system optimized for good network I/O performance.
CPU: 64-bit quad-core or better with good overall performance. Our current setup uses an Intel Xeon E3-1290 V2 (Ivy Bridge) CPU, but a CPU with more cores or a multi-CPU setup might work well. This classification may be useful: PassMark - CPU Mark High End CPUs.
Network: two network cards, one for WAN and another one dedicated for exchanging traffic with the network emulator. The dedicated card should run at the same speed as the dedicated card of the network emulator.
OS: Ubuntu 12.04.4 (LTS) 64-bit server.
The machine should not be used for anything else, and should be connected to a secure network, since we are going to disable the firewall.
This machine is used for configuring the experiments, and controlling remotely the router emulator and the traffic generator.
CPU: 64-bit, with modest performance.
Network: network connectivity is required in order to run experiments.
OS: Ubuntu 12.04.4 (LTS) 64-bit desktop.
These instructions assume that the system is running a clean Ubuntu 12.04.4 (LTS) 64-bit server installation. The instructions should be executed as root (run sudo su).
Configure password-less SSH authentication with keys for root
See https://help.ubuntu.com/community/SSH/OpenSSH/Keys.
Hyper-threading does not work well with our low-latency configuration, and is not particularly useful, so we disable it.
apt-get install build-essential
# In /etc/network/interfaces:
auto eth7
iface eth7 inet static
address 192.168.77.1
netmask 255.255.255.0
apt-get install wget mc lm-sensors ipmitool htop stress zip
apt-get install bc
apt-get install traceroute ethtool irqbalance strace tcpdump bwm-ng netcat pciutils
apt-get install iptables-dev libdb4.8-dev
wget https://www.kernel.org/pub/linux/utils/net/iproute2/iproute2-3.11.0.tar.xz
tar -xJf iproute2-3.11.0.tar.xz
cd iproute2-3.11.0
./configure --prefix=/usr
make -j7
make install
cd ..
wget https://www.kernel.org/pub/software/network/ethtool/ethtool-3.11.tar.xz
tar -xJf ethtool-3.11.tar.xz
cd ethtool-3.11
./configure --prefix=/usr
make -j7
make install
rm -f /sbin/ethttool
cp /usr/sbin/ethtool /sbin/ethttool
cd ..
rm /etc/modprobe.d/ipmi.conf
apt-get install software-properties-common python-software-properties
add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get update
apt-cache search "g\+\+"
apt-get install gcc-4.7 g++-4.7
ls -lh /usr/bin/g++
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7
update-alternatives --config gcc
# Select 4.7, then check:
g++ --version
apt-get install libssl-dev flex bison gperf libicu-dev libxslt-dev ruby libcups2-dev libglib2.0-dev
pushd .
cd /tmp
rm -rf qt-everywhere-opensource-src-4.8.5
wget http://master.qt-project.org/official_releases/qt/4.8/4.8.5/qt-everywhere-opensource-src-4.8.5.tar.gz
tar xzf qt-everywhere-opensource-src-4.8.5.tar.gz
cd qt-everywhere-opensource-src-4.8.5
./configure --prefix=/usr/local --opensource --confirm-license --graphicssystem=raster --release -no-gui -nomake examples -nomake demos
make -j4
make install
cd ..
popd
apt-get install libnuma-dev flex bison
apt-get install libunwind7-dev
wget https://gperftools.googlecode.com/files/gperftools-2.1.tar.gz
tar xzf gperftools-2.1.tar.gz
cd gperftools-2.1
./configure --prefix=/usr
make -j4
make install
cd ..
apt-get install libnl-dev libev-dev
Run these commands as non-root:
sudo apt-get install ncurses-dev kernel-package
mkdir ~/src
cd ~/src
apt-get source linux-image-$(uname -r)
cd linux-...
make menuconfig
Edit the following options:
- enable CONFIG_NO_HZ_FULL (General setup -> Timers subsystem -> Timer tick handling -> Full dynticks system (tickless));
- enable CONFIG_NO_HZ_FULL_ALL (General setup -> Timers subsystem -> Timer tick handling -> Full dynticks system on all CPUs by default);
- enable CONFIG_PREEMPT (Processor type and features -> Preemption model -> Preemptible kernel (Low-Latency Desktop));
- select your CPU model instead of generic x86_64 (Processor type and features -> Processor family -> Core 2/newer Xeon).
export CONCURRENCY_LEVEL=4
time fakeroot make-kpkg --initrd --append-to-version=-line kernel-image kernel-headers
cd ..
sudo dpkg -i linux-image-* linux-headers-*
service ufw stop
echo "manual" > /etc/init/ufw.override
echo "blacklist iptable_filter" >> /etc/modprobe.d/blacklist.conf
echo "blacklist ip_tables" >> /etc/modprobe.d/blacklist.conf
echo "blacklist x_tables" >> /etc/modprobe.d/blacklist.conf
for d in /lib/modules/* ; do mv $d/kernel/net/ipv4/netfilter $d/kernel/net/ipv4/netfilter-old || /bin/true ; done
reboot
# In /etc/default/grub, add to GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX:
isolcpus=1,2,3 nohz_full=1,2,3 rcu_nocbs=1,2,3 acpi=noirq nosoftlockup intel_idle.max_cstate=0 processor.max_cstate=1 idle=halt mce=ignore_ce memtest
Update the bootloader:
update-grub
Parameters explained:
- isolcpus=1,2,3: isolates cores 1, 2 and 3 from the rest of the system (they will be dedicated for LINE, i.e. no other applications will be scheduled to run on these cores); core 0 will be kept for housekeeping;
- nohz_full=1,2,3: disables the local timer on cores 1, 2 and 3 (to reduce latency/jitter);
- rcu_nocbs=1,2,3: disables RCU callback execution on cores 1, 2 and 3 (to reduce latency/jitter);
- acpi=noirq: whether this is useful is highly dependent on your motherboard and/or BIOS. We used it because a large number of "Rescheduled interrupts" was listed in /proc/interrupts. YMMV. See ReschedulingInterrupts;
- nosoftlockup: disables the kernel soft-lockup detector. Setting this is recommended in the low-latency configuration manual of our HP server (included in servers/HP Proliant Low Latency Guide.pdf);
- intel_idle.max_cstate=0 processor.max_cstate=1 idle=halt mce=ignore_ce: do not allow any core to go into a sleep state (reduces latency/jitter);
- memtest: runs a memory test at boot. Optional.
# Set ENABLED=0 in /etc/default/irqbalance
reboot
These instructions assume that the system is running a clean Ubuntu 12.04.4 (LTS) 64-bit server installation. The instructions should be executed as root (run sudo su).
See https://help.ubuntu.com/community/SSH/OpenSSH/Keys.
apt-get install build-essential
# In /etc/network/interfaces:
auto eth7
iface eth7 inet static
address 192.168.77.2
netmask 255.255.255.0
apt-get install wget mc lm-sensors ipmitool htop stress zip
apt-get install bc
apt-get install traceroute ethtool irqbalance strace tcpdump bwm-ng netcat pciutils
apt-get install iptables-dev libdb4.8-dev
wget https://www.kernel.org/pub/linux/utils/net/iproute2/iproute2-3.11.0.tar.xz
tar -xJf iproute2-3.11.0.tar.xz
cd iproute2-3.11.0
./configure --prefix=/usr
make -j7
make install
cd ..
wget https://www.kernel.org/pub/software/network/ethtool/ethtool-3.11.tar.xz
tar -xJf ethtool-3.11.tar.xz
cd ethtool-3.11
./configure --prefix=/usr
make -j7
make install
rm -f /sbin/ethttool
cp /usr/sbin/ethtool /sbin/ethttool
cd ..
rm /etc/modprobe.d/ipmi.conf
apt-get install software-properties-common python-software-properties
add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get update
apt-cache search "g\+\+"
apt-get install gcc-4.7 g++-4.7
ls -lh /usr/bin/g++
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7
update-alternatives --config gcc
# Select 4.7, then check:
g++ --version
apt-get install libssl-dev flex bison gperf libicu-dev libxslt-dev ruby libcups2-dev libglib2.0-dev
pushd .
cd /tmp
rm -rf qt-everywhere-opensource-src-4.8.5
wget http://master.qt-project.org/official_releases/qt/4.8/4.8.5/qt-everywhere-opensource-src-4.8.5.tar.gz
tar xzf qt-everywhere-opensource-src-4.8.5.tar.gz
cd qt-everywhere-opensource-src-4.8.5
./configure --prefix=/usr/local --opensource --confirm-license --graphicssystem=raster --release -no-gui -nomake examples -nomake demos
make -j4
make install
cd ..
popd
apt-get install libunwind7-dev
wget https://gperftools.googlecode.com/files/gperftools-2.1.tar.gz
tar xzf gperftools-2.1.tar.gz
cd gperftools-2.1
./configure --prefix=/usr
make -j4
make install
cd ..
apt-get install libnl-dev libev-dev
service ufw stop
echo "manual" > /etc/init/ufw.override
echo "blacklist iptable_filter" >> /etc/modprobe.d/blacklist.conf
echo "blacklist ip_tables" >> /etc/modprobe.d/blacklist.conf
echo "blacklist x_tables" >> /etc/modprobe.d/blacklist.conf
for d in /lib/modules/* ; do mv $d/kernel/net/ipv4/netfilter $d/kernel/net/ipv4/netfilter-old || /bin/true ; done
reboot
This machine requires a graphical desktop and password-less SSH access as root to the network emulator machine and the traffic generator machine.
apt-get install build-essential git
apt-get install libqt4-dev qt4-dev-tools
apt-get install libunwind7-dev
cd line
cp remote-config-template.sh remote-config.sh
Edit remote-config.sh and set the following variables:
- REMOTE_HOST_ROUTER: The IP address of the network emulator;
- REMOTE_DEDICATED_IF_ROUTER: The name of the dedicated network interface on the network emulator (in our network configuration example from above, eth7);
- REMOTE_HOST_HOSTS: The IP address of the traffic generator;
- REMOTE_DEDICATED_IF_HOSTS: The name of the dedicated network interface on the traffic generator (in our network configuration example from above, eth7).
mkdir build-line-gui
cd build-line-gui
qmake ../line-gui/line-gui.pro -spec linux-g++-64 CONFIG+=debug CONFIG+=declarative_debug
make -j4
cd ..
mkdir build-line-runner
cd build-line-runner
qmake ../line-runner/line-runner.pro -spec linux-g++-64 CONFIG+=debug CONFIG+=declarative_debug
make -j4
cd ..
cd line-router
./make-remote.sh
cd ..
cd line-traffic
./make-remote.sh
cd ..
cd build-line-gui
./line-gui
Video: doc/testing-the-setup.webm
Video: doc/editing-the-topology.webm
Video: doc/generating-traffic.webm
Video: doc/inspecting-the-results.webm
LINE uses the following third party libraries/components/applications:
A cross-platform application and UI framework using C++ or QML. http://qt.nokia.com License: LGPLv2.1
A network socket that dramatically improves the packet capture speed. http://www.ntop.org/products/pf_ring License: GPLv2
A full-featured and high-performance event loop that is loosely modelled after libevent. http://software.schmorp.de/pkg/libev.html License: BSD license
A project to collect data network topologies from around the world. http://www.topology-zoo.org
A network topology generator. http://www.cs.bu.edu/brite BRITE is not included nor directly used in LINE; however, LINE is able to import BRITE files.
http://gephi.org License: The majority of the Gephi Plateform code is available under the dual license CDDL 1.0 and GNU General Public License (GPL) v3. Gephi and the layout program are included separately from LINE in source and binary forms, and the binary can be executed automatically by LINE.
Force-Directed layout algorithm for real-world large-scale undirected graphs. http://gephi.org/plugins/openord-layout License: GPLv3 Gephi and the layout program are included separately from LINE in source and binary forms, and the binary can be executed automatically by LINE.
nicelabel.cpp, adapted from label.c by by Paul Heckbert from "Graphics Gems", Academic Press, 1990. License: public domain IP/TCP/UDP checksum code from the Linux kernel https://www.kernel.org License: GPLv2
A fast malloc library with thread-caching. https://code.google.com/p/gperftools License: New BSD License
ProducerConsumerQueue is a one producer and one consumer queue without locks. https://github.com/facebook/folly/blob/master/folly/ProducerConsumerQueue.h License: Apache License, Version 2.0
https://github.com/cameron314/readerwriterqueue http://moodycamel.com/blog/2013/a-fast-lock-free-queue-for-c++ License: Simplified BSD license
http://software.intel.com/en-us/articles/single-producer-single-consumer-queue http://www.1024cores.net/home/lock-free-algorithms/queues/unbounded-spsc-queue License: Simplified BSD license
http://dygraphs.com License: MIT license
http://mootools.net License: MIT license
All images from openclipart.org are licensed under CC0 1.0 Universal (CC0 1.0) - Public Domain Dedication.
- http://www.openclipart.org/detail/17668/rgtaylor_csc_net_computer
- http://www.openclipart.org/detail/1918/router-by-juanjo
- http://www.openclipart.org/detail/5012/laptop-by-metalmarious
- http://www.openclipart.org/detail/113191/ftline-line-arrow-end-by-anonymous
- http://www.openclipart.org/detail/143629/edit-pencil-icon-by-darth_schmoo The icons from the nuoveXT.2.2 theme are licensed under LGPL.
- http://gnome-look.org/content/show.php/nuoveXT+2?content=56625