3. Optimizing the Host PC for NPAP Evaluation
As a highly customizable IP Core subsystem, NPAP delivers line-rate throughput and short, deterministic latency. Software-based TCP/IP stacks often compensate higher latencies with bigger buffer sizes to achieve the maximum line-rate throughput. Sometimes these higher latencies cause the throughput to drop in NPAP to host PC setups, due to the fact that FPGAs only include a limited amount of available memory, setting a hard limit to the available buffer size. Consequently, when evaluating NPAP, the bottleneck often shifts to the latency of the software-based TCP/IP stacks.
To accurately measure the hardware-accelerated capabilities of NPAP, you must tune your host environment to keep pace with the FPGA. This chapter outlines example steps to optimize your host PC in a Board-to-PC setup for maximum throughput and minimum latency. The commands in this section were tested on our setup and might need to be adjusted depending on the hardware used. Feel free to contact us, if any of the commands below fail.
The setup we used a workstation running Ubuntu 18.04 with an Intel Xeon CPU E5-1620 and an Intel Corporation Ethernet Server Adapter X520-2 (Intel 10-Gigabit-Ethernet-Controller 82599). The workstation was connected to a Dell S5148 switch that provided the connection to an AUB15P board with NPAP ERD 15 (ERD 15 (1x10G)).
3.1. Theoretical Baseline: Bandwidth-Delay Product (BDP)
The BDP defines the maximum amount of data currently “in flight” across the network link. If session buffers are smaller than the BDP, the transmitter can only use a portion of the available network link as it needs to buffer every packet “in flight”. In TCP term this means that the transmission window cannot scale effectively, which leads to link underutilization. The theoretical maximum buffer footprint is expressed using the following formula:
For instance, on a point-to-point link operating at 10 Gbps with a round-trip time latency of approximately 0.16 ms, the baseline volume is calculated as:
To ensure continuous line-rate saturation, maximum buffer dimensions assigned at board and host PC should always exceed this threshold. For further information check out one of this Tutorial on Buffer and Window Sizing Based on the BDP from Inventive.
Note
Ensure that your host NIC as well as all components along the path are configured with an MTU of 9000 and that the IPv4 address is assigned correctly, as detailed in How to evaluate NPAP using the ERD.
3.2. Optimizing Host PC Latency and Throughput
To match the deterministic execution of the NPAP core, you must optimize software interrupt handling and eliminate CPU frequency scaling delays on the host PC.
3.2.1. Interrupt Coalescing
Network interface cards batch interrupts to save CPU cycles, which inherently adds latency.
On our Intel NIC tuning the rx-usecs parameter (the number of microseconds the NIC waits before issuing an interrupt) allows you to balance CPU load against latency.
0: Lowest latency, highest CPU load.1000: Higher latency, lower CPU load.
Note
In our experience the optimal value for the rx-usecs in most setups is somewhere between 3us and 10us.
$ sudo ethtool -C <interface> rx-usecs 5
3.2.2. Interrupt Affinity (SMP Affinity)
By default, some NICs offer Receive-Side Scaling (RSS) and daemons like irqbalance to distribute network interrupts across multiple CPU cores.
While this benefits general-purpose server workloads, it introduces context-switching overhead and degrades L1/L2 cache locality during high-speed, single-flow tests with NPAP.
To further reduce the latency of your setup, you should pin the NIC’s hardware interrupts to the exact same CPU core that will run your benchmarking application (e.g., netperf).
First, temporarily disable the irqbalance daemon to prevent the OS from overriding your manual configuration:
$ sudo systemctl stop irqbalance
Next, identify the hardware IRQ numbers associated with your network interface’s receive and transmit queues:
$ grep <interface> /proc/interrupts
Once you have identified the specific IRQ numbers (often listed in the first column), pin each of them to your target CPU core using the smp_affinity_list file.
In this example, we pin the interrupt to CPU Core 1:
$ echo 1 | sudo tee /proc/irq/<IRQ_NUMBER>/smp_affinity_list
Note
For multi-queue NICs, you will see multiple IRQs for a single interface.
For simple netperf testing you can repeat the echo command for every queue associated with your interface, mapping all lanes to one core.
In a production environment it might be more efficient to only steer the IRQs relevant for the application to the core running the applications.
3.2.3. Setting CPU Frequency Governor
Modern operating systems dynamically scale CPU frequencies to save power, which can introduce unpredictable latency spikes during network tests. Lock the CPU core(s) handling the network traffic to maximum performance:
$ echo "performance" | sudo tee /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
3.2.4. Network Buffer Expansion
Linux default network buffer sizes are sometimes too small to saturate high-speed (10G/25G/100G) links, leading to dropped packets at the host OS level rather than the FPGA level.
Expand the read (rmem) and write (wmem) buffers, and increase the transmission queue length:
# Expand core networking buffers
$ sudo sysctl -w net.core.rmem_max=16777216
$ sudo sysctl -w net.core.smem_max=16777216
$ sudo sysctl -w net.core.wmem_max=16777216
$ sudo sysctl -w net.core.wmem_default=16777216
$ sudo sysctl -w net.core.rmem_default=16777216
# Expand TCP-specific IPv4 buffers (min, default, max)
$ sudo sysctl -w net.ipv4.tcp_rmem='4096 87380 33554432'
$ sudo sysctl -w net.ipv4.tcp_wmem='4096 65536 33554432'
# Increase the transmit queue length on the interface
$ sudo ip link set dev <interface> txqueuelen 3000
You can verify current ring parameters using ethtool -g <interface>.
3.2.5. CPU Affinity
To further reduce the host PCs latency, bind the process to the specific CPU core you optimized (e.g., Core 1) using taskset. This prevents context switching and ensures optimal cache utilization.
$ taskset -c 1 netperf -t TCP_STREAM -H 192.168.15.1 -l 10 -- -m 8192