OPNSense performance optimization for gigabit speed

Many of our customers choose to run OPNsense on their routers. 

There are several "system tunables" in OPNsense that unlock extra performance on compatible hardware.

The default settings we get after OPNSense installation aren't optimal for intel NICs. All routers we offer in the store are have intel network interfaces, therefore it makes sense to tweak few settings to unlock extra performance. 

By default routing performance on APU2-APU4 will be at around 650Mbit/s on OPNsense. If you follow the steps below you will get a full gigabit speed. 

First things first:

Upgrade BIOS on APU routers to get CPU boost

(this guidance applies only to APU2, APU3 and APU4 routers). 

APU routers have AMD GX-412TC CPU which has 1.4GHz boost frequency. To enable the boost, you need to upgrade your BIOS to version v4.9.0.2 or later. See this other article for the BIOS upgrade instructions.

Enable TCP Offload Engine

Intel NICs are able to use the "hardware TCP segmentation offload". To turn this option on, go to System -> Settings -> Tunables and find "TCP Offload Engine".

Setting net.inet.tcp.tso to 1 will enable hardware segmentation offloading (TSO, TSO4, TSO6). TSO causes the NIC to handle splitting up packets into MTU-sized chunks rather than handling that at the OS level. 

In our tests this setting increased throughput. Depending on the configuration of your firewall, you may want to experiment if this setting works equally well for your load.

Enable Hardware Checksum Offloading

Similar to the setting above, Intel NICs can calculate the packet checksums in the hardware rather than at the OS level. This offloads the CPU and increases bandwith. 

To turn this option on, go to System -> Settings -> Tunables and find "UDP Checksums".

net.inet.udp.checksum should be set to 1. Checksum offloading is usually beneficial as it allows the checksum to be calculated (outgoing) or verified (incoming) in hardware at a much faster rate than it could be handled in software. 

 

Enable multi queue processing

APU routers have very performant Intel  and Network Interfaces. These NICs have 4/2 transmit and receive queues, being able to work simultaneusly on 4/2 connections. With some fine tuning, OPNSense can take advantage of this and route at 1Gbit when using more than one connection.

To enable multi queue support we want to add the following two settings, enabling multi processing on both transimitng and receiving queues.

hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
legal.intel_igb.license_ack="1"

The easiest way to do that is to add new system tunables as shown on the screenshot below.

 

Gigabit throughput verification

After these settings have been applied, your should be able to get a full gigabit throughput when using more than one connection at a time. See the test below that verifies this setup.

root@homedesktop:/home/sniku# iperf3 -c 192.168.5.175 -t 20 -P 2
Connecting to host 192.168.5.175, port 5201
[  5] local 192.168.1.102 port 51756 connected to 192.168.5.175 port 5201
[  7] local 192.168.1.102 port 51758 connected to 192.168.5.175 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  66.4 MBytes   557 Mbits/sec    0    465 KBytes       
[  7]   0.00-1.00   sec  49.2 MBytes   413 Mbits/sec    0    389 KBytes       
[SUM]   0.00-1.00   sec   116 MBytes   969 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  57.5 MBytes   483 Mbits/sec    0    551 KBytes       
[  7]   1.00-2.00   sec  55.8 MBytes   468 Mbits/sec    0    529 KBytes       
[SUM]   1.00-2.00   sec   113 MBytes   951 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  56.2 MBytes   472 Mbits/sec    0    580 KBytes       
[  7]   2.00-3.00   sec  56.3 MBytes   472 Mbits/sec    0    554 KBytes       
[SUM]   2.00-3.00   sec   113 MBytes   944 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  55.4 MBytes   464 Mbits/sec    0    580 KBytes       
[  7]   3.00-4.00   sec  55.9 MBytes   469 Mbits/sec    0    554 KBytes       
[SUM]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0             
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-4.33   sec  18.8 MBytes   473 Mbits/sec    0    580 KBytes       
[  7]   4.00-4.33   sec  19.0 MBytes   477 Mbits/sec    0    554 KBytes       
[SUM]   4.00-4.33   sec  37.8 MBytes   950 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-4.33   sec   254 MBytes   492 Mbits/sec    0             sender
[  5]   0.00-4.33   sec  0.00 Bytes  0.00 bits/sec                  receiver
[  7]   0.00-4.33   sec   236 MBytes   457 Mbits/sec    0             sender
[  7]   0.00-4.33   sec  0.00 Bytes  0.00 bits/sec                  receiver
[SUM]   0.00-4.33   sec   491 MBytes   950 Mbits/sec    0             sender
[SUM]   0.00-4.33   sec  0.00 Bytes  0.00 bits/sec                  receiver