OPNSense performance optimization for gigabit speed
Update 2022-11-26: We have received reports that these instructions don't work on the latest OPNSense 22.7 release. We will investigate in the future (as time allows) and update the instructions if possible.
Many of our customers choose to run OPNsense on their routers.
There are several "system tunables" in OPNsense that unlock extra performance on compatible hardware.
The default settings we get after OPNSense installation aren't optimal for intel NICs. All routers we offer in the store are have intel network interfaces, therefore it makes sense to tweak few settings to unlock extra performance.
By default routing performance on APU2-APU6 will be at around 650Mbit/s on OPNsense 21.1 and ~560Mbit/s on OPNSense 22.1. If you follow the steps below you will get a full gigabit speed.
First things first:
Upgrade BIOS on APU routers to get CPU boost
(this guidance applies only to APU2, APU3 and APU4 routers).
APU routers have AMD GX-412TC CPU which has 1.4GHz boost frequency. To enable the boost, you need to upgrade your BIOS to version v4.9.0.2 or later. See this other article for the BIOS upgrade instructions.
Enable TCP Offload Engine
Intel NICs are able to use the "hardware TCP segmentation offload". To turn this option on, go to System -> Settings -> Tunables and find "TCP Offload Engine".
Setting net.inet.tcp.tso to 1 will enable hardware segmentation offloading (TSO, TSO4, TSO6). TSO causes the NIC to handle splitting up packets into MTU-sized chunks rather than handling that at the OS level.
In our tests this setting increased throughput. Depending on the configuration of your firewall, you may want to experiment if this setting works equally well for your load.
Enable Hardware Checksum Offloading
Similar to the setting above, Intel NICs can calculate the packet checksums in the hardware rather than at the OS level. This offloads the CPU and increases bandwith.
To turn this option on, go to System -> Settings -> Tunables and find "UDP Checksums".
net.inet.udp.checksum should be set to 1. Checksum offloading is usually beneficial as it allows the checksum to be calculated (outgoing) or verified (incoming) in hardware at a much faster rate than it could be handled in software.
Enable multi queue processing
APU routers have very performant Intel and Network Interfaces. These NICs have 4/2 transmit and receive queues, being able to work simultaneusly on 4/2 connections. With some fine tuning, OPNSense can take advantage of this and route at 1Gbit when using more than one connection.
To enable multi queue support we want to add the following two settings, enabling multi processing on both transimitng and receiving queues.
hw.igb.rx_process_limit="-1" hw.igb.tx_process_limit="-1" legal.intel_igb.license_ack="1"
The easiest way to do that is to add new system tunables as shown on the screenshot below.
Gigabit throughput verification
After these settings have been applied, your should be able to get a full gigabit throughput when using more than one connection at a time. See the test below that verifies this setup.
root@homedesktop:/home/sniku# iperf3 -c 192.168.5.175 -t 20 -P 2 Connecting to host 192.168.5.175, port 5201 [ 5] local 192.168.1.102 port 51756 connected to 192.168.5.175 port 5201 [ 7] local 192.168.1.102 port 51758 connected to 192.168.5.175 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 66.4 MBytes 557 Mbits/sec 0 465 KBytes [ 7] 0.00-1.00 sec 49.2 MBytes 413 Mbits/sec 0 389 KBytes [SUM] 0.00-1.00 sec 116 MBytes 969 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 57.5 MBytes 483 Mbits/sec 0 551 KBytes [ 7] 1.00-2.00 sec 55.8 MBytes 468 Mbits/sec 0 529 KBytes [SUM] 1.00-2.00 sec 113 MBytes 951 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 0 580 KBytes [ 7] 2.00-3.00 sec 56.3 MBytes 472 Mbits/sec 0 554 KBytes [SUM] 2.00-3.00 sec 113 MBytes 944 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 55.4 MBytes 464 Mbits/sec 0 580 KBytes [ 7] 3.00-4.00 sec 55.9 MBytes 469 Mbits/sec 0 554 KBytes [SUM] 3.00-4.00 sec 111 MBytes 934 Mbits/sec 0 ^C- - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-4.33 sec 18.8 MBytes 473 Mbits/sec 0 580 KBytes [ 7] 4.00-4.33 sec 19.0 MBytes 477 Mbits/sec 0 554 KBytes [SUM] 4.00-4.33 sec 37.8 MBytes 950 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-4.33 sec 254 MBytes 492 Mbits/sec 0 sender [ 5] 0.00-4.33 sec 0.00 Bytes 0.00 bits/sec receiver [ 7] 0.00-4.33 sec 236 MBytes 457 Mbits/sec 0 sender [ 7] 0.00-4.33 sec 0.00 Bytes 0.00 bits/sec receiver [SUM] 0.00-4.33 sec 491 MBytes 950 Mbits/sec 0 sender [SUM] 0.00-4.33 sec 0.00 Bytes 0.00 bits/sec receiver