Configuring Networking Options
The standard network configuration on Ubuntu causes bottlenecks in the networking of the Node and Gateway communication. The following instructions detail how to modify it to achieve better networking performance.
Configuring Congestion Windows
The TCP congestion window size limits the maximum amount of data sent out to a network after a time of little operation. The default size needs to be increased to remove a bottleneck in the xx network.
This change is necessary for the xx network because the cMix protocol transmits in short bursts. As a result, the congestion windows contract between transmissions, causing them to reopen on every transmission, significantly slowing down the network in high latency environments.
The following instructions are required to be completed on both the Node and Gateway machines.
First, to prevent the congestion windows from shrinking unnecessarily when the connection is idle, disable
tcp_slow_start_after_idle
.To make these settings persist across reboots, store them in the sysctl configuration file.
Modify the TCP congestion control algorithm (CCA) to use TCP Bottleneck Bandwidth and RRT (BBR).
This should output the following.
Modify the default queuing discipline to be
fq
; this is required to use BBR.This should output the following.
Apply these two options to sysctl.conf so that they persist on reboot.
Optional: If you want to verify that the correct values were added to /etc/sysctl.conf, then print it to the console.
This will print the entire file to the terminal. The last three lines should match the following.
Restart the service(s) for the appropriate binary.
For Node:
For Gateway:
Optional Configuration to Make Initial Connection Not Slow
While disabling tcp_slow_start_after_idle
will keep the congestion windows large after connections are used a few times, you can optionally modify the initial congestion window size using the following instructions to make the minimum congestion window size large enough for cMix batches.
First, set congestion windows size on sending and receiving to be 700 times the maximum segment size.
Store the above command in /etc/rc.local, to make it run on boot.
Note, if this fails to work or causes problems, such as increased timeouts or instability, remove the file and reboot.
Discussion |
---|
The first command sets the congestion windows on sending and receiving to 700 times the maximum segment size (MSS), which is set based on the detected maximum transmission unit of the connection, typically around 1,500 bytes. Our message sizes are 8,192 bits, or 1,024 bytes. The usable part of the MSS is typically 1,480 bytes. Thus, with batch sizes of 1,000, we need approximately (1,000 × 1,024) / 1,480 ≈ 692 times the MSS, and we round up to 700 to account for overhead. |
Configuring Congestion Control Algorithm
By default, Ubuntu uses Reno and CUBIC congestion control algorithms (CCA). However, Bottleneck Bandwidth and RRT (BBR) is a newer CCA that reduces latency and increases bandwidth.
Modify the TCP congestion control algorithm to use TCP Bottleneck Bandwidth and RRT (BBR).
Modify the default queuing discipline to be fq; this is required to use BBR.
Apply these two options to sysctl.conf so they persist on reboot.