Configuring Networking Options

From xx network wiki
Jump to navigation Jump to search
This is a team contributed page

The standard network configuration on Ubuntu causes bottlenecks in the networking of the Node and Gateway communication. The following instructions detail how to modify it to achieve better networking performance.

Configuring Congestion Windows

The TCP congestion window size limits the maximum amount of data sent out to a network after a time of little operation. The default size needs to be increased to remove a bottleneck in the xx network.

This change is necessary for the xx network because the cMix protocol transmits in short bursts. As a result, the congestion windows contract between transmissions, causing them to reopen on every transmission, significantly slowing down the network in high latency environments.

The following instructions are required to be completed on both the Node and Gateway machines.

  1. First, to prevent the congestion windows from shrinking unnecessarily when the connection is idle, disable tcp_slow_start_after_idle.

  2. To make these settings persist across reboots, store them in the sysctl configuration file.

  3. Modify the TCP congestion control algorithm (CCA) to use TCP Bottleneck Bandwidth and RRT (BBR).

    This should output the following.

  4. Modify the default queuing discipline to be fq; this is required to use BBR.

    This should output the following.

  5. Apply these two options to sysctl.conf so that they persist on reboot.

  6. Optional: If you want to verify that the correct values were added to /etc/sysctl.conf, then print it to the console.

    This will print the entire file to the terminal. The last three lines should match the following.

  7. Restart the service(s) for the appropriate binary.

    For Node:

    For Gateway:

Optional Configuration to Make Initial Connection Not Slow

These changes can have negative effects in other areas of network performance, so it is generally not recommended to use these settings and you should only change these options if and only if you are certain that the rest of your network will work with these new settings.

While disabling tcp_slow_start_after_idle will keep the congestion windows large after connections are used a few times, you can optionally modify the initial congestion window size using the following instructions to make the minimum congestion window size large enough for cMix batches.

  1. First, set congestion windows size on sending and receiving to be 700 times the maximum segment size.

  2. Store the above command in /etc/rc.local, to make it run on boot.

Note, if this fails to work or causes problems, such as increased timeouts or instability, remove the file and reboot.

Discussion

The first command sets the congestion windows on sending and receiving to 700 times the maximum segment size (MSS), which is set based on the detected maximum transmission unit of the connection, typically around 1,500 bytes. Our message sizes are 8,192 bits, or 1,024 bytes. The usable part of the MSS is typically 1,480 bytes. Thus, with batch sizes of 1,000, we need approximately (1,000 × 1,024) / 1,480 ≈ 692 times the MSS, and we round up to 700 to account for overhead.

Configuring Congestion Control Algorithm

By default, Ubuntu uses Reno and CUBIC congestion control algorithms (CCA). However, Bottleneck Bandwidth and RRT (BBR) is a newer CCA that reduces latency and increases bandwidth.

  1. Modify the TCP congestion control algorithm to use TCP Bottleneck Bandwidth and RRT (BBR).

  2. Modify the default queuing discipline to be fq; this is required to use BBR.

  3. Apply these two options to sysctl.conf so they persist on reboot.