Node Handbook

From xx network wiki
Jump to navigation Jump to search
This is a team contributed page

The Node Handbook provides detailed instructions for Node operators on how to deploy a Node and Gateway.

Background

This manual is designed to provide the necessary information to deploy, debug, and operate a Node and Gateway on the xx network BetaNet. A Node in the network will participate in the two fundamental tasks that comprise the xx network: mixing communication through Elixxir’s cMix protocol and executing consensus through the Praxxis xx consensus.

To learn more about the xx network, Elixxir, and Praxxis, refer to the following papers.

For more information, further discussion, and additional help with this guide or general participation in the BetaNet, use the following contacts.

Continue reading below for general information about how the software works. To skip to the instructions for setting up a Node and Gateway, skip to Node and Gateway Set Up.

Architecture of the BetaNet

Currently, the Permissioning server is used to define the network and orchestrate its activities. This allows the engineering team to tune the network and bring the highly decentralized variant of cMix protocol to a workable state for MainNet. As the BetaNet software matures, xx consensus will take over roles currently held by the Permissioning server.

General Architecture

The initial BetaNet is made up of three main entities:

  1. Nodes: The core operators of the network; they execute the cMix protocol.
  2. Gateways: The public facing components of Nodes, one exists per Node. They store received messages and provide public access to data.
  3. Clients: Users use Clients to communicate on the network and are generally deployed on mobile devices.
    The Client repository is now public and community testing of the xx messenger is ongoing.

Hierarchy

The hierarchy of the BetaNet.
Figure 1: The hierarchy of the BetaNet.

BetaNet was built with a tiered Scheduler-Worker design, with all components controlled either directly or recursively by a central Permissioning server. Each member polls the entity above them in the hierarchy for information, as shown in Figure 1. Bi-directional communication only exists within the same level.

The Network Definition File (NDF) contains all the connection information for the entities in the network and is provided by the Permissioning server through the hierarchy shown. The Nodes provide the hash of their current NDF to Permissioning; if they differ then the updated NDF is provided to the Node. The Gateways poll the Nodes and Clients poll Gateways for the new NDF in the same fashion. However, the NDF provided to the Clients is stripped of Node IP addresses. In the current implementation of Clients, they currently get the NDF from Permissioning. This is expected to change by their release.

Nodes, Gateways, and Clients also receive scheduling instructions from the Permissioning server. These instructions are contained within RoundInfo structures, which are both prescriptive and descriptive of changes to rounds, which groups a set of Nodes to anonymize communications. A round is created when RoundInfo is issued to start a round’s precomputation. When the Nodes finish the precomputation, Permissioning issues a new RoundInfo that schedules it for real time, which can be delayed depending on the number of queued rounds. A further RoundInfo is issued when a round completes or fails.

Network Membership

In the network, all trust is derived from the Permissioning server. Currently, Permissioning is an information collator, but it will be replaced by the xx network consensus mechanism in the near future.

Identities in the network are defined by an asymmetric keypair. Most entities will hold this keypair within a TLS certificate, but some entities will just have a keypair. Nodes and Gateways prove their membership to the network by the inclusion of membership information in a signed NDF.

Initial Run

When first started, a Node will require two TLS certificates and a unique 256-bit randomly generated registration code provided by xx network to join the network. The certificates must be generated using RSA keys and the registration code will be provided in Base64 string format.

On first run, the Node will generate a cryptographic ID and register it with the network, via the Permissioning server. Most of the initial files and generated files are critical and must be preserved by the Node operator. For more information, refer to Cryptographic and Network Primitives.

Hardware Requirements

Below are the hardware requirements for running a Node and Gateway. Note that there are additional requirements when running a Node and Gateway on the same machine.

Node Hardware Requirements

Nodes are high powered machines that use both a CPU and GPU. The software does support full CPU Nodes, but higher core counts are required. As the software becomes more mature and more power is extracted from the GPU, it is likely that the hardware requirements for such Nodes will increase.

CPU

High core count modern CPU

Capable of meeting a multithreaded PassMark score of 15,500 or a Cinebench R5 multi score of 1,750.
Examples: AMD Ryzen 7 2700x, AMD Ryzen 5 3600x, Intel Core i9-9980HK
GPU

Nvidia Turing: Nvidia GeForce RTX 2070 or greater

Nvidia Ampere: Nvidia GeForce RTX 3060 Ti or greater

RAM

16 GB DDR4 or more

An upgrade path to 32 GB is recommended.
Storage

1 TB High Speed Enterprise NVMe (PCI) SSD

Recommended speed: 500,000/500,000 IOPS
Recommended reliability: 1.5 million hours MTBF
Example: Samsung 970 PRO SSD 1TB – M.2 NVMe
SSD reliability is not as important for the BetaNet when there will not be many transitions. It is likely that lower endurance SSDs will be fine but will need to be replaced when it comes time for MainNet. If an SSD fails, it could take substantial time before a Node is able to get back online.
Full SSD utilization will not occur at launch but will be fully utilized by xx consensus as the BetaNet matures.
Bandwidth 100 Mbps upload / 100 Mbps download

Gateway Hardware Requirements

Gateways require relatively low powered machines. Every Node must have a Gateway.

CPU

Dual core modern CPU, cloud deployment possible

Example: AMD Ryzen 3 2200G
GPU Not required
RAM 2 GB or more
Storage

250 GB

Used for database instances.
Bandwidth

100 Mbps upload / 100 Mbps download

The Gateway bandwidth requirements are separate from the Node bandwidth requirements.

Node and Gateway Hardware Requirements

It is possible to run a Gateway off the same machine as a Node; however, it results in a loss of security when they are run off the same IP address and network connection. The Gateway structure makes it more difficult for a Node to be maliciously removed from the network.

The specifications for a machine to run both a Node and Gateway differ slightly and are as follows.

CPU

High core count modern CPU

Capable of meeting a multithreaded PassMark score of 15,500 or a Cinebench R5 multi score of 1,750.
Examples: AMD Ryzen 7 2700x, AMD Ryzen 5 3600x, Intel Core i9-9980HK
GPU

Nvidia Turing: Nvidia GeForce RTX 2070 or greater

Nvidia Ampere: Nvidia GeForce RTX 3060 Ti or greater

RAM

16 GB DDR4 or more

An upgrade path to 32 GB is recommended.
Storage 1

1 TB High Speed Enterprise NVMe (PCI) SSD

Recommended speed: 500,000/500,000 IOPS
Recommended reliability: 1.5 million hours MTBF
Example: Samsung 970 PRO SSD 1TB – M.2 NVMe
SSD reliability is not as important for the BetaNet when there will not be many transitions. It is likely that lower endurance SSDs will be fine but will need to be replaced when it comes time for MainNet. If an SSD fails, it could take substantial time before a Node is able to get back online.
Full SSD utilization will not occur at launch but will be fully utilized by xx consensus as the BetaNet matures.
Storage 2

240 GB High Speed Enterprise SSD

Recommended speed: 100,000/10,000 IOPS
Recommended reliability: 1.5 million hours MTBF
Example: 883 DCT 240GB
SSD reliability is not as important for the BetaNet when there will not be many transitions. It is likely that lower endurance SSDs will be fine but will need to be replaced when it comes time for MainNet.
Bandwidth 150 Mbps upload / 200 Mbps download

Turing or Ampere GPU Requirement

The xx network BetaNet requires a Nvidia Turing or Ampere GPU due to upgrades to the microarchitecture of the shaders granting an order of magnitude increase in performance for the specific workloads used. The mul units have been increased to 32 bits, which greatly increases the speed of the modular exponentiation and modular multiplication which are some of the core cryptographic operations of the cMix protocol.

Software Overview

Figure 2: Overview of the codebase showing relationships and dependencies. Arrows indicate dependencies. The source code for Primitives, Crypto, Comms, GPUMaths, Server, Gateway, Client, User Discovery, Integration, Local Environment, and the Wrapper Script are available on the Elixxir and xx network gits.

The xx network BetaNet codebase is combined between xx network’s consensus software and Elixxir’s cMix protocol. At launch, xx network will be testing and tuning Elixxir software with xx consensus coming online at a later date.

Elixxir software is organized into four groups of repositories as shown in Figure 2: Core, Services, Clients, and Tools. The Core libraries handle functionality across Clients and Services, which contain data structures, interfaces, and cryptography common to many of the xx network components. Services implements various messenger features, as well as authorizing Clients and Nodes to join the network. Clients interact with the network using a shared Client API (xxDK). Tools provide several interfaces and utilities to deploy, test, and debug the network.

Most of the code found in the Core and Services is written in the Go programming language. The repositories listed as public below can be found on the public Elixxir and xx network GitLab pages.

To follow the set up guide, most Node operators will likely elect to download the prepackaged tarballs containing the compiled binaries. To learn more, refer to the Node and Gateway Software section.

xx network Codebase

Core

xx network/Primitives
name: primitives project: xx network availability: public git: gitlab.com/xx_network/primitives

The xx network Primitives repository contains the basic data structures and utilities that are used in the Elixxir and Praxxis codebases. This includes the ID structure for Nodes, Gateways, and users and the NDF structure. Additional generic structures and utilities are also contained in Primitives, such as file access and rate limiting utilities.

xx network/Crypto
name: crypto project: xx network availability: public git: gitlab.com/xx_network/crypto

The xx network Crypto repository contains cryptographic primitives that are shared between Elixxir and Praxxis codebases. Primarily lower level primitives, randomness generation, TLS handling, and nonce handling.

xx network/Comms
name: comms project: xx network availability: public git: gitlab.com/xx_network/comms

The Comms repository handles all network communication functionality as well as all of the core connectivity logic. It includes generic comms handlers and protocols and not specific implementations, which are implemented in the respective Comm branches for each project.

Elixxir Codebase

Core

Elixxir/Primitives
name: primitives project: elixxir availability: public git: gitlab.com/elixxir/primitives

The Elixxir Primitives repository contains the basic data structures and utilities that are used by all Elixxir repositories. This includes the user fact structure, the cMix message structure, and the version object.

Elixxir/Crypto
name: crypto project: elixxir availability: public git: gitlab.com/elixxir/crypto

The Crypto repository encompasses all of the base cryptographic functionality found in the code base. Relying heavily on Go’s big integer implementation, Crypto features a cryptographically secure random number generator implementation, libraries for working with large integers in modulo cyclic groups for cMix operations, and basic encrypt/decrypt functionality. Like Primitives, Crypto only contains code that is generic to the larger system. A core approach to this repository is to supply wrappers for operations that may require migration to other implementations in the future.

Elixxir/Comms
name: comms project: elixxir availability: public git: gitlab.com/elixxir/comms

Comms builds on the generic utilities of xx network Comms to provide specific functionality for Elixxir. It holds a gRPC protocol file along with a thin Client/Server implementation. The repository currently uses TLS certificates with RSA keys for encryption and identification, which will migrate to xx consensus-based quantum secure authenticated channels as development progresses.

Elixxir/GPUMaths
name: gpumathsgo project: elixxir availability: public git: gitlab.com/elixxir/gpumathsgo

The GPUMaths repository accelerates the math used by Server, especially for precomputations. It provides a subset of the math implemented in the Crypto repository but accelerated on GPUs. Underlying the acceleration is a publicly available CUDA arbitrary-precision math library, CGBN2. Components of this written in CUDA are in a separate repository gpumathsnative.

Services

Elixxir/Server
name: server project: elixxir availability: public git: gitlab.com/elixxir/server

The Server repository implements the core cMix functionality and is the software that a Node runs. It performs precomputation and real time computation and processes messages. It receives batches of messages from the Gateways as well as performs network team operations.

Elixxir/Gateway
name: gateway project: elixxir availability: public git: gitlab.com/elixxir/gateway

The Gateway repository contains the API for Clients to interact with the network. Every Node runs a Gateway and the Gateways collect and store messages for Clients. Gateway is designed to be a scalable front-end to the xx network.

Elixxir/Permissioning
name: registration project: elixxir availability: private

Permissioning, also referred to as Registration, manages the NDF for Clients and Servers and schedules cMix rounds within the network. Eventually this functionality will be managed by the distributed xx consensus. For now, this code handles admission, manages which Nodes are part of the network, and orchestrates when Nodes operate.

Clients

Elixxir/Client API (xxDK)
name: client project: elixxir availability: public git: gitlab.com/elixxir/client

All Clients use the Client API to interact and send messages with the cMix network. It uses Go mobile to produce a library compatible with iOS and Android.

Elixxir/User Discovery (UD)
name: user-discovery-bot project: elixxir availability: public git: gitlab.com/elixxir/user-discovery-bot

User Discovery helps users make first contact with other users. It facilitates user search and lookup in a private manner.

Elixxir/Mobile Clients
name: client-ios project: elixxir availability: private
name: client-android project: elixxir availability: private

Currently, clients exist for iOS and Android operating systems. These both use the Go mobile libraries produced in the Client API.

Tools

Elixxir/Wrapper Script
name: wrapper project: elixxir availability: public git: gitlab.com/elixxir/wrapper

The Wrapper Script is a Node and Gateway management script that simplifies the running of the xx network software. The script automates the management of the xx network software log files. For easy management or in the event of an error, it starts, stops, and restarts the software without the operator having to revisit the command line. Optionally, it can be set to automatically update the Node and Gateway with the latest xx network binaries and configuration files. To learn more, refer to the Wrapper Script Arguments section.

Elixxir/DevOps

(deployment)

name: deployment project: elixxir availability: private

DevOps is a deployment platform for Microsoft Azure, Google Cloud Platform, and Amazon Web Services (AWS) written in Terraform. DevOps allows for the deployment of test networks, management of deployments of individual Nodes, and the deployment of multi-cloud implementations of xx network.

Elixxir/Integration integration [public]
name: integration project: elixxir availability: public git: gitlab.com/elixxir/integration

The Integration repository is a series of end-to-end tests designed to test different functionality of the cMix protocol. Several tests focus on different batch sizes with Nodes only. Another test covers all the Client-level interaction within the network.

Elixxir/Local Environment
name: LocalEnvironment project: elixxir availability: public git: gitlab.com/elixxir/LocalEnvironment

The Local Environment repository contains a set of scripts designed to allow the testing of the entire platform on a single machine.

Version Scheme

Both Gateway and Node binaries have a version string embedded in them, which consists of a major version, a minor version, and a patch string, separated by a period.

Major Minor Patch
┌┴┐ ┌┴┐ ┌─┴─┐
4 . 2 . 6ab

To be able to participate in the network, a Node or Gateway has to have a compatible version with the required version reported by the Permissioning server. For a version to be compatible, the major version must be equal to the required major version and the minor version must be greater than or equal to the required minor version. The patch can be anything, but it will always be present.

Management Tools

Management Scripts Features

The management scripts are designed to make administration of Nodes and Gateways easier and less time consuming as well as provide data back to the xx network about the functionality of the Node. Many features that it provides require trusting the xx network. The usage of the management scripts is not required and can be disabled. The following are the primary features of these scripts.

  • Binary restart: In the event that the binary process stops due to a handled error, the Wrapper Script will restart it. The process can crash because, in the current implementation, some errors are not fully handled when caused by Node failures on the network. It is highly recommended that this feature is utilized.
  • Binary update: The Wrapper Script has the ability to accept automatic binary and Wrapper Script updates provided by the xx network. In theory, the xx network can push any code it wants if this feature is activated, but it greatly simplifies the task of running a Node. Included in this feature is the ability to send stop and start commands to the Node. All commands are signed and can be proved to have come from the xx network. Running without this feature is fully supported but requires manual updates.
  • Log upload: This features uploads logs to xx network for debugging. It can be disabled and is not mandatory but is useful to the xx network for development purposes.

The Management Script is two scripts combined, the service file and the actual Wrapper Script. The service file is a systemd script that starts the Wrapper Script and ensures it stays running. The wrapper script is written in Python and does most of the heavy lifting. Both can be found in the Elixxir wrapper repository.

Service File Arguments

The service files maintain the Node and Gateway processes running the background. There is a different service for each. By default, these files (xxnetwork-node.service and xxnetwork-gateway.service) are located in /opt/xxnetwork/ and are soft linked to /etc/systemd/system/. They are the same script just configured differently for Node and Gateway. These scripts call the Wrapper Script with different options. To see the full details of the options, refer to the Wrapper Script Arguments section below.

Node Service File Arguments

The following are the available options in the service file configured for Node. The sections in red should never be modified and should maintain the original values as provided. The items in blue can be modified.

ExecStart=/bin/bash -c '/opt/xxnetwork/xxnetwork-wrapper.py
 --logpath /opt/xxnetwork/node-logs/node.log
 --binary /opt/xxnetwork/bin/xxnetwork-node
 --s3path server
 --s3logbucket alphanet-logs-prod
 --s3managementbucket alphanet-management-prod
 --s3accesskey ${s3_access_key_id}
 --s3secret ${s3_access_key_secret}
 --s3region us-west-1
 --configdir /opt/xxnetwork/
 --erroutputpath /opt/xxnetwork/node-logs/node-err.log
 --tmpdir /tmp/xxnetwork/node
 --idpath /opt/xxnetwork/node-logs/nodeIDF.json >> /opt/xxnetwork/node-logs/xxnetwork-wrapper.log 2>&1'

Gateway Service File Arguments

The following are the available options in the service file configured for Gateway. The sections in red should never be modified and should maintain the original values as provided. The items in blue can be modified.

ExecStart=/bin/bash -c '/opt/xxnetwork/xxnetwork-wrapper.py
 --logpath /opt/xxnetwork/gateway-logs/gateway.log
 --binary /opt/xxnetwork/bin/xxnetwork-gateway
 --s3path gateway
 --s3logbucket alphanet-logs-prod
 --s3managementbucket alphanet-management-prod
 --s3accesskey ${s3_access_key_id}
 --s3secret ${s3_access_key_secret}
 --s3region us-west-1
 --configdir /opt/xxnetwork/
 --erroutputpath /opt/xxnetwork/gateway-logs/gateway-err.log
 --tmpdir /tmp/xxnetwork/gateway
 --idpath /opt/xxnetwork/gateway-logs/gatewayIDF.json >> /opt/xxnetwork/gateway-logs/xxnetwork-wrapper.log 2>&1'

Wrapper Script Arguments

The following arguments should be modified inside the service file described above.

The s3* options are set per the deployment package sent to you and should not be changed. The other options can be changed per your installation, but it is highly recommended that you go with the defaults sent with the install package.
usage: wrapper.py [-h] [-d] [-l LOGPATH] [-i IDPATH] -b BINARY [--gpulib GPULIB] [--gpubin GPUBIN] [--disable-consensus] [--consensus-binary CONSENSUS_BINARY] [--consensus-config CONSENSUS_CONFIG] [--consensus-state CONSENSUS_STATE] [--consensus-log CONSENSUS_LOG] [--consensus-cw-group CONSENSUS_CW_GROUP] [-c CONFIGDIR] -s S3PATH [-m S3MANAGEMENTBUCKET] [--disable-cloudwatch] [--cloudwatch-log-group CLOUDWATCH_LOG_GROUP] --s3accesskey S3ACCESSKEY --s3secret S3SECRET --s3region S3REGION [--tmpdir TMPDIR] [--cmdlogdir CMDLOGDIR] [--erroutputpath ERROUTPUTPATH] [--configoverride CONFIGOVERRIDE]
Flag Default Required
-h, --help False
Show this help message and exits.
-d, --disableupdates False False
Disables automatic updates.

Used to disable the download and automatic installation of new binaries, the default configuration file, the wrapper script itself, and the security certificate used for verifying wrapper script commands.

-l LOGPATH, --logpath LOGPATH /opt/xxnetwork/logs/xx.log False
The path to store logs, e.g., /opt/xxnetwork/node-logs/node.log.
-i IDPATH, --idpath IDPATH /opt/xxnetwork/logs/IDF.json False
Node ID path, e.g., /opt/xxnetwork/logs/nodeIDF.json.

The Node stores the ID file at this path.

-b BINARY, --binary BINARY True
Path of the binary to be run by the wrapper.
--gpulib GPULIB /opt/xxnetwork/lib/libpowmosm75.so False
Path to the GPU exponentiation library.
--gpubin GPUBIN /opt/xxnetwork/lib/libpow.fatbin False
Path to the GPU bin file.
--disable-consensus False False
Disable consensus binary.
--consensus-binary CONSENSUS_BINARY /opt/xxnetwork/bin/xxnetwork-consensus False
Path to the consensus binary.
--consensus-config CONSENSUS_CONFIG /opt/xxnetwork/consensus.yaml False
Path to the consensus config file.
--consensus-state CONSENSUS_STATE /opt/xxnetwork/consensus.tar.gz False
Path to the consensus state tarball.
--consensus-log CONSENSUS_LO /opt/xxnetwork/consensus-logs/consensus.log False
Path to the consensus log file.
--consensus-cw-group CONSENSUS_CW_GROUP xxnetwork-consensus-prod False
CW log group for consensus logs
-c CONFIGDIR, --configdir CONFIGDIR /opt/xxnetwork/ False
Path to the config directory, e.g., ~/.xxnetwork/.
--configoverride CONFIGOVERRIDE False
Override for config file path.

If this flag is set, then the script ignores the --configdir flag and uses this file path instead.

-s S3PATH, --s3path S3PATH True
Path to the S3 management directory.
Do not modify.
-m S3MANAGEMENTBUCKET, --s3managementbucket S3MANAGEMENTBUCKET False
Path to the S3 management bucket name.
Do not modify.
--disable-cloudwatch False False
Disable uploading log events to CloudWatch.
--cloudwatch-log-group CLOUDWATCH_LOG_GROUP xxnetwork-logs-prod False
Log group for CloudWatch logging.
--s3accesskey S3ACCESSKEY True
S3 access key.
Do not modify.
--s3secret S3SECRET True
S3 access key secret.
Do not modify.
--s3region S3REGION
S3 region.
Do not modify.
--tmpdir TMPDIR /tmp False
Directory for temporary files.
--cmdlogdir CMDLOGDIR /opt/xxnetwork/cmdlog False
Directory for commands log.
--erroutputpath ERROUTPUTPATH False
Path to recovered error path.

The Node stores recoverable errors at this path.

Deployment Schedule

While BetaNet is a functioning network, it is still in development and requires consistent updates. If the Wrapper Script is being used in its default configuration, then Node and Gateway will automatically update.

However, for Node operators with automatic updates turned off, xx network is committed to the following update schedule to allow them ample time to deploy updates.

Scheduled Updates

Every week, there is a single window where major releases and breaking minor releases can be downloaded and installed. The source code will be released on Tuesday before the update goes live on Thursday. Once available on Thursday, any Nodes not up to date cannot participate in the network but will be considered online for the purpose of compensation. Node operators will have until Monday to install the release, after which time they will be considered offline for the purpose of compensation. Refer to the following timeline for specific times and descriptions.

All updates will be posted to the Elixxir GitLab and Node operators will be informed by email. The xx network will strive to post notice of updates before the Tuesday release, if possible.

Tuesday
1:00 pm PT – Release of source code.
Wednesday
Thursday
1:00 PM PT – Push to Permissioning server, Nodes with out of date code stop functioning.
1:00 PM PT – Push to Nodes with automatic updates enabled.
1:00 PM PT – Grace period for Nodes to update begins.
Friday
Saturday
Sunday
Monday
1:00 PM PT – Grace period for Nodes to update ends, Nodes that have not updated will be considered offline for the purpose of compensation.

Critical Unscheduled Updates

Critical updates may be released with no notice in the event of catastrophic failure or immediate danger to the network.

Nodes will be given 5 business days to update their Node for the purpose of compensation in the event of such an update.

Cryptographic and Network Primitives

Two TLS credentials (a certificate and private key pair) and an identification file define the identity of a Node. These files are extremely important to back up and store securely. Loss of these files will make it impossible to reconnect a Node to the network. In following the default guide, these files are named node_cert.crt, node_key.key, gateway_cert.crt, gateway_key.key, and nodeIDF.json. Refer to the Lost Certificate and Important Files Policy for more information.

There are various cryptographic primitives used in the xx network; some are provided by the xx network and some are generated by the Node operator.

Primitives Provided by xx network

The following primitives are provided to you by xx network, either emailed directly or they are downloaded from public repositories.

Registration Code Sent via email The registration code is a 256-bit random number encoded in Base64. It is a single-use code used only on initial registration after which, the Node’s ID and TLS certificate can be used to identify it within the network. This will be provided to each Node operator prior to the network going live.
Permissioning TLS Certificate Provided inside the tarball A TLS certificate is provided to be used to identify the Permissioning server.
Permissioning Server Address Provided inside the tarball The DNS address of the Permissioning server is pre-populated inside the YAML files for Node and Gateway.
Wrapper Script Bucket Address Provided inside the tarball The IP address the Wrapper Script uses to update the binaries. It is pre-populated in the Wrapper Script.
Log Bucket Address Provided inside the tarball The IP address the Wrapper Script uses to upload logs to. It is pre-populated in the Wrapper Script.
cMix Cyclic Group Provided as part of the NDF on initial registration The cMix cyclic group is the modulo cyclic group in which cMix operations are conducted. At launch, this will be defined by a 2048-bit strong and safe prime with a generator of 2.
E2E Cyclic Group Provided as part of the NDF on initial registration The E2E cyclic group is the modulo cyclic group in which end to end encryption operations by clients will be conducted. At launch, this will be defined by a 3192-bit strong and safe prime with a generator of 2.

Primitives Created by the Node Operator

The following primitives are generated prior to and during the initial registration of the Node. More details about generation can be found in Generate TLS Credentials.

TLS Credentials

The TLS credentials are generated as X.509, SHA-256, RSA 4096-bits, and are recommended to last 730 days (2 years). The generated credentials include:

  • Node TLS certificate
  • Node private key
  • Gateway TLS certificate
  • Gateway private key

ID File (IDF)

IDs are 264 bits and consist of a 256-bit hash (BLAKE2b) of a salt and the RSA public key from the Node’s TLS certificate appended with a 8 bits that describe the ID type.

ID = Hash(Node_PubKey, Salt) + TYPE

Due to the construction of the ID, the ownership of the ID can be proved under RSA’s cryptographic assumptions. Due to the hash, IDs are unpredictably generated in a large sparse space so it is overwhelmingly improbable that independently generated IDs will ever collide, so no central checking of ID generation is necessary.

The last byte of the ID is a type byte which describes what type of entity it belongs to. The options for type are:

Type Hex Note
Generic 0x00 Used for one-off entities such as the Permissioning server, the notifications server, and others
Gateway 0x01 The Gateways in the network
Node 0x02 The Nodes in the network
User 0x03 Unique client ID

On first run, the Node automatically generates the ID and saves it to a JSON file with the following structure.

{
	"id": "1Mo9im6HHpoTDt/lTYjSkWV7dAD0Eh+18xUFfxlm4I4C",

The ID of the Node or Gateway, presented as a base 64 encoded string.

	"type": "node",

The type of ID is shown here for readability.

	"salt": [119, 234, 246, 209, 166, 166, 72, 17, 253, 196, 172, 187, 230, 2, 132, 137, 49, 219, 142, 58, 82, 169, 60, 230, 112, 17, 30, 112, 30, 68, 217, 92],

The salt used to generate the ID.

	"idBytes": [212, 202, 61, 138, 110, 135, 30, 154, 19, 14, 223, 229, 77, 136, 210, 145, 101, 123, 116, 0, 244, 18, 31, 181, 243, 21, 5, 127, 25, 102, 224, 142, 2],
}

The ID of the Node or Gateway, presented as an array of bytes.

Lost Certificate and Important Files Policy

As of April 27, 2021, the following guidelines and policies are in effect.

  • The Node and Gateway identities are defined by the certificates, their private keys, and the IDF files. Loss of these files will render your Node and Gateway unable to prove their identity and unable to rejoin the network.
  • Backup and retention of these files are the sole responsibility of the Node operator.
  • The xx network team nor anyone else can recover a Node or Gateway if these files are lost.

Node and Gateway Set Up

Below you will find the instructions for setting up a Node and Gateway on the xx network for the first time. These instructions detail how to install both a Node and Gateway on separate machines as well as installing them together on the same machine. Follow the instructions once for each machine.

The installation instructions assume that your machine has an active internet connection and an empty storage drive or a drive that can be written over with enough space as outlined in the Hardware Requirements section.

Overview

Setting up an xx Node and Gateway is not particularly difficult, but due to the number of components, can take up to 2 hours. The steps for setting up a Node and Gateway are almost identical and the instructions can be followed for both. The general steps for setting up a Node are:

  1. Install the operating system
  2. Synchronize the system clock (NTP)
  3. Configuring the local network
  4. Install the GPU drivers (if setting up a Node with GPU)
  5. Install and configure the Node and Gateway software
  6. Generate TLS credentials
  7. Install and configure the databases
  8. Set up and start the services
  9. Backup important files
    Make sure to not skip this important step!
Some Tips for Inexperienced Users

If this is your first time using a command line interface or you do not remember how to use it, the following are some tips to make using the interface a little easier.

  • In this document, anytime code is presented in a black box with monospaced font it means that it is command line input or output. Commands prefixed by a $ are commands that you should enter into your command prompt (do not include $ in the command). Any lines without that prefix are output from the system.
  • The sudo command is often prepended to commands found in these instructions. This enables commands to be run with elevated privileges. When used, the system will ask for your password to continue running the command.
  • Whenever the system asks for a password to continue, no characters will appear when typing, but type in your password and press Enter and it will work.
  • When typing a command or path, use the Tab key to auto complete a partially written statement.

Installing the Operating System

The xx network software has been tested only on Ubuntu Server 18.04. We cannot guarantee support if a different operating system version is used, although no decisions have been made to specifically preclude any operating systems.

  1. Download Ubuntu Server install image from the Official Ubuntu website.

    Click on the link "64-bit PC (AMD64) server install image" to download the Ubuntu Server image.
  2. Next, a bootable disk with Linux needs to be created. This can be done by writing it to a DVD or more commonly, a flash drive. Follow one of the following tutorials on how to do so depending on your current operating system and chosen media.

  3. Once your flash drive or DVD are ready, follow the Tutorial on Installing Ubuntu Server.

    1. In step 6, make sure you select the first option Install Ubuntu.

    2. In step 7, make sure to configure your internet connection and get an IP address.

    3. In step 8, ensure that you select Use an Entire Disk.

    4. In step 12, pick a server name that does not have any personal identifying information and select a strong password.

      Create a strong but memorable password. It is recommended that it is longer than 12 characters. Store this password in a safe and secure location. Never share this password with anyone.
  4. Make sure the machine has turned back on and login using the credentials created in the previous step. Sometimes extra text will be printed and you will not be able to see the login prompt. The prompt should still be there; just type your username and press Enter to continue.

Check Internet Connection

The rest of the instructions require internet access. Follow these steps to ensure the machine is connected.

  1. Check your current local connection status and local IP address.

    This should result in a similar output to below. The machine should have a valid local IP address.

  2. Finally, check that the machine has access to the internet by pinging another server. We chose to use xx.network, but you can use any domain.

    If the machine is connected to the internet, the output should look similar to the following example.

Updating Software and Installing Dependencies

To ensure all the software works correctly, the system needs to be updated. In addition, an additional dependency for the Wrapper Script to function.

  1. Before continuing, check for updates. This will print many lines about what software is being checked.

  2. Once the check is done, install the updates.

  3. Reboot the machine to ensure all updates are installed fully. Once the machine starts up, log back in.

  4. Install the Python package installer.

  5. Update the package installer.

  6. Install the boto3 and pyOpenSSL dependencies. The Wrapper Script uses the first package to read commands and send logs to xx network through AWS and the second is used to authenticate them.

Setting Up UFW

Uncomplicated Firewall (UFW) is the default firewall configuration tool for Ubuntu. UFW should already be installed; the following instructions will describe how to enable it and allow the correct ports to be open.

  1. First, ensure that UFW is disabled so that it can be configured.

    This should result in the following output.

  2. Allow the port that the Node will use to communicate on over TCP. Only do this for machines running the Node software.

    Port 11420 is the default port in the provided Node configuration file. A different port may be used, but it must be configured in node.yaml, which is downloaded in a future step.
  3. Allow the port that the Gateway will use to communicate over TCP. Only do this for machines running the Gateway software.

    Port 22840 is the default port in the provided Gateway configuration file. A different port may be used, but it must be configured in gateway.yaml, which is downloaded in a future step.
  4. If you want to manage your node remotely, you may want to allow port 22 over TCP for SSH. SSH should only be enabled with key authentication. Improper SSH setup can allow unwanted access so take care when setting up remote access. If you do not want to use SSH or do not know how to, then skip this step.

  5. Finally, enable UFW.

    If you are connected over SSH, you may be prompted to continue, press Y. Note that you may be disconnected when doing so.

Clock Synchronization (NTP)

Commands received from the Permissioning server are time-stamped and a synchronized clock is important to properly interpret them. To do so, NTP (Network Time Protocol) must be set up and synchronized.

In Ubuntu Server 18.04, this is done through timedatectl. In most installations, it is already running and Node operators only need to check that it is correctly configured. However, the process for other operating systems may be different and it will need to be enabled.

  1. Check if the time synchronization service is running and that the clock is synchronized by entering the following command.

    This will print the following output.

    1. If System clock synchronized is set to yes and systemd-timesyncd.service active is set to yes, then no further action is needed. Skip to the next section Modifying Max Number of Processes and Files. Otherwise, continue to the next step.

    2. If System clock synchronized is set to no and systemd-timesyncd.service active is set to yes, then the service has had insufficient time to synchronize the clock. Skip to the next section Modifying Max Number of Processes and Files but make sure to check that the clock is synchronized before starting the Node or Gateway software. Otherwise, continue to the next step.

      Ensure the time is synchronized before starting the software in the System Services section.
    3. If System clock synchronized is set to no and systemd-timesyncd.service active is set to no, then the service must be manually started in the next step.

  2. Enter in the following command to get a list of time zones.

    This will print a list of time zones. Use the up key and down key to navigate the list and find the time zone that the machine is in. Once found, make a note of the time zone, and press Q to exit.

  3. Once the correct time zone for the machine is found, use the following command to set it.

  4. Using the date command, ensure that the correct time zone was selected. If the printed time is incorrect, return to step 2 to select a new time zone.

  5. Begin the clock synchronization service by entering the following command.

  6. Ensure that the service is running by calling timedatectl again.

    If systemd-timesyncd.service active is set to yes, then it has been successful. If System clock synchronized is set to yes, then this section is done. If it is set to {{inline-code|no||, then it may take up to 30 minutes for the clock to synchronize. You can continue to the following section, but make sure to check that the clock is synchronized before starting the Node or Gateway software.

Modifying Max Number of Processes and Files

By default, there is a maximum number of processes and files that can be opened at once. To prevent the Node and Gateway from encountering this limit, it must be removed for all users.

  1. To remove this limit for users, open the limits configuration file using nano or your favorite text editor.

    The following process of using the nano text editor to modify a file is used elsewhere in this document. Refer back here for detailed steps on how to use it.
  2. Once the file is open, use the down arrow key to go to the second to last line above where it says # End of file. Add the following four lines above that line.

  3. To save the file, press Ctrl+X and when prompted to save the buffer, press Y.

  4. Finally, when prompted with the file name, press Enter to exit.

  5. Once the change has been made, reboot the system.

Configuring TCP Networking Options

The congestion window size limits the maximum amount of data sent out to a network after a time of little operation. The default size needs to be increased to remove a bottleneck in the xx network.

This change is necessary for the xx network because the cMix protocol transmits in short bursts. As a result, the congestion windows contract between transmissions, causing them to need to reopen on every transmission, significantly slowing down the network in high latency environments.

The following instructions are required to be completed on both the Node and Gateway machines.

  1. First, to prevent the congestion windows from shrinking unnecessarily when the connection is idle, disable tcp_slow_start_after_idle.

  2. To make these settings persist across reboots, store them in the sysctl configuration file.

  3. Modify the TCP congestion control algorithm (CCA) to use TCP Bottleneck Bandwidth and RRT (BBR).

  4. Modify the default queuing discipline to be fq; this is required to use BBR.

  5. Apply these two options to sysctl.conf so they persist on reboot.

Optional Configuration to Make Initial Connection Not Slow

While disabling tcp_slow_start_after_idle will keep the congestion windows large after connections are used a few times, you can optionally modify the initial congestion window size using the following instructions to make the minimum congestion window size large enough for cMix batches.

Please note that these changes can have negative effects in other areas of network performance, so it is generally not recommended to use these settings and you should only change these options if and only if you are certain that the rest of your network will work with these new settings.
  1. First, set congestion windows size on sending and receiving to be 700 times the maximum segment size.

  2. Store the above command in /etc/rc.local, to make it run on boot.

Note, if this fails to work or causes problems, such as increased timeouts or instability, remove the file and reboot.

Configuring Local Network (Port Forwarding)

To ensure that the machine can be accessed from outside the local network, the network’s router or gateway must be configured to allow external access to the machine on ports configured above. Three main pieces of information are needed for this part: (1) the port numbers to forward (the defaults are 11420 and 22840), (2) the protocol to use (TCP), and (3) the local IP address of the machine, which is retrieved below.

  1. Get the local IP address of the machine. If Node and Gateway are being run off separate machines on the same network, then run this on both machines.

    The local IP address will be printed; it will be in the form of 0.0.0.0. Make sure to make note of this for the later steps.

    If the machine has multiple network interfaces or an IPv6 address, they will also appear in this list. Ensure that only the correct internal IPv4 address is used.


The following section describes how to configure the networking equipment on your network. Because of the varying type of equipment configurations, these instructions are generic and may not be accurate for your hardware. Please refer to the manufacturer’s instructions for more detailed and accurate information. Configuration of the network will most likely occur from a different machine on the network.
  1. To access the router to configure, its IP address is needed.

    This will output the following line, where the first address printed is the router IP address.

  2. Go to this IP address in a browser (on a different machine) and login using the router credentials. These credentials are either set up by the network administrator or are the default credentials located on the router or found online.

  3. Locate the port forwarding options (occasionally called virtual server). These options are sometimes found under the advanced section.

    Before forwarding the port found above, you may want to provide a static local IP address to the machine. However, that is outside the scope of these instructions.
  4. Add the new ports to forward for the Node and Gateway. For each, create a new entry and enter the IP address found in step 1, set the port to the chosen ports (by default it is 11420 for Node and 22840 for Gateway), and select the TCP protocol. Make sure to save or apply the changes.

GPU Drivers

The Node software requires a Nvidia RTX graphic processor and the installation of its drivers.

  1. Install the Nvidia driver.

  2. Once the installation is complete, reboot the system.

  3. Once the system reboots, log back into the computer.

Verifying the Driver Installation

  1. Check that the system has claimed the device.

    This should result in a similar output to the following.

  2. Next, check the driver and state information.

    This should result in a similar output to the following.

Node and Gateway Software

  1. Change the working directory to /opt/.

  2. Download the tarball for Node and/or Gateway using curl with the following URLs. The newest versions can also be found on the xx network website.

    For Node:

    For Gateway:

  3. Next, the downloaded files need to be extracted to the /opt/ directory. Use tar extract them with the flags -xf so that the directory structure remains intact. This will create a new /opt/xxnetwork/ directory with the extracted files inside.

    For Node:

    For Gateway:

    Both node.tar.gz and gateway.tar.gz contain some files that are the same (e.g., xxnetwork-wrapper.py). When extracting both files, it is expected that these files will be overwritten. Order of extraction does not matter.
    If this command produces an error similar to not in gzip format, then that means the file did not download correctly in the previous step. Try downloading the file again and ensure the URL is correct.

    The file needed to install the software are in the newly created /opt/xxnetwork/ directory. Those files are:

    Node Gateway Note
    bin/xxnetwork-node bin/xxnetwork-gateway The binary for Node or Gateway.
    node.yaml gateway.yaml The configuration file for the binary.
    xxnetwork-node.service xxnetwork-gateway.service systemd service file.
    xxnetwork-wrapper.py xxnetwork-wrapper.py A management script for the binary.
    generate_certs.py generate_certs.py TLS certificate/key generator script.
    creds/network_management.crt creds/network_management.crt The TLS certificate used to authenticate commands sent to the Wrapper Script.
    creds/permissioning_cert.crt creds/permissioning_cert.crt The TLS certificate for Permissioning.
    lib/libpowmosm75.so GPU dependency (for Node only).
    lib/libpow.fatbin GPU dependency (for Node only).

Generate TLS Credentials

TLS credentials for the Node and Gateway are critical to its identity in the network. For more information refer to Cryptographic and Network Primitives. If running Node and Gateway on different machines, then generate the certificates on the Node machine and copy the necessary credentials to the Gateway server.

The two TLS credentials (a certificate and private key pair) generated in the following section are extremely important to back up and store securely. Loss of these files will make it impossible to reconnect a Node to the network. In following the default configuration, these files are named node_cert.crt, node_key.key, gateway_cert.crt, and gateway_key.key. In addition, it is important to back up the file nodeIDF.json, which is generated on the initial run. Refer to the Lost Certificate and Important Files Policy for more information.
  1. Using the provided generate_certs.py script, generate the TLS certificates and keys for Node and Gateway. It will provide prompts for all the information required. Start the script using Python 3. Note that this must be run inside the /opt/xxnetwork/ directory.

    You do not need to use this script to generate the required TLS credentials. Use the requirements in the TLS Credentials section to ensure the generated certificates meet the network requirements.
  2. The script will launch and query for various information to create the credentials for Node and Gateway. If you choose not to provide any details, default values will be used. After every answer press Enter.

    Once all the details have been provided, the certificates will be generated and moved into creds/.

  3. To check that all the correct files are in the creds/ directory, use ls to list the files.

  4. If running Gateway on a separate machine from Node, then copy gateway_cert.crt, gateway_key.key, and node_cert.crt to the Gateway machine and delete gateway_key.key from the Node machine.

Configuring Node and Gateway

Configuring Node and Gateway requires modifying their respective YAML files retrieved in the previous steps. These instructions go over the basic configuration to get a Node and Gateway working in the default state. Refer to the Server and Gateway README files to learn more. Note that all addresses must be IPv4; IPv6 is not currently supported.

If you are currently configuring the Node on its own machine, then proceed to Node Configuration.
Gateway Configuration

The default gateway.yaml file only requires setting the address of its Node. However, there are other optional flags for non-standard configurations. Database information is set in a later step.

  1. Get the IP address of the Node. If the Node is being run on the same machine as Gateway, then this IP address is 0.0.0.0:port (where the default port is 11420). If the Node and Gateway are installed on separate machines, then you will need to get the Node’s public IP address, by running the following command on the Node.

    This should be run on the Node machine.
  2. The Gateway configuration file has to be modified to include this IP address. To do this, on the Gateway machine, open gateway.yaml in nano or your favorite text editor.

  3. Once the file is open, use the down arrow key to get to the line that says nodeAddress and replace the placeholder with the one found in step 1. Make sure to include the chosen port, as shown.

  4. For non-default setups and experienced users, the following options can be used.

    1. By default, the Gateway runs on port 22840; however, this can be modified using the port flag.

    2. The internal listening address of the Gateway defaults to 0.0.0.0. However, a different internal address can be set via listeningAddress. This expects an IPv4 address without a port; it uses the port from the port flag.

    3. The public IP of the Gateway, as reported to the network, is determined by pinging an external service. To override this or to avoid this outside communication, the public IP and port can be manually set via overridePublicIP. If no port is set, then the port from the port flag is used. This expects an IPv4 address only.

    4. If you placed the TLS certificate and key files in a different location than the default one in the Generate TLS Credentials, then modify the location via keypath, certPath, serverCertPath, and permissioningCertPath.

  5. Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press Enter.

Node Configuration

The default node.yaml file only requires setting your registration code. However, there are other optional flags for non-standard configurations. Database information is set in a later step.

  1. Every Node operator will receive a unique registration code from [email protected] in order to join the network. The registration code will look similar to the following example.

    Example Registration Code: UvyGCZU8WP18PmmIdcpVmx00QA3xNe7sEB9Hixkk=

    The Node configuration file has to be modified to include this registration code. To do this, open node.yaml in nano or your favorite text editor.

  2. Enter this code into the registrationCode field between the two quotes "".

  3. For non-default setups and experienced users, the following options can be used.

    1. If the Node does not have a GPU, then set useGPU to false.

      Running a Node without a GPU is not a recommended configuration. Only do so if the CPU in the machine is very powerful.
    2. By default, the Node runs on port 11420; however, this can be modified using the port flag.

    3. The internal listening address of the Node defaults to 0.0.0.0. However, a different internal address can be set via listeningAddress. This expects an IPv4 address without a port; it uses the port from the port flag.

    4. The public IP of the Node, as reported to the network, is determined by pinging an external service. To override this or to avoid this outside communication, the public IP and port can be manually set via overridePublicIP. If no port is set, then the port from the port flag is used. This expects an IPv4 address only.

    5. If you placed the TLS certificate and key files in a different location than the default one in the Generate TLS Credentials, then modify the location via node > paths > cert, node > paths > key, gateway > paths > cert, and permissioning > paths > cert

  4. Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press Enter.

Configure Service Files

To ensure that the Node and Gateway services are run as the correct user, they will need to be modified.

  1. The following steps require the machine’s username. For most users, this will be the username created and installation of the operating system. If you do not remember your username, use whoami.

  2. Open the service file for the binary being run. If both Node and Gateway are being run, this modification will need to be made with both files. Open either xxnetwork-node.service or xxnetwork-gateway.service with nano or your favorite text editor.

    For Node:

    For Gateway:

  3. Under the [Service] heading, after the User= field, modify the username to the one found in step 1.

  4. Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press Enter.

  5. Repeat step 1 through step 4 for the other service file if both Node and Gateway are being run.

Databases

Both the Node and Gateway require PostgreSQL databases. The Node database stores key pairs with clients while the Gateway database is used to store encrypted messages for users. The following instructions detail how to install PostgreSQL and set up the required databases for the two possible setups: running the Node and Gateway on the same machine and on separate machines. Follow the case pertaining to your setup and at the end make sure to Verify the Database.

Case 1: Node and Gateway Run on the Same Machine

These instructions are for Node operators who run the Node and Gateway software on the same machine. Note that a separate drive for the Gateway database is required that meets the hardware specifications. Before starting, prepare the second disk for the Gateway database by mounting and partitioning it. To learn more, refer to Adding the Gateway Database (One Computer Configuration) post on the xx network forum.

  1. Install PostgreSQL and its dependencies.

  2. Once the installation is complete, enable the PostgreSQL service.

  3. Next, start the service.

  4. Create a database user with the username cmix, which is used to access the database.

  5. You will be asked to set a password for the cmix user. Create a long and secure password but note that it will only be saved into the Node and Gateway configuration files in the following steps and you will not need to remember it once everything is configured.

    Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.
Node Database Setup
  1. Create the required database with the name cmix_node.

  2. Next, modify the Node configuration file to use the newly generated password in step 5 above. To do this, open node.yaml in nano or your favorite text editor.

  3. Under the field database > password, enter in the new password. If the database was created with a different username or database name, then update those values too.

  4. Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press Enter.

Gateway Database Setup
  1. Determine the location to place the Gateway database. The following directions use /mnt/ts_gateway as an example path; replace it with the location on the secondary drive.

  2. Create a directory on the secondary drive to store the database in using mkdir.

  3. Change the ownership of the directory.

  4. Switch to the postgres user account.

  5. Run the PostgreSQL client server via psql.

    The command prompt has changed from $ to #.
  6. Once at the postgres prompt, create the tablespace and point it to the directory on the second disk drive (enter everything after the postgres-# part).

  7. Enter \q to exit the session.

  8. Enter exit to return back to your user.

  9. Create the Gateway database with the name cmix_gateway with the previously made tablespace.

  10. Finally, modify the Gateway configuration file to use the password generated when setting up the database user cmix. To do this, open gateway.yaml in nano or your favorite text editor.

  11. Under the field dbPassword, enter in the new password. If the database was created with a different username or database name, then update those values too.

  12. Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press Enter.

Case 2: Node and Gateway are Run on Separate Machines

These instructions are for operators who run the Node and Gateway software on the separate machines.

Node Database Setup on Node Machine

Follow the instructions below on the machine running the Node software.

  1. Install PostgreSQL and its dependencies.

  2. Once the installation is complete, enable the PostgreSQL service.

  3. Next, start the service.

  4. Create a database user with the username cmix, which is used to access the database.

  5. You will be asked to set a password for the cmix user. Create a long and secure password but note that it will only be saved into the Node configuration file in the following steps and you will not need to remember it once everything is configured.

    Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.
  6. Create the required database with the name cmix_node.

  7. Next, modify the Node configuration file to use the newly generated password in step 5 above. To do this, open node.yaml in nano or your favorite text editor.

  8. Under the field database > password, enter in the new password. If the database was created with a different username or database name, then update those values too.

  9. Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press Enter.

Gateway Database Setup on Gateway Machine

Follow the instructions below on the machine running the Gateway software.

  1. Install PostgreSQL and its dependencies.

  2. Once the installation is complete, enable the PostgreSQL service.

  3. Next, start the service.

  4. Create a database user with the username cmix, which is used to access the database.

  5. You will be asked to set a password for the cmix user. Create a long and secure password but note that it will only be saved into the Gateway configuration file in the following steps and you will not need to remember it once everything is configured.

    Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.
  6. Create the required database with the name cmix_gateway.

  7. Next, modify the Gateway configuration file to use the newly generated password in step 5 above. To do this, open gateway.yaml in nano or your favorite text editor.

  8. Under the field dbPassword, enter in the new password. If the database was created with a different username or database name, then update those values too.

  9. Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press Enter.

Verify the Database

The following steps will describe how to ensure the database was created correctly. This section is optional but highly recommended.

  1. Login with the postgres account.

  2. Run the PostgreSQL client server via psql.

    The command prompt has changed from $ to #.
  3. Once at the postgres prompt, enter in \l to get a list of databases (do not enter the postgres-# part).

    Ensure that the correct databases with the correct owners show up. If Node and Gateway are on the same machine, then the output should appear similar to below. If each is being run separately, then only the cmix_node or cmix_gateway should be displayed.

  4. If Node and Gateway are run off the same machine, verify the gateway tablespace points to the directory on the second disk drive by typing \db to get a list of available tablespaces. Ensure that the tablespace ts_gateway has the correct location set.

  5. Enter \q to exit the command line.

  6. Enter exit to return back to your prompt.

System Services

This section describes how to install the service files for Node and Gateway. Follow only the instructions related to the type of machine you are deploying. Follow both Node and Gateway instructions if running both off of the same machine.

Warning: The following steps will start the Node and/or Gateway software and will have it join the xx network. If you do not want to do so currently, then do not do the following steps until you are ready.
  1. Before starting the Node or Gateway service, ensure that the clock is synchronized by entering the following command.

    This should result in the following output.

  2. If both System clock synchronized and systemd-timesyncd.service active are set to yes, then proceed to the next step. Otherwise, refer back to the Clock Synchronization (NTP) section.

  3. Next, soft link the systemd service file(s) to the systemd directory /etc/systemd/system/.

    For Node:

    For Gateway:

  4. Reload the systemd service files.

  5. To ensure that your user has the correct permissions, run the following command. Make sure to replace both instances of [user] with your username.

  6. Next, enable the service(s) for the appropriate binary.

    For Node:

    This should result in the following output.

    For Gateway:

    This should result in the following output.

  7. Then, start the services.

    For Node:

    For Gateway:

Verify the Service has Started

  1. First, check the status of the service to ensure it is running.

    For Node:

    This should result in the following output. Press Q to exit the output.

    For Gateway:

    This should result in the following output. Press Q to exit the output.

  2. Next, check that the process is running. If the commands result in the expected output, then the Node or Gateway process is running.

    This will print a list of running xx network processes. If xxnetwork-gatew appears on the list, then the Gateway process is running. If xxnetwork-node appears on the list, then the Node process is running.

    The PID (the first number printed) and the elapsed time will be different.

Backup Certificates and Important Files

Ambox warning pn.svg ATTENTION Ambox warning pn.svg


Lost Certificates Cannot be Recovered

Once the Node and Gateway are properly functioning, Node operators must backup some important files. Loss of these files will irreparably disconnect a Node from the network. Follow the instructions on the next page detailing how to do so.

Warning: The following instructions are extremely important to follow to ensure that your Node and Gateway can stay operating on the network. Failure to follow these instructions can lead to your Node or Gateway being permanently unable to operate on the network.
Warning: The files described below are sensitive as they identify your Node and Gateway uniquely. Anybody who gains access to these files can appear on the network as your Node or Gateway. It is extremely important to keep these files secure as any third party access can lead to the compromising of your Node and Gateway’s identity.

There are six files that your Node/Gateway requires to authenticate itself on the network. Loss of these files will lead to your Node or Gateway being unable to participate in the network and regeneration of these files is contingent on the policy outlined above. Therefore, every Node Operator must backup these files to a secure location not on the same machine. It is recommended that these files be backed up to multiple secure locations. For example, a flash drive or secure cloud storage.

Backing Up Files

  1. Locate the six files that must be backed up. If you used the default paths in the configuration instructions, then these files can be found according to the table below.

    File Default Name Default Location
    1. Node’s public certificate node_cert.crt /opt/xxnetwork/creds/
    2. Node’s private key node_key.key /opt/xxnetwork/creds/
    3. Node’s ID file (IDF) nodeIDF.json /opt/xxnetwork/node-logs/
    4. Gateway’s public certificate gateway_cert.crt /opt/xxnetwork/creds/
    5. Gateway’s private key gateway_key.key /opt/xxnetwork/creds/
    6. Gateway’s ID file (IDF) gatewayIDF.json /opt/xxnetwork/gateway-logs/

    If you cannot locate the files, then their paths can be found in the Node and Gateway YAML files as shown in the table below.

    File YAML File Flag
    1. Node’s public certificate node.yaml node.paths.cert
    2. Node’s private key node.yaml node.paths.key
    3. Node’s ID file (IDF) node.yaml node.paths.idf
    4. Gateway’s public certificate gateway.yaml certPath
    5. Gateway’s private key gateway.yaml keyPath
    6. Gateway’s ID file (IDF) gateway.yaml idfPath
  2. The easiest method to copy these files off of the machine is to use a USB flash drive; you can use the same one that you may have used to install the operating system. First, insert the flash drive and then run lsblk to find the mount point of the flash drive.

    This should output a list of mount points on your computer. Locate the name of the flash drive. This can be done by looking at the size and finding which mount point has the same size as your flash drive.

  3. Copy all six files to the flash drive path found in the previous step. If your files are in the default locations, it will be similar to the following command.

    To copy one file at a time, use the following command.

  4. Once the files are copied, remove the flash drive, and save it in a secure location. We highly recommend that you back up these files in at least one other location.

    Ensure that these files only exist in secure locations. If anyone gains access to these files, your Node and Gateway’s identity on the network can be compromised.

Manually Deploying Binaries

After following the Node and Gateway Set Up, it is possible to use alternate binaries than the ones supplied by the Wrapper Script. The source code to compile the binaries can be downloaded from the Elixxir GitLab page.

  1. To disable automatic update via the Wrapper Script, add --disableupdates to the call in the service files located in /opt/xxnetwork/xxnetwork-node.service for Node and /opt/xxnetwork/xxnetwork-gateway.service for Gateway.

  2. Replace the binaries in /opt/xxnetwork/bin/ with the self-compiled ones generated in the following steps. Then restart the service.

    For Node:

    For Gateway:

Compile Binaries

A second option to manually deploying binaries is to download the source code and compile it yourself. Go version 1.16 is required for both Server and Gateway.

  1. Download and install the latest version of Go version 1.16. Refer to the Environment Set Up section for detailed instructions.

Gateway Binary

Compiling Gateway is a straightforward process. For additional information, refer to the Gateway .gitlab-ci.yaml file.

  1. Download the latest master branch of Gateway.

  2. Compile Gateway using the following command.

    It is also possible to compile it for other systems, though support cannot be guaranteed.

  3. Rename the file to xxnetwork-gateway and move it to /opt/xxnetwork/bin. Then restart the service.

Node Binary

If the Node is expected to run with GPU acceleration, compiling Server requires an extra step to compile. This setup assumes a working Nvidia installation. For additional information, refer to the Server .gitlab-ci.yaml file.

  1. First, install the CUDA Toolkit 11.2. Nvidia provides a list of commands to run that install all the necessary software onto the machine. Simply run all the commands in the provided order.

    Note that this process can be slow.
  2. Once the installation is complete, reboot the system.

  3. Next, install libgmp-dev, which is a dependency the GPU code needs to compile.

  4. Install the latest version of the gpumathsnative repository. This is where the underlying CUDA implementations of the mathematical operations live is what allows operations to be done on the GPU.

    Skip this step if the Node is being run without the GPU.
  5. Download the latest master branch of Server.

  6. Finally, compile the Server using the following command.

  7. Name the file xxnetwork-node and move it to /opt/xxnetwork/bin. Then restart the service.

Testing

A majority of the code in the Elixxir codebase is tested. Node operators can run the tests on any public repository themselves using the instructions in this section.

Environment Set Up

Prior to running any tests, set up a development environment with Go 1.16 and some additional software.

These instructions describe how to run tests in Ubuntu Server 18.04. However, almost all tests run successfully on Windows, macOS, and other Linux distributions.
  1. First, install the build-essential package, which contains a number of tools for compiling binaries. This step is only necessary if running tests on Server with the GPU.

    If the GPU drivers were previously installed, build-essential may be already installed too.
  2. Download the latest version of Go version 1.16. Refer to the Go install page for more information.

  3. Extract the archive into /usr/local.

  4. The .profile file for the current user must be modified to include /usr/local/go/bin in the PATH environment variable. This can be done manually by opening the file and modifying it. Alternatively, use the following command to automatically append the correct line to the .profile file.

  5. Once the file is saved, to apply the changes immediately, run the following command.

  6. To ensure that everything is working and that Go is the correct version, run the following command.

    It should result in a similar output to this:

  7. Finally, create a directory to place the Go source code that will be downloaded in the following sections. Any name can be chosen.

Setting Up Node for Testing

To run some of the tests on Node, a GPU is required and the gpumathsnative repository must be downloaded. This is what allows operations to be done on the GPU and is where the underlying CUDA implementations of the mathematical operations live.

  1. First, ensure that the GPU Drivers are set up.

  2. Next, install the CUDA Toolkit 11.2. Nvidia provides a list of commands to run that install all the necessary software onto the machine. Simply run all the commands in the provided order.

    Note that this process can be slow.
  3. Once the installation is complete, reboot the system.

  4. Install libgmp-de, a dependency for the GPU code to run.

  5. Download the latest master version of the gpumathsnative repository.

  6. Run the following commands to go to the correct directory and execute the make command.

  7. When running tests with GPU on Node, include the flag -tags gpu.

Running Tests

Running tests in Go for most Elixxir repositories is a straightforward process. Server has an extra step that is described below. To run a test, simply download a repository and use go test.

  1. Go to the Elixxir or xx network GitLab page and select a repository to download. Alternatively, select one from the following list of public repositories:

    • elixxir/primitives
    • elixxir/crypto
    • elixxir/comms
    • elixxir/gpumathsgo
    • elixxir/server
    • elixxir/gateway
    • elixxir/client
    • elixxir/user-discovery-bot
    • elixxir/LocalEnvironment
    • elixxir/integration
    • elixxir/wrapper
    • xx_network/primitives
    • xx_network/crypto
    • xx_network/comms
  2. Use git clone or any other method to download the repository.

    In the following steps, Primitives will be used as an example.

  3. Change directories to the root of the downloaded repository.

  4. To compile and run all the tests for this repository, use the following command.

    The first time the code is compiled, all of the dependencies will download first.

    Running the tests can take a while, but once they are finished, a summary of the tests and their elapsed time will be printed out. If any tests fail, those errors will be printed too.

  5. To run tests located in a specific directory, include the directory name. Also, including the -v flag will print more details about which test is being run and the debug logs from it.

    The result will be a list of the tests run and their elapsed time. At the bottom is the total time for all the tests.

  6. To run only one test or multiple tests matching a regular expression, use the -run flag.

Appendix

Paths and Files

There are a number of file paths that need to be specified for a Node or Gateway to run. Some of these paths point to existing files, others point to where a file should be saved. Many of these are modified through the Node and Gateway configuration files, which must be explicitly specified when running the software.

Below is an example directory structure if the xx network software was installed following the instructions in this manual. The files that are required at startup are marked with a red asterisk () and the files that are generated are marked with a blue dagger ().

Legend
file required at startup
file generated
  • /opt/xxnetwork/
    • bin
      • xxnetwork-gateway
      • xxnetwork-node
    • cmdlod
      • command_1591809548.jsonl
      • command_1591856646.jsonl
      • version_1591166323.jsonl
    • creds
      • gateway_cert.crt
      • gateway_key.key
      • network_management.crt
      • node_cert.crt
      • node_key.key
      • permissioning_cert.crt
    • gateway-logs
      • gateway.log
      • gatewayIDF.json
      • xxnetwork-wrapper.log
    • gateway.yaml
    • generate_certs.py
    • lib
      • libpow.fatbin
      • libpowmosm75.so
    • node-logs
      • metrics.log
      • node-err.log
      • node.log
      • nodeIDF.json
      • xxnetwork-wrapper.log
    • node.yaml
    • xxnetwork-gateway.service
    • xxnetwork-node.service
    • xxnetwork-wrapper.py

Node Paths and Files

The Node requires four files to run:

  1. The Node private key
  2. The Node TLS certificate
  3. The Gateway TLS certificate
  4. The Permissioning TLS certificate
node:
  paths:
    # Path to the self-signed TLS certificate for Node. Expects PEM format. (Required)
    cert: "/opt/xxnetwork/creds/node_cert.crt"
    # Path to the private key associated with the self-signed TLS certificate. (Required)
    key: "/opt/xxnetwork/creds/node_key.key"
gateways:
  paths:
    # Path to the self-signed TLS certificate for Gateway. Expects PEM format. (Required)
    cert: "/opt/xxnetwork/creds/gateway_cert.crt"

permissioning:
  paths:
    # Path to the self-signed TLS certificate for the Permissioning server. Expects PEM
    # format. (Required)
    cert: "/opt/xxnetwork/creds/permissioning_cert.crt"

There are three different paths for logs: the normal Node log, the error log, and the metrics logs as well as one for the Node IDF file.

node:
  paths:
    # Path where an error file will be placed in the event of a fatal error. This path is
    # used by the Wrapper Script. (Required)
    errOutput: "/opt/xxnetwork/node-logs/node-err.log"
    # Path to where the identity file (IDF) is saved. The IDF stores the Node's network
    # identity. This is used by the wrapper management script. (Required)
    idf: "/opt/xxnetwork/node-logs/nodeIDF.json"
    # Path where log file will be saved.
    log: "/opt/xxnetwork/node-logs/node.log"
metrics:
  # Path to store metrics logs.
  log: "/opt/xxnetowkr/server-logs/metrics.log"

Gateway Paths and Files

To run, Gateway requires four files:

  1. The Gateway private key
  2. The Gateway TLS certificate
  3. The Node TLS certificate
  4. The Permissioning TLS certificate

These appear in gateway.yaml as follows.

# Path to the private key associated with the self-signed TLS certificate. (Required)
keyPath: "/opt/xxnetwork/creds/gateway_key.key"

# Path to the self-signed TLS certificate for Gateway. Expects PEM format. (Required)
certPath: "/opt/xxnetwork/creds/gateway_cert.crt"

# Path to the self-signed TLS certificate for Node. Expects PEM format. (Required)
serverCertPath: "/opt/xxnetwork/creds/node_cert.crt"

# Path to the self-signed TLS certificate for the Permissioning server. Expects PEM format.
# (Required)
permissioningCertPath: "/opt/xxnetwork/creds/permissioning_cert.crt

In addition, Gateway has an optional IDF path field and an optional log field. If no paths are supplied, defaults will be used. When the Gateway runs, it will create files at these paths.

# Path where log file will be saved. (Default "./gateway-logs/gateway.log")
log: "/opt/xxnetwork/gateway-logs/gateway.log"

# Path to where the identity file (IDF) is saved. The IDF stores the Gateway's Node's
# network identity. This is used by the wrapper management script. (Required)
idfPath: "/opt/xxnetwork/gateway-logs/gatewayIDF.json"

Network Definition File (NDF)

The NDF is a JSON file with a predefined structure that matches the internal NetworkDefinition structure.

Some objects on the NDF must have data that matches predefined formats. These are outlined below.

  • Id: must be a byte array 33 bytes long that matches the id.ID object. IDs must be generated using Crypto and cannot be created any other way.
  • Tls_certificate: must be a TLS certificate in PEM format. All new lines should be replaced with Unix escape sequence \n.


{
	"Timestamp": "YYYY-MM-DDTHH:MM:SS.0000000+00:00",

Timestamp in RFC3339 format.

	"Gateways": [
		
		{
			"Id": "dGVzdDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB",
			"Address": "0.0.0.0:80000",
			"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
		},
		
	],

Array of Gateway objects in the network. Each Gateway has an ID, an address string (containing the IP address and port), and a TLS certificate.

	"Nodes": [
		
		{
			"Id": "dGVzdDMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC",
			"Address": "0.0.0.1:40000",
			"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
		},
		
	],

Array of Node objects in the network. Each Node has an ID, address, and TLS certificate in the same format as the Gateway object.

	"Registration": {
		"Address": "0.0.0.3:18000",
		"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
	},

The Registration field has information about the Permissioning server.

	"Notification": {
		"Address": "0.0.0.7",
		"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
	},

The Notification field has information about Notification Bot.

	"Udb": {
		"Id": "dGVzdDYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD"
		"Cert": "-----BEGIN CERTIFICATE-----...",
		"Address": "0.0.0.52:18001",
		"DhPubKey": "eyJWYWx1ZSI6NTEwOTQzODAxNTcyOTQ4NjUzN...",
	},

The Udb field contains the ID, certificate, address, and Diffie–Hellman key of User Discovery

	"E2e": {
		"Prime": "E2EE983D031DC1DB6F1A7A67DF0E9A8E5561DB8E...",
		"Generator": "2"
	},
	"Cmix": {
		"Prime": "F6FAC7E480EE519354C0F856AEBDC43AD6014...",
		"Generator": "2"
	}
}

Both the E2e and Cmix fields define the cyclic groups that messaging and end to end encryption operate within. The E2e group is based on a 3192-bit strong and safe prime and the Cmix group is a 2048 strong and safe prime.

Bandwidth Limiting

The cMix protocol uses bandwidth in unique ways. It has short bursts (10~100 milliseconds) of medium-high usage and extended periods (250 milliseconds to 5 seconds) of minimal usage. Overall, at the most taxing configuration testing shows it will saturate 20% to 30% of a 100 Mbps connection.

That 20% to 30% figure is dependent on network conditions and other Nodes it is operating with, but in rare circumstances, it may be 20% to 30% of whatever bandwidth is available. If a Node operator has greater than 100 Mbps available and does not want to allow the network to consume beyond the stated requirements, they can limit bandwidth utilization on the Node.

To do this, Linux traffic control (tc) and Hierarchy Token Bucket (HTB), similar to QoS settings on a router, can be used to limit the bandwidth of the Node and/or Gateway to the desired amount. Configuration requires three pieces of information: (1) the port number(s) of the Node and/or Gateway, (2) the network interface name, and (3) the desired bandwidth cap. The following instructions will describe how to get these values and how to configure traffic control.

  1. First, get the port number. By default, the port for Node is 11420 and for Gateway it is 22840. If the port numbers were changed during set up, they can be found in the following locations.

    1. Node: The default location is /opt/xxnetwork/node.yaml. The port is under node > port.

    2. Gateway: The default location is /opt/xxnetwork/gateway.yaml. The port is under port.

  2. Next, determine the network interface that the machine is operating on. For most machines, the following command should print the currently used network interface.

    Alternatively, manually find the correct network interface by printing out all available interfaces.

    This should result in a similar output to below. Find the name of the network interface that is currently being used. It will have the correct local IP address.

  3. Add the qdisc that will limit the bandwidth for the network interface found.

  4. Set the max bandwidth for either Node or Gateway. The bandwidth must be set to at least 100 Mbit if running a Node or Gateway. If running them together, Node must be set to a minimum of 100 Mbit and Gateway a minimum of 50 Mbit.

    Warning: Creating a bandwidth limit lower than specified can result in underperforming and eventual removal from the network.

    For Node:

    For Gateway:

  5. Set the port number that the bandwidth will be limited on. Make sure to use the correct port found in the previous step.

    For Node:

    For Gateway:

  6. To check that the rules were set correctly, list all the classes.

    This should print a similar output to below.

To stop limiting the bandwidth, the rule can be deleted using the following command.