Difference between revisions of "Node Handbook"
Line 1,398: | Line 1,398: | ||
| {{mono|generate_certs.py}} | | {{mono|generate_certs.py}} | ||
| TLS certificate/key generator script. | | TLS certificate/key generator script. | ||
|- | |||
| {{mono|creds/network_management.crt}} | |||
| {{mono|creds/network_management.crt}} | |||
| The TLS certificate used to authenticate commands sent to the Wrapper Script. | |||
|- | |- | ||
| {{mono|creds/permissioning_cert.crt}} | | {{mono|creds/permissioning_cert.crt}} | ||
| {{mono|creds/permissioning_cert.crt}} | | {{mono|creds/permissioning_cert.crt}} | ||
| The TLS certificate for Permissioning. | | The TLS certificate for Permissioning. | ||
|- | |- | ||
| {{mono|lib/libpowmosm75.so}} | | {{mono|lib/libpowmosm75.so}} |
Revision as of 21:20, 21 June 2021
Beta Node Handbook
Instructions for Node operators on how to deploy a Node.688x58px
Background
This manual is designed to provide the necessary information to deploy, debug, and operate a Node and Gateway on the xx network BetaNet. A Node in the network will participate in the two fundamental tasks that comprise the xx network: mixing communication through Elixxir’s cMix protocol and executing consensus through the Praxxis xx consensus.
To learn more about the xx network, Elixxir, and Praxxis, refer to the following papers.
- xx network Whitepaper
- xx consensus Whitepaper
- xx cMix Whitepaper
- Academic Paper – cMix: Mixing with Minimal Real-Time Asymmetric Cryptographic Operations
For more information, further discussion, and additional help with this guide or general participation in the BetaNet, use the following contacts.
Continue reading below for general information about how the software works. To skip to the instructions for setting up a Node and Gateway, skip to Node and Gateway Set Up.
Architecture of the BetaNet
General Architecture
The initial BetaNet is made up of three main entities:
- Nodes: The core operators of the network; they execute the cMix protocol.
- Gateways: The public facing components of Nodes, one exists per Node. They store received messages and provide public access to data.
- Clients: Users use Clients to communicate on the network and are generally deployed on mobile devices.The Client repository is now public and community testing of the xx messenger is ongoing.
Hierarchy
BetaNet was built with a tiered Scheduler-Worker design, with all components controlled either directly or recursively by a central Permissioning server. Each member polls the entity above them in the hierarchy for information, as shown in Figure 1. Bi-directional communication only exists within the same level.
The Network Definition File (NDF) contains all the connection information for the entities in the network and is provided by the Permissioning server through the hierarchy shown. The Nodes provide the hash of their current NDF to Permissioning; if they differ then the updated NDF is provided to the Node. The Gateways poll the Nodes and Clients poll Gateways for the new NDF in the same fashion. However, the NDF provided to the Clients is stripped of Node IP addresses. In the current implementation of Clients, they currently get the NDF from Permissioning. This is expected to change by their release.
Nodes, Gateways, and Clients also receive scheduling instructions from the Permissioning server. These instructions are contained within RoundInfo
structures, which are both prescriptive and descriptive of changes to rounds, which groups a set of Nodes to anonymize communications. A round is created when RoundInfo
is issued to start a round’s precomputation. When the Nodes finish the precomputation, Permissioning issues a new RoundInfo
that schedules it for real time, which can be delayed depending on the number of queued rounds. A further RoundInfo
is issued when a round completes or fails.
Network Membership
In the network, all trust is derived from the Permissioning server. Currently, Permissioning is an information collator, but it will be replaced by the xx network consensus mechanism in the near future.
Identities in the network are defined by an asymmetric keypair. Most entities will hold this keypair within a TLS certificate, but some entities will just have a keypair. Nodes and Gateways prove their membership to the network by the inclusion of membership information in a signed NDF.
Initial Run
When first started, a Node will require two TLS certificates and a unique 256-bit randomly generated registration code provided by xx network to join the network. The certificates must be generated using RSA keys and the registration code will be provided in Base64 string format.
On first run, the Node will generate a cryptographic ID and register it with the network, via the Permissioning server. Most of the initial files and generated files are critical and must be preserved by the Node operator. For more information, refer to Cryptographic and Network Primitives.
Hardware Requirements
Below are the hardware requirements for running a Node and Gateway. Note that there are additional requirements when running a Node and Gateway on the same machine.
Node Hardware Requirements
Nodes are high powered machines that use both a CPU and GPU. The software does support full CPU Nodes, but higher core counts are required. As the software becomes more mature and more power is extracted from the GPU, it is likely that the hardware requirements for such Nodes will increase.
CPU |
High core count modern CPU
|
---|---|
GPU |
Nvidia Turing: Nvidia GeForce RTX 2070 or greater Nvidia Ampere: Nvidia GeForce RTX 3060 Ti or greater |
RAM |
16 GB DDR4 or more
|
Storage |
1 TB High Speed Enterprise NVMe (PCI) SSD
|
Bandwidth | 100 Mbps upload / 100 Mbps download |
Gateway Hardware Requirements
Gateways require relatively low powered machines. Every Node must have a Gateway.
CPU |
Dual core modern CPU, cloud deployment possible
|
---|---|
GPU | Not required |
RAM | 2 GB or more |
Storage |
250 GB
|
Bandwidth |
100 Mbps upload / 100 Mbps download
|
Node and Gateway Hardware Requirements
It is possible to run a Gateway off the same machine as a Node; however, it results in a loss of security when they are run off the same IP address and network connection. The Gateway structure makes it more difficult for a Node to be maliciously removed from the network.
The specifications for a machine to run both a Node and Gateway differ slightly and are as follows.
CPU |
High core count modern CPU
|
---|---|
GPU |
Nvidia Turing: Nvidia GeForce RTX 2070 or greater Nvidia Ampere: Nvidia GeForce RTX 3060 Ti or greater |
RAM |
16 GB DDR4 or more
|
Storage 1 |
1 TB High Speed Enterprise NVMe (PCI) SSD
|
Storage 2 |
240 GB High Speed Enterprise SSD
|
Bandwidth | 150 Mbps upload / 200 Mbps download |
Turing or Ampere GPU Requirement
The xx network BetaNet requires a Nvidia Turing or Ampere GPU due to upgrades to the microarchitecture of the shaders granting an order of magnitude increase in performance for the specific workloads used. The mul units have been increased to 32 bits, which greatly increases the speed of the modular exponentiation and modular multiplication which are some of the core cryptographic operations of the cMix protocol.
Software Overview
The xx network BetaNet codebase is combined between xx network’s consensus software and Elixxir’s cMix protocol. At launch, xx network will be testing and tuning Elixxir software with xx consensus coming online at a later date.
Elixxir software is organized into four groups of repositories as shown in Figure 2: Core, Services, Clients, and Tools. The Core libraries handle functionality across Clients and Services, which contain data structures, interfaces, and cryptography common to many of the xx network components. Services implements various messenger features, as well as authorizing Clients and Nodes to join the network. Clients interact with the network using a shared Client API (xxDK). Tools provide several interfaces and utilities to deploy, test, and debug the network.
Most of the code found in the Core and Services is written in the Go programming language. The repositories listed as public below can be found on the public Elixxir and xx network GitLab pages.
To follow the set up guide, most Node operators will likely elect to download the prepackaged tarballs containing the compiled binaries. To learn more, refer to the Node and Gateway Software section.
xx network Codebase
Core
xx network/Primitives
name: | primitives
|
project: | xx network | availability: | public | git: | gitlab.com/xx_network/primitives |
The xx network Primitives repository contains the basic data structures and utilities that are used in the Elixxir and Praxxis codebases. This includes the ID structure for Nodes, Gateways, and users and the NDF structure. Additional generic structures and utilities are also contained in Primitives, such as file access and rate limiting utilities.
xx network/Crypto
name: | crypto
|
project: | xx network | availability: | public | git: | gitlab.com/xx_network/crypto |
The xx network Crypto repository contains cryptographic primitives that are shared between Elixxir and Praxxis codebases. Primarily lower level primitives, randomness generation, TLS handling, and nonce handling.
xx network/Comms
name: | comms
|
project: | xx network | availability: | public | git: | gitlab.com/xx_network/comms |
The Comms repository handles all network communication functionality as well as all of the core connectivity logic. It includes generic comms handlers and protocols and not specific implementations, which are implemented in the respective Comm branches for each project.
Elixxir Codebase
Core
Elixxir/Primitives
name: | primitives
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/primitives |
The Elixxir Primitives repository contains the basic data structures and utilities that are used by all Elixxir repositories. This includes the user fact structure, the cMix message structure, and the version object.
Elixxir/Crypto
name: | crypto
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/crypto |
The Crypto repository encompasses all of the base cryptographic functionality found in the code base. Relying heavily on Go’s big integer implementation, Crypto features a cryptographically secure random number generator implementation, libraries for working with large integers in modulo cyclic groups for cMix operations, and basic encrypt/decrypt functionality. Like Primitives, Crypto only contains code that is generic to the larger system. A core approach to this repository is to supply wrappers for operations that may require migration to other implementations in the future.
Elixxir/Comms
name: | comms
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/comms |
Comms builds on the generic utilities of xx network Comms to provide specific functionality for Elixxir. It holds a gRPC protocol file along with a thin Client/Server implementation. The repository currently uses TLS certificates with RSA keys for encryption and identification, which will migrate to xx consensus-based quantum secure authenticated channels as development progresses.
Elixxir/GPUMaths
name: | gpumathsgo
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/gpumathsgo |
The GPUMaths repository accelerates the math used by Server, especially for precomputations. It provides a subset of the math implemented in the Crypto repository but accelerated on GPUs. Underlying the acceleration is a publicly available CUDA arbitrary-precision math library, CGBN2. Components of this written in CUDA are in a separate repository gpumathsnative.
Services
Elixxir/Server
name: | server
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/server |
The Server repository implements the core cMix functionality and is the software that a Node runs. It performs precomputation and real time computation and processes messages. It receives batches of messages from the Gateways as well as performs network team operations.
Elixxir/Gateway
name: | gateway
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/gateway |
The Gateway repository contains the API for Clients to interact with the network. Every Node runs a Gateway and the Gateways collect and store messages for Clients. Gateway is designed to be a scalable front-end to the xx network.
Elixxir/Permissioning
name: | registration
|
project: | elixxir | availability: | private |
Permissioning, also referred to as Registration, manages the NDF for Clients and Servers and schedules cMix rounds within the network. Eventually this functionality will be managed by the distributed xx consensus. For now, this code handles admission, manages which Nodes are part of the network, and orchestrates when Nodes operate.
Clients
Elixxir/Client API (xxDK)
name: | client
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/client |
All Clients use the Client API to interact and send messages with the cMix network. It uses Go mobile to produce a library compatible with iOS and Android.
Elixxir/User Discovery (UD)
name: | user-discovery-bot
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/user-discovery-bot |
User Discovery helps users make first contact with other users. It facilitates user search and lookup in a private manner.
Elixxir/Mobile Clients
name: | client-ios
|
project: | elixxir | availability: | private |
name: | client-android
|
project: | elixxir | availability: | private |
Currently, clients exist for iOS and Android operating systems. These both use the Go mobile libraries produced in the Client API.
Tools
Elixxir/Wrapper Script
name: | wrapper
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/wrapper |
The Wrapper Script is a Node and Gateway management script that simplifies the running of the xx network software. The script automates the management of the xx network software log files. For easy management or in the event of an error, it starts, stops, and restarts the software without the operator having to revisit the command line. Optionally, it can be set to automatically update the Node and Gateway with the latest xx network binaries and configuration files. To learn more, refer to the Wrapper Script Arguments section.
Elixxir/DevOps
(deployment)
name: | deployment
|
project: | elixxir | availability: | private |
DevOps is a deployment platform for Microsoft Azure, Google Cloud Platform, and Amazon Web Services (AWS) written in Terraform. DevOps allows for the deployment of test networks, management of deployments of individual Nodes, and the deployment of multi-cloud implementations of xx network.
Elixxir/Integration integration
[public]
name: | integration
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/integration |
The Integration repository is a series of end-to-end tests designed to test different functionality of the cMix protocol. Several tests focus on different batch sizes with Nodes only. Another test covers all the Client-level interaction within the network.
Elixxir/Local Environment
name: | LocalEnvironment
|
project: | elixxir | availability: | public | git: | gitlab.com/elixxir/LocalEnvironment |
The Local Environment repository contains a set of scripts designed to allow the testing of the entire platform on a single machine.
Version Scheme
Both Gateway and Node binaries have a version string embedded in them, which consists of a major version, a minor version, and a patch string, separated by a period.
Major | Minor | Patch | ||
---|---|---|---|---|
┌┴┐ | ┌┴┐ | ┌─┴─┐ | ||
4 | . | 2 | . | 6ab |
To be able to participate in the network, a Node or Gateway has to have a compatible version with the required version reported by the Permissioning server. For a version to be compatible, the major version must be equal to the required major version and the minor version must be greater than or equal to the required minor version. The patch can be anything, but it will always be present.
Management Tools
Management Scripts Features
The management scripts are designed to make administration of Nodes and Gateways easier and less time consuming as well as provide data back to the xx network about the functionality of the Node. Many features that it provides require trusting the xx network. The usage of the management scripts is not required and can be disabled. The following are the primary features of these scripts.
- Binary restart: In the event that the binary process stops due to a handled error, the Wrapper Script will restart it. The process can crash because, in the current implementation, some errors are not fully handled when caused by Node failures on the network. It is highly recommended that this feature is utilized.
- Binary update: The Wrapper Script has the ability to accept automatic binary and Wrapper Script updates provided by the xx network. In theory, the xx network can push any code it wants if this feature is activated, but it greatly simplifies the task of running a Node. Included in this feature is the ability to send stop and start commands to the Node. All commands are signed and can be proved to have come from the xx network. Running without this feature is fully supported but requires manual updates.
- Log upload: This features uploads logs to xx network for debugging. It can be disabled and is not mandatory but is useful to the xx network for development purposes.
The Management Script is two scripts combined, the service file and the actual Wrapper Script. The service file is a systemd script that starts the Wrapper Script and ensures it stays running. The wrapper script is written in Python and does most of the heavy lifting. Both can be found in the Elixxir wrapper repository.
Service File Arguments
The service files maintain the Node and Gateway processes running the background. There is a different service for each. By default, these files (xxnetwork-node.service and xxnetwork-gateway.service) are located in /opt/xxnetwork/ and are soft linked to /etc/systemd/system/. They are the same script just configured differently for Node and Gateway. These scripts call the Wrapper Script with different options. To see the full details of the options, refer to the Wrapper Script Arguments section below.
Node Service File Arguments
The following are the available options in the service file configured for Node. The sections in red should never be modified and should maintain the original values as provided. The items in blue can be modified.
ExecStart=/bin/bash -c '/opt/xxnetwork/xxnetwork-wrapper.py --logpath /opt/xxnetwork/node-logs/node.log --binary /opt/xxnetwork/bin/xxnetwork-node --s3path server --s3logbucket alphanet-logs-prod --s3managementbucket alphanet-management-prod --s3accesskey ${s3_access_key_id} --s3secret ${s3_access_key_secret} --s3region us-west-1 --configdir /opt/xxnetwork/ --erroutputpath /opt/xxnetwork/node-logs/node-err.log --tmpdir /tmp/xxnetwork/node --idpath /opt/xxnetwork/node-logs/nodeIDF.json >> /opt/xxnetwork/node-logs/xxnetwork-wrapper.log 2>&1'
Gateway Service File Arguments
The following are the available options in the service file configured for Gateway. The sections in red should never be modified and should maintain the original values as provided. The items in blue can be modified.
ExecStart=/bin/bash -c '/opt/xxnetwork/xxnetwork-wrapper.py --logpath /opt/xxnetwork/gateway-logs/gateway.log --binary /opt/xxnetwork/bin/xxnetwork-gateway --s3path gateway --s3logbucket alphanet-logs-prod --s3managementbucket alphanet-management-prod --s3accesskey ${s3_access_key_id} --s3secret ${s3_access_key_secret} --s3region us-west-1 --configdir /opt/xxnetwork/ --erroutputpath /opt/xxnetwork/gateway-logs/gateway-err.log --tmpdir /tmp/xxnetwork/gateway --idpath /opt/xxnetwork/gateway-logs/gatewayIDF.json >> /opt/xxnetwork/gateway-logs/xxnetwork-wrapper.log 2>&1'
Wrapper Script Arguments
The following arguments should be modified inside the service file described above.
usage: wrapper.py [-h] [-d] [-l LOGPATH] [-i IDPATH] -b BINARY [--gpulib GPULIB] [--gpubin GPUBIN] [--disable-consensus] [--consensus-binary CONSENSUS_BINARY] [--consensus-config CONSENSUS_CONFIG] [--consensus-state CONSENSUS_STATE] [--consensus-log CONSENSUS_LOG] [--consensus-cw-group CONSENSUS_CW_GROUP] [-c CONFIGDIR] -s S3PATH [-m S3MANAGEMENTBUCKET] [--disable-cloudwatch] [--cloudwatch-log-group CLOUDWATCH_LOG_GROUP] --s3accesskey S3ACCESSKEY --s3secret S3SECRET --s3region S3REGION [--tmpdir TMPDIR] [--cmdlogdir CMDLOGDIR] [--erroutputpath ERROUTPUTPATH] [--configoverride CONFIGOVERRIDE]
Flag | Default | Required |
---|---|---|
-h, --help | False | |
Show this help message and exits. | ||
-d, --disableupdates | False | False |
Disables automatic updates.
Used to disable the download and automatic installation of new binaries, the default configuration file, the wrapper script itself, and the security certificate used for verifying wrapper script commands. | ||
-l LOGPATH, --logpath LOGPATH | /opt/xxnetwork/logs/xx.log | False |
The path to store logs, e.g., /opt/xxnetwork/node-logs/node.log. | ||
-i IDPATH, --idpath IDPATH | /opt/xxnetwork/logs/IDF.json | False |
Node ID path, e.g., /opt/xxnetwork/logs/nodeIDF.json.
The Node stores the ID file at this path. | ||
-b BINARY, --binary BINARY | True | |
Path of the binary to be run by the wrapper. | ||
--gpulib GPULIB | /opt/xxnetwork/lib/libpowmosm75.so | False |
Path to the GPU exponentiation library. | ||
--gpubin GPUBIN | /opt/xxnetwork/lib/libpow.fatbin | False |
Path to the GPU bin file. | ||
--disable-consensus | False | False |
Disable consensus binary. | ||
--consensus-binary CONSENSUS_BINARY | /opt/xxnetwork/bin/xxnetwork-consensus | False |
Path to the consensus binary. | ||
--consensus-config CONSENSUS_CONFIG | /opt/xxnetwork/consensus.yaml | False |
Path to the consensus config file. | ||
--consensus-state CONSENSUS_STATE | /opt/xxnetwork/consensus.tar.gz | False |
Path to the consensus state tarball. | ||
--consensus-log CONSENSUS_LO | /opt/xxnetwork/consensus-logs/consensus.log | False |
Path to the consensus log file. | ||
--consensus-cw-group CONSENSUS_CW_GROUP | xxnetwork-consensus-prod | False |
CW log group for consensus logs | ||
-c CONFIGDIR, --configdir CONFIGDIR | /opt/xxnetwork/ | False |
Path to the config directory, e.g., ~/.xxnetwork/. | ||
--configoverride CONFIGOVERRIDE | False | |
Override for config file path.
If this flag is set, then the script ignores the --configdir flag and uses this file path instead. | ||
-s S3PATH, --s3path S3PATH | True | |
Path to the S3 management directory.
Do not modify.
| ||
-m S3MANAGEMENTBUCKET, --s3managementbucket S3MANAGEMENTBUCKET | False | |
Path to the S3 management bucket name.
Do not modify.
| ||
--disable-cloudwatch | False | False |
Disable uploading log events to CloudWatch. | ||
--cloudwatch-log-group CLOUDWATCH_LOG_GROUP | xxnetwork-logs-prod | False |
Log group for CloudWatch logging. | ||
--s3accesskey S3ACCESSKEY | True | |
S3 access key.
Do not modify.
| ||
--s3secret S3SECRET | True | |
S3 access key secret.
Do not modify.
| ||
--s3region S3REGION | ||
S3 region.
Do not modify.
| ||
--tmpdir TMPDIR | /tmp | False |
Directory for temporary files. | ||
--cmdlogdir CMDLOGDIR | /opt/xxnetwork/cmdlog | False |
Directory for commands log. | ||
--erroutputpath ERROUTPUTPATH | False | |
Path to recovered error path.
The Node stores recoverable errors at this path. |
Deployment Schedule
While BetaNet is a functioning network, it is still in development and requires consistent updates. If the Wrapper Script is being used in its default configuration, then Node and Gateway will automatically update.
However, for Node operators with automatic updates turned off, xx network is committed to the following update schedule to allow them ample time to deploy updates.
Scheduled Updates
Every week, there is a single window where major releases and breaking minor releases can be downloaded and installed. The source code will be released on Tuesday before the update goes live on Thursday. Once available on Thursday, any Nodes not up to date cannot participate in the network but will be considered online for the purpose of compensation. Node operators will have until Monday to install the release, after which time they will be considered offline for the purpose of compensation. Refer to the following timeline for specific times and descriptions.
All updates will be posted to the Elixxir GitLab and Node operators will be informed by email. The xx network will strive to post notice of updates before the Tuesday release, if possible.
Tuesday | 1:00 pm PT – Release of source code.
|
---|---|
Wednesday | |
Thursday |
1:00 PM PT – Push to Permissioning server, Nodes with out of date code stop functioning.
1:00 PM PT – Push to Nodes with automatic updates enabled.
1:00 PM PT – Grace period for Nodes to update begins.
|
Friday | |
Saturday | |
Sunday | |
Monday | 1:00 PM PT – Grace period for Nodes to update ends, Nodes that have not updated will be considered offline for the purpose of compensation.
|
Critical Unscheduled Updates
Critical updates may be released with no notice in the event of catastrophic failure or immediate danger to the network.
Nodes will be given 5 business days to update their Node for the purpose of compensation in the event of such an update.
Cryptographic and Network Primitives
There are various cryptographic primitives used in the xx network; some are provided by the xx network and some are generated by the Node operator.
Primitives Provided by xx network
The following primitives are provided to you by xx network, either emailed directly or they are downloaded from public repositories.
Registration Code | Sent via email | The registration code is a 256-bit random number encoded in Base64. It is a single-use code used only on initial registration after which, the Node’s ID and TLS certificate can be used to identify it within the network. This will be provided to each Node operator prior to the network going live. |
---|---|---|
Permissioning TLS Certificate | Provided inside the tarball | A TLS certificate is provided to be used to identify the Permissioning server. |
Permissioning Server Address | Provided inside the tarball | The DNS address of the Permissioning server is pre-populated inside the YAML files for Node and Gateway. |
Wrapper Script Bucket Address | Provided inside the tarball | The IP address the Wrapper Script uses to update the binaries. It is pre-populated in the Wrapper Script. |
Log Bucket Address | Provided inside the tarball | The IP address the Wrapper Script uses to upload logs to. It is pre-populated in the Wrapper Script. |
cMix Cyclic Group | Provided as part of the NDF on initial registration | The cMix cyclic group is the modulo cyclic group in which cMix operations are conducted. At launch, this will be defined by a 2048-bit strong and safe prime with a generator of 2. |
E2E Cyclic Group | Provided as part of the NDF on initial registration | The E2E cyclic group is the modulo cyclic group in which end to end encryption operations by clients will be conducted. At launch, this will be defined by a 3192-bit strong and safe prime with a generator of 2. |
Primitives Created by the Node Operator
The following primitives are generated prior to and during the initial registration of the Node. More details about generation can be found in Generate TLS Credentials.
TLS Credentials
The TLS credentials are generated as X.509, SHA-256, RSA 4096-bits, and are recommended to last 730 days (2 years). The generated credentials include:
- Node TLS certificate
- Node private key
- Gateway TLS certificate
- Gateway private key
ID File (IDF)
IDs are 264 bits and consist of a 256-bit hash (BLAKE2b) of a salt and the RSA public key from the Node’s TLS certificate appended with a 8 bits that describe the ID type.
ID = Hash(Node_PubKey, Salt) + TYPE
Due to the construction of the ID, the ownership of the ID can be proved under RSA’s cryptographic assumptions. Due to the hash, IDs are unpredictably generated in a large sparse space so it is overwhelmingly improbable that independently generated IDs will ever collide, so no central checking of ID generation is necessary.
The last byte of the ID is a type byte which describes what type of entity it belongs to. The options for type are:
Type | Hex | Note |
---|---|---|
Generic | 0x00 | Used for one-off entities such as the Permissioning server, the notifications server, and others |
Gateway | 0x01 | The Gateways in the network |
Node | 0x02 | The Nodes in the network |
User | 0x03 | Unique client ID |
On first run, the Node automatically generates the ID and saves it to a JSON file with the following structure.
{
"id": "1Mo9im6HHpoTDt/lTYjSkWV7dAD0Eh+18xUFfxlm4I4C",
|
The ID of the Node or Gateway, presented as a base 64 encoded string. |
"type": "node",
|
The type of ID is shown here for readability. |
"salt": [119, 234, 246, 209, 166, 166, 72, 17, 253, 196, 172, 187, 230, 2, 132, 137, 49, 219, 142, 58, 82, 169, 60, 230, 112, 17, 30, 112, 30, 68, 217, 92],
|
The salt used to generate the ID. |
"idBytes": [212, 202, 61, 138, 110, 135, 30, 154, 19, 14, 223, 229, 77, 136, 210, 145, 101, 123, 116, 0, 244, 18, 31, 181, 243, 21, 5, 127, 25, 102, 224, 142, 2],
}
|
The ID of the Node or Gateway, presented as an array of bytes. |
Lost Certificate and Important Files Policy
As of April 27, 2021, the following guidelines and policies are in effect.
- The Node and Gateway identities are defined by the certificates, their private keys, and the IDF files. Loss of these files will render your Node and Gateway unable to prove their identity and unable to rejoin the network.
- Backup and retention of these files are the sole responsibility of the Node operator.
- The xx network team nor anyone else can recover a Node or Gateway if these files are lost.
Node and Gateway Set Up
Below you will find the instructions for setting up a Node and Gateway on the xx network for the first time. These instructions detail how to install both a Node and Gateway on separate machines as well as installing them together on the same machine. Follow the instructions once for each machine.
The installation instructions assume that your machine has an active internet connection and an empty storage drive or a drive that can be written over with enough space as outlined in the Hardware Requirements section.
Overview
Setting up an xx Node and Gateway is not particularly difficult, but due to the number of components, can take up to 2 hours. The steps for setting up a Node and Gateway are almost identical and the instructions can be followed for both. The general steps for setting up a Node are:
- Install the operating system
- Synchronize the system clock (NTP)
- Configuring the local network
- Install the GPU drivers (if setting up a Node with GPU)
- Install and configure the Node and Gateway software
- Generate TLS credentials
- Install and configure the databases
- Set up and start the services
- Backup important files Make sure to not skip this important step!
Some Tips for Inexperienced Users
If this is your first time using a command line interface or you do not remember how to use it, the following are some tips to make using the interface a little easier.
- In this document, anytime code is presented in a black box with monospaced font it means that it is command line input or output. Commands prefixed by a
$
are commands that you should enter into your command prompt (do not include$
in the command). Any lines without that prefix are output from the system. - The
sudo
command is often prepended to commands found in these instructions. This enables commands to be run with elevated privileges. When used, the system will ask for your password to continue running the command.
- Whenever the system asks for a password to continue, no characters will appear when typing, but type in your password and press ↵ Enter and it will work.
- When typing a command or path, use the Tab ↹ key to auto complete a partially written statement.
Installing the Operating System
The xx network software has been tested only on Ubuntu Server 18.04. We cannot guarantee support if a different operating system version is used, although no decisions have been made to specifically preclude any operating systems.
Download Ubuntu Server install image from the Official Ubuntu website.
Next, a bootable disk with Linux needs to be created. This can be done by writing it to a DVD or more commonly, a flash drive. Follow one of the following tutorials on how to do so depending on your current operating system and chosen media.
Once your flash drive or DVD are ready, follow the Tutorial on Installing Ubuntu Server.
In step 6, make sure you select the first option
Install Ubuntu
.In step 7, make sure to configure your internet connection and get an IP address.
In step 8, ensure that you select
Use an Entire Disk
.In step 12, pick a server name that does not have any personal identifying information and select a strong password.
Create a strong but memorable password. It is recommended that it is longer than 12 characters. Store this password in a safe and secure location. Never share this password with anyone.
Make sure the machine has turned back on and login using the credentials created in the previous step. Sometimes extra text will be printed and you will not be able to see the login prompt. The prompt should still be there; just type your username and press ↵ Enter to continue.
Check Internet Connection
The rest of the instructions require internet access. Follow these steps to ensure the machine is connected.
Check your current local connection status and local IP address.
This should result in a similar output to below. The machine should have a valid local IP address.
Finally, check that the machine has access to the internet by pinging another server. We chose to use xx.network, but you can use any domain.
If the machine is connected to the internet, the output should look similar to the following example.
Updating Software and Installing Dependencies
To ensure all the software works correctly, the system needs to be updated. In addition, an additional dependency for the Wrapper Script to function.
Before continuing, check for updates. This will print many lines about what software is being checked.
Once the check is done, install the updates.
Reboot the machine to ensure all updates are installed fully. Once the machine starts up, log back in.
Install the Python package installer.
Update the package installer.
Install the
boto3
andpyOpenSSL
dependencies. The Wrapper Script uses the first package to read commands and send logs to xx network through AWS and the second is used to authenticate them.
Setting Up UFW
Uncomplicated Firewall (UFW) is the default firewall configuration tool for Ubuntu. UFW should already be installed; the following instructions will describe how to enable it and allow the correct ports to be open.
First, ensure that UFW is disabled so that it can be configured.
This should result in the following output.
Allow the port that the Node will use to communicate on over TCP. Only do this for machines running the Node software.
Port 11420 is the default port in the provided Node configuration file. A different port may be used, but it must be configured innode.yaml
, which is downloaded in a future step.Allow the port that the Gateway will use to communicate over TCP. Only do this for machines running the Gateway software.
Port 22840 is the default port in the provided Gateway configuration file. A different port may be used, but it must be configured ingateway.yaml
, which is downloaded in a future step.If you want to manage your node remotely, you may want to allow port 22 over TCP for SSH. SSH should only be enabled with key authentication. Improper SSH setup can allow unwanted access so take care when setting up remote access. If you do not want to use SSH or do not know how to, then skip this step.
Finally, enable UFW.
If you are connected over SSH, you may be prompted to continue, press Y. Note that you may be disconnected when doing so.
Clock Synchronization (NTP)
Commands received from the Permissioning server are time-stamped and a synchronized clock is important to properly interpret them. To do so, NTP (Network Time Protocol) must be set up and synchronized.
In Ubuntu Server 18.04, this is done through timedatectl
. In most installations, it is already running and Node operators only need to check that it is correctly configured. However, the process for other operating systems may be different and it will need to be enabled.
Check if the time synchronization service is running and that the clock is synchronized by entering the following command.
This will print the following output.
If
System clock synchronized
is set toyes
andsystemd-timesyncd.service active
is set toyes
, then no further action is needed. Skip to the next section Modifying Max Number of Processes and Files. Otherwise, continue to the next step.If
System clock synchronized
is set tono
andsystemd-timesyncd.service active
is set toyes
, then the service has had insufficient time to synchronize the clock. Skip to the next section Modifying Max Number of Processes and Files but make sure to check that the clock is synchronized before starting the Node or Gateway software. Otherwise, continue to the next step.Ensure the time is synchronized before starting the software in the System Services section.If
System clock synchronized
is set tono
andsystemd-timesyncd.service active
is set tono
, then the service must be manually started in the next step.
Enter in the following command to get a list of time zones.
This will print a list of time zones. Use the up key ↑ and down key ↓ to navigate the list and find the time zone that the machine is in. Once found, make a note of the time zone, and press Q to exit.
Once the correct time zone for the machine is found, use the following command to set it.
Using the date command, ensure that the correct time zone was selected. If the printed time is incorrect, return to step 2 to select a new time zone.
Begin the clock synchronization service by entering the following command.
Ensure that the service is running by calling
timedatectl
again.If
systemd-timesyncd.service active
is set toyes
, then it has been successful. IfSystem clock synchronized
is set toyes
, then this section is done. If it is set to {{inline-code|no||, then it may take up to 30 minutes for the clock to synchronize. You can continue to the following section, but make sure to check that the clock is synchronized before starting the Node or Gateway software.
Modifying Max Number of Processes and Files
By default, there is a maximum number of processes and files that can be opened at once. To prevent the Node and Gateway from encountering this limit, it must be removed for all users.
To remove this limit for users, open the limits configuration file using nano or your favorite text editor.
The following process of using the nano text editor to modify a file is used elsewhere in this document. Refer back here for detailed steps on how to use it.Once the file is open, use the down arrow key ↓ to go to the second to last line above where it says
# End of file
. Add the following four lines above that line.To save the file, press Ctrl+X and when prompted to save the buffer, press Y.
Finally, when prompted with the file name, press ↵ Enter to exit.
Once the change has been made, reboot the system.
Configuring TCP Networking Options
The congestion window size limits the maximum amount of data sent out to a network after a time of little operation. The default size needs to be increased to remove a bottleneck in the xx network.
This change is necessary for the xx network because the cMix protocol transmits in short bursts. As a result, the congestion windows contract between transmissions, causing them to need to reopen on every transmission, significantly slowing down the network in high latency environments.
The following instructions are required to be completed on both the Node and Gateway machines.
First, to prevent the congestion windows from shrinking unnecessarily when the connection is idle, disable
tcp_slow_start_after_idle
.To make these settings persist across reboots, store them in the sysctl configuration file.
Modify the TCP congestion control algorithm (CCA) to use TCP Bottleneck Bandwidth and RRT (BBR).
Modify the default queuing discipline to be fq; this is required to use BBR.
Apply these two options to sysctl.conf so they persist on reboot.
Optional Configuration to Make Initial Connection Not Slow
While disabling tcp_slow_start_after_idle
will keep the congestion windows large after connections are used a few times, you can optionally modify the initial congestion window size using the following instructions to make the minimum congestion window size large enough for cMix batches.
First, set congestion windows size on sending and receiving to be 700 times the maximum segment size.
Store the above command in /etc/rc.local, to make it run on boot.
Note, if this fails to work or causes problems, such as increased timeouts or instability, remove the file and reboot.
Configuring Local Network (Port Forwarding)
To ensure that the machine can be accessed from outside the local network, the network’s router or gateway must be configured to allow external access to the machine on ports configured above. Three main pieces of information are needed for this part: (1) the port numbers to forward (the defaults are 11420 and 22840), (2) the protocol to use (TCP), and (3) the local IP address of the machine, which is retrieved below.
Get the local IP address of the machine. If Node and Gateway are being run off separate machines on the same network, then run this on both machines.
The local IP address will be printed; it will be in the form of 0.0.0.0. Make sure to make note of this for the later steps.
If the machine has multiple network interfaces or an IPv6 address, they will also appear in this list. Ensure that only the correct internal IPv4 address is used.
- The following section describes how to configure the networking equipment on your network. Because of the varying type of equipment configurations, these instructions are generic and may not be accurate for your hardware. Please refer to the manufacturer’s instructions for more detailed and accurate information. Configuration of the network will most likely occur from a different machine on the network.
To access the router to configure, its IP address is needed.
This will output the following line, where the first address printed is the router IP address.
Go to this IP address in a browser (on a different machine) and login using the router credentials. These credentials are either set up by the network administrator or are the default credentials located on the router or found online.
Locate the port forwarding options (occasionally called virtual server). These options are sometimes found under the advanced section.
Before forwarding the port found above, you may want to provide a static local IP address to the machine. However, that is outside the scope of these instructions.Add the new ports to forward for the Node and Gateway. For each, create a new entry and enter the IP address found in step 1, set the port to the chosen ports (by default it is 11420 for Node and 22840 for Gateway), and select the TCP protocol. Make sure to save or apply the changes.
GPU Drivers
The Node software requires a Nvidia RTX graphic processor and the installation of its drivers.
Install the Nvidia driver.
Once the installation is complete, reboot the system.
Once the system reboots, log back into the computer.
Verifying the Driver Installation
Check that the system has claimed the device.
This should result in a similar output to the following.
Next, check the driver and state information.
This should result in a similar output to the following.
Node and Gateway Software
Change the working directory to /opt/.
Download the tarball for Node and/or Gateway using
curl
with the following URLs. The newest versions can also be found on the xx network website.For Node:
For Gateway:
Next, the downloaded files need to be extracted to the /opt/ directory. Use
tar
extract them with the flags-xf
so that the directory structure remains intact. This will create a new /opt/xxnetwork/ directory with the extracted files inside.For Node:
For Gateway:
Both node.tar.gz and gateway.tar.gz contain some files that are the same (e.g., xxnetwork-wrapper.py). When extracting both files, it is expected that these files will be overwritten. Order of extraction does not matter.If this command produces an error similar tonot in gzip format
, then that means the file did not download correctly in the previous step. Try downloading the file again and ensure the URL is correct.The file needed to install the software are in the newly created /opt/xxnetwork/ directory. Those files are:
Node Gateway Note bin/xxnetwork-node bin/xxnetwork-gateway The binary for Node or Gateway. node.yaml gateway.yaml The configuration file for the binary. xxnetwork-node.service xxnetwork-gateway.service systemd service file. xxnetwork-wrapper.py xxnetwork-wrapper.py A management script for the binary. generate_certs.py generate_certs.py TLS certificate/key generator script. creds/network_management.crt creds/network_management.crt The TLS certificate used to authenticate commands sent to the Wrapper Script. creds/permissioning_cert.crt creds/permissioning_cert.crt The TLS certificate for Permissioning. lib/libpowmosm75.so GPU dependency (for Node only). lib/libpow.fatbin GPU dependency (for Node only).
Generate TLS Credentials
TLS credentials for the Node and Gateway are critical to its identity in the network. For more information refer to Cryptographic and Network Primitives. If running Node and Gateway on different machines, then generate the certificates on the Node machine and copy the necessary credentials to the Gateway server.
gateway_cert.crt
, and gateway_key.key. In addition, it is important to back up the file nodeIDF.json, which is generated on the initial run. Refer to the Lost Certificate and Important Files Policy for more information.Using the provided generate_certs.py script, generate the TLS certificates and keys for Node and Gateway. It will provide prompts for all the information required. Start the script using Python 3. Note that this must be run inside the /opt/xxnetwork/ directory.
You do not need to use this script to generate the required TLS credentials. Use the requirements in the TLS Credentials section to ensure the generated certificates meet the network requirements.The script will launch and query for various information to create the credentials for Node and Gateway. If you choose not to provide any details, default values will be used. After every answer press ↵ Enter.
Once all the details have been provided, the certificates will be generated and moved into creds/.
To check that all the correct files are in the creds/ directory, use
ls
to list the files.If running Gateway on a separate machine from Node, then copy gateway_cert.crt, gateway_key.key, and node_cert.crt to the Gateway machine and delete gateway_key.key from the Node machine.
Configuring Node and Gateway
Configuring Node and Gateway requires modifying their respective YAML files retrieved in the previous steps. These instructions go over the basic configuration to get a Node and Gateway working in the default state. Refer to the Server and Gateway README files to learn more. Note that all addresses must be IPv4; IPv6 is not currently supported.
Gateway Configuration
The default gateway.yaml file only requires setting the address of its Node. However, there are other optional flags for non-standard configurations. Database information is set in a later step.
Get the IP address of the Node. If the Node is being run on the same machine as Gateway, then this IP address is 0.0.0.0:port (where the default port is 11420). If the Node and Gateway are installed on separate machines, then you will need to get the Node’s public IP address, by running the following command on the Node.
This should be run on the Node machine.The Gateway configuration file has to be modified to include this IP address. To do this, on the Gateway machine, open gateway.yaml in nano or your favorite text editor.
Once the file is open, use the down arrow key ↓ to get to the line that says
nodeAddress
and replace the placeholder with the one found in step 1. Make sure to include the chosen port, as shown.For non-default setups and experienced users, the following options can be used.
By default, the Gateway runs on port 22840; however, this can be modified using the port flag.
The internal listening address of the Gateway defaults to 0.0.0.0. However, a different internal address can be set via
listeningAddress
. This expects an IPv4 address without a port; it uses the port from theport
flag.The public IP of the Gateway, as reported to the network, is determined by pinging an external service. To override this or to avoid this outside communication, the public IP and port can be manually set via
overridePublicIP
. If no port is set, then the port from theport
flag is used. This expects an IPv4 address only.If you placed the TLS certificate and key files in a different location than the default one in the Generate TLS Credentials, then modify the location via
keypath
,certPath
,serverCertPath
, andpermissioningCertPath
.
Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press ↵ Enter.
Node Configuration
The default node.yaml file only requires setting your registration code. However, there are other optional flags for non-standard configurations. Database information is set in a later step.
Every Node operator will receive a unique registration code from [email protected] in order to join the network. The registration code will look similar to the following example.
Example Registration Code:UvyGCZU8WP18PmmIdcpVmx00QA3xNe7sEB9Hixkk=
The Node configuration file has to be modified to include this registration code. To do this, open node.yaml in nano or your favorite text editor.
Enter this code into the
registrationCode
field between the two quotes""
.For non-default setups and experienced users, the following options can be used.
If the Node does not have a GPU, then set
useGPU
tofalse
.Running a Node without a GPU is not a recommended configuration. Only do so if the CPU in the machine is very powerful.By default, the Node runs on port 11420; however, this can be modified using the
port
flag.The internal listening address of the Node defaults to 0.0.0.0. However, a different internal address can be set via
listeningAddress
. This expects an IPv4 address without a port; it uses the port from theport
flag.The public IP of the Node, as reported to the network, is determined by pinging an external service. To override this or to avoid this outside communication, the public IP and port can be manually set via
overridePublicIP
. If no port is set, then the port from theport
flag is used. This expects an IPv4 address only.If you placed the TLS certificate and key files in a different location than the default one in the Generate TLS Credentials, then modify the location via node > paths > cert, node > paths > key, gateway > paths > cert, and permissioning > paths > cert
Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press ↵ Enter.
Configure Service Files
To ensure that the Node and Gateway services are run as the correct user, they will need to be modified.
The following steps require the machine’s username. For most users, this will be the username created and installation of the operating system. If you do not remember your username, use whoami.
Open the service file for the binary being run. If both Node and Gateway are being run, this modification will need to be made with both files. Open either xxnetwork-node.service or xxnetwork-gateway.service with nano or your favorite text editor.
For Node:
For Gateway:
Under the
[Service]
heading, after theUser=
field, modify the username to the one found in step 1.Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press ↵ Enter.
Repeat step 1 through step 4 for the other service file if both Node and Gateway are being run.
Databases
Both the Node and Gateway require PostgreSQL databases. The Node database stores key pairs with clients while the Gateway database is used to store encrypted messages for users. The following instructions detail how to install PostgreSQL and set up the required databases for the two possible setups: running the Node and Gateway on the same machine and on separate machines. Follow the case pertaining to your setup and at the end make sure to Verify the Database.
Case 1: Node and Gateway Run on the Same Machine
These instructions are for Node operators who run the Node and Gateway software on the same machine. Note that a separate drive for the Gateway database is required that meets the hardware specifications. Before starting, prepare the second disk for the Gateway database by mounting and partitioning it. To learn more, refer to Adding the Gateway Database (One Computer Configuration) post on the xx network forum.
Install PostgreSQL and its dependencies.
Once the installation is complete, enable the PostgreSQL service.
Next, start the service.
Create a database user with the username
cmix
, which is used to access the database.You will be asked to set a password for the cmix user. Create a long and secure password but note that it will only be saved into the Node and Gateway configuration files in the following steps and you will not need to remember it once everything is configured.
Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.
Node Database Setup
Create the required database with the name
cmix_node
.Next, modify the Node configuration file to use the newly generated password in step 5 above. To do this, open node.yaml in nano or your favorite text editor.
Under the field
database
>password
, enter in the new password. If the database was created with a different username or database name, then update those values too.Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press ↵ Enter.
Gateway Database Setup
Determine the location to place the Gateway database. The following directions use /mnt/ts_gateway as an example path; replace it with the location on the secondary drive.
Create a directory on the secondary drive to store the database in using
mkdir
.Change the ownership of the directory.
Switch to the
postgres
user account.Run the PostgreSQL client server via
psql
.The command prompt has changed from$
to#
.Once at the
postgres
prompt, create the tablespace and point it to the directory on the second disk drive (enter everything after thepostgres-#
part).Enter
\q
to exit the session.Enter
exit
to return back to your user.Create the Gateway database with the name
cmix_gateway
with the previously made tablespace.Finally, modify the Gateway configuration file to use the password generated when setting up the database user cmix. To do this, open gateway.yaml in nano or your favorite text editor.
Under the field
dbPassword
, enter in the new password. If the database was created with a different username or database name, then update those values too.Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press ↵ Enter.
Case 2: Node and Gateway are Run on Separate Machines
These instructions are for operators who run the Node and Gateway software on the separate machines.
Node Database Setup on Node Machine
Follow the instructions below on the machine running the Node software.
Install PostgreSQL and its dependencies.
Once the installation is complete, enable the PostgreSQL service.
Next, start the service.
Create a database user with the username
cmix
, which is used to access the database.You will be asked to set a password for the cmix user. Create a long and secure password but note that it will only be saved into the Node configuration file in the following steps and you will not need to remember it once everything is configured.
Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.Create the required database with the name
cmix_node
.Next, modify the Node configuration file to use the newly generated password in step 5 above. To do this, open node.yaml in nano or your favorite text editor.
Under the field
database
>password
, enter in the new password. If the database was created with a different username or database name, then update those values too.Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press ↵ Enter.
Gateway Database Setup on Gateway Machine
Follow the instructions below on the machine running the Gateway software.
Install PostgreSQL and its dependencies.
Once the installation is complete, enable the PostgreSQL service.
Next, start the service.
Create a database user with the username
cmix
, which is used to access the database.You will be asked to set a password for the cmix user. Create a long and secure password but note that it will only be saved into the Gateway configuration file in the following steps and you will not need to remember it once everything is configured.
Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.Create the required database with the name
cmix_gateway
.Next, modify the Gateway configuration file to use the newly generated password in step 5 above. To do this, open gateway.yaml in nano or your favorite text editor.
Under the field
dbPassword
, enter in the new password. If the database was created with a different username or database name, then update those values too.Once the change is made, save the file by pressing Ctrl+X and when prompted to save buffer, press Y. Finally, when prompted with the file name, press ↵ Enter.
Verify the Database
The following steps will describe how to ensure the database was created correctly. This section is optional but highly recommended.
Login with the
postgres
account.Run the PostgreSQL client server via
psql
.The command prompt has changed from$
to#
.Once at the postgres prompt, enter in
\l
to get a list of databases (do not enter thepostgres-#
part).Ensure that the correct databases with the correct owners show up. If Node and Gateway are on the same machine, then the output should appear similar to below. If each is being run separately, then only the cmix_node or cmix_gateway should be displayed.
If Node and Gateway are run off the same machine, verify the gateway tablespace points to the directory on the second disk drive by typing
\db
to get a list of available tablespaces. Ensure that the tablespace ts_gateway has the correct location set.Enter
\q
to exit the command line.Enter
exit
to return back to your prompt.
System Services
This section describes how to install the service files for Node and Gateway. Follow only the instructions related to the type of machine you are deploying. Follow both Node and Gateway instructions if running both off of the same machine.
Before starting the Node or Gateway service, ensure that the clock is synchronized by entering the following command.
This should result in the following output.
If both
System clock synchronized
andsystemd-timesyncd.service active
are set toyes
, then proceed to the next step. Otherwise, refer back to the Clock Synchronization (NTP) section.Next, soft link the systemd service file(s) to the systemd directory /etc/systemd/system/.
For Node:
For Gateway:
Reload the systemd service files.
To ensure that your user has the correct permissions, run the following command. Make sure to replace both instances of
[user]
with your username.Next, enable the service(s) for the appropriate binary.
For Node:
This should result in the following output.
For Gateway:
This should result in the following output.
Then, start the services.
For Node:
For Gateway:
Verify the Service has Started
First, check the status of the service to ensure it is running.
For Node:
This should result in the following output. Press Q to exit the output.
For Gateway:
This should result in the following output. Press Q to exit the output.
Next, check that the process is running. If the commands result in the expected output, then the Node or Gateway process is running.
This will print a list of running xx network processes. If
xxnetwork-gatew
appears on the list, then the Gateway process is running. Ifxxnetwork-node
appears on the list, then the Node process is running.The PID (the first number printed) and the elapsed time will be different.
Backup Certificates and Important Files
Once the Node and Gateway are properly functioning, Node operators must backup some important files. Loss of these files will irreparably disconnect a Node from the network. Follow the instructions on the next page detailing how to do so.
There are six files that your Node/Gateway requires to authenticate itself on the network. Loss of these files will lead to your Node or Gateway being unable to participate in the network and regeneration of these files is contingent on the policy outlined above. Therefore, every Node Operator must backup these files to a secure location not on the same machine. It is recommended that these files be backed up to multiple secure locations. For example, a flash drive or secure cloud storage.
Backing Up Files
Locate the six files that must be backed up. If you used the default paths in the configuration instructions, then these files can be found according to the table below.
File Default Name Default Location 1. Node’s public certificate node_cert.crt /opt/xxnetwork/creds/ 2. Node’s private key node_key.key /opt/xxnetwork/creds/ 3. Node’s ID file (IDF) nodeIDF.json /opt/xxnetwork/node-logs/ 4. Gateway’s public certificate gateway_cert.crt /opt/xxnetwork/creds/ 5. Gateway’s private key gateway_key.key /opt/xxnetwork/creds/ 6. Gateway’s ID file (IDF) gatewayIDF.json /opt/xxnetwork/gateway-logs/ If you cannot locate the files, then their paths can be found in the Node and Gateway YAML files as shown in the table below.
File YAML File Flag 1. Node’s public certificate node.yaml node.paths.cert 2. Node’s private key node.yaml node.paths.key 3. Node’s ID file (IDF) node.yaml node.paths.idf 4. Gateway’s public certificate gateway.yaml certPath 5. Gateway’s private key gateway.yaml keyPath 6. Gateway’s ID file (IDF) gateway.yaml idfPath The easiest method to copy these files off of the machine is to use a USB flash drive; you can use the same one that you may have used to install the operating system. First, insert the flash drive and then run
lsblk
to find the mount point of the flash drive.This should output a list of mount points on your computer. Locate the name of the flash drive. This can be done by looking at the size and finding which mount point has the same size as your flash drive.
Copy all six files to the flash drive path found in the previous step. If your files are in the default locations, it will be similar to the following command.
To copy one file at a time, use the following command.
Once the files are copied, remove the flash drive, and save it in a secure location. We highly recommend that you back up these files in at least one other location.
Ensure that these files only exist in secure locations. If anyone gains access to these files, your Node and Gateway’s identity on the network can be compromised.
Manually Deploying Binaries
After following the Node and Gateway Set Up, it is possible to use alternate binaries than the ones supplied by the Wrapper Script. The source code to compile the binaries can be downloaded from the Elixxir GitLab page.
To disable automatic update via the Wrapper Script, add
--disableupdates
to the call in the service files located in /opt/xxnetwork/xxnetwork-node.service for Node and /opt/xxnetwork/xxnetwork-gateway.service for Gateway.Replace the binaries in /opt/xxnetwork/bin/ with the self-compiled ones generated in the following steps. Then restart the service.
For Node:
For Gateway:
Compile Binaries
A second option to manually deploying binaries is to download the source code and compile it yourself. Go version 1.16 is required for both Server and Gateway.
- Download and install the latest version of Go version 1.16. Refer to the Environment Set Up section for detailed instructions.
Gateway Binary
Compiling Gateway is a straightforward process. For additional information, refer to the Gateway .gitlab-ci.yaml file.
Download the latest master branch of Gateway.
Compile Gateway using the following command.
It is also possible to compile it for other systems, though support cannot be guaranteed.
Rename the file to xxnetwork-gateway and move it to /opt/xxnetwork/bin. Then restart the service.
Node Binary
If the Node is expected to run with GPU acceleration, compiling Server requires an extra step to compile. This setup assumes a working Nvidia installation. For additional information, refer to the Server .gitlab-ci.yaml file.
First, install the CUDA Toolkit 11.2. Nvidia provides a list of commands to run that install all the necessary software onto the machine. Simply run all the commands in the provided order.
Note that this process can be slow.Once the installation is complete, reboot the system.
Next, install
libgmp-dev
, which is a dependency the GPU code needs to compile.Install the latest version of the gpumathsnative repository. This is where the underlying CUDA implementations of the mathematical operations live is what allows operations to be done on the GPU.
Skip this step if the Node is being run without the GPU.Download the latest master branch of Server.
Finally, compile the Server using the following command.
Name the file xxnetwork-node and move it to /opt/xxnetwork/bin. Then restart the service.
Testing
A majority of the code in the Elixxir codebase is tested. Node operators can run the tests on any public repository themselves using the instructions in this section.
Environment Set Up
Prior to running any tests, set up a development environment with Go 1.16 and some additional software.
First, install the build-essential package, which contains a number of tools for compiling binaries. This step is only necessary if running tests on Server with the GPU.
If the GPU drivers were previously installed, build-essential may be already installed too.Download the latest version of Go version 1.16. Refer to the Go install page for more information.
Extract the archive into /usr/local.
The .profile file for the current user must be modified to include /usr/local/go/bin in the PATH environment variable. This can be done manually by opening the file and modifying it. Alternatively, use the following command to automatically append the correct line to the .profile file.
Once the file is saved, to apply the changes immediately, run the following command.
To ensure that everything is working and that Go is the correct version, run the following command.
It should result in a similar output to this:
Finally, create a directory to place the Go source code that will be downloaded in the following sections. Any name can be chosen.
Setting Up Node for Testing
To run some of the tests on Node, a GPU is required and the gpumathsnative repository must be downloaded. This is what allows operations to be done on the GPU and is where the underlying CUDA implementations of the mathematical operations live.
First, ensure that the GPU Drivers are set up.
Next, install the CUDA Toolkit 11.2. Nvidia provides a list of commands to run that install all the necessary software onto the machine. Simply run all the commands in the provided order.
Note that this process can be slow.Once the installation is complete, reboot the system.
Install
libgmp-de
, a dependency for the GPU code to run.Download the latest master version of the gpumathsnative repository.
Run the following commands to go to the correct directory and execute the make command.
When running tests with GPU on Node, include the flag
-tags gpu
.
Running Tests
Running tests in Go for most Elixxir repositories is a straightforward process. Server has an extra step that is described below. To run a test, simply download a repository and use go test.
Go to the Elixxir or xx network GitLab page and select a repository to download. Alternatively, select one from the following list of public repositories:
- elixxir/primitives
- elixxir/crypto
- elixxir/comms
- elixxir/gpumathsgo
- elixxir/server
- elixxir/gateway
- elixxir/client
- elixxir/user-discovery-bot
- elixxir/LocalEnvironment
- elixxir/integration
- elixxir/wrapper
- xx_network/primitives
- xx_network/crypto
- xx_network/comms
Use
git clone
or any other method to download the repository.In the following steps, Primitives will be used as an example.
Change directories to the root of the downloaded repository.
To compile and run all the tests for this repository, use the following command.
The first time the code is compiled, all of the dependencies will download first.Running the tests can take a while, but once they are finished, a summary of the tests and their elapsed time will be printed out. If any tests fail, those errors will be printed too.
To run tests located in a specific directory, include the directory name. Also, including the
-v
flag will print more details about which test is being run and the debug logs from it.The result will be a list of the tests run and their elapsed time. At the bottom is the total time for all the tests.
To run only one test or multiple tests matching a regular expression, use the
-run
flag.
Appendix
Paths and Files
There are a number of file paths that need to be specified for a Node or Gateway to run. Some of these paths point to existing files, others point to where a file should be saved. Many of these are modified through the Node and Gateway configuration files, which must be explicitly specified when running the software.
Below is an example directory structure if the xx network software was installed following the instructions in this manual. The files that are required at startup are marked with a red asterisk (✱) and the files that are generated are marked with a blue dagger (†).
✱ | file required at startup |
† | file generated |
- /opt/xxnetwork/
- bin
- xxnetwork-gateway
- xxnetwork-node
- cmdlod
- command_1591809548.jsonl†
- command_1591856646.jsonl†
- version_1591166323.jsonl†
- creds
- gateway_cert.crt✱
- gateway_key.key✱
- network_management.crt✱
- node_cert.crt✱
- node_key.key✱
- permissioning_cert.crt✱
- gateway-logs
- gateway.log†
- gatewayIDF.json†
- xxnetwork-wrapper.log†
- gateway.yaml✱
- generate_certs.py
- lib
- libpow.fatbin✱
- libpowmosm75.so✱
- node-logs
- metrics.log†
- node-err.log†
- node.log†
- nodeIDF.json†
- xxnetwork-wrapper.log†
- node.yaml✱
- xxnetwork-gateway.service
- xxnetwork-node.service
- xxnetwork-wrapper.py
- bin
Node Paths and Files
The Node requires four files to run:
- The Node private key
- The Node TLS certificate
- The Gateway TLS certificate
- The Permissioning TLS certificate
node:
paths:
# Path to the self-signed TLS certificate for Node. Expects PEM format. (Required)
cert: "/opt/xxnetwork/creds/node_cert.crt"
# Path to the private key associated with the self-signed TLS certificate. (Required)
key: "/opt/xxnetwork/creds/node_key.key"
gateways:
paths:
# Path to the self-signed TLS certificate for Gateway. Expects PEM format. (Required)
cert: "/opt/xxnetwork/creds/gateway_cert.crt"
permissioning:
paths:
# Path to the self-signed TLS certificate for the Permissioning server. Expects PEM
# format. (Required)
cert: "/opt/xxnetwork/creds/permissioning_cert.crt"
There are three different paths for logs: the normal Node log, the error log, and the metrics logs as well as one for the Node IDF file.
node:
paths:
# Path where an error file will be placed in the event of a fatal error. This path is
# used by the Wrapper Script. (Required)
errOutput: "/opt/xxnetwork/node-logs/node-err.log"
# Path to where the identity file (IDF) is saved. The IDF stores the Node's network
# identity. This is used by the wrapper management script. (Required)
idf: "/opt/xxnetwork/node-logs/nodeIDF.json"
# Path where log file will be saved.
log: "/opt/xxnetwork/node-logs/node.log"
metrics:
# Path to store metrics logs.
log: "/opt/xxnetowkr/server-logs/metrics.log"
Gateway Paths and Files
To run, Gateway requires four files:
- The Gateway private key
- The Gateway TLS certificate
- The Node TLS certificate
- The Permissioning TLS certificate
These appear in gateway.yaml
as follows.
# Path to the private key associated with the self-signed TLS certificate. (Required)
keyPath: "/opt/xxnetwork/creds/gateway_key.key"
# Path to the self-signed TLS certificate for Gateway. Expects PEM format. (Required)
certPath: "/opt/xxnetwork/creds/gateway_cert.crt"
# Path to the self-signed TLS certificate for Node. Expects PEM format. (Required)
serverCertPath: "/opt/xxnetwork/creds/node_cert.crt"
# Path to the self-signed TLS certificate for the Permissioning server. Expects PEM format.
# (Required)
permissioningCertPath: "/opt/xxnetwork/creds/permissioning_cert.crt
In addition, Gateway has an optional IDF path field and an optional log field. If no paths are supplied, defaults will be used. When the Gateway runs, it will create files at these paths.
# Path where log file will be saved. (Default "./gateway-logs/gateway.log")
log: "/opt/xxnetwork/gateway-logs/gateway.log"
# Path to where the identity file (IDF) is saved. The IDF stores the Gateway's Node's
# network identity. This is used by the wrapper management script. (Required)
idfPath: "/opt/xxnetwork/gateway-logs/gatewayIDF.json"
Network Definition File (NDF)
The NDF is a JSON file with a predefined structure that matches the internal NetworkDefinition structure.
Some objects on the NDF must have data that matches predefined formats. These are outlined below.
- Id: must be a byte array 33 bytes long that matches the id.ID object. IDs must be generated using Crypto and cannot be created any other way.
- Tls_certificate: must be a TLS certificate in PEM format. All new lines should be replaced with Unix escape sequence
\n
.
{
"Timestamp": "YYYY-MM-DDTHH:MM:SS.0000000+00:00",
|
Timestamp in RFC3339 format. |
"Gateways": [
⋮
{
"Id": "dGVzdDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB",
"Address": "0.0.0.0:80000",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},
⋮
],
|
Array of Gateway objects in the network. Each Gateway has an ID, an address string (containing the IP address and port), and a TLS certificate. |
"Nodes": [
⋮
{
"Id": "dGVzdDMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC",
"Address": "0.0.0.1:40000",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},
⋮
],
|
Array of Node objects in the network. Each Node has an ID, address, and TLS certificate in the same format as the Gateway object. |
"Registration": {
"Address": "0.0.0.3:18000",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},
|
The |
"Notification": {
"Address": "0.0.0.7",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},
|
The |
"Udb": {
"Id": "dGVzdDYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD"
"Cert": "-----BEGIN CERTIFICATE-----...",
"Address": "0.0.0.52:18001",
"DhPubKey": "eyJWYWx1ZSI6NTEwOTQzODAxNTcyOTQ4NjUzN...",
},
|
The |
"E2e": {
"Prime": "E2EE983D031DC1DB6F1A7A67DF0E9A8E5561DB8E...",
"Generator": "2"
},
"Cmix": {
"Prime": "F6FAC7E480EE519354C0F856AEBDC43AD6014...",
"Generator": "2"
}
}
|
Both the |
Bandwidth Limiting
The cMix protocol uses bandwidth in unique ways. It has short bursts (10~100 milliseconds) of medium-high usage and extended periods (250 milliseconds to 5 seconds) of minimal usage. Overall, at the most taxing configuration testing shows it will saturate 20% to 30% of a 100 Mbps connection.
That 20% to 30% figure is dependent on network conditions and other Nodes it is operating with, but in rare circumstances, it may be 20% to 30% of whatever bandwidth is available. If a Node operator has greater than 100 Mbps available and does not want to allow the network to consume beyond the stated requirements, they can limit bandwidth utilization on the Node.
To do this, Linux traffic control (tc) and Hierarchy Token Bucket (HTB), similar to QoS settings on a router, can be used to limit the bandwidth of the Node and/or Gateway to the desired amount. Configuration requires three pieces of information: (1) the port number(s) of the Node and/or Gateway, (2) the network interface name, and (3) the desired bandwidth cap. The following instructions will describe how to get these values and how to configure traffic control.
First, get the port number. By default, the port for Node is 11420 and for Gateway it is 22840. If the port numbers were changed during set up, they can be found in the following locations.
Node: The default location is /opt/xxnetwork/node.yaml. The port is under
node
>port
.Gateway: The default location is /opt/xxnetwork/gateway.yaml. The port is under
port
.
Next, determine the network interface that the machine is operating on. For most machines, the following command should print the currently used network interface.
Alternatively, manually find the correct network interface by printing out all available interfaces.
This should result in a similar output to below. Find the name of the network interface that is currently being used. It will have the correct local IP address.
Add the qdisc that will limit the bandwidth for the network interface found.
Set the max bandwidth for either Node or Gateway. The bandwidth must be set to at least 100 Mbit if running a Node or Gateway. If running them together, Node must be set to a minimum of 100 Mbit and Gateway a minimum of 50 Mbit.
Warning: Creating a bandwidth limit lower than specified can result in underperforming and eventual removal from the network.For Node:
For Gateway:
Set the port number that the bandwidth will be limited on. Make sure to use the correct port found in the previous step.
For Node:
For Gateway:
To check that the rules were set correctly, list all the classes.
This should print a similar output to below.
To stop limiting the bandwidth, the rule can be deleted using the following command.