Difference between revisions of "Node Handbook"

From xx network wiki
Jump to navigation Jump to search
m (Protected "Node Handbook" ([Edit=Allow only administrators] (indefinite) [Move=Allow only administrators] (indefinite)))
(48 intermediate revisions by one other user not shown)
Line 1: Line 1:
Beta Node Handbook
This manual is designed to provide the necessary information to deploy, debug, and operate the cMix, Gateway, and xx chain software on the xx network. A Node in the network will participate in the two fundamental tasks that comprise the xx network: mixing communication through cMix protocol and executing consensus through xx consensus.


Instructions for Node operators on how to deploy a Node.[[File:media/image1.emf|688x58px]]
Refer to the sections below for getting started with the cMix, Gateway, and xx chain software.


== Background ==
=== Getting Started ===


This manual is designed to provide the necessary information to deploy, debug, and operate a Node and Gateway on the xx network BetaNet. A Node in the network will participate in the two fundamental tasks that comprise the xx network: mixing communication through Elixxir’s cMix protocol and executing consensus through the Praxxis xx consensus.
First-time Node operators must set up their hardware, download and install the xx network software, and stake their Node.


To learn more about the xx network, Elixxir, and Praxxis, refer to the following papers.
# Follow instructions for [[Operating System Installation and Configuration]] on the Node machine
# Follow instructions for [[Operating System Installation and Configuration]] on the Gateway machine
# Do [[Node Set Up]] on the Node machine
# Do [[Gateway Set Up]] on the Gateway machine
# Follow instructions for [[Staking a Node]]


* [https://xx.network/xx-whitepaper-v2.0.pdf xx network Whitepaper]
{{Node Handbook Main}}
* [https://xx.network/xx-consensus-whitepaper.pdf xx consensus Whitepaper]
* [https://xx.network/xxcMixwhitepaper.pdf xx cMix Whitepaper]
* [https://xx.network/cmix-whitepaper.pdf Academic Paper – cMix: Mixing with Minimal Real-Time Asymmetric Cryptographic Operations]
 
For more information, further discussion, and additional help with this guide or general participation in the BetaNet, use the following contacts.
 
* [mailto:[email protected] Contact email]
* [https://forum.xx.network/ BetaNet Forum]
* [https://discord.gg/Y8pCkbK Discord]
* [https://t.me/xxnetwork Telegram]
 
Continue reading below for general information about how the software works. To skip to the instructions for setting up a Node and Gateway, skip to [[#Node_and_Gateway_Set_Up|Node and Gateway Set Up]].
 
=== Architecture of the BetaNet ===
 
{{Note|Currently, the Permissioning server is used to define the network and orchestrate its activities. This allows the engineering team to tune the network and bring the highly decentralized variant of cMix protocol to a workable state for MainNet. As the BetaNet software matures, xx consensus will take over roles currently held by the Permissioning server.}}
 
==== General Architecture ====
 
The initial BetaNet is made up of three main entities:
 
# '''Nodes:''' The core operators of the network; they execute the cMix protocol.
# '''Gateways:''' The public facing components of Nodes, one exists per Node. They store received messages and provide public access to data.
# '''Clients:''' Users use Clients to communicate on the network and are generally deployed on mobile devices.<br />{{Note|inline=1|small=1|The [https://gitlab.com/elixxir/client Client repository] is now public and community testing of the xx messenger is ongoing.}}
 
==== Hierarchy ====
 
[[File:BetaNet_Hierarchy.svg|frame|right|alt=The hierarchy of the BetaNet.|Figure 1: The hierarchy of the BetaNet.]]
 
BetaNet was built with a tiered Scheduler-Worker design, with all components controlled either directly or recursively by a central ''Permissioning'' server. Each member polls the entity above them in the hierarchy for information, as shown in Figure 1. Bi-directional communication only exists within the same level.
 
The [[#Network_Definition_File_.28NDF.29|Network Definition File (NDF)]] contains all the connection information for the entities in the network and is provided by the Permissioning server through the hierarchy shown. The Nodes provide the hash of their current NDF to Permissioning; if they differ then the updated NDF is provided to the Node. The Gateways poll the Nodes and Clients poll Gateways for the new NDF in the same fashion. However, the NDF provided to the Clients is stripped of Node IP addresses. In the current implementation of Clients, they currently get the NDF from Permissioning. This is expected to change by their release.
 
Nodes, Gateways, and Clients also receive scheduling instructions from the Permissioning server. These instructions are contained within <code>RoundInfo</code> structures, which are both prescriptive and descriptive of changes to rounds, which groups a set of Nodes to anonymize communications. A round is created when <code>RoundInfo</code> is issued to start a round’s precomputation. When the Nodes finish the precomputation, Permissioning issues a new <code>RoundInfo</code> that schedules it for real time, which can be delayed depending on the number of queued rounds. A further <code>RoundInfo</code> is issued when a round completes or fails.
 
==== Network Membership ====
 
In the network, all trust is derived from the Permissioning server. Currently, Permissioning is an information collator, but it will be replaced by the xx network consensus mechanism in the near future.
 
Identities in the network are defined by an asymmetric keypair. Most entities will hold this keypair within a TLS certificate, but some entities will just have a keypair. Nodes and Gateways prove their membership to the network by the inclusion of membership information in a signed NDF.
 
==== Initial Run ====
 
When first started, a Node will require two TLS certificates and a unique 256-bit randomly generated registration code provided by xx network to join the network. The certificates must be generated using RSA keys and the registration code will be provided in Base64 string format.
 
On first run, the Node will generate a cryptographic ID and register it with the network, via the Permissioning server. Most of the initial files and generated files are critical and must be preserved by the Node operator. For more information, refer to [[#Cryptographic_and_Network_Primitives|Cryptographic and Network Primitives]].
 
== Hardware Requirements ==
 
Below are the hardware requirements for running a Node and Gateway. Note that there are additional requirements when running a Node and Gateway on the same machine.
 
=== Node Hardware Requirements ===
 
Nodes are high powered machines that use both a CPU and GPU. The software does support full CPU Nodes, but higher core counts are required. As the software becomes more mature and more power is extracted from the GPU, it is likely that the hardware requirements for such Nodes will increase.
 
{| class="wikitable"
! CPU
|
High core count modern CPU
 
: Capable of meeting a multithreaded PassMark score of 15,500 or a Cinebench R5 multi score of 1,750.
 
: '''Examples:''' AMD Ryzen 7 2700x, AMD Ryzen 5 3600x, Intel Core i9-9980HK
|-
! GPU
|
Nvidia Turing: Nvidia GeForce RTX 2070 or greater
 
Nvidia Ampere: Nvidia GeForce RTX 3060 Ti or greater
|-
! RAM
|
16 GB DDR4 or more
 
: {{Note|inline=1|small=1|An upgrade path to 32 GB is recommended.}}
|-
! Storage
|
1 TB High Speed Enterprise NVMe (PCI) SSD
 
: '''Recommended speed:''' 500,000/500,000 IOPS
 
: '''Recommended reliability:''' 1.5 million hours MTBF
 
: '''Example:''' Samsung 970 PRO SSD 1TB – M.2 NVMe
 
: {{Note|inline=1|small=1|SSD reliability is not as important for the BetaNet when there will not be many transitions. It is likely that lower endurance SSDs will be fine but will need to be replaced when it comes time for MainNet. If an SSD fails, it could take substantial time before a Node is able to get back online.}}
 
: {{Note|inline=1|small=1|Full SSD utilization will not occur at launch but will be fully utilized by xx consensus as the BetaNet matures.}}
|-
! Bandwidth
| 100 Mbps upload / 100 Mbps download
|}
 
=== Gateway Hardware Requirements ===
 
Gateways require relatively low powered machines. Every Node must have a Gateway.
 
{| class="wikitable"
! CPU
|
Dual core modern CPU, cloud deployment possible
 
: '''Example:''' AMD Ryzen 3 2200G
|-
! GPU
| Not required
|-
! RAM
| 2 GB or more
|-
! Storage
|
250 GB
 
: Used for database instances.
|-
! Bandwidth
|
100 Mbps upload / 100 Mbps download
 
: {{Note|inline=1|small=1|The Gateway bandwidth requirements are separate from the Node bandwidth requirements.}}
|}
 
=== Node and Gateway Hardware Requirements ===
 
It is possible to run a Gateway off the same machine as a Node; however, it results in a loss of security when they are run off the same IP address and network connection. The Gateway structure makes it more difficult for a Node to be maliciously removed from the network.
 
The specifications for a machine to run both a Node and Gateway differ slightly and are as follows.
 
{| class="wikitable"
! CPU
|
High core count modern CPU
 
: Capable of meeting a multithreaded PassMark score of 15,500 or a Cinebench R5 multi score of 1,750.
 
: '''Examples:''' AMD Ryzen 7 2700x, AMD Ryzen 5 3600x, Intel Core i9-9980HK
|-
! GPU
|
Nvidia Turing: Nvidia GeForce RTX 2070 or greater
 
Nvidia Ampere: Nvidia GeForce RTX 3060 Ti or greater
|-
! RAM
|
16 GB DDR4 or more
 
: {{Note|inline=1|small=1|An upgrade path to 32 GB is recommended.}}
|-
! Storage 1
|
1 TB High Speed Enterprise NVMe (PCI) SSD
 
: '''Recommended speed:''' 500,000/500,000 IOPS
 
: '''Recommended reliability:''' 1.5 million hours MTBF
 
: '''Example:''' Samsung 970 PRO SSD 1TB – M.2 NVMe
 
: {{Note|inline=1|small=1|SSD reliability is not as important for the BetaNet when there will not be many transitions. It is likely that lower endurance SSDs will be fine but will need to be replaced when it comes time for MainNet. If an SSD fails, it could take substantial time before a Node is able to get back online.}}
 
: {{Note|inline=1|small=1|Full SSD utilization will not occur at launch but will be fully utilized by xx consensus as the BetaNet matures.}}
|-
! Storage 2
|
240 GB High Speed Enterprise SSD
 
: '''Recommended speed:''' 100,000/10,000 IOPS
 
: '''Recommended reliability:''' 1.5 million hours MTBF
 
: '''Example:''' 883 DCT 240GB
 
: {{Note|inline=1|small=1|SSD reliability is not as important for the BetaNet when there will not be many transitions. It is likely that lower endurance SSDs will be fine but will need to be replaced when it comes time for MainNet.}}
|-
! Bandwidth
| 150 Mbps upload / 200 Mbps download
|}
 
=== Turing or Ampere GPU Requirement ===
 
The xx network BetaNet requires a Nvidia Turing or Ampere GPU due to upgrades to the microarchitecture of the shaders granting an order of magnitude increase in performance for the specific workloads used. The mul units have been increased to 32 bits, which greatly increases the speed of the modular exponentiation and modular multiplication which are some of the core cryptographic operations of the cMix protocol.
 
== Software Overview ==
 
[[File:Codebase Overview.svg|frame|Figure 2: Overview of the codebase showing relationships and dependencies. Arrows indicate dependencies. The source code for Primitives, Crypto, Comms, GPUMaths, Server, Gateway, Client, User Discovery, Integration, Local Environment, and the Wrapper Script are available on the Elixxir and xx network gits.]]
 
The xx network BetaNet codebase is combined between xx network’s consensus software and Elixxir’s cMix protocol. At launch, xx network will be testing and tuning Elixxir software with xx consensus coming online at a later date.
 
Elixxir software is organized into four groups of repositories as shown in ''Figure 2'': Core, Services, Clients, and Tools. The ''Core'' libraries handle functionality across Clients and Services, which contain data structures, interfaces, and cryptography common to many of the xx network components. ''Services'' implements various messenger features, as well as authorizing Clients and Nodes to join the network. ''Clients'' interact with the network using a shared Client API (xxDK). ''Tools'' provide several interfaces and utilities to deploy, test, and debug the network.
 
Most of the code found in the Core and Services is written in the Go programming language. The repositories listed as public below can be found on the public [https://gitlab.com/elixxir Elixxir] and [https://gitlab.com/xx_network xx network] GitLab pages.
 
To follow the set up guide, most Node operators will likely elect to download the prepackaged tarballs containing the compiled binaries. To learn more, refer to the ''Node and Gateway Software'' section.
 
=== xx network Codebase ===
 
==== Core ====
 
===== xx network/Primitives =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>primitives</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |xx network
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/xx_network/primitives gitlab.com/xx_network/primitives]
|}
 
The xx network Primitives repository contains the basic data structures and utilities that are used in the Elixxir and Praxxis codebases. This includes the ID structure for Nodes, Gateways, and users and the NDF structure. Additional generic structures and utilities are also contained in Primitives, such as file access and rate limiting utilities.
 
===== xx network/Crypto =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>crypto</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |xx network
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/xx_network/crypto gitlab.com/xx_network/crypto]
|}
 
The xx network Crypto repository contains cryptographic primitives that are shared between Elixxir and Praxxis codebases. Primarily lower level primitives, randomness generation, TLS handling, and nonce handling.
 
===== xx network/Comms =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>comms</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |xx network
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/xx_network/comms gitlab.com/xx_network/comms]
|}
 
The Comms repository handles all network communication functionality as well as all of the core connectivity logic. It includes generic comms handlers and protocols and not specific implementations, which are implemented in the respective Comm branches for each project.
 
=== Elixxir Codebase ===
 
==== Core ====
 
===== Elixxir/Primitives =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>primitives</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/primitives gitlab.com/elixxir/primitives]
|}
 
The Elixxir Primitives repository contains the basic data structures and utilities that are used by all Elixxir repositories. This includes the user fact structure, the cMix message structure, and the version object.
 
===== Elixxir/Crypto =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>crypto</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/crypto gitlab.com/elixxir/crypto]
|}
 
The Crypto repository encompasses all of the base cryptographic functionality found in the code base. Relying heavily on Go’s ''big integer'' implementation, Crypto features a cryptographically secure random number generator implementation, libraries for working with large integers in modulo cyclic groups for cMix operations, and basic encrypt/decrypt functionality. Like Primitives, Crypto only contains code that is generic to the larger system. A core approach to this repository is to supply wrappers for operations that may require migration to other implementations in the future.
 
===== Elixxir/Comms =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>comms</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/comms gitlab.com/elixxir/comms]
|}
 
Comms builds on the generic utilities of xx network Comms to provide specific functionality for Elixxir. It holds a gRPC protocol file along with a thin Client/Server implementation. The repository currently uses TLS certificates with RSA keys for encryption and identification, which will migrate to xx consensus-based quantum secure authenticated channels as development progresses.
 
===== Elixxir/GPUMaths =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>gpumathsgo</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/gpumathsgo gitlab.com/elixxir/gpumathsgo]
|}
 
The GPUMaths repository accelerates the math used by Server, especially for precomputations. It provides a subset of the math implemented in the Crypto repository but accelerated on GPUs. Underlying the acceleration is a publicly available CUDA arbitrary-precision math library, CGBN2. Components of this written in CUDA are in a separate repository [https://gitlab.com/elixxir/gpumathsnative gpumathsnative].
 
==== Services ====
 
===== Elixxir/Server =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>server</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/server gitlab.com/elixxir/server]
|}
 
The Server repository implements the core cMix functionality and is the software that a Node runs. It performs precomputation and real time computation and processes messages. It receives batches of messages from the Gateways as well as performs network team operations.
 
===== Elixxir/Gateway =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>gateway</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/gateway gitlab.com/elixxir/gateway]
|}
 
The Gateway repository contains the API for Clients to interact with the network. Every Node runs a Gateway and the Gateways collect and store messages for Clients. Gateway is designed to be a scalable front-end to the xx network.
 
===== Elixxir/Permissioning =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>registration</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |private
|}
 
Permissioning, also referred to as Registration, manages the NDF for Clients and Servers and schedules cMix rounds within the network. Eventually this functionality will be managed by the distributed xx consensus. For now, this code handles admission, manages which Nodes are part of the network, and orchestrates when Nodes operate.
 
==== Clients ====
 
===== Elixxir/Client API (xxDK) =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>client</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/client gitlab.com/elixxir/client]
|}
 
All Clients use the Client API to interact and send messages with the cMix network. It uses Go mobile to produce a library compatible with iOS and Android.
 
===== Elixxir/User Discovery (UD) =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>user-discovery-bot</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/user-discovery-bot gitlab.com/elixxir/user-discovery-bot]
|}
 
User Discovery helps users make first contact with other users. It facilitates user search and lookup in a private manner.
 
===== Elixxir/Mobile Clients =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>client-ios</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |private
 
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>client-android</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |private
|}
 
Currently, clients exist for iOS and Android operating systems. These both use the Go mobile libraries produced in the Client API.
 
==== Tools ====
 
===== Elixxir/Wrapper Script =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>wrapper</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/wrapper gitlab.com/elixxir/wrapper]
|}
 
The Wrapper Script is a Node and Gateway management script that simplifies the running of the xx network software. The script automates the management of the xx network software log files. For easy management or in the event of an error, it starts, stops, and restarts the software without the operator having to revisit the command line. Optionally, it can be set to automatically update the Node and Gateway with the latest xx network binaries and configuration files. To learn more, refer to the Wrapper Script Arguments section.
 
===== Elixxir/DevOps =====
 
(deployment)
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>deployment</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |private
|}
 
DevOps is a deployment platform for Microsoft Azure, Google Cloud Platform, and Amazon Web Services (AWS) written in Terraform. DevOps allows for the deployment of test networks, management of deployments of individual Nodes, and the deployment of multi-cloud implementations of xx network.
 
===== Elixxir/Integration [https://gitlab.com/elixxir/integration <code>integration</code>] <small>[public]</small> =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>integration</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/integration gitlab.com/elixxir/integration]
|}
 
The Integration repository is a series of end-to-end tests designed to test different functionality of the cMix protocol. Several tests focus on different batch sizes with Nodes only. Another test covers all the Client-level interaction within the network.
 
===== Elixxir/Local Environment =====
 
{| class="wikitable" style="font-size: small;"
|-
| style="border: none; font-weight: bold;" | name:
| style="border: none;" |<code>LocalEnvironment</code>
| style="border-width: 0 0 0 1px; font-weight: bold;" | project:
| style="border: none" |elixxir
| style="border-width: 0 0 0 1px; font-weight: bold;" | availability:
| style="border: none" |public
| style="border-width: 0 0 0 1px; font-weight: bold;" | git:
| style="border: none" |[https://gitlab.com/elixxir/LocalEnvironment gitlab.com/elixxir/LocalEnvironment]
|}
 
The Local Environment repository contains a set of scripts designed to allow the testing of the entire platform on a single machine.
 
=== Version Scheme ===
 
Both Gateway and Node binaries have a version string embedded in them, which consists of a major version, a minor version, and a patch string, separated by a period.
 
{| style="text-align: center;"
! Major
!
! Minor
!
! Patch
|-
| ┌┴┐
|
| ┌┴┐
|
| ┌─┴─┐
|- style="font-family: monospace;font-size: xx-large;"
| 4
| .
| 2
| .
| 6ab
|}
 
To be able to participate in the network, a Node or Gateway has to have a compatible version with the required version reported by the Permissioning server. For a version to be compatible, the major version must be equal to the required major version and the minor version must be greater than or equal to the required minor version. The patch can be anything, but it will always be present.
 
== Management Tools ==
 
=== Management Scripts Features ===
 
The management scripts are designed to make administration of Nodes and Gateways easier and less time consuming as well as provide data back to the xx network about the functionality of the Node. Many features that it provides require trusting the xx network. The usage of the management scripts is not required and can be disabled. The following are the primary features of these scripts.
 
* '''Binary restart:''' In the event that the binary process stops due to a handled error, the Wrapper Script will restart it. The process can crash because, in the current implementation, some errors are not fully handled when caused by Node failures on the network. It is highly recommended that this feature is utilized.
* '''Binary update:''' The Wrapper Script has the ability to accept automatic binary and Wrapper Script updates provided by the xx network. In theory, the xx network can push any code it wants if this feature is activated, but it greatly simplifies the task of running a Node. Included in this feature is the ability to send stop and start commands to the Node. All commands are signed and can be proved to have come from the xx network. Running without this feature is fully supported but requires manual updates.
* '''Log upload:''' This features uploads logs to xx network for debugging. It can be disabled and is not mandatory but is useful to the xx network for development purposes.
 
The Management Script is two scripts combined, the service file and the actual Wrapper Script. The service file is a systemd script that starts the Wrapper Script and ensures it stays running. The wrapper script is written in Python and does most of the heavy lifting. Both can be found in the [https://gitlab.com/elixxir/wrapper Elixxir wrapper repository].
 
=== Service File Arguments ===
 
The service files maintain the Node and Gateway processes running the background. There is a different service for each. By default, these files ({{mono|xxnetwork-node.service}} and {{mono|xxnetwork-gateway.service}}) are located in {{mono|/opt/xxnetwork/}} and are soft linked to {{mono|/etc/systemd/system/}}. They are the same script just configured differently for Node and Gateway. These scripts call the Wrapper Script with different options. To see the full details of the options, refer to the [[#Wrapper_Script_Arguments|Wrapper Script Arguments]] section below.
 
==== Node Service File Arguments ====
 
The following are the available options in the service file configured for Node. The sections in {{font color|red||red}} should never be modified and should maintain the original values as provided. The items in {{font color|#00B0F0||blue}} can be modified.
 
{{font color|red||ExecStart{{=}}/bin/bash -c '/opt/xxnetwork/xxnetwork-wrapper.py}}
  {{font color|#00B0F0||--logpath /opt/xxnetwork/node-logs/node.log
  --binary /opt/xxnetwork/bin/xxnetwork-node}}
  {{font color|red||--s3path server
  --s3logbucket alphanet-logs-prod
  --s3managementbucket alphanet-management-prod
  --s3accesskey ${s3_access_key_id}
  --s3secret ${s3_access_key_secret}
  --s3region us-west-1}}
  {{font color|#00B0F0||--configdir /opt/xxnetwork/
  --erroutputpath /opt/xxnetwork/node-logs/node-err.log
  --tmpdir /tmp/xxnetwork/node
  --idpath /opt/xxnetwork/node-logs/nodeIDF.json &gt;&gt; /opt/xxnetwork/node-logs/xxnetwork-wrapper.log 2&gt;&amp;1'}}
 
==== Gateway Service File Arguments ====
 
The following are the available options in the service file configured for Gateway. The sections in {{font color|red||red}} should never be modified and should maintain the original values as provided. The items in {{font color|#00B0F0||blue}} can be modified.
 
{{font color|red||ExecStart{{=}}/bin/bash -c '/opt/xxnetwork/xxnetwork-wrapper.py}}
  {{font color|#00B0F0||--logpath /opt/xxnetwork/gateway-logs/gateway.log
  --binary /opt/xxnetwork/bin/xxnetwork-gateway}}
  {{font color|red||--s3path gateway
  --s3logbucket alphanet-logs-prod
  --s3managementbucket alphanet-management-prod
  --s3accesskey ${s3_access_key_id}
  --s3secret ${s3_access_key_secret}
  --s3region us-west-1}}
  {{font color|#00B0F0||--configdir /opt/xxnetwork/
  --erroutputpath /opt/xxnetwork/gateway-logs/gateway-err.log
  --tmpdir /tmp/xxnetwork/gateway
  --idpath /opt/xxnetwork/gateway-logs/gatewayIDF.json &gt;&gt; /opt/xxnetwork/gateway-logs/xxnetwork-wrapper.log 2&gt;&amp;1'}}
 
=== Wrapper Script Arguments ===
 
The following arguments should be modified inside the service file described above.
 
{{Note|inline=1|The {{mono|s3*}} options are set per the deployment package sent to you and should not be changed. The other options can be changed per your installation, but it is highly recommended that you go with the defaults sent with the install package.}}
 
<div class="mw-highlight mw-content-ltr" dir="ltr"><pre style="padding-left:calc(18ch + 1em);text-indent:-18ch;">usage: wrapper.py [-h] [-d] [-l LOGPATH] [-i IDPATH] -b BINARY [--gpulib GPULIB] [--gpubin GPUBIN] [--disable-consensus] [--consensus-binary CONSENSUS_BINARY] [--consensus-config CONSENSUS_CONFIG] [--consensus-state CONSENSUS_STATE] [--consensus-log CONSENSUS_LOG] [--consensus-cw-group CONSENSUS_CW_GROUP] [-c CONFIGDIR] -s S3PATH [-m S3MANAGEMENTBUCKET] [--disable-cloudwatch] [--cloudwatch-log-group CLOUDWATCH_LOG_GROUP] --s3accesskey S3ACCESSKEY --s3secret S3SECRET --s3region S3REGION [--tmpdir TMPDIR] [--cmdlogdir CMDLOGDIR] [--erroutputpath ERROUTPUTPATH] [--configoverride CONFIGOVERRIDE]</pre></div>
 
{| class="wikitable zebra2r-head no-border2r-head"
! Flag
! Default
! Required
|-
| {{mono|-h}}, {{mono|--help}}
|
| False
|-
|colspan="3"|Show this help message and exits.
|-
| {{mono|-d}}, {{mono|--disableupdates}}
| False
| False
|-
|colspan="3"|Disables automatic updates.
 
Used to disable the download and automatic installation of new binaries, the default configuration file, the wrapper script itself, and the security certificate used for verifying wrapper script commands.
|-
| {{mono|-l LOGPATH}}, {{mono|--logpath LOGPATH}}
| {{mono|/opt/xxnetwork/logs/xx.log}}
| False
|-
|colspan="3"|The path to store logs, e.g., {{mono|/opt/xxnetwork/node-logs/node.log}}.
|-
| {{mono|-i IDPATH}}, {{mono|--idpath IDPATH}}
| {{mono|/opt/xxnetwork/logs/IDF.json}}
| False
|-
|colspan="3"|Node ID path, e.g., {{mono|/opt/xxnetwork/logs/nodeIDF.json}}.
 
The Node stores the ID file at this path.
|-
| {{mono|-b BINARY}}, {{mono|--binary BINARY}}
|
| True
|-
|colspan="3"|Path of the binary to be run by the wrapper.
|-
| {{mono|--gpulib GPULIB}}
| {{mono|/opt/xxnetwork/lib/libpowmosm75.so}}
| False
|-
|colspan="3"|Path to the GPU exponentiation library.
|-
| {{mono|--gpubin GPUBIN}}
| {{mono|/opt/xxnetwork/lib/libpow.fatbin}}
| False
|-
|colspan="3"|Path to the GPU bin file.
|-
| {{mono|--disable-consensus}}
| False
| False
|-
|colspan="3"|Disable consensus binary.
|-
| {{mono|--consensus-binary CONSENSUS_BINARY}}
| {{mono|/opt/xxnetwork/bin/xxnetwork-consensus}}
| False
|-
|colspan="3"|Path to the consensus binary.
|-
| {{mono|--consensus-config CONSENSUS_CONFIG}}
| {{mono|/opt/xxnetwork/consensus.yaml}}
| False
|-
|colspan="3"|Path to the consensus config file.
|-
| {{mono|--consensus-state CONSENSUS_STATE}}
| {{mono|/opt/xxnetwork/consensus.tar.gz}}
| False
|-
|colspan="3"|Path to the consensus state tarball.
|-
| {{mono|--consensus-log CONSENSUS_LO}}
| {{mono|/opt/xxnetwork/consensus-logs/consensus.log}}
| False
|-
|colspan="3"|Path to the consensus log file.
|-
| {{mono|--consensus-cw-group CONSENSUS_CW_GROUP}}
| {{mono|xxnetwork-consensus-prod}}
| False
|-
|colspan="3"|CW log group for consensus logs
|-
| {{mono|-c CONFIGDIR}}, {{mono|--configdir CONFIGDIR}}
| {{mono|/opt/xxnetwork/}}
| False
|-
|colspan="3"|Path to the config directory, e.g., {{mono|~/.xxnetwork/}}.
|-
| {{mono|--configoverride CONFIGOVERRIDE}}
|
| False
|-
|colspan="3"|Override for config file path.
 
If this flag is set, then the script ignores the {{mono|--configdir}} flag and uses this file path instead.
|-
| {{mono|-s S3PATH}}, {{mono|--s3path S3PATH}}
|
| True
|-
|colspan="3"|Path to the S3 management directory.
<br />{{Note|inline=1|small=1|Do not modify.|warn}}
|-
| {{mono|-m S3MANAGEMENTBUCKET}}, {{mono|--s3managementbucket S3MANAGEMENTBUCKET}}
|
| False
|-
|colspan="3"|Path to the S3 management bucket name.
<br />{{Note|inline=1|small=1|Do not modify.|warn}}
|-
| {{mono|--disable-cloudwatch}}
| False
| False
|-
|colspan="3"|Disable uploading log events to CloudWatch.
|-
| {{mono|--cloudwatch-log-group CLOUDWATCH_LOG_GROUP}}
| {{mono|xxnetwork-logs-prod}}
| False
|-
|colspan="3"|Log group for CloudWatch logging.
|-
| {{mono|--s3accesskey S3ACCESSKEY}}
|
| True
|-
|colspan="3"|S3 access key.
<br />{{Note|inline=1|small=1|Do not modify.|warn}}
|-
| {{mono|--s3secret S3SECRET}}
|
| True
|-
|colspan="3"|S3 access key secret.
<br />{{Note|inline=1|small=1|Do not modify.|warn}}
|-
| {{mono|--s3region S3REGION}}
|
|
|-
|colspan="3"|S3 region.
<br />{{Note|inline=1|small=1|Do not modify.|warn}}
|-
| {{mono|--tmpdir TMPDIR}}
| {{mono|/tmp}}
| False
|-
|colspan="3"|Directory for temporary files.
|-
| {{mono|--cmdlogdir CMDLOGDIR}}
| {{mono|/opt/xxnetwork/cmdlog}}
| False
|-
|colspan="3"|Directory for commands log.
|-
| {{mono|--erroutputpath ERROUTPUTPATH}}
|
| False
|-
|colspan="3"|Path to recovered error path.
 
The Node stores recoverable errors at this path.
|}
 
== Deployment Schedule ==
 
While BetaNet is a functioning network, it is still in development and requires consistent updates. If the Wrapper Script is being used in its default configuration, then Node and Gateway will automatically update.
 
However, for Node operators with automatic updates turned off, xx network is committed to the following update schedule to allow them ample time to deploy updates.
 
=== Scheduled Updates ===
 
Every week, there is a single window where major releases and breaking minor releases can be downloaded and installed. The source code will be released on Tuesday before the update goes live on Thursday. Once available on Thursday, any Nodes not up to date cannot participate in the network but will be considered online for the purpose of compensation. Node operators will have until Monday to install the release, after which time they will be considered offline for the purpose of compensation. Refer to the following timeline for specific times and descriptions.
 
All updates will be posted to the [https://gitlab.com/elixxir/ Elixxir GitLab] and Node operators will be informed by email. The xx network will strive to post notice of updates before the Tuesday release, if possible.
 
{| class="wikitable left-head valign-top"
! Tuesday
| {{Hanging indent|text=1:00 pm PT – Release of source code.}}
|-
! Wednesday
|
|-
! Thursday
|
{{Hanging indent|text=1:00 PM PT – Push to Permissioning server, Nodes with out of date code stop functioning.}}
 
{{Hanging indent|text=1:00 PM PT – Push to Nodes with automatic updates enabled.}}
 
{{Hanging indent|text=1:00 PM PT – Grace period for Nodes to update begins.}}
|-
! Friday
|
|-
! Saturday
|
|-
! Sunday
|
|-
! Monday
| {{Hanging indent|text=1:00 PM PT – Grace period for Nodes to update ends, Nodes that have not updated will be considered offline for the purpose of compensation.}}
|}
 
=== Critical Unscheduled Updates ===
 
Critical updates may be released with no notice in the event of catastrophic failure or immediate danger to the network.
 
Nodes will be given 5 business days to update their Node for the purpose of compensation in the event of such an update.
 
== Cryptographic and Network Primitives ==
 
{{Note|Two TLS credentials (a certificate and private key pair) and an identification file define the identity of a Node. These files are extremely important to back up and store securely. Loss of these files will make it impossible to reconnect a Node to the network. In following the default guide, these files are named {{mono|node_cert.crt}}, {{mono|node_key.key}}, {{mono|gateway_cert.crt}}, {{mono|gateway_key.key}}, and {{mono|nodeIDF.json}}. Refer to the [[#Lost_Certificate_and_Important_Files_Policy|Lost Certificate and Important Files Policy]] for more information.|warn}}
 
There are various cryptographic primitives used in the xx network; some are provided by the xx network and some are generated by the Node operator.
 
=== Primitives Provided by xx network ===
 
The following primitives are provided to you by xx network, either emailed directly or they are downloaded from public repositories.
{| class="wikitable left-head valign-top"
|-
! Registration Code
| Sent via email
| The registration code is a 256-bit random number encoded in Base64. It is a single-use code used only on initial registration after which, the Node’s ID and TLS certificate can be used to identify it within the network. This will be provided to each Node operator prior to the network going live.
|-
! Permissioning TLS Certificate
| Provided inside the tarball || A TLS certificate is provided to be used to identify the Permissioning server.
|-
! Permissioning Server Address
| Provided inside the tarball || The DNS address of the Permissioning server is pre-populated inside the YAML files for Node and Gateway.
|-
! Wrapper Script Bucket Address
| Provided inside the tarball || The IP address the Wrapper Script uses to update the binaries. It is pre-populated in the Wrapper Script.
|-
! Log Bucket Address
| Provided inside the tarball || The IP address the Wrapper Script uses to upload logs to. It is pre-populated in the Wrapper Script.
|-
! cMix Cyclic Group
| Provided as part of the NDF on initial registration || The cMix cyclic group is the modulo cyclic group in which cMix operations are conducted. At launch, this will be defined by a 2048-bit strong and safe prime with a generator of 2.
|-
! E2E Cyclic Group
| Provided as part of the NDF on initial registration || The E2E cyclic group is the modulo cyclic group in which end to end encryption operations by clients will be conducted. At launch, this will be defined by a 3192-bit strong and safe prime with a generator of 2.
|}
 
=== Primitives Created by the Node Operator ===
 
The following primitives are generated prior to and during the initial registration of the Node. More details about generation can be found in [[#Generate_TLS_Credentials|Generate TLS Credentials]].
 
==== TLS Credentials ====
 
The TLS credentials are generated as X.509, SHA-256, RSA 4096-bits, and are recommended to last 730 days (2 years). The generated credentials include:
 
* Node TLS certificate
* Node private key
* Gateway TLS certificate
* Gateway private key
 
==== ID File (IDF) ====
 
IDs are 264 bits and consist of a 256-bit hash (BLAKE2b) of a salt and the RSA public key from the Node’s TLS certificate appended with a 8 bits that describe the ID type.
 
{{code|lang=none|ID {{=}} Hash(Node_PubKey, Salt) + TYPE}}
 
Due to the construction of the ID, the ownership of the ID can be proved under RSA’s cryptographic assumptions. Due to the hash, IDs are unpredictably generated in a large sparse space so it is overwhelmingly improbable that independently generated IDs will ever collide, so no central checking of ID generation is necessary.
 
The last byte of the ID is a type byte which describes what type of entity it belongs to. The options for type are:
 
{| class="wikitable"
! Type
! Hex
! Note
|-
| {{mono|Generic}}
| {{mono|0x00}}
| Used for one-off entities such as the Permissioning server, the notifications server, and others
|-
| Gateway
| {{mono|0x01}}
| The Gateways in the network
|-
| {{mono|Node}}
| {{mono|0x02}}
| The Nodes in the network
|-
| {{mono|User}}
| {{mono|0x03}}
| Unique client ID
|}
 
On first run, the Node automatically generates the ID and saves it to a JSON file with the following structure.
{| class="codetable"
| style="width: 50%" | {{Codesample|lang=json|line=1|code={
"id": "1Mo9im6HHpoTDt/lTYjSkWV7dAD0Eh+18xUFfxlm4I4C",}}
| <p>The ID of the Node or Gateway, presented as a base 64 encoded string.</p>
|-
| {{Codesample|lang=json|line=1|start=3|code=<nowiki> </nowiki>"type": "node",}}
| <p>The type of ID is shown here for readability.</p>
|-
| {{Codesample|lang=json|line=1|start=4|code=<nowiki> </nowiki>"salt": [119, 234, 246, 209, 166, 166, 72, 17, 253, 196, 172, 187, 230, 2, 132, 137, 49, 219, 142, 58, 82, 169, 60, 230, 112, 17, 30, 112, 30, 68, 217, 92],}}
| <p>The salt used to generate the ID.</p>
|-
| {{Codesample|lang=json|line=1|start=5|code=<nowiki> </nowiki>"idBytes": [212, 202, 61, 138, 110, 135, 30, 154, 19, 14, 223, 229, 77, 136, 210, 145, 101, 123, 116, 0, 244, 18, 31, 181, 243, 21, 5, 127, 25, 102, 224, 142, 2],
<nowiki>}</nowiki>}}
| <p>The ID of the Node or Gateway, presented as an array of bytes.</p>
|}
 
== Lost Certificate and Important Files Policy ==
 
As of April 27, 2021, the following guidelines and policies are in effect.
 
* The Node and Gateway identities are defined by the certificates, their private keys, and the IDF files. Loss of these files will render your Node and Gateway unable to prove their identity and unable to rejoin the network.
* Backup and retention of these files are the sole responsibility of the Node operator.
* The xx network team nor anyone else can recover a Node or Gateway if these files are lost.
 
== Node and Gateway Set Up ==
 
Below you will find the instructions for setting up a Node and Gateway on the xx network for the first time. These instructions detail how to install both a Node and Gateway on separate machines as well as installing them together on the same machine. Follow the instructions once for each machine.
 
The installation instructions assume that your machine has an active internet connection and an empty storage drive or a drive that can be written over with enough space as outlined in the [[#Hardware_Requirements|Hardware Requirements]] section.
 
=== Overview ===
 
Setting up an xx Node and Gateway is not particularly difficult, but due to the number of components, can take up to 2 hours. The steps for setting up a Node and Gateway are almost identical and the instructions can be followed for both. The general steps for setting up a Node are:
 
# Install the operating system
# Synchronize the system clock (NTP)
# Configuring the local network
# Install the GPU drivers (if setting up a Node with GPU)
# Install and configure the Node and Gateway software
# Generate TLS credentials
# Install and configure the databases
# Set up and start the services
# Backup important files {{Note|inline=1|Make sure to not skip this important step!|reminder}}
 
===== Some Tips for Inexperienced Users =====
 
If this is your first time using a command line interface or you do not remember how to use it, the following are some tips to make using the interface a little easier.
 
* In this document, anytime code is presented in a {{font color|#BBB|#000|{{mono|black box with monospaced font}}}} it means that it is command line input or output. Commands prefixed by a <code>'''$'''</code> are commands that you should enter into your command prompt (do not include <code>'''$'''</code> in the command). Any lines without that prefix are output from the system.
* The <code>sudo</code> command is often prepended to commands found in these instructions. This enables commands to be run with elevated privileges. When used, the system will ask for your password to continue running the command.
 
{{block indent|left=1|text=
{{terminal|skin=noborder|text=
'''$''' sudo nano gateway.yaml
[sudo] password for user: <span class="blink" style="color:lime;">█</span>}}}}
 
* Whenever the system asks for a password to continue, no characters will appear when typing, but type in your password and press {{key press|enter}} and it will work.
* When typing a command or path, use the {{key press|tab}} key to auto complete a partially written statement.
 
=== Installing the Operating System ===
 
The xx network software has been tested only on Ubuntu Server 18.04. We cannot guarantee support if a different operating system version is used, although no decisions have been made to specifically preclude any operating systems.
 
<ol style="list-style-type: decimal;">
 
<li><p>Download Ubuntu Server install image from the [https://releases.ubuntu.com/18.04/ Official Ubuntu website].</p>
 
[[File:Download Ubuntu Server.png|800px|center|alt=Click on the link "64-bit PC (AMD64) server install image" to download the Ubuntu Server image.]]
</li>
 
<li><p>Next, a bootable disk with Linux needs to be created. This can be done by writing it to a DVD or more commonly, a flash drive. Follow one of the following tutorials on how to do so depending on your current operating system and chosen media.</p>
 
{{div col}}
<ul>
<li><p>[https://ubuntu.com/tutorials/create-a-usb-stick-on-ubuntu Create a bootable USB stick on Ubuntu]</p></li>
<li><p>[https://ubuntu.com/tutorials/create-a-usb-stick-on-windows Create a bootable USB stick on Windows]</p></li>
<li><p>[https://ubuntu.com/tutorials/create-a-usb-stick-on-macos Create a bootable USB stick on macOS]</p></li>
<li><p>[https://ubuntu.com/tutorials/burn-a-dvd-on-ubuntu How to burn a DVD on Ubuntu]</p></li>
<li><p>[https://ubuntu.com/tutorials/burn-a-dvd-on-windows How to burn a DVD on Windows]</p></li>
<li><p>[https://ubuntu.com/tutorials/burn-a-dvd-on-macos How to burn a DVD on macOS]</p></li>
</ul>
{{div col end}}
</li>
<li><p>Once your flash drive or DVD are ready, follow the [https://ubuntu.com/tutorials/install-ubuntu-server Tutorial on Installing Ubuntu Server].</p>
<ol style="list-style-type: lower-alpha;">
<li><p>In step 6, make sure you select the first option <code>Install Ubuntu</code>.</p></li>
<li><p>In step 7, make sure to configure your internet connection and get an IP address.</p></li>
<li><p>In step 8, ensure that you select <code>Use an Entire Disk</code>.</p></li>
<li><p>In step 12, pick a server name that does not have any personal identifying information and select a strong password.</p>
{{Note|inline=1|small=1|Create a strong but memorable password. It is recommended that it is longer than 12 characters. Store this password in a safe and secure location. Never share this password with anyone.|warn}}
 
</li>
</ol>
</li>
<li><p>Make sure the machine has turned back on and login using the credentials created in the previous step. Sometimes extra text will be printed and you will not be able to see the login prompt. The prompt should still be there; just type your username and press {{key press|enter}} to continue.</p></li></ol>
 
==== Check Internet Connection ====
 
The rest of the instructions require internet access. Follow these steps to ensure the machine is connected.
 
<ol style="list-style-type: decimal;">
<li><p>Check your current local connection status and local IP address.</p>
{{terminal|skin=noborder|text='''$''' ip addr}}
<p>This should result in a similar output to below. The machine should have a valid local IP address.</p>
{{terminal|skin=noborder|text=
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:f9:b7:83 brd ff:ff:ff:ff:ff:ff
inet <span style="background:#595935;position:relative;">192.168.127.138<span style="position:absolute;font-family:sans-serif;color:#fff;white-space:nowrap;padding:0 0.45em;line-height:1.5em;font-size:85%;top:-2.8em;margin-left:4em;font-weight:bold;background:rgba(0,0,0,0.75);"><span style="position:absolute;left:-5em;bottom:-0.85em;">[[File:Curve Arrow 1.svg|none|link=]]</span>Your local IP address.</span></span>/24 brd 192.168.127.255 scope global dynamic enp1s0
valid_lft 1574sec preferred_lft 1574sec
inet6 fe80::20c:29ff:fef9:b783/64 scope link
valid_lft forever preferred_lft forever}}
</li>
<li><p>Finally, check that the machine has access to the internet by pinging another server. We chose to use {{mono|xx.network}}, but you can use any domain.</p>
{{terminal|skin=noborder|text='''$''' ping -c 4 xx.network}}
<p>If the machine is connected to the internet, the output should look similar to the following example.</p>
 
{{terminal|skin=noborder|text=
PING xx.network (104.26.11.97) 56(84) bytes of data.
64 bytes from 104.26.11.97: icmp_seq=1 ttl=54 time=18.5 ms
64 bytes from 104.26.11.97: icmp_seq=2 ttl=54 time=16.3 ms
64 bytes from 104.26.11.97: icmp_seq=3 ttl=54 time=18.8 ms
64 bytes from 104.26.11.97: icmp_seq=4 ttl=54 time=18.5 ms
 
--- xx.network ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 16.323/18.066/18.854/1.018 ms
 
}}
</li></ol>
 
==== Updating Software and Installing Dependencies ====
 
To ensure all the software works correctly, the system needs to be updated. In addition, an additional dependency for the Wrapper Script to function.
 
<ol style="list-style-type: decimal;">
<li><p>Before continuing, check for updates. This will print many lines about what software is being checked.</p>
{{terminal|skin=noborder|text='''$''' sudo apt -y update}}
</li>
<li><p>Once the check is done, install the updates.</p>
{{terminal|skin=noborder|text='''$''' sudo apt -y upgrade}}
</li>
<li><p>Reboot the machine to ensure all updates are installed fully. Once the machine starts up, log back in.</p>
{{terminal|skin=noborder|text='''$''' sudo reboot now}}
</li>
<li><p>Install the Python package installer.</p>
{{terminal|skin=noborder|text='''$''' sudo apt install -y python3-pip}}
</li>
<li><p>Update the package installer.</p>
{{terminal|skin=noborder|text='''$''' sudo pip3 install -U pip}}
</li>
<li><p>Install the <code>boto3</code> and <code>pyOpenSSL</code> dependencies. The Wrapper Script uses the first package to read commands and send logs to xx network through AWS and the second is used to authenticate them.</p>
{{terminal|skin=noborder|text='''$''' pip3 install --user -U boto3 pyOpenSSL}}
</li>
</ol>
 
==== Setting Up UFW ====
 
Uncomplicated Firewall (UFW) is the default firewall configuration tool for Ubuntu. UFW should already be installed; the following instructions will describe how to enable it and allow the correct ports to be open.
 
<ol style="list-style-type: decimal;">
<li><p>First, ensure that UFW is disabled so that it can be configured.</p>
{{terminal|skin=noborder|text='''$''' sudo ufw disable}}
<p>This should result in the following output.</p>
{{terminal|skin=noborder|text=Firewall stopped and disabled on system startup}}
</li>
<li><p>Allow the port that the Node will use to communicate on over TCP. Only do this for machines running the Node software.</p>
 
{{terminal|skin=noborder|text='''$''' sudo ufw allow 11420/tcp
Rules updated
Rules updated (v6)}}
{{Note|inline=1|small=1|Port {{mono|11420}} is the default port in the provided Node configuration file. A different port may be used, but it must be configured in <code>node.yaml</code>, which is downloaded in a future step.}}
</li>
<li><p>Allow the port that the Gateway will use to communicate over TCP. Only do this for machines running the Gateway software.</p>
{{terminal|skin=noborder|text='''$''' sudo ufw allow 22840/tcp
Rules updated
Rules updated (v6)}}
{{Note|inline=1|small=1|Port {{mono|22840}} is the default port in the provided Gateway configuration file. A different port may be used, but it must be configured in <code>gateway.yaml</code>, which is downloaded in a future step.}}
</li>
<li><p>If you want to manage your node remotely, you may want to allow port {{mono|22}} over TCP for SSH. SSH should only be enabled with key authentication. Improper SSH setup can allow unwanted access so take care when setting up remote access. If you do not want to use SSH or do not know how to, then skip this step.</p>
{{terminal|skin=noborder|text='''$''' sudo ufw allow 22/tcp
Rules updated
Rules updated (v6)}}
</li>
<li><p>Finally, enable UFW.</p>
 
{{terminal|skin=noborder|text='''$''' sudo ufw enable}}
If you are connected over SSH, you may be prompted to continue, press {{key press|Y}}. Note that you may be disconnected when doing so.
{{terminal|skin=noborder|text=Command may disrupt existing ssh connections. Proceed with operation (y<nowiki>|</nowiki>n)? {{highlight|y|#595935}}
Firewall is active and enabled on system startup}}
</li></ol>
 
=== Clock Synchronization (NTP) ===
 
Commands received from the Permissioning server are time-stamped and a synchronized clock is important to properly interpret them. To do so, NTP (Network Time Protocol) must be set up and synchronized.
 
In Ubuntu Server 18.04, this is done through {{inline-code|timedatectl}}. In most installations, it is already running and Node operators only need to check that it is correctly configured. However, the process for other operating systems may be different and it will need to be enabled.
 
<ol style="list-style-type: decimal;">
<li><p>Check if the time synchronization service is running and that the clock is synchronized by entering the following command.</p>
{{terminal|skin=noborder|text='''$''' timedatectl status}}
<p>This will print the following output.</p>
{{terminal|skin=noborder|text=
<nowiki>                      </nowiki>Local time: Sat 2020-06-13 20:43:41 UTC
                  Universal time: Sat 2020-06-13 20:43:41 UTC
                        RTC time: Sat 2020-06-13 20:43:41
                      Time zone: Etc/UTC (UTC, +0000)
      System clock synchronized: yes
systemd-timesyncd.service active: yes
                RTC in local TZ: no}}
<ol style="list-style-type: lower-alpha;">
<li><p>If {{inline-code|System clock synchronized}} is set to {{inline-code|yes}} and {{inline-code|systemd-timesyncd.service active}} is set to {{inline-code|yes}}, then no further action is needed. Skip to the next section [[#Modifying_Max_Number_of_Processes_and_Files|Modifying Max Number of Processes and Files]]. Otherwise, continue to the next step.</p>
{{terminal|skin=noborder|text=
<nowiki>      </nowiki>System clock synchronized: {{highlight|yes|#595935}}
systemd-timesyncd.service active: {{highlight|yes|#595935}}}}
</li>
<li><p>If {{inline-code|System clock synchronized}} is set to {{inline-code|no}} and {{inline-code|systemd-timesyncd.service active}} is set to {{inline-code|yes}}, then the service has had insufficient time to synchronize the clock. Skip to the next section [[#Modifying_Max_Number_of_Processes_and_Files|Modifying Max Number of Processes and Files]] but make sure to check that the clock is synchronized before starting the Node or Gateway software. Otherwise, continue to the next step.</p>
{{terminal|skin=noborder|text=
<nowiki>      </nowiki>System clock synchronized: {{highlight|no|#595935}}
systemd-timesyncd.service active: {{highlight|yes|#595935}}}}
{{Note|inline=1|small=1|Ensure the time is synchronized before starting the software in the [[#System_Services|System Services section]].}}
</li>
<li><p>If {{inline-code|System clock synchronized}} is set to {{inline-code|no}} and {{inline-code|systemd-timesyncd.service active}} is set to {{inline-code|no}}, then the service must be manually started in the [[#Clock_Synchronization_.28NTP.29_step02|next step]].</p>
 
{{terminal|skin=noborder|text=
<nowiki>      </nowiki>System clock synchronized: {{highlight|no|#595935}}
systemd-timesyncd.service active: {{highlight|no|#595935}}}}
</li>
</ol>
<li id="Clock_Synchronization_.28NTP.29_step02"><p>Enter in the following command to get a list of time zones.</p>
{{terminal|skin=noborder|text='''$''' timedatectl list-timezones}}
<p>This will print a list of time zones. Use the up key {{key press|up}} and down key {{key press|down}} to navigate the list and find the time zone that the machine is in. Once found, make a note of the time zone, and press {{key press|Q}} to exit.</p>
 
{{terminal|skin=noborder|text=America/Kentucky/Louisville
America/Kentucky/Monticello
America/Kralendijk
America/La_Paz
America/Lima
America/Los_Angeles
America/Lower_Princes
America/Maceio
America/Managua
America/Manaus
<span style="background: #BBB; color: #000">lines 86-137</span>}}
</li>
<li><p>Once the correct time zone for the machine is found, use the following command to set it.</p>
{{terminal|skin=noborder|text='''$''' sudo timedatectl set-timezone {{highlight|[your selected time zone from [[#Clock_Synchronization_.28NTP.29_step02|{{font color|#BBB|step 2}}]]]|#595935}}}}
</li>
<li><p>Using the date command, ensure that the correct time zone was selected. If the printed time is incorrect, return to [[#Clock_Synchronization_.28NTP.29_step02|step 2]] to select a new time zone.</p>
 
{{terminal|skin=noborder|text=
'''$''' date
Sat Jun 13 13:45:31 PDT 2020}}
</li>
<li><p>Begin the clock synchronization service by entering the following command.</p>
 
{{terminal|skin=noborder|text='''$''' sudo timedatectl set-ntp on}}
</li>
<li><p>Ensure that the service is running by calling {{inline-code|timedatectl}} again.</p>
 
{{terminal|skin=noborder|text='''$''' timedatectl status}}
<p>If {{inline-code|systemd-timesyncd.service active}} is set to {{inline-code|yes}}, then it has been successful. If {{inline-code|System clock synchronized}} is set to {{inline-code|yes}}, then this section is done. If it is set to {{inline-code|no||, then it may take up to 30 minutes for the clock to synchronize. You can continue to the following section, but make sure to check that the clock is synchronized before starting the Node or Gateway software.</p>
 
{{terminal|skin=noborder|text=
<nowiki>                      </nowiki>Local time: Sat 2020-06-13 20:43:41 UTC
                  Universal time: Sat 2020-06-13 20:43:41 UTC
                        RTC time: Sat 2020-06-13 20:43:41
                      Time zone: Etc/UTC (UTC, +0000)
      System clock synchronized: {{highlight|no|#595935}}
systemd-timesyncd.service active: {{highlight|yes|#595935}}
                RTC in local TZ: no
}}
</li></ol>
 
=== Modifying Max Number of Processes and Files ===
 
By default, there is a maximum number of processes and files that can be opened at once. To prevent the Node and Gateway from encountering this limit, it must be removed for all users.
<ol style="list-style-type: decimal;">
<li><p>To remove this limit for users, open the limits configuration file using nano or your favorite text editor.</p>
{{terminal|skin=noborder|text='''$''' sudo nano /etc/security/limits.conf}}
{{Note|inline=1|small=1|The following process of using the nano text editor to modify a file is used elsewhere in this document. Refer back here for detailed steps on how to use it.}}
</li>
<li><p>Once the file is open, use the down arrow key {{key press|down}} to go to the second to last line above where it says {{inline-code|lang=none|# End of file}}. Add the following four lines above that line.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|etc/security/limits.conf}}{{right|Modified  }}<br /></div>
<span style="background:#595935;position:relative;">*  soft    nofile  unlimited
*  hard    nofile  unlimited
*  soft    nproc    unlimited
*  hard    nproc    unlimited<span style="position:absolute;font-family:sans-serif;color:#fff;white-space:nowrap;padding:0 0.45em;line-height:1.5em;font-size:85%;top:40%;margin-left:5.75em;font-weight:bold;background:rgba(0,0,0,0.75);"><span style="position:absolute;left:-6em;top:-3.1em;">[[File:Bracket Arrow 1.svg|none|link=]]</span>Add these four lines.</span></span>
{{font color | #00BBBB | # End of file }}
 
<span style="column-count:6; width:100%;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</span>}}</li>
<li><p>To save the file, press {{key press|Ctrl|X}} and when prompted to save the buffer, press {{key press|Y}}.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|etc/security/limits.conf}}{{right|Modified  }}<br /></div>
*  soft    nofile  unlimited
*  hard    nofile  unlimited
*  soft    nproc    unlimited
*  hard    nproc    unlimited
{{font color | #00BBBB | # End of file }}
 
<div style="background:#BBB; color:#000;">Save modified buffer?  (Answering "No" will DISCARD changes.)</div>{{font color|#000|#BBB|&nbsp;Y}} Yes
{{font color|#000|#BBB|&nbsp;N}} No{{in5}}{{font color|#000|#BBB|^C}} Cancel}}
</li>
<li><p>Finally, when prompted with the file name, press {{key press|Enter}} to exit.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|etc/security/limits.conf}}{{right|Modified  }}<br /></div>
*  soft    nofile  unlimited
*  hard    nofile  unlimited
*  soft    nproc    unlimited
*  hard    nproc    unlimited
{{font color | #00BBBB | # End of file }}
 
<div style="background:#BBB; color:#000;">File Name to Write: etc/security/limits.conf</div><span style="column-count: 6; width:100%">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^C}} Cancel
{{font color|#000|#BBB|M-D}} DOS Format
{{font color|#000|#BBB|M-M}} Mac Format
{{font color|#000|#BBB|M-A}} Append
{{font color|#000|#BBB|M-P}} Prepend
{{font color|#000|#BBB|M-B}} Backup File
{{font color|#000|#BBB|^T}} To Files</span>}}
</li>
<li><p>Once the change has been made, reboot the system.</p>
{{terminal|skin=noborder|text='''$''' sudo reboot now}}
</li></ol>
 
=== Configuring TCP Networking Options ===
 
The congestion window size limits the maximum amount of data sent out to a network after a time of little operation. The default size needs to be increased to remove a bottleneck in the xx network.
 
This change is necessary for the xx network because the cMix protocol transmits in short bursts. As a result, the congestion windows contract between transmissions, causing them to need to reopen on every transmission, significantly slowing down the network in high latency environments.
 
The following instructions are required to be completed on both the Node and Gateway machines.
 
<ol style="list-style-type: decimal;">
<li><p>First, to prevent the congestion windows from shrinking unnecessarily when the connection is idle, disable {{inline-code|tcp_slow_start_after_idle}}.</p>
 
{{terminal|skin=noborder|text='''$''' sudo /bin/bash -c &quot;echo 0 &gt; /proc/sys/net/ipv4/tcp_slow_start_after_idle&quot;}}
</li>
<li><p>To make these settings persist across reboots, store them in the sysctl configuration file.</p>
 
{{terminal|skin=noborder|text='''$''' sudo /bin/bash -c 'echo &quot;net.ipv4.tcp_slow_start_after_idle=0&quot; &gt;&gt; /etc/sysctl.conf'}}
</li>
<li><p>Modify the TCP congestion control algorithm (CCA) to use TCP Bottleneck Bandwidth and RRT (BBR).</p>
 
{{terminal|skin=noborder|text='''$''' sudo sysctl -w net.ipv4.tcp_congestion_control=bbr}}
</li>
<li><p>Modify the default queuing discipline to be fq; this is required to use BBR.</p>
 
{{terminal|skin=noborder|text='''$''' sudo sysctl -w net.core.default_qdisc=fq}}
</li>
<li><p>Apply these two options to {{mono|sysctl.conf}} so they persist on reboot.</p>
 
{{terminal|skin=noborder|text='''$''' sudo /bin/bash -c 'echo &quot;net.ipv4.tcp_congestion_control=bbr&quot; &gt;&gt; /etc/sysctl.conf'
 
'''$''' sudo /bin/bash -c 'echo &quot;net.core.default_qdisc=fq&quot; &gt;&gt; /etc/sysctl.conf'}}
</li></ol>
===== Optional Configuration to Make Initial Connection Not Slow =====
 
While disabling {{inline-code|tcp_slow_start_after_idle}} will keep the congestion windows large after connections are used a few times, you can optionally modify the initial congestion window size using the following instructions to make the minimum congestion window size large enough for cMix batches.
 
{{Note|Please note that these changes can have negative effects in other areas of network performance, so it is generally not recommended to use these settings and you should only change these options if and only if you are certain that the rest of your network will work with these new settings.|warn}}
 
<ol style="list-style-type: decimal;">
<li><p>First, set congestion windows size on sending and receiving to be 700 times the maximum segment size.</p>
 
{{terminal|skin=noborder|text='''$''' sudo ip route change `ip route <nowiki>|</nowiki> grep &quot;^default&quot; <nowiki>|</nowiki> head -1` initcwnd 700 initrwnd 700}}
</li>
<li><p>Store the above command in {{mono|/etc/rc.local}}, to make it run on boot.</p>
 
{{terminal|skin=noborder|text='''$''' sudo /bin/bash -c 'echo -e &quot;#!/bin/bash\nip route change \`ip route <nowiki>|</nowiki> grep \&quot;^default\&quot; <nowiki>|</nowiki> head -1\` initcwnd 700 initrwnd 700\nexit 0&quot; &gt; /etc/rc.local' &amp;&amp; sudo chmod +x /etc/rc.local}}
</li></ol>
Note, if this fails to work or causes problems, such as increased timeouts or instability, remove the file and reboot.
 
{{terminal|skin=noborder|text='''$''' sudo rm /etc/rc.local &amp;&amp; sudo reboot}}
=== Configuring Local Network (Port Forwarding) ===
 
To ensure that the machine can be accessed from outside the local network, the network’s router or gateway must be configured to allow external access to the machine on ports configured above. Three main pieces of information are needed for this part: (1) the port numbers to forward (the defaults are {{mono|11420}} and {{mono|22840}}), (2) the protocol to use (TCP), and (3) the local IP address of the machine, which is retrieved below.
 
<ol style="list-style-type: decimal;">
<li id="Configuring_Local_Network_step01"><p>Get the local IP address of the machine. If Node and Gateway are being run off separate machines on the same network, then run this on both machines.</p>
{{terminal|skin=noborder|text='''$''' hostname -I}}
<p>The local IP address will be printed; it will be in the form of {{mono|0.0.0.0}}. Make sure to make note of this for the later steps.</p>
{{terminal|skin=noborder|text=<span style="background:#595935;position:relative;">[your internal IP address]<span style="position:absolute;font-family:sans-serif;color:#fff;white-space:nowrap;padding:0 0.45em;line-height:1.5em;font-size:85%;margin-left:5.5em;font-weight:bold;background:rgba(0,0,0,0.75);"><span style="position:absolute;left:-5em;">[[File:Straight Arrow 1.svg|none|link=]]</span>Make note of this address.</span></span>}}
 
{{Note|inline=1|small=1|If the machine has multiple network interfaces or an IPv6 address, they will also appear in this list. Ensure that only the correct internal IPv4 address is used.}}
</li>
</ol>
 
: {{Note|The following section describes how to configure the networking equipment on your network. Because of the varying type of equipment configurations, these instructions are generic and may not be accurate for your hardware. Please refer to the manufacturer’s instructions for more detailed and accurate information. Configuration of the network will most likely occur from a different machine on the network.}}
 
<ol start="2" style="list-style-type: decimal;">
<li><p>To access the router to configure, its IP address is needed.</p>
{{terminal|skin=noborder|text='''$''' ip r <nowiki>|</nowiki> grep default}}
<p>This will output the following line, where the first address printed is the router IP address.</p>
{{terminal|skin=noborder|text=default via {{highlight|192.168.1.1|#595935}} dev enp1s0 proto dhcp src 192.168.1.37 metric 100}}
</li>
<li><p>Go to this IP address in a browser (on a different machine) and login using the router credentials. These credentials are either set up by the network administrator or are the default credentials located on the router or found online.</p></li>
<li><p>Locate the ''port forwarding'' options (occasionally called ''virtual server''). These options are sometimes found under the advanced section.</p>
{{Note|inline=1|small=1|Before forwarding the port found above, you may want to provide a static local IP address to the machine. However, that is outside the scope of these instructions.}}
</li>
<li><p>Add the new ports to forward for the Node and Gateway. For each, create a new entry and enter the IP address found in [[#Configuring_Local_Network_step01|step 1]], set the port to the chosen ports (by default it is {{mono|11420}} for Node and {{mono|22840}} for Gateway), and select the TCP protocol. Make sure to save or apply the changes.</p></li></ol>
 
=== GPU Drivers ===
 
The Node software requires a Nvidia RTX graphic processor and the installation of its drivers.
<ol style="list-style-type: decimal;">
<li><p>Install the Nvidia driver.</p>
{{terminal|skin=noborder|text='''$''' sudo apt install -y nvidia-driver-460}}
</li>
<li><p>Once the installation is complete, reboot the system.</p>
 
{{terminal|skin=noborder|text='''$''' sudo reboot now}}
</li>
<li><p>Once the system reboots, log back into the computer.</p></li></ol>
 
==== Verifying the Driver Installation ====
<ol style="list-style-type: decimal;">
<li><p>Check that the system has claimed the device.</p>
{{terminal|skin=noborder|text='''$''' sudo lshw -c display}}
<p>This should result in a similar output to the following.</p>
 
{{terminal|skin=noborder|text=
<nowiki>  </nowiki>*-display
    description: VGA compatible controller
    product: NVIDIA Corporation
    vendor: NVIDIA Corporation
    physical id: 0
    bus info: pci@0000:06:00.0
    version: a1
    width: 64 bits
    clock: 33MHz
    capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
    configuration: driver=nvidia latency=0
    resources: irq:56 memory:f6000000-f6ffffff memory:e0000000-efffffff memory:f0000000-f1ffffff ioport:e000(size=128) memory:c0000-dffff
}}
</li>
<li><p>Next, check the driver and state information.</p>
 
{{terminal|skin=noborder|text='''$''' nvidia-smi}}
This should result in a similar output to the following.
 
{{terminal|skin=noborder|text=
<nowiki>+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82      Driver Version: 440.82      CUDA Version: 10.2    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|        Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|  0  GeForce RTX 2070    Off  | 00000000:06:00.0 Off |                  N/A |
| 41%  51C    P0    28W / 175W |      0MiB /  7979MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
 
+-----------------------------------------------------------------------------+
| Processes:                                                      GPU Memory |
|  GPU      PID  Type  Process name                            Usage      |
|=============================================================================|
|  No running processes found                                                |
+-----------------------------------------------------------------------------+</nowiki>
}}
</li></ol>
 
=== Node and Gateway Software ===
<ol style="list-style-type: decimal;">
<li><p>Change the working directory to {{mono|/opt/}}.</p>
{{terminal|skin=noborder|text='''$''' cd /opt/}}
</li>
<li><p>Download the tarball for Node and/or Gateway using {{inline-code|curl}} with the following URLs. The newest versions can also be found on the [https://xx.network/nodes/run xx network website].</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo curl -L -O [https://xx.network/codebase/node.tar.gz {{font color|#BBB|https://xx.network/codebase/node.tar.gz}}]}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo curl -L -O [https://xx.network/codebase/gateway.tar.gz {{font color|#BBB|https://xx.network/codebase/gateway.tar.gz}}]}}
</li>
<li><p>Next, the downloaded files need to be extracted to the {{mono|/opt/}} directory. Use {{inline-code|tar}} extract them with the flags {{inline-code|-xf}} so that the directory structure remains intact. This will create a new {{mono|/opt/xxnetwork/}} directory with the extracted files inside.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo tar -xvf node.tar.gz}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo tar -xvf gateway.tar.gz}}
{{Note|inline=1|small=1|Both {{mono|node.tar.gz}} and {{mono|gateway.tar.gz}} contain some files that are the same (e.g., {{mono|xxnetwork-wrapper.py}}). When extracting both files, it is expected that these files will be overwritten. Order of extraction does not matter.}}
{{Note|inline=1|small=1|If this command produces an error similar to {{inline-code|lang=none|not in gzip format}}, then that means the file did not download correctly in the previous step. Try downloading the file again and ensure the URL is correct.}}
<p>The file needed to install the software are in the newly created {{mono|/opt/xxnetwork/}} directory. Those files are:</p>
{| class="wikitable zebra-head"
! Node
! Gateway
! Note
|-
| {{mono|bin/xxnetwork-node}}
| {{mono|bin/xxnetwork-gateway}}
| The binary for Node or Gateway.
|-
| {{mono|node.yaml}}
| {{mono|gateway.yaml}}
| The configuration file for the binary.
|-
| {{mono|xxnetwork-node.service}}
| {{mono|xxnetwork-gateway.service}}
| systemd service file.
|-
| {{mono|xxnetwork-wrapper.py}}
| {{mono|xxnetwork-wrapper.py}}
| A management script for the binary.
|-
| {{mono|generate_certs.py}}
| {{mono|generate_certs.py}}
| TLS certificate/key generator script.
|-
| {{mono|creds/permissioning_cert.crt}}
| {{mono|creds/permissioning_cert.crt}}
| The TLS certificate for Permissioning.
|-
| {{mono|creds/network_management.crt}}
| {{mono|creds/network_management.crt}}
| The TLS certificate used to authenticate commands sent to the Wrapper Script.
|-
| {{mono|lib/libpowmosm75.so}}
|
| GPU dependency (for Node only).
|-
| {{mono|lib/libpow.fatbin}}
|
| GPU dependency (for Node only).
|}
</li></ol>
==== Generate TLS Credentials ====
 
TLS credentials for the Node and Gateway are critical to its identity in the network. For more information refer to [[#Cryptographic_and_Network_Primitives|Cryptographic and Network Primitives]]. If running Node and Gateway on different machines, then generate the certificates on the Node machine and copy the necessary credentials to the Gateway server.
 
{{Note|The two TLS credentials (a certificate and private key pair) generated in the following section are extremely important to back up and store securely. Loss of these files will make it impossible to reconnect a Node to the network. In following the default configuration, these files are named {{mono|node_cert.crt}}, {{mono|node_key.key}}, {{inline-code|gateway_cert.crt}}, and {{mono|gateway_key.key}}. In addition, it is important to back up the file {{mono|nodeIDF.json}}, which is generated on the initial run. Refer to the [[#Lost_Certificate_and_Important_Files_Policy|Lost Certificate and Important Files Policy]] for more information.|warn}}
 
<ol style="list-style-type: decimal;">
<li><p>Using the provided {{mono|generate_certs.py}} script, generate the TLS certificates and keys for Node and Gateway. It will provide prompts for all the information required. Start the script using Python 3. Note that this must be run inside the {{mono|/opt/xxnetwork/}} directory.</p>
{{terminal|skin=noborder|text='''$''' cd /opt/xxnetwork/ &amp;&amp; python3 generate_certs.py}}
{{Note|inline=1|small=1|You do not need to use this script to generate the required TLS credentials. Use the requirements in the [[#TLS_Credentials|TLS Credentials section]] to ensure the generated certificates meet the network requirements.}}
</li>
<li><p>The script will launch and query for various information to create the credentials for Node and Gateway. If you choose not to provide any details, default values will be used. After every answer press {{key press|Enter}}.</p>
{{terminal|skin=noborder|text=
This script will ask you to input information to be used in key generation.
If you do not wish to enter any given field, a default will be provided, attributed to the xx network.
Country (default: 'KY (Cayman Islands)'): {{highlight|US|#595935}}
State/province (default: ''): {{highlight|CA|#595935}}
Locality (default: 'George Town'): {{highlight|Claremont|#595935}}
Organization (default: 'xxnetwork'):
Organizational unit (default: 'nodes'): {{highlight|testNodes|#595935}}
Email (default: '[email protected]'):
Domain (default: 'xx.network'):}}
<p>Once all the details have been provided, the certificates will be generated and moved into {{mono|creds/}}.</p>
{{terminal|skin=noborder|text=
Generating a RSA private key
............... ++++
....++++
writing new private key to '{{highlight|creds/node_key.key|#595935}}'
<nowiki>-----
~~~~~</nowiki>
Generating a RSA private key
..................................++++
............++++
writing new private key to '{{highlight|creds/gateway_key.key|#595935}}'
<nowiki>-----</nowiki>}}
</li>
<li><p>To check that all the correct files are in the {{mono|creds/}} directory, use {{inline-code|ls}} to list the files.</p>
{{terminal|skin=noborder|text=
'''$''' ls -1 creds/
gateway_cert.crt
gateway_key.key
network_management.crt
node_cert.crt
node_key.key
permissioning_cert.crt}}
</li>
<li><p>If running Gateway on a separate machine from Node, then copy {{mono|gateway_cert.crt}}, {{mono|gateway_key.key}}, and {{mono|node_cert.crt}} to the Gateway machine and delete {{mono|gateway_key.key}} from the Node machine.</p></li></ol>
 
==== Configuring Node and Gateway ====
 
Configuring Node and Gateway requires modifying their respective YAML files retrieved in the previous steps. These instructions go over the basic configuration to get a Node and Gateway working in the default state. Refer to the [https://gitlab.com/elixxir/server/-/blob/release/README.md Server] and [https://gitlab.com/elixxir/gateway/-/blob/release/README.md Gateway] README files to learn more. Note that all addresses must be IPv4; IPv6 is not currently supported.
 
{{Note|If you are currently configuring the Node on its own machine, then proceed to [[#Node_Configuration|Node Configuration]].}}
 
===== Gateway Configuration =====
 
The default {{mono|gateway.yaml}} file only requires setting the address of its Node. However, there are other optional flags for non-standard configurations. Database information is set in a later step.
 
<ol style="list-style-type: decimal;">
<li id="Configuring_Node_and_Gateway_step01"><p>Get the IP address of the Node. If the Node is being run on the same machine as Gateway, then this IP address is {{mono|0.0.0.0:''port''}} (where the default port is {{mono|11420}}). If the Node and Gateway are installed on separate machines, then you will need to get the Node’s ''public'' IP address, by running the following command on the Node.</p>
{{terminal|skin=noborder|text=
'''$''' curl ipinfo.io/ip
{{highlight|[your IP address]|#595935}}}}
{{Note|inline=1|small=1|This should be run on the Node machine.|reminder}}
</li>
<li><p>The Gateway configuration file has to be modified to include this IP address. To do this, on the Gateway machine, open {{mono|gateway.yaml}} in nano or your favorite text editor.</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/gateway.yaml}}
</li>
<li><p>Once the file is open, use the down arrow key {{key press|down}} to get to the line that says {{inline-code|nodeAddress}} and replace the placeholder with the one found in [[#Configuring_Node_and_Gateway_step01|step 1]]. Make sure to include the chosen port, as shown.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|gateway.yaml}}{{right|Modified  }}<br /></div>{{font color|#00BBBB|# The IP address of the Node that the Gateway communicates with. Expects an IPv4
# address with a port. (Required)}}
nodeAddress: "{{highlight|[node IP address]:11420|#595935}}"
 
<span style="column-count:6; width:100%;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</span>}}
{{terminal|skin=noborder|text=
GNU nano 2.9.3 gateway.yaml Modified
# The IP address of the Node that the Gateway communicates with. Expects an IPv4
# address with a port. (Required)
nodeAddress: &quot;[node IP address]:11420&quot;
^G Get Help ^O Write Out ^W Where Is ^K Cut Text ^J Justify ^C Cur Pos
^X Exit ^R Read File ^\ Replace ^U Uncut Text ^T To Spell ^_ Go To Line}}
</li>
<li><p>For non-default setups and experienced users, the following options can be used.</p>
<ol style="list-style-type: lower-alpha;">
<li><p>By default, the Gateway runs on port {{mono|22840}}; however, this can be modified using the port flag.</p></li>
<li><p>The internal listening address of the Gateway defaults to {{mono|0.0.0.0}}. However, a different internal address can be set via {{inline-code|listeningAddress}}. This expects an IPv4 address without a port; it uses the port from the {{inline-code|port}} flag.</p></li>
<li><p>The public IP of the Gateway, as reported to the network, is determined by pinging an external service. To override this or to avoid this outside communication, the public IP and port can be manually set via {{inline-code|overridePublicIP}}. If no port is set, then the port from the {{inline-code|port}} flag is used. This expects an IPv4 address only.</p></li>
<li><p>If you placed the TLS certificate and key files in a different location than the default one in the [[#Generate_TLS_Credentials|Generate TLS Credentials]], then modify the location via {{inline-code|keypath}}, {{inline-code|certPath}}, {{inline-code|serverCertPath}}, and {{inline-code|permissioningCertPath}}.</p></li>
</ol>
</li>
<li><p>Once the change is made, save the file by pressing {{key press|Ctrl|X}} and when prompted to save buffer, press {{key press|Y}}. Finally, when prompted with the file name, press {{key press|Enter}}.</p></li>
</ol>
 
===== Node Configuration =====
 
The default {{mono|node.yaml}} file only requires setting your registration code. However, there are other optional flags for non-standard configurations. Database information is set in a later step.
<ol style="list-style-type: decimal;">
<li><p>Every Node operator will receive a unique registration code from [mailto:[email protected] [email protected]] in order to join the network. The registration code will look similar to the following example.</p>
<p>{{align|center|'''Example Registration Code:''' <big>{{inline-code|<nowiki>UvyGCZU8WP18PmmIdcpVmx00QA3xNe7sEB9Hixkk=</nowiki>}}</big>}}</p>
<p>The Node configuration file has to be modified to include this registration code. To do this, open {{mono|node.yaml}} in nano or your favorite text editor.</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/node.yaml}}
</li>
<li><p>Enter this code into the {{inline-code|registrationCode}} field between the two quotes {{inline-code|""}}.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|node.yaml}}{{right|Modified  }}<br /></div>{{font color|#00BBBB|# Registration code used for first time registration. This is a unique code
# provided by xx network. (Required)}}
registrationCode: "{{highlight|[your registration code]|#595935}}"
 
<span style="column-count:6; width:100%;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</span>}}
</li>
<li><p>For non-default setups and experienced users, the following options can be used.</p>
<ol style="list-style-type: lower-alpha;">
<li><p>If the Node does not have a GPU, then set {{inline-code|useGPU}} to {{inline-code|lang=none|false}}.</p>
{{Note|inline=1|small=1|Running a Node without a GPU is not a recommended configuration. Only do so if the CPU in the machine is very powerful.}}
</li>
<li><p>By default, the Node runs on port {{mono|11420}}; however, this can be modified using the {{inline-code|port}} flag.</p></li>
<li><p>The internal listening address of the Node defaults to {{mono|0.0.0.0}}. However, a different internal address can be set via {{inline-code|listeningAddress}}. This expects an IPv4 address without a port; it uses the port from the {{inline-code|port}} flag.</p></li>
<li><p>The public IP of the Node, as reported to the network, is determined by pinging an external service. To override this or to avoid this outside communication, the public IP and port can be manually set via {{inline-code|overridePublicIP}}. If no port is set, then the port from the {{inline-code|port}} flag is used. This expects an IPv4 address only.</p></li>
<li><p>If you placed the TLS certificate and key files in a different location than the default one in the [[#Generate_TLS_Credentials|Generate TLS Credentials]], then modify the location via {{mono|node}} &gt; {{mono|paths}} &gt; {{mono|cert}}, {{mono|node}} &gt; {{mono|paths}} &gt; {{mono|key}}, {{mono|gateway}} &gt; {{mono|paths}} &gt; {{mono|cert}}, and {{mono|permissioning}} &gt; {{mono|paths}} &gt; {{mono|cert}}</p></li></ol>
</li>
<li><p>Once the change is made, save the file by pressing {{key press|Ctrl|X}} and when prompted to save buffer, press {{key press|Y}}. Finally, when prompted with the file name, press {{key press|Enter}}.</p></li></ol>
 
==== Configure Service Files ====
 
To ensure that the Node and Gateway services are run as the correct user, they will need to be modified.
 
<ol style="list-style-type: decimal;">
<li id="Configure_Service_Files_step01"><p>The following steps require the machine’s username. For most users, this will be the username created and installation of the operating system. If you do not remember your username, use whoami.</p>
{{terminal|skin=noborder|text=
'''$''' whoami
{{highlight|[your username]|#595935}}}}
</li>
<li><p>Open the service file for the binary being run. If both Node and Gateway are being run, this modification will need to be made with both files. Open either {{mono|xxnetwork-node.service}} or {{mono|xxnetwork-gateway.service}} with nano or your favorite text editor.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/xxnetwork-node.service}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/xxnetwork-gateway.service}}
</li>
<li><p>Under the {{inline-code|[Service]}} heading, after the {{inline-code|User{{=}}}} field, modify the username to the one found in [[#Configure_Service_Files_step01|step 1]].</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|xxnetwork-node.service}}{{right|Modified  }}<br /></div>[Service]
User={{highlight|ubuntu|#595935}}
Type=simple
 
<span style="column-count:6; width:100%;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</span>}}
</li>
<li id="Configure_Service_Files_step04"><p>Once the change is made, save the file by pressing {{key press|Ctrl|X}} and when prompted to save buffer, press {{key press|Y}}. Finally, when prompted with the file name, press {{key press|Enter}}.</p></li>
<li><p>Repeat [[#Configure_Service_Files_step01|step 1]] through [[#Configure_Service_Files_step04|step 4]] for the other service file if both Node and Gateway are being run.</p></li></ol>
 
=== Databases ===
 
Both the Node and Gateway require PostgreSQL databases. The Node database stores key pairs with clients while the Gateway database is used to store encrypted messages for users. The following instructions detail how to install PostgreSQL and set up the required databases for the two possible setups: running the Node and Gateway on the same machine and on separate machines. Follow the case pertaining to your setup and at the end make sure to [[#Verify_the_Database|Verify the Database]].
 
==== Case 1: Node and Gateway Run on the Same Machine ====
 
These instructions are for Node operators who run the Node and Gateway software on the same machine. Note that a separate drive for the Gateway database is required that meets the [[#Gateway_Hardware_Requirements|hardware specifications]]. Before starting, prepare the second disk for the Gateway database by mounting and partitioning it. To learn more, refer to [https://forum.xx.network/t/adding-the-gateway-database-one-computer-configuration/4769 Adding the Gateway Database (One Computer Configuration)] post on the [https://forum.xx.network/ xx network forum].
<ol style="list-style-type: decimal;">
<li><p>Install PostgreSQL and its dependencies.</p>
{{terminal|skin=noborder|text='''$''' sudo apt install -y postgresql-client postgresql postgresql-contrib}}
</li>
<li><p>Once the installation is complete, enable the PostgreSQL service.</p>
{{terminal|skin=noborder|text='''$''' sudo update-rc.d postgresql enable}}
</li>
<li><p>Next, start the service.</p>
{{terminal|skin=noborder|text='''$''' sudo service postgresql start}}
</li>
<li><p>Create a database user with the username {{inline-code|cmix}}, which is used to access the database.</p>
{{terminal|skin=noborder|text='''$''' sudo -u postgres createuser --createdb --pwprompt cmix}}
</li>
<li id="Case_1:_Node_and_Gateway_Run_on_the_Same_Machine_step05"><p>You will be asked to set a password for the {{mono|cmix}} user. Create a long and secure password but note that it will only be saved into the Node and Gateway configuration files in the following steps and you will not need to remember it once everything is configured.</p>
{{terminal|skin=noborder|text=
Enter password for new role:
Enter it again:}}
{{Note|inline=1|small=1|Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.|warn}}
</li>
</ol>
===== Node Database Setup =====
 
<ol style="list-style-type: decimal;">
<li><p>Create the required database with the name {{inline-code|cmix_node}}.</p>
{{terminal|skin=noborder|text='''$''' sudo -u postgres createdb -O cmix cmix_node}}
</li>
<li><p>Next, modify the Node configuration file to use the newly generated password in [[#Case_1:_Node_and_Gateway_Run_on_the_Same_Machine_step05|step 5]] above. To do this, open {{mono|node.yaml}} in nano or your favorite text editor.</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/node.yaml}}
</li>
<li><p>Under the field {{inline-code|database}} &gt; {{inline-code|password}}, enter in the new password. If the database was created with a different username or database name, then update those values too.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|node.yaml}}{{right|Modified  }}<br /></div>{{font color | #00BBBB | # Information to connect to the Postgres database storing keys. (Required) }}
database:
  name: "cmix_node"
  address: "0.0.0.0:5432"
  username: "cmix"
  password: "{{highlight|[password]|#595935}}"
 
<span style="column-count:6; width:100%;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</span>}}
</li>
<li><p>Once the change is made, save the file by pressing {{key press|Ctrl|X}} and when prompted to save buffer, press {{key press|Y}}. Finally, when prompted with the file name, press {{key press|Enter}}.</p></li>
</ol>
 
===== Gateway Database Setup =====
 
<ol style="list-style-type: decimal;">
<li><p>Determine the location to place the Gateway database. The following directions use {{mono|/mnt/ts_gateway}} as an example path; replace it with the location on the secondary drive.</p></li>
<li><p>Create a directory on the secondary drive to store the database in using {{inline-code|mkdir}}.</p>
{{terminal|skin=noborder|text='''$''' sudo mkdir {{highlight|/mnt/ts_gateway|#595935}}}}
</li>
<li><p>Change the ownership of the directory.</p>
{{terminal|skin=noborder|text='''$''' sudo chown -v postgres:postgres {{highlight|/mnt/ts_gateway|#595935}}}}
</li>
<li><p>Switch to the {{inline-code|postgres}} user account.</p>
{{terminal|skin=noborder|text='''$''' sudo su postgres}}
</li>
<li><p>Run the PostgreSQL client server via {{inline-code|psql}}.</p>
{{terminal|skin=noborder|text=
'''$''' psql
psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
Type "help" for help.}}
{{Note|inline=1|small=1|The command prompt has changed from <code>'''$'''</code> to <code>'''#'''</code>.}}
</li>
<li><p>Once at the {{inline-code|postgres}} prompt, create the tablespace and point it to the directory on the second disk drive (enter everything after the {{inline-code|postgres-#}} part).</p>
{{terminal|skin=noborder|text=
'''postgres=#''' CREATE TABLESPACE ts_gateway LOCATION '{{highlight|/mnt/ts_gateway|#595935}}';
CREATE TABLESPACE}}
</li>
<li><p>Enter {{inline-code|\q}} to exit the session.</p>
{{terminal|skin=noborder|text='''postgres=#''' \q}}
</li>
<li><p>Enter {{inline-code|lang=none|exit}} to return back to your user.</p>
{{terminal|skin=noborder|text='''$''' exit}}
</li>
<li><p>Create the Gateway database with the name {{inline-code|cmix_gateway}} with the previously made tablespace.</p>
{{terminal|skin=noborder|text='''$''' sudo -u postgres createdb -D <u>ts_gateway</u> -O cmix {{highlight|cmix_gateway|#595935}}}}
</li>
<li><p>Finally, modify the Gateway configuration file to use the password generated when setting up the database user {{mono|cmix}}. To do this, open {{mono|gateway.yaml}} in nano or your favorite text editor.</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/gateway.yaml}}
</li>
<li><p>Under the field {{inline-code|dbPassword}}, enter in the new password. If the database was created with a different username or database name, then update those values too.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|gateway.yaml}}{{right|Modified  }}<br /></div>{{font color|#00BBBB|# Database connection information. (Required)}}
dbName: "cmix_gateway"
dbAddress: "0.0.0.0:5432"
dbUsername: "cmix"
dbPassword: "{{highlight|[password]|#595935}}"
 
<span style="column-count:6; width:100%;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</span>}}
</li>
<li><p>Once the change is made, save the file by pressing {{key press|Ctrl|X}} and when prompted to save buffer, press {{key press|Y}}. Finally, when prompted with the file name, press {{key press|Enter}}.</p></li>
</ol>
 
==== Case 2: Node and Gateway are Run on Separate Machines ====
 
These instructions are for operators who run the Node and Gateway software on the separate machines.
 
===== Node Database Setup on Node Machine =====
 
Follow the instructions below on the machine running the Node software.
<ol style="list-style-type: decimal;">
<li><p>Install PostgreSQL and its dependencies.</p>
{{terminal|skin=noborder|text='''$''' sudo apt install -y postgresql-client postgresql postgresql-contrib}}
</li>
<li><p>Once the installation is complete, enable the PostgreSQL service.</p>
{{terminal|skin=noborder|text='''$''' sudo update-rc.d postgresql enable}}
</li>
<li><p>Next, start the service.</p>
{{terminal|skin=noborder|text='''$''' sudo service postgresql start}}
</li>
<li><p>Create a database user with the username {{inline-code|cmix}}, which is used to access the database.</p>
{{terminal|skin=noborder|text='''$''' sudo -u postgres createuser --createdb --pwprompt {{highlight|cmix|#595935}}}}
</li>
<li id="Node_Database_Setup_step05"><p>You will be asked to set a password for the {{mono|cmix}} user. Create a long and secure password but note that it will only be saved into the Node configuration file in the following steps and you will not need to remember it once everything is configured.</p>
{{terminal|skin=noborder|text=
Enter password for new role:
Enter it again:}}
{{Note|inline=1|small=1|Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.|warn}}
</li>
<li><p>Create the required database with the name {{inline-code|cmix_node}}.</p>
{{terminal|skin=noborder|text='''$''' sudo -u postgres createdb -O cmix {{highlight|cmix_node|#595935}}}}
</li>
<li><p>Next, modify the Node configuration file to use the newly generated password in [[#Node_Database_Setup_step05|step 5]] above. To do this, open {{mono|node.yaml}} in nano or your favorite text editor.</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/node.yaml}}
</li>
<li><p>Under the field {{inline-code|database}} &gt; {{inline-code|password}}, enter in the new password. If the database was created with a different username or database name, then update those values too.</p>
{{terminal|skin=noborder|text=
<table style="width:100%; background:#BBB; color:#000;">
<tr>
<td style="width:30%; text-align:left; padding-left:2em;">GNU nano 2.9.3
</td>
<td style="width:30%; text-align:center;">node.yaml
</td>
<td style="width:30%; text-align:right; padding-right:2em;">Modified
</td></tr></table>{{font color | #00BBBB | # Information to connect to the Postgres database storing keys. (Required) }}
database:
  name: "cmix_node"
  address: "0.0.0.0:5432"
  username: "cmix"
  password: "{{highlight|[password]|#595935}}"
 
<div style="column-count: 6;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</div>}}
</li>
<li><p>Once the change is made, save the file by pressing {{key press|Ctrl|X}} and when prompted to save buffer, press {{key press|Y}}. Finally, when prompted with the file name, press {{key press|Enter}}.</p></li>
</ol>
 
===== Gateway Database Setup on Gateway Machine =====
 
Follow the instructions below on the machine running the Gateway software.
 
<ol style="list-style-type: decimal;">
<li><p>Install PostgreSQL and its dependencies.</p>
{{terminal|skin=noborder|text='''$''' sudo apt install -y postgresql-client postgresql postgresql-contrib}}
</li>
<li><p>Once the installation is complete, enable the PostgreSQL service.</p>
{{terminal|skin=noborder|text='''$''' sudo update-rc.d postgresql enable}}
</li>
<li><p>Next, start the service.</p>
{{terminal|skin=noborder|text='''$''' sudo service postgresql start}}
</li>
<li><p>Create a database user with the username {{inline-code|cmix}}, which is used to access the database.</p>
{{terminal|skin=noborder|text='''$''' sudo -u postgres createuser --createdb --pwprompt {{highlight|cmix|#595935}}}}
</li>
<li id="Gateway_Database_Setup_step05"><p>You will be asked to set a password for the {{mono|cmix}} user. Create a long and secure password but note that it will only be saved into the Gateway configuration file in the following steps and you will not need to remember it once everything is configured.</p>
{{terminal|skin=noborder|text=
Enter password for new role:
Enter it again:}}
{{Note|inline=1|small=1|Never store this password digitally unless directed to do so by xx network. Never provide this password to anyone.|warn}}
</li>
<li><p>Create the required database with the name {{inline-code|cmix_gateway}}.</p>
{{terminal|skin=noborder|text='''$''' sudo -u postgres createdb -O {{highlight|cmix cmix_gateway|#595935}}}}
</li>
<li><p>Next, modify the Gateway configuration file to use the newly generated password in [[#Gateway_Database_Setup_step05|step 5]] above. To do this, open {{mono|gateway.yaml}} in nano or your favorite text editor.</p>
{{terminal|skin=noborder|text='''$''' sudo nano /opt/xxnetwork/gateway.yaml}}
</li>
<li><p>Under the field {{inline-code|dbPassword}}, enter in the new password. If the database was created with a different username or database name, then update those values too.</p>
{{terminal|skin=noborder|text=
<div style="column-count: 3; background:#BBB; color:#000;">{{left|  GNU nano 2.9.3}}<br />{{center|gateway.yaml}}{{right|Modified  }}<br /></div>{{font color|#00BBBB|# Database connection information. (Required)}}
dbName: "cmix_gateway"
dbAddress: "0.0.0.0:5432"
dbUsername: "cmix"
dbPassword: "{{highlight|[password]|#595935}}"
 
<span style="column-count:6; width:100%;">{{font color|#000|#BBB|^G}} Get Help
{{font color|#000|#BBB|^X}} Exit
{{font color|#000|#BBB|^O}} Write Out
{{font color|#000|#BBB|^R}} Read File
{{font color|#000|#BBB|^W}} Where Is
{{font color|#000|#BBB|^\}} Replace
{{font color|#000|#BBB|^K}} Cut Text
{{font color|#000|#BBB|^U}} Uncut Text
{{font color|#000|#BBB|^J}} Justify
{{font color|#000|#BBB|^T}} To Spell
{{font color|#000|#BBB|^C}} Cur Pos
{{font color|#000|#BBB|^_}} Go To Line</span>}}
</li>
<li><p>Once the change is made, save the file by pressing {{key press|Ctrl|X}} and when prompted to save buffer, press {{key press|Y}}. Finally, when prompted with the file name, press {{key press|Enter}}.</p></li>
</ol>
 
==== Verify the Database ====
 
The following steps will describe how to ensure the database was created correctly. This section is optional but highly recommended.
<ol style="list-style-type: decimal;">
<li><p>Login with the {{inline-code|postgres}} account.</p>
{{terminal|skin=noborder|text='''$''' sudo su postgres}}
</li>
<li><p>Run the PostgreSQL client server via {{inline-code|psql}}.</p>
{{terminal|skin=noborder|text=
'''$''' psql
psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
Type "help" for help.}}
{{Note|inline=1|small=1|The command prompt has changed from <code>'''$'''</code> to <code>'''#'''</code>.}}
</li>
<li><p>Once at the {{mono|postgres}} prompt, enter in {{inline-code|\l}} to get a list of databases (do not enter the {{inline-code|lang=none|postgres-#}} part).</p>
{{terminal|skin=noborder|text='''postgres-#''' \l}}
<p>Ensure that the correct databases with the correct owners show up. If Node and Gateway are on the same machine, then the output should appear similar to below. If each is being run separately, then only the {{mono|cmix_node}} or {{mono|cmix_gateway}} should be displayed.</p>
{{terminal|skin=noborder|text=
<nowiki>                                List of databases
    Name    |  Owner  | Encoding |  Collate  |    Ctype    | Access privileges
--------------+----------+----------+-------------+-------------+--------------------</nowiki>
{{highlight|cmix_gateway|#595935}} <nowiki>|</nowiki> {{highlight|cmix|#595935}}<nowiki>    | UTF8    | C.UTF-8    | C.UTF-8    |</nowiki>
{{highlight|cmix_node|#595935}}    <nowiki>|</nowiki> {{highlight|cmix|#595935}}<nowiki>    | UTF8    | en_US.UTF-8 | en_US.UTF-8 |
postgres    | postgres | UTF8    | en_US.UTF-8 | en_US.UTF-8 |
template0    | postgres | UTF8    | en_US.UTF-8 | en_US.UTF-8 | =c/postgres      +
              |          |          |            |            | postgres=CTc/postgres
template1    | postgres | UTF8    | en_US.UTF-8 | en_US.UTF-8 | =c/postgres      +
              |          |          |            |            | postgres=CTc/postgres
(4 rows)</nowiki>}}
</li>
<li><p>If Node and Gateway are run off the same machine, verify the gateway tablespace points to the directory on the second disk drive by typing {{inline-code|\db}} to get a list of available tablespaces. Ensure that the tablespace {{mono|ts_gateway}} has the correct location set.</p>
{{terminal|skin=noborder|text=
postgres=# \db
<nowiki>              List of tablespaces
    Name    |  Owner  |        Location
------------+----------+-------------------------
pg_default | postgres |
pg_global  | postgres |
</nowiki>{{highlight|ts_gateway|#595935}} <nowiki>| postgres |</nowiki> {{highlight|/mnt/ts_gateway|#595935}}
(3 rows)}}
</li>
<li><p>Enter {{inline-code|\q}} to exit the command line.</p>
{{terminal|skin=noborder|text='''postgres-#''' \q}}
</li>
<li><p>Enter {{inline-code|lang=none|exit}} to return back to your prompt.</p>
{{terminal|skin=noborder|text='''$''' exit}}
</li></ol>
 
=== System Services ===
 
This section describes how to install the service files for Node and Gateway. Follow only the instructions related to the type of machine you are deploying. Follow both Node and Gateway instructions if running both off of the same machine.
 
{{Note|'''Warning:''' The following steps will start the Node and/or Gateway software and will have it join the xx network. If you do not want to do so currently, then do not do the following steps until you are ready.|warn}}
 
<ol style="list-style-type: decimal;">
<li><p>Before starting the Node or Gateway service, ensure that the clock is synchronized by entering the following command.</p>
{{terminal|skin=noborder|text='''$''' timedatectl status}}
<p>This should result in the following output.</p>
{{terminal|skin=noborder|text=
<nowiki>                      </nowiki>Local time: Sat 2020-06-13 20:43:41 UTC
                  Universal time: Sat 2020-06-13 20:43:41 UTC
                        RTC time: Sat 2020-06-13 20:43:41
                      Time zone: Etc/UTC (UTC, +0000)
      System clock synchronized: {{highlight|yes|#595935}}
systemd-timesyncd.service active: {{highlight|yes|#595935}}
                RTC in local TZ: no}}
</li>
<li><p>If both {{inline-code|System clock synchronized}} and {{inline-code|systemd-timesyncd.service active}} are set to {{inline-code|yes}}, then proceed to the next step. Otherwise, refer back to the [[#Clock_Synchronization_.28NTP.29|Clock Synchronization (NTP) section]].</p></li>
<li><p>Next, soft link the systemd service file(s) to the systemd directory {{mono|/etc/systemd/system/}}.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo ln -s /opt/xxnetwork/xxnetwork-node.service /etc/systemd/system}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo ln -s /opt/xxnetwork/xxnetwork-gateway.service /etc/systemd/system}}
</li>
<li><p>Reload the systemd service files.</p>
{{terminal|skin=noborder|text='''$''' sudo systemctl daemon-reexec}}
</li>
<li><p>To ensure that your user has the correct permissions, run the following command. Make sure to replace both instances of {{inline-code|[user]}} with your username.</p>
{{terminal|skin=noborder|text='''$''' sudo chown {{highlight|[user]|#595935}}:{{highlight|[user]|#595935}} -Rv /opt/xxnetwork}}
</li>
<li><p>Next, enable the service(s) for the appropriate binary.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo systemctl enable xxnetwork-node.service}}
<p>This should result in the following output.</p>
{{terminal|skin=noborder|text=Created symlink /etc/systemd/system/multi-user.target.wants/xxnetwork-node.service → /opt/xxnetwork/xxnetwork-node.service.}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo systemctl enable xxnetwork-gateway.service}}
<p>This should result in the following output.</p>
{{terminal|skin=noborder|text=Created symlink /etc/systemd/system/multi-user.target.wants/xxnetwork-gateway.service → /opt/xxnetwork/xxnetwork-gateway.service.}}
</li>
<li><p>Then, start the services.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo systemctl start xxnetwork-node.service}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo systemctl start xxnetwork-gateway.service}}
</li>
</ol>
 
==== Verify the Service has Started ====
 
<ol style="list-style-type: decimal;">
<li><p>First, check the status of the service to ensure it is running.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo systemctl status xxnetwork-node.service}}
<p>This should result in the following output. Press {{key press|Q}} to exit the output.</p>
{{terminal|skin=noborder|text=
{{font color|#55FF55|●}} xxnetwork-node.service - Job that starts the Wrapper service
  Loaded: loaded (/opt/xxnetwork/xxnetwork-node.service; enabled; vendor preset: enabled)
  Active: {{font color|#55FF55|active (running)}} since Tue 2020-06-23 15:36:32 PDT; 10s ago
Main PID: 6295 (bash)
    Tasks: 8 (limit: 4915)
  CGroup: /system.slice/xxnetwork-node.service
          ├─6295 /bin/bash -c /opt/xxnetwork/xxnetwork-wrapper.py --logpath /opt/xxnetw...
          ├─6296 python3 /opt/xxnetwork/xxnetwork-wrapper.py --logpath /opt/xxnetwork/n...
          └─67181 /opt/xxnetwork/bin/xxnetwork-node}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo systemctl status xxnetwork-gateway.service}}
<p>This should result in the following output. Press {{key press|Q}} to exit the output.</p>
{{terminal|skin=noborder|text=
{{font color|#55FF55|●}} xxnetwork-gateway.service - Job that starts the Wrapper service
  Loaded: loaded (/opt/xxnetwork/xxnetwork-gateway.service; enabled; vendor preset: enabled)
  Active: {{font color|#55FF55|active (running)}} since Tue 2020-06-23 15:36:37 PDT; 2min 0s ago
Main PID: 6343 (bash)
    Tasks: 12 (limit: 4915)
  CGroup: /system.slice/xxnetwork-gateway.service
          ├─6343 /bin/bash -c /opt/xxnetwork/xxnetwork-wrapper.py --logpath /opt/xxnetw...
          ├─6344 python3 /opt/xxnetwork/xxnetwork-wrapper.py --logpath /opt/xxnetwork/g...
          └─6418 /opt/xxnetwork/bin/xxnetwork-gateway}}
</li>
<li><p>Next, check that the process is running. If the commands result in the expected output, then the Node or Gateway process is running.</p>
{{terminal|skin=noborder|text='''$''' ps -A | grep xxnetwork}}
<p>This will print a list of running xx network processes. If {{inline-code|xxnetwork-gatew}} appears on the list, then the Gateway process is running. If {{inline-code|xxnetwork-node}} appears on the list, then the Node process is running.</p>
 
{{terminal|skin=noborder|text=
<nowiki> 2433 ?        00:00:17 xxnetwork-gatew
9915 ?        00:00:22 xxnetwork-node</nowiki>}}
{{Note|inline=1|small=1|The PID (the first number printed) and the elapsed time will be different.}}
</li>
</ol>
 
=== Backup Certificates and Important Files ===
 
{{box|{{center|
[[File:Ambox warning pn.svg|50px|frameless]] <big><big><big><big>'''ATTENTION'''</big></big></big></big> [[File:Ambox warning pn.svg|50px|frameless]]
 
 
<big><big>'''Lost Certificates Cannot be Recovered'''</big></big>
}}
<p>Once the Node and Gateway are properly functioning, Node operators must backup some important files. Loss of these files will irreparably disconnect a Node from the network. Follow the instructions on the next page detailing how to do so.</p>|background=#fee7e6|border color=#c33|padding=0.5em}}
 
{{Note|'''Warning:''' The following instructions are extremely important to follow to ensure that your Node and Gateway can stay operating on the network. Failure to follow these instructions can lead to your Node or Gateway being permanently unable to operate on the network.|warn}}
 
{{Note|'''Warning:''' The files described below are sensitive as they identify your Node and Gateway uniquely. Anybody who gains access to these files can appear on the network as your Node or Gateway. It is extremely important to keep these files secure as any third party access can lead to the compromising of your Node and Gateway’s identity.|warn}}
 
There are six files that your Node/Gateway requires to authenticate itself on the network. Loss of these files will lead to your Node or Gateway being unable to participate in the network and regeneration of these files is contingent on the policy outlined above. Therefore, every Node Operator ''must'' backup these files to a secure location not on the same machine. It is recommended that these files be backed up to multiple secure locations. For example, a flash drive or secure cloud storage.
 
==== Backing Up Files ====
 
<ol style="list-style-type: decimal;">
<li><p>Locate the six files that must be backed up. If you used the default paths in the configuration instructions, then these files can be found according to the table below.</p>
{{row numbers|<nowiki>
{| class="wikitable zebra-head"
!
! File
! Default Name
! Default Location
|-
|style="text-align:right;"|_row_count.
| Node’s public certificate
| {{mono|node_cert.crt}}
| {{mono|/opt/xxnetwork/creds/}}
|-
|style="text-align:right;"|_row_count.
| Node’s private key
| {{mono|node_key.key}}
| {{mono|/opt/xxnetwork/creds/}}
|-
|style="text-align:right;"|_row_count.
| Node’s ID file (IDF)
| {{mono|nodeIDF.json}}
| {{mono|/opt/xxnetwork/node-logs/}}
|-
|style="text-align:right;"|_row_count.
| Gateway’s public certificate
| {{mono|gateway_cert.crt}}
| {{mono|/opt/xxnetwork/creds/}}
|-
|style="text-align:right;"|_row_count.
| Gateway’s private key
| {{mono|gateway_key.key}}
| {{mono|/opt/xxnetwork/creds/}}
|-
|style="text-align:right;"|_row_count.
| Gateway’s ID file (IDF)
| {{mono|gatewayIDF.json}}
| {{mono|/opt/xxnetwork/gateway-logs/}}
|}</nowiki>}}
<p>If you cannot locate the files, then their paths can be found in the Node and Gateway YAML files as shown in the table below.</p>
{{row numbers|<nowiki>
{| class="wikitable zebra-head"
!
! File
! YAML File
! Flag
|-
|style="text-align:right;"|_row_count.
| Node’s public certificate
| {{mono|node.yaml}}
| {{mono|node.paths.cert}}
|-
|style="text-align:right;"|_row_count.
| Node’s private key
| {{mono|node.yaml}}
| {{mono|node.paths.key}}
|-
|style="text-align:right;"|_row_count.
| Node’s ID file (IDF)
| {{mono|node.yaml}}
| {{mono|node.paths.idf}}
|-
|style="text-align:right;"|_row_count.
| Gateway’s public certificate
| {{mono|gateway.yaml}}
| {{mono|certPath}}
|-
|style="text-align:right;"|_row_count.
| Gateway’s private key
| {{mono|gateway.yaml}}
| {{mono|keyPath}}
|-
|style="text-align:right;"|_row_count.
| Gateway’s ID file (IDF)
| {{mono|gateway.yaml}}
| {{mono|idfPath}}
|}</nowiki>}}
</li>
<li><p>The easiest method to copy these files off of the machine is to use a USB flash drive; you can use the same one that you may have used to install the operating system. First, insert the flash drive and then run {{inline-code|lsblk}} to find the mount point of the flash drive.</p>
{{terminal|skin=noborder|text='''$''' lsblk}}
<p>This should output a list of mount points on your computer. Locate the name of the flash drive. This can be done by looking at the size and finding which mount point has the same size as your flash drive.</p>
{{terminal|skin=noborder|text=
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop1                      7:1    0    97M  1 loop /snap/core/9665
loop2                      7:2    0  96.6M  1 loop /snap/core/9804
nvme0n1                  259:0    0 232.9G  0 disk
├─nvme0n1p1              259:1    0  512M  0 part /boot/efi
├─nvme0n1p2              259:2    0    1G  0 part /boot
└─nvme0n1p3              259:3    0 231.4G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0 115.7G  0 lvm  /
sdb                        8:16  1  7.2G  0 disk
└─sdb1                      8:18  1    7G  0 part  {{highlight|/media/[username]/[flash drive name]|#595935}}
}}
</li>
<li><p>Copy all six files to the flash drive path found in the previous step. If your files are in the default locations, it will be similar to the following command.</p>
{{terminal|skin=noborder|text='''$''' cp /opt/xxnetwork/creds/{node_cert.crt, node_key.key, gateway_cert.crt, gateway_key.key} /opt/xxnetwork/node-logs/nodeIDF.json /opt/xxnetwork/gateway-logs/gatewayIDF.json /media/[''username'']/[''flash drive name'']}}
<p>To copy one file at a time, use the following command.</p>
{{terminal|skin=noborder|text=$ cp [''path of file to copy''] [''path of flash drive'']}}
</li>
<li><p>Once the files are copied, remove the flash drive, and save it in a secure location. We highly recommend that you back up these files in at least one other location.</p>
<p>Ensure that these files only exist in secure locations. If anyone gains access to these files, your Node and Gateway’s identity on the network can be compromised.</p></li></ol>
 
== Manually Deploying Binaries ==
 
After following the [[#Node_and_Gateway_Set_Up|Node and Gateway Set Up]], it is possible to use alternate binaries than the ones supplied by the Wrapper Script. The source code to compile the binaries can be downloaded from the [https://gitlab.com/elixxir/server/-/blob/release/.gitlab-ci.yml Elixxir GitLab] page.
 
<ol style="list-style-type: decimal;">
<li><p>To disable automatic update via the Wrapper Script, add {{inline-code|--disableupdates}} to the call in the service files located in {{mono|/opt/xxnetwork/xxnetwork-node.service}} for Node and {{mono|/opt/xxnetwork/xxnetwork-gateway.service}} for Gateway.</p></li>
<li><p>Replace the binaries in {{mono|/opt/xxnetwork/bin/}} with the self-compiled ones generated in the following steps. Then restart the service.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' systemctl restart xxnetwork-node.service}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' systemctl restart xxnetwork-gateway.service}}
</li></ol>
=== Compile Binaries ===
 
A second option to manually deploying binaries is to download the source code and compile it yourself. Go version 1.16 is required for both Server and Gateway.
 
# Download and install the latest version of [https://golang.org/dl/#go1.16.4 Go version 1.16]. Refer to the [[#Environment_Set_Up|Environment Set Up section]] for detailed instructions.
 
==== Gateway Binary ====
 
Compiling Gateway is a straightforward process. For additional information, refer to the Gateway [https://gitlab.com/elixxir/gateway/-/blob/release/.gitlab-ci.yml {{mono|.gitlab-ci.yaml}}] file.
 
<ol style="list-style-type: decimal;">
<li><p>Download the latest [https://gitlab.com/elixxir/gateway/-/tree/master master branch] of Gateway.</p></li>
<li><p>Compile Gateway using the following command.</p>
{{terminal|skin=noborder|text='''$''' GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags '-w -s' -o xxnetwork-gateway main.go}}
<p>It is also possible to compile it for other systems, though support cannot be guaranteed.</p>
{{terminal|skin=noborder|text=
'''$''' GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -ldflags '-w -s' -o xxnetwork-gateway.win64 main.go
'''$''' GOOS=windows GOARCH=386 CGO_ENABLED=0 go build -ldflags '-w -s' -o xxnetwork-gateway.win32 main.go
'''$''' GOOS=darwin GOARCH=amd64 CGO_ENABLED=0 go build -ldflags '-w -s' -o xxnetwork-gateway.darwin64 main.go}}
</li>
<li><p>Rename the file to {{mono|xxnetwork-gateway}} and move it to {{mono|/opt/xxnetwork/bin}}. Then restart the service.</p>
{{terminal|skin=noborder|text='''$''' systemctl restart xxnetwork-gateway.service}}
</li></ol>
 
==== Node Binary ====
 
If the Node is expected to run with GPU acceleration, compiling Server requires an extra step to compile. This setup assumes a working Nvidia installation. For additional information, refer to the Server [https://gitlab.com/elixxir/server/-/blob/release/.gitlab-ci.yml {{mono|.gitlab-ci.yaml}}] file.
 
<ol style="list-style-type: decimal;">
<li><p>First, install the CUDA Toolkit 11.2. Nvidia provides a [https://developer.nvidia.com/cuda-10.2-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal list of commands] to run that install all the necessary software onto the machine. Simply run all the commands in the provided order.</p>
{{terminal|skin=noborder|text=
'''$''' wget [https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin {{font color|#BBB|https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin}}]
 
'''$''' sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
 
'''$''' wget [https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda-repo-ubuntu1804-11-2-local_11.2.2-460.32.03-1_amd64.deb {{font color|#BBB|https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda-repo-ubuntu1804-11-2-local_11.2.2-460.32.03-1_amd64.deb}}]
 
'''$''' sudo dpkg -i cuda-repo-ubuntu1804-11-2-local_11.2.2-460.32.03-1_amd64.deb
 
'''$''' sudo apt-key add /var/cuda-repo-ubuntu1804-11-2-local/7fa2af80.pub
 
'''$''' sudo apt-get update
 
'''$''' sudo apt-get -y install cuda}}
{{Note|inline=1|small=1|Note that this process can be slow.}}
</li>
<li><p>Once the installation is complete, reboot the system.</p>
{{terminal|skin=noborder|text='''$''' sudo reboot now}}
</li>
<li><p>Next, install {{inline-code|libgmp-dev}}, which is a dependency the GPU code needs to compile.</p>
{{terminal|skin=noborder|text=
'''$''' sudo apt update
'''$''' sudo apt install -y libgmp-dev cuda}}
</li>
<li><p>Install the latest version of the [https://gitlab.com/elixxir/gpumathsnative gpumathsnative repository]. This is where the underlying CUDA implementations of the mathematical operations live is what allows operations to be done on the GPU.</p>
{{terminal|skin=noborder|text=
'''$''' git clone -b master [https://gitlab.com/elixxir/gpumathsnative.git {{font color|#BBB|https://gitlab.com/elixxir/gpumathsnative.git}}]
 
'''$''' pushd gpumathsnative/cgbnBindings/powm
 
'''$''' make fatbin
 
'''$''' sudo make install
 
'''$''' popd}}
{{Note|inline=1|small=1|Skip this step if the Node is being run without the GPU.}}
</li>
<li><p>Download the latest [https://gitlab.com/elixxir/server/-/tree/master master branch] of Server.</p></li>
<li><p>Finally, compile the Server using the following command.</p>
{{terminal|skin=noborder|text='''$''' GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -tags gpu -ldflags '-w -s -L /opt/xxnetwork/lib' -o xxnetwork-node main.go}}
</li>
<li><p>Name the file {{mono|xxnetwork-node}} and move it to {{mono|/opt/xxnetwork/bin}}. Then restart the service.</p>
{{terminal|skin=noborder|text='''$''' systemctl restart xxnetwork-node.service}}
</li></ol>
 
== Testing ==
 
A majority of the code in the Elixxir codebase is tested. Node operators can run the tests on any [https://gitlab.com/elixxir public repository] themselves using the instructions in this section.
 
=== Environment Set Up ===
 
Prior to running any tests, set up a development environment with Go 1.16 and some additional software.
 
{{Note|inline=1|These instructions describe how to run tests in Ubuntu Server 18.04. However, almost all tests run successfully on Windows, macOS, and other Linux distributions.}}
 
<ol style="list-style-type: decimal;">
<li><p>First, install the {{mono|build-essential}} package, which contains a number of tools for compiling binaries. This step is only necessary if running tests on Server with the GPU.</p>
{{terminal|skin=noborder|text=
'''$''' sudo apt-get update
'''$''' sudo apt-get install build-essential}}
{{Note|inline=1|small=1|If the GPU drivers were previously installed, {{mono|build-essential}} may be already installed too.}}
</li>
<li><p>Download the latest version of [https://golang.org/dl/#go1.16.4 Go version 1.16]. Refer to the [https://golang.org/doc/install Go install page] for more information.</p>
{{terminal|skin=noborder|text='''$''' curl -L -O [https://golang.org/dl/go1.16.4.linux-amd64.tar.gz {{font color|#BBB|https://golang.org/dl/go1.16.4.linux-amd64.tar.gz}}]}}
</li>
<li><p>Extract the archive into {{mono|/usr/local}}.</p>
{{terminal|skin=noborder|text='''$''' sudo tar -C /usr/local -xzf go1.16.4.linux-amd64.tar.gz}}
</li>
<li><p>The {{mono|.profile}} file for the current user must be modified to include {{mono|/usr/local/go/bin}} in the {{mono|PATH}} environment variable. This can be done manually by opening the file and modifying it. Alternatively, use the following command to automatically append the correct line to the {{mono|.profile}} file.</p>
{{terminal|skin=noborder|text='''$''' echo 'export PATH=$PATH:/usr/local/go/bin' &gt;&gt;~/.profile}}
</li>
<li><p>Once the file is saved, to apply the changes immediately, run the following command.</p>
{{terminal|skin=noborder|text='''$''' source ~/.profile}}
</li>
<li><p>To ensure that everything is working and that Go is the correct version, run the following command.</p>
{{terminal|skin=noborder|text='''$''' go version}}
<p>It should result in a similar output to this:</p>
{{terminal|skin=noborder|text=go version go1.16.4 linux/amd64}}
</li>
<li><p>Finally, create a directory to place the Go source code that will be downloaded in the following sections. Any name can be chosen.</p>
{{terminal|skin=noborder|text='''$''' mkdir ~/dev}}
</li></ol>
 
==== Setting Up Node for Testing ====
 
To run some of the tests on Node, a GPU is required and the [https://gitlab.com/elixxir/gpumathsnative gpumathsnative] repository must be downloaded. This is what allows operations to be done on the GPU and is where the underlying CUDA implementations of the mathematical operations live.
 
<ol style="list-style-type: decimal;">
<li><p>First, ensure that the [[#GPU_Drivers|GPU Drivers]] are set up.</p></li>
<li><p>Next, install the CUDA Toolkit 11.2. Nvidia provides a [https://developer.nvidia.com/cuda-10.2-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal list of commands] to run that install all the necessary software onto the machine. Simply run all the commands in the provided order.</p>
{{terminal|skin=noborder|text=
'''$''' wget [https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin {{font color|#BBB|https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin}}]
 
'''$''' sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
 
'''$''' wget [https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda-repo-ubuntu1804-11-2-local_11.2.2-460.32.03-1_amd64.deb {{font color|#BBB|https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda-repo-ubuntu1804-11-2-local_11.2.2-460.32.03-1_amd64.deb}}]
 
'''$''' sudo dpkg -i cuda-repo-ubuntu1804-11-2-local_11.2.2-460.32.03-1_amd64.deb
 
'''$''' sudo apt-key add /var/cuda-repo-ubuntu1804-11-2-local/7fa2af80.pub
 
'''$''' sudo apt-get update
 
'''$''' sudo apt-get -y install cuda}}
{{Note|inline=1|small=1|Note that this process can be slow.}}
</li>
<li><p>Once the installation is complete, reboot the system.</p>
{{terminal|skin=noborder|text='''$''' sudo reboot now}}
</li>
<li><p>Install {{inline-code|libgmp-de}}, a dependency for the GPU code to run.</p>
{{terminal|skin=noborder|text=
'''$''' sudo apt update
'''$''' sudo apt install -y libgmp-dev}}
</li>
<li><p>Download the latest master version of the gpumathsnative repository.</p>
{{terminal|skin=noborder|text='''$''' git clone -b master [https://gitlab.com/elixxir/gpumathsnative.git {{font color|#BBB|https://gitlab.com/elixxir/gpumathsnative.git}}]}}
</li>
<li><p>Run the following commands to go to the correct directory and execute the make command.</p>
{{terminal|skin=noborder|text=
'''$''' pushd gpumathsnative/cgbnBindings/powm
 
'''$''' make turing
 
'''$''' sudo make install
 
'''$''' popd}}
</li>
<li><p>When running tests with GPU on Node, include the flag {{inline-code|-tags gpu}}.</p>
{{terminal|skin=noborder|text='''$''' go test ./... -tags gpu}}
</li></ol>
 
=== Running Tests ===
 
Running tests in Go for most Elixxir repositories is a straightforward process. Server has an extra step that is described [[#Setting_Up_Node_for_Testing|below]]. To run a test, simply download a repository and use go test.
 
<ol style="list-style-type: decimal;">
<li id="Running_Tests_step01"><p>Go to the [https://gitlab.com/elixxir Elixxir] or [https://gitlab.com/xx_network xx network] GitLab page and select a repository to download. Alternatively, select one from the following list of public repositories:</p>
{{Div col|colwidth=30ch}}
* {{mono|elixxir/primitives}}
* {{mono|elixxir/crypto}}
* {{mono|elixxir/comms}}
* {{mono|elixxir/gpumathsgo}}
* {{mono|elixxir/server}}
* {{mono|elixxir/gateway}}
* {{mono|elixxir/client}}
* {{mono|elixxir/user-discovery-bot}}
* {{mono|elixxir/LocalEnvironment}}
* {{mono|elixxir/integration}}
* {{mono|elixxir/wrapper}}
* {{mono|xx_network/primitives}}
* {{mono|xx_network/crypto}}
* {{mono|xx_network/comms}}
{{Div col end}}
</li>
<li><p>Use {{inline-code|lang=none|git clone}} or any other method to download the repository.</p>
{{terminal|skin=noborder|text='''$''' git clone -b master <nowiki>https://gitlab.com/elixxir/</nowiki>{{highlight|[repository name from [[#Running_Tests_step01|{{font color|#BBB|step 1}}]]]|#595935}}.git}}
<p>In the following steps, Primitives will be used as an example.</p>
{{terminal|skin=noborder|text='''$''' git clone -b master [https://gitlab.com/elixxir/primitives.git {{font color|#BBB|https://gitlab.com/elixxir/primitives.git}}]}}
</li>
<li><p>Change directories to the root of the downloaded repository.</p>
{{terminal|skin=noborder|text='''$''' cd primitives/}}
</li>
<li><p>To compile and run all the tests for this repository, use the following command.</p>
{{terminal|skin=noborder|text='''$''' go test ./...}}
{{Note|inline=1|small=1|The first time the code is compiled, all of the dependencies will download first.}}
<p>Running the tests can take a while, but once they are finished, a summary of the tests and their elapsed time will be printed out. If any tests fail, those errors will be printed too.</p>
 
{{terminal|skin=noborder|text=
ok      gitlab.com/elixxir/primitives/current  0.016s
ok      gitlab.com/elixxir/primitives/format    0.025s
ok      gitlab.com/elixxir/primitives/id        0.023s
ok      gitlab.com/elixxir/primitives/id/idf    0.134s
ok      gitlab.com/elixxir/primitives/ndf      0.031s
ok      gitlab.com/elixxir/primitives/rateLimiting      113.118s
ok      gitlab.com/elixxir/primitives/ring      0.020s
ok      gitlab.com/elixxir/primitives/states    0.017s
ok      gitlab.com/elixxir/primitives/switchboard      4.344s
ok      gitlab.com/elixxir/primitives/utils    0.093s
ok      gitlab.com/elixxir/primitives/version  0.008s}}
</li>
<li><p>To run tests located in a specific directory, include the directory name. Also, including the {{inline-code|-v}} flag will print more details about which test is being run and the debug logs from it.</p>
{{terminal|skin=noborder|text='''$''' go test -v ./utils}}
<p>The result will be a list of the tests run and their elapsed time. At the bottom is the total time for all the tests.</p>
{{terminal|skin=noborder|text=
==== RUN  TestUnmarshal_DataLengthError
--- PASS: TestUnmarshal_DataLengthError (0.00s)
=== RUN  TestType_String
--- PASS: TestType_String (0.00s)
=== RUN  TestType_String_Error
--- PASS: TestType_String_Error (0.00s)
PASS
ok      gitlab.com/elixxir/primitives/id        0.065s}}
</li>
<li><p>To run only one test or multiple tests matching a regular expression, use the {{inline-code|-run}} flag.</p>
{{terminal|skin=noborder|text='''$''' go test -run {{highlight|[test name or regex]|#595935}}}}
</li></ol>
 
== Appendix ==
 
=== Paths and Files ===
 
There are a number of file paths that need to be specified for a Node or Gateway to run. Some of these paths point to existing files, others point to where a file should be saved. Many of these are modified through the Node and Gateway configuration files, which must be explicitly specified when running the software.
 
Below is an example directory structure if the xx network software was installed following the instructions in this manual. The files that are required at startup are marked with a red asterisk ({{font color|#F00|✱}}) and the files that are generated are marked with a blue dagger ({{font color|#00F|'''†'''}}).
 
{| class="infobox" style="text-align:left;"
|+ style="font-weight:bold;" | Legend
|-
| {{font color|#F00|✱}} || file required at startup
|- style="padding-top:1em;"
| {{font color|#00F|'''†'''}} || file generated
|}
 
{{Tree list}}
* {{mono|/opt/xxnetwork/}}
** {{mono|bin}}
*** {{mono|xxnetwork-gateway}}
*** {{mono|xxnetwork-node}}
** {{mono|cmdlod}}
*** {{mono|command_1591809548.jsonl}}{{font color|#00F|'''†'''}}
*** {{mono|command_1591856646.jsonl}}{{font color|#00F|'''†'''}}
*** {{mono|version_1591166323.jsonl}}{{font color|#00F|'''†'''}}
** {{mono|creds}}
*** {{mono|gateway_cert.crt}}{{font color|#F00|✱}}
*** {{mono|gateway_key.key}}{{font color|#F00|✱}}
*** {{mono|network_management.crt}}{{font color|#F00|✱}}
*** {{mono|node_cert.crt}}{{font color|#F00|✱}}
*** {{mono|node_key.key}}{{font color|#F00|✱}}
*** {{mono|permissioning_cert.crt}}{{font color|#F00|✱}}
** {{mono|gateway-logs}}
*** {{mono|gateway.log}}{{font color|#00F|'''†'''}}
*** {{mono|gatewayIDF.json}}{{font color|#00F|'''†'''}}
*** {{mono|xxnetwork-wrapper.log}}{{font color|#00F|'''†'''}}
** {{mono|gateway.yaml}}{{font color|#F00|✱}}
** {{mono|generate_certs.py}}
** {{mono|lib}}
*** {{mono|libpow.fatbin}}{{font color|#F00|✱}}
*** {{mono|libpowmosm75.so}}{{font color|#F00|✱}}
** {{mono|node-logs}}
*** {{mono|metrics.log}}{{font color|#00F|'''†'''}}
*** {{mono|node-err.log}}{{font color|#00F|'''†'''}}
*** {{mono|node.log}}{{font color|#00F|'''†'''}}
*** {{mono|nodeIDF.json}}{{font color|#00F|'''†'''}}
*** {{mono|xxnetwork-wrapper.log}}{{font color|#00F|'''†'''}}
** {{mono|node.yaml}}{{font color|#F00|✱}}
** {{mono|xxnetwork-gateway.service}}
** {{mono|xxnetwork-node.service}}
** {{mono|xxnetwork-wrapper.py}}
{{Tree list/end}}
 
==== Node Paths and Files ====
 
The Node requires four files to run:
 
# The Node private key
# The Node TLS certificate
# The Gateway TLS certificate
# The Permissioning TLS certificate
 
{{code|lang=yaml|
node:
  paths:
    # Path to the self-signed TLS certificate for Node. Expects PEM format. (Required)
    cert: "/opt/xxnetwork/creds/node_cert.crt"
    # Path to the private key associated with the self-signed TLS certificate. (Required)
    key: "/opt/xxnetwork/creds/node_key.key"}}
{{code|lang=yaml|
gateways:
  paths:
    # Path to the self-signed TLS certificate for Gateway. Expects PEM format. (Required)
    cert: "/opt/xxnetwork/creds/gateway_cert.crt"
 
permissioning:
  paths:
    # Path to the self-signed TLS certificate for the Permissioning server. Expects PEM
    # format. (Required)
    cert: "/opt/xxnetwork/creds/permissioning_cert.crt"}}
 
There are three different paths for logs: the normal Node log, the error log, and the metrics logs as well as one for the Node IDF file.
 
{{code|lang=yaml|
node:
  paths:
    # Path where an error file will be placed in the event of a fatal error. This path is
    # used by the Wrapper Script. (Required)
    errOutput: "/opt/xxnetwork/node-logs/node-err.log"
    # Path to where the identity file (IDF) is saved. The IDF stores the Node's network
    # identity. This is used by the wrapper management script. (Required)
    idf: "/opt/xxnetwork/node-logs/nodeIDF.json"
    # Path where log file will be saved.
    log: "/opt/xxnetwork/node-logs/node.log"}}
{{code|lang=yaml|
metrics:
  # Path to store metrics logs.
  log: "/opt/xxnetowkr/server-logs/metrics.log"}}
 
==== Gateway Paths and Files ====
 
To run, Gateway requires four files:
 
# The Gateway private key
# The Gateway TLS certificate
# The Node TLS certificate
# The Permissioning TLS certificate
 
These appear in {{inline-code|gateway.yaml}} as follows.
 
{{code|lang=yaml|
# Path to the private key associated with the self-signed TLS certificate. (Required)
keyPath: "/opt/xxnetwork/creds/gateway_key.key"
 
# Path to the self-signed TLS certificate for Gateway. Expects PEM format. (Required)
certPath: "/opt/xxnetwork/creds/gateway_cert.crt"
 
# Path to the self-signed TLS certificate for Node. Expects PEM format. (Required)
serverCertPath: "/opt/xxnetwork/creds/node_cert.crt"
 
# Path to the self-signed TLS certificate for the Permissioning server. Expects PEM format.
# (Required)
permissioningCertPath: "/opt/xxnetwork/creds/permissioning_cert.crt
}}
 
In addition, Gateway has an optional IDF path field and an optional log field. If no paths are supplied, defaults will be used. When the Gateway runs, it will create files at these paths.
 
{{code|lang=yaml|
# Path where log file will be saved. (Default "./gateway-logs/gateway.log")
log: "/opt/xxnetwork/gateway-logs/gateway.log"
 
# Path to where the identity file (IDF) is saved. The IDF stores the Gateway's Node's
# network identity. This is used by the wrapper management script. (Required)
idfPath: "/opt/xxnetwork/gateway-logs/gatewayIDF.json"}}
 
=== Network Definition File (NDF) ===
 
The NDF is a JSON file with a predefined structure that matches the internal {{mono|NetworkDefinition}} structure.
 
Some objects on the NDF must have data that matches predefined formats. These are outlined below.
 
* {{mono|'''Id:'''}} must be a byte array 33 bytes long that matches the {{mono|id.ID}} object. IDs must be generated using Crypto and cannot be created any other way.
* {{mono|'''Tls_certificate:'''}} must be a TLS certificate in PEM format. All new lines should be replaced with Unix escape sequence {{inline-code|\n}}.
<br />
{| class="codetable"
| style="width:50%;" |{{Codesample|lang=json|line=1|code=
{
"Timestamp": "YYYY-MM-DDTHH:MM:SS.0000000+00:00",}}
| <p>Timestamp in [https://tools.ietf.org/html/rfc3339 RFC3339] format.</p>
|-
| {{Codesample|lang=json|line=1|start=3|code=
<nowiki> </nowiki>"Gateways": [
{
"Id": "dGVzdDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB",
"Address": "0.0.0.0:80000",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},
],}}
| <p>Array of {{mono|Gateway}} objects in the network. Each Gateway has an ID, an address string (containing the IP address and port), and a TLS certificate.</p>
|-
| {{Codesample|lang=json|line=1|start=12|code=
<nowiki> </nowiki>"Nodes": [
{
"Id": "dGVzdDMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC",
"Address": "0.0.0.1:40000",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},
],}}
| <p>Array of {{mono|Node}} objects in the network. Each Node has an ID, address, and TLS certificate in the same format as the Gateway object.</p>
|-
| {{Codesample|lang=json|line=1|start=21|code=
<nowiki> </nowiki>"Registration": {
"Address": "0.0.0.3:18000",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},}}
| <p>The {{inline-code|Registration}} field has information about the Permissioning server.</p>
|-
| {{Codesample|lang=json|line=1|start=25|code=
<nowiki> </nowiki>"Notification": {
"Address": "0.0.0.7",
"Tls_certificate": "-----BEGIN CERTIFICATE-----..."
},}}
| <p>The {{inline-code|Notification}} field has information about Notification Bot.</p>
|-
| {{Codesample|lang=json|line=1|start=29|code=
<nowiki> </nowiki>"Udb": {
"Id": "dGVzdDYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD"
"Cert": "-----BEGIN CERTIFICATE-----...",
"Address": "0.0.0.52:18001",
"DhPubKey": "eyJWYWx1ZSI6NTEwOTQzODAxNTcyOTQ4NjUzN...",
},}}
| <p>The {{inline-code|Udb}} field contains the ID, certificate, address, and Diffie–Hellman key of User Discovery</p>
|-
| {{Codesample|lang=json|line=1|start=35|code=
<nowiki> </nowiki>"E2e": {
"Prime": "E2EE983D031DC1DB6F1A7A67DF0E9A8E5561DB8E...",
"Generator": "2"
},
"Cmix": {
"Prime": "F6FAC7E480EE519354C0F856AEBDC43AD6014...",
"Generator": "2"
}
<nowiki>}</nowiki>}}
| <p>Both the {{inline-code|E2e}} and {{inline-code|Cmix}} fields define the cyclic groups that messaging and end to end encryption operate within. The {{inline-code|E2e}} group is based on a 3192-bit strong and safe prime and the {{inline-code|Cmix}} group is a 2048 strong and safe prime.</p>
|}
 
=== Bandwidth Limiting ===
 
The cMix protocol uses bandwidth in unique ways. It has short bursts (10~100 milliseconds) of medium-high usage and extended periods (250 milliseconds to 5 seconds) of minimal usage. Overall, at the most taxing configuration testing shows it will saturate 20% to 30% of a 100 Mbps connection.
 
That 20% to 30% figure is dependent on network conditions and other Nodes it is operating with, but in rare circumstances, it may be 20% to 30% of whatever bandwidth is available. If a Node operator has greater than 100 Mbps available and does not want to allow the network to consume beyond the stated requirements, they can limit bandwidth utilization on the Node.
 
To do this, [https://linux.die.net/man/8/tc-htb Linux traffic control ({{mono|tc}}) and Hierarchy Token Bucket (HTB)], similar to QoS settings on a router, can be used to limit the bandwidth of the Node and/or Gateway to the desired amount. Configuration requires three pieces of information: (1) the port number(s) of the Node and/or Gateway, (2) the network interface name, and (3) the desired bandwidth cap. The following instructions will describe how to get these values and how to configure traffic control.
 
<ol style="list-style-type: decimal;">
<li><p>First, get the port number. By default, the port for Node is {{mono|11420}} and for Gateway it is {{mono|22840}}. If the port numbers were changed during set up, they can be found in the following locations.</p>
<ol style="list-style-type: lower-alpha;">
<li><p>'''Node:''' The default location is {{mono|/opt/xxnetwork/node.yaml}}. The port is under {{inline-code|node}} &gt; {{inline-code|port}}.</p></li>
<li><p>'''Gateway:''' The default location is {{mono|/opt/xxnetwork/gateway.yaml}}. The port is under {{inline-code|port}}.</p></li></ol>
</li>
<li><p>Next, determine the network interface that the machine is operating on. For most machines, the following command should print the currently used network interface.</p>
{{terminal|skin=noborder|text='''$''' ip addr show | awk '/inet.*brd/{print $NF; exit}'}}
<p>Alternatively, manually find the correct network interface by printing out all available interfaces.</p>
{{terminal|skin=noborder|text='''$''' ip addr}}
<p>This should result in a similar output to below. Find the name of the network interface that is currently being used. It will have the correct local IP address.</p>
{{terminal|skin=noborder|text=
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: {{highlight|enp1s0|#595935}}: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:f9:b7:83 brd ff:ff:ff:ff:ff:ff
inet <u>192.168.127.138</u>/24 brd 192.168.127.255 scope global dynamic enp1s0
valid_lft 1574sec preferred_lft 1574sec
inet6 fe80::20c:29ff:fef9:b783/64 scope link
valid_lft forever preferred_lft forever
}}
</li>
<li><p>Add the qdisc that will limit the bandwidth for the network interface found.</p>
{{terminal|skin=noborder|text='''$''' sudo tc qdisc add dev {{highlight|[interface name]|#595935}} root handle 1: htb default 30}}
</li>
<li><p>Set the max bandwidth for either Node or Gateway. The bandwidth must be set to at least 100 Mbit if running a Node or Gateway. If running them together, Node must be set to a minimum of 100 Mbit and Gateway a minimum of 50 Mbit.</p>
{{Note|'''Warning:''' Creating a bandwidth limit lower than specified can result in underperforming and eventual removal from the network.|warn}}
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo tc class add dev {{highlight|[interface name]|#595935}} parent 1:1 classid '''1:10''' htb rate {{highlight|100Mbit|#595935}}}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo tc class add dev {{highlight|[interface name]|#595935}} parent 1:1 classid '''1:20''' htb rate {{highlight|50Mbit|#595935}}}}
</li>
<li><p>Set the port number that the bandwidth will be limited on. Make sure to use the correct port found in the previous step.</p>
<p>'''For Node:'''</p>
{{terminal|skin=noborder|text='''$''' sudo tc filter add dev {{highlight|[interface name]|#595935}} parent 1:0 protocol ip prio 1 u32 match ip dport {{highlight|[Node port]|#595935}} 0xffff flowid '''1:10'''}}
<p>'''For Gateway:'''</p>
{{terminal|skin=noborder|text='''$''' sudo tc filter add dev {{highlight|[interface name]|#595935}} parent 1:0 protocol ip prio 1 u32 match ip dport {{highlight|[Gateway port]|#595935}} 0xffff flowid '''1:20'''}}
</li>
<li><p>To check that the rules were set correctly, list all the classes.</p>
{{terminal|skin=noborder|text='''$''' tc class show dev {{highlight|[interface name]|#595935}}}}
<p>This should print a similar output to below.</p>
{{terminal|skin=noborder|text=
class htb 1:10 root prio rate 100Mbit ceil 100Mbit burst 15 cburst 15
class htb 1:20 root prio rate 50Mbit ceil 50Mbit burst 15 cburst 15}}
</li>
</ol>
 
To stop limiting the bandwidth, the rule can be deleted using the following command.
{{terminal|skin=noborder|text='''$''' sudo tc qdisc del dev {{highlight|[interface name]|#595935}} root}}

Revision as of 17:02, 15 October 2021

This manual is designed to provide the necessary information to deploy, debug, and operate the cMix, Gateway, and xx chain software on the xx network. A Node in the network will participate in the two fundamental tasks that comprise the xx network: mixing communication through cMix protocol and executing consensus through xx consensus.

Refer to the sections below for getting started with the cMix, Gateway, and xx chain software.

Getting Started

First-time Node operators must set up their hardware, download and install the xx network software, and stake their Node.

  1. Follow instructions for Operating System Installation and Configuration on the Node machine
  2. Follow instructions for Operating System Installation and Configuration on the Gateway machine
  3. Do Node Set Up on the Node machine
  4. Do Gateway Set Up on the Gateway machine
  5. Follow instructions for Staking a Node

More Information

To learn more about the xx network, Elixxir, and Praxxis, refer to the following papers.

For more information, further discussion, and additional help with this guide or general participation in the BetaNet, use the following contacts.

Updates

7/26/2021
<translate> Maintenance update</translate> Updated Hardware Requirements for MainNet.
6/24/2021
<translate> Maintenance update</translate> Updated Wrapper Script Arguments for consensus.
6/22/2021
<translate> Maintenance update</translate> The complete Node Handbook has been added to the xx network wiki.

More news