System Considerations

The Keep Network expects certain capabilities for each node running on the network. To help attain these capabilities consider the following criteria:

  • It is paramount that Keep nodes remain available to the Keep Network. We strongly encourage a stable and redundant internet connection. See Network section.

  • A connection to a production grade self-hosted or third party Ethereum node deployment. See Ethereum API section.

  • Persistent and redundant storage that will survive a VM or container rotation, and a disk failure. See Persistance section.

  • Each node running on the network requires a unique Ethereum Operator Account. The account has to maintain a positive Ether balance at all times. See Operator Account section.

  • Each node running on the network requires a unique IP address or a unique application port running under the same IP. See Network section.

Your operating environment will ultimately dictate what machine type to go with. This is particularly relevant if you’re running a containerized solution where multiple applications are sharing VM resources. The below types are sufficient for running at least one instance of the Keep Node.

Cloud Provider Machine Type

Google Cloud

n2-highcpu-2

AWS

c5.large

Azure

F2s v2

Self-hosted

2 vCPU / 2 GiB RAM / 1 GiB Persistent Storage

Starting a Client

Configuration

The client expects configuration options to be specified in a config file or passed as CLI flags. If you specify an option by using a parameter on the command line, it will override the value read from the configuration file.

Operator Account

The client requires an Ethereum Key File of an Operator Account to connect to the Ethereum chain.

The Ethereum Key File is expected to be encrypted with a password. The password has to be provided in a prompt after the client starts or configured as a KEEP_ETHEREUM_PASSWORD environment variable.

The Operator Account has to maintain a positive Ether balance at all times. We strongly advice you monitor the account and top-up when its balance gets below 0,5 Ether.

Ethereum API

A Keep Node requires a connection to a WebSocket Ethereum API. You should obtain a WS API URL from a service provider (e.g. Alchemy, Infura) or run your own Ethereum node (e.g. Geth).

CLI Options

$ keep-client start --help
Starts the Keep Client in the foreground

Usage:
  keep-client start [flags]

Flags:
      --ethereum.url string                        WS connection URL for Ethereum client.
      --ethereum.keyFile string                    The local filesystem path to Keep operator account keyfile.
      --ethereum.miningCheckInterval duration      The time interval in seconds in which transaction mining status is checked. If the transaction is not mined within this time, the gas price is increased and transaction is resubmitted. (default 1m0s)
      --ethereum.maxGasFeeCap wei                  The maximum gas fee the client is willing to pay for the transaction to be mined. If reached, no resubmission attempts are performed. (default 500 gwei)
      --ethereum.requestPerSecondLimit int         Request per second limit for all types of Ethereum client requests. (default 150)
      --ethereum.concurrencyLimit int              The maximum number of concurrent requests which can be executed against Ethereum client. (default 30)
      --ethereum.balanceAlertThreshold wei         The minimum balance of operator account below which client starts reporting errors in logs. (default 500000000 gwei)
      --network.bootstrap                          Run the client in bootstrap mode.
      --network.peers strings                      Addresses of the network bootstrap nodes.
  -p, --network.port int                           Keep client listening port. (default 3919)
      --network.announcedAddresses strings         Overwrites the default Keep client address announced in the network. Should be used for NAT or when more advanced firewall rules are applied.
      --network.disseminationTime int              Specifies courtesy message dissemination time in seconds for topics the node is not subscribed to. Should be used only on selected bootstrap nodes. (0 = none)
      --storage.dir string                         Location to store the Keep client key shares and other sensitive data.
      --clientInfo.port int                        Client Info HTTP server listening port. (default 9601)
      --clientInfo.networkMetricsTick duration     Client Info network metrics check tick in seconds. (default 1m0s)
      --clientInfo.ethereumMetricsTick duration    Client info Ethereum metrics check tick in seconds. (default 10m0s)
      --tbtc.preParamsPoolSize int                 tECDSA pre-parameters pool size. (default 1000)
      --tbtc.preParamsGenerationTimeout duration   tECDSA pre-parameters generation timeout. (default 2m0s)
      --tbtc.preParamsGenerationDelay duration     tECDSA pre-parameters generation delay. (default 10s)
      --tbtc.preParamsGenerationConcurrency int    tECDSA pre-parameters generation concurrency. (default 1)
      --tbtc.keyGenerationConcurrency int          tECDSA key generation concurrency. (default number of cores)
      --developer.bridgeAddress string             Address of the Bridge smart contract
      --developer.randomBeaconAddress string       Address of the RandomBeacon smart contract
      --developer.tokenStakingAddress string       Address of the TokenStaking smart contract
      --developer.walletRegistryAddress string     Address of the WalletRegistry smart contract

Global Flags:
  -c, --config string   Path to the configuration file. Supported formats: TOML, YAML, JSON.
      --developer       Developer network
      --goerli          Görli network
      --mainnet         Mainnet network

Environment variables:
    KEEP_ETHEREUM_PASSWORD    Password for Keep operator account keyfile decryption.
    LOG_LEVEL                 Space-delimited set of log level directives; set to "help" for help.

Config File

Application configuration can be stored in a file and passed to the application with the --config flag.

Example:

./keep-client --config /path/to/your/config.toml start

Configuration files in formats TOML, YAML and JSON are supported.

Sample configuration file:

# This is a sample TOML configuration file for the Keep client.

[ethereum]
URL = "ws://127.0.0.1:8546"
KeyFile = "/Users/someuser/ethereum/data/keystore/UTC--2018-03-11T01-37-33.202765887Z--AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA8AAAAAAAAA"

# Uncomment to override the defaults for transaction status monitoring.

# MiningCheckInterval is the interval in which transaction
# mining status is checked. If the transaction is not mined within this
# time, the gas price is increased and transaction is resubmitted.
#
# MiningCheckInterval = 60  # 60 sec (default value)

# MaxGasFeeCap specifies the maximum gas fee cap the client is
# willing to pay for the transaction to be mined. The offered transaction
# gas cost can not be higher than the max gas fee cap value. If the maximum
# allowed gas fee cap is reached, no further resubmission attempts are
# performed. This property should be set only for Ethereum. In case of
# legacy non-EIP-1559 transactions, this field works in the same way as
# `MaxGasPrice` property.
#
# MaxGasFeeCap = "500 Gwei" # 500 Gwei (default value)

# Uncomment to enable Ethereum node rate limiting. Both properties can be
# used together or separately.
#
# RequestsPerSecondLimit sets the maximum average number of requests
# per second which can be executed against the Ethereum node.
# All types of Ethereum node requests are rate-limited,
# including view function calls.
#
# RequestsPerSecondLimit = 150

# ConcurrencyLimit sets the maximum number of concurrent requests which
# can be executed against the Ethereum node at the same time.
# This limit affects all types of Ethereum node requests,
# including view function calls.
#
# ConcurrencyLimit = 30

# BalanceAlertThreshold defines a minimum value of the operator's account
# balance below which the client will start reporting errors in logs.
# A value can be provided in `wei`, `Gwei` or `ether`, e.g. `7.5 ether`,
# `7500000000 Gwei`.
#
# BalanceAlertThreshold = "0.5 ether" # 0.5 ether (default value)

[network]
Bootstrap = false
Peers = [
	"/ip4/127.0.0.1/tcp/3919/ipfs/16Uiu2HAmFRJtCWfdXhZEZHWb4tUpH1QMMgzH1oiamCfUuK6NgqWX",
]
Port = 3920

# Uncomment to override the node's default addresses announced in the network
# AnnouncedAddresses = ["/dns4/example.com/tcp/3919", "/ip4/80.70.60.50/tcp/3919"]

# Uncomment to enable courtesy message dissemination for topics this node is
# not subscribed to. Messages will be forwarded to peers for the duration
# specified as a value in seconds.
# Message dissemination is disabled by default and should be enabled only
# on selected bootstrap nodes. It is not a good idea to enable dissemination
# on non-bootstrap node as it may clutter communication and eventually lead
# to blacklisting the node. The maximum allowed value is 90 seconds.
#
# DisseminationTime = 90

[storage]
Dir = "/my/secure/location"

# ClientInfo exposes metrics and diagnostics modules.
# 
# Metrics collects and exposes information useful for external monitoring tools usually
# operating on time series data.
# All values exposed by metrics module are quantifiable or countable.
#
# The following metrics are available:
# - connected peers count
# - connected bootstraps count
# - eth client connectivity status
# 
# Diagnostics module exposes the following information:
# - list of connected peers along with their network id and ethereum operator address
# - information about the client's network id and ethereum operator address
[clientInfo]
Port = 9601
# NetworkMetricsTick = 60
# EthereumMetricsTick = 600

# Uncomment to overwrite default values for TBTC config.
#
# [tbtc]
# PreParamsPoolSize = 3000
# PreParamsGenerationTimeout = "2m"
# PreParamsGenerationDelay = "10s"
# PreParamsGenerationConcurrency = 1
# KeyGenConcurrency = 1

# Developer options to work with locally deployed contracts
#
# [developer]
# TokenStakingAddress = "0xBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
# RandomBeaconAddress = "0xBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
# WalletRegistryAddress = "0xBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
# BridgeAddress = "0xBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

Persistance

A client will produce and store data on disk. The directory should be provided to the client under storage.Dir (flag: --storage.dir) configuration property.

It is crucial to ensure the data directory is persisted and backed up on a regular basis.

There will be two subdirectories created in the storage directory: - keystore, - work.

keystore

The keystore subdirectory contains sensitive key material data generated by the client. Loosing the keystore data is a serious protocol offense and leads to slashing and potentially losing funds.

It is the operator’s responsibility to ensure the keystore data are not lost under any circumstances.
work

The work directory contains data generated by the client that should persist the client restarts or relocations. If the work data are lost the client will be able to recreate them, but it is inconvenient due to the time needed for the operation to complete and may lead to losing rewards.

Network

The node has to be accessible publicly to establish and maintain connections with bootstrap nodes and discovered peers.

The node exposes metrics and diagnostics services for monitoring.

Ports

A Network Port has to be exposed publicly, so the peers can connect to your node. A Diagnostics Port has to be exposed publicly, for the Rewards Allocation.

Please update your firewall rules if necessary.
Name Config Property Direction Type Default

Network

network.port

Egress/Ingress

TCP

3919

Client Info

clientInfo.port

Egress

TCP

9601

Announced Addresses

An Announced Address is a layered addressing information (multiaddress/multiaddr) announced to the Keep Network that is used by peers to connect with your node, e.g.: /dns4/bootstrap-0.test.keep.network/tcp/3919 or /ip4/104.154.61.116/tcp/3919.

If the machine you’re running your node is not exposing a public IP (e.g. it is behind NAT) you should set the network.AnnouncedAddresses (flag: --network.announcedAddresses) configuration property to an addresses (ip4 or dns4) under which your node is reachable for the public.

To read more about multiaddress see the libp2p docummentation.

Minimum Required Configuration

The minimum required configuration for the client to start covers setting:

Installation

Build from Source

See our developer docs.

Run

Sample commands to run the Keep Client.

Instead of passing the configuration properties as command arguments you can point to a configuration file with --config flag. See Config File section for details.

Binary

export KEEP_ETHEREUM_PASSWORD="<Ethereum Key File Password>"

keep-client \
    start \
    --ethereum.url "<Ethereum API WS URL>" \
    --ethereum.keyFile "<Operator Key File Path>" \
    --storage.dir "<Storage Data Directory>"

Docker

ETHEREUM_WS_URL="<Ethereum API WS URL>"

OPERATOR_KEY_FILE_NAME="<Operator Key File Name>"
OPERATOR_KEY_FILE_PASSWORD="<Ethereum Key File Password>"

CONFIG_DIR=$(pwd)/config
STORAGE_DIR=$(pwd)/storage

docker run --detach \
    --volume $CONFIG_DIR:/mnt/keep/config \
    --volume $STORAGE_DIR:/mnt/keep/storage \
    --env KEEP_ETHEREUM_PASSWORD=$OPERATOR_KEY_FILE_PASSWORD \
    --env LOG_LEVEL=debug \
    --log-opt max-size=100m \
    --log-opt max-file=3 \
    -p 3919:3919 \
    -p 9601:9601 \
    keepnetwork/keep-client:latest \
    start \
    --ethereum.url $ETHEREUM_WS_URL \
    --ethereum.keyFile /mnt/keep/config/$OPERATOR_KEY_FILE_NAME \
    --storage.dir /mnt/keep/storage

Logging

Configuration

Logging can be configured with environment variables. Please see sample settings:

LOG_LEVEL=DEBUG
IPFS_LOGGING_FMT=nocolor
GOLOG_FILE=/var/log/keep/keep.log
GOLOG_TRACING_FILE=/var/log/keep/trace.json

Startup

Below are some of the key things to look out for to make sure you’re booted and connected to the network:

▓▓▌ ▓▓ ▐▓▓ ▓▓▓▓▓▓▓▓▓▓▌▐▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▄
▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓▌▐▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
  ▓▓▓▓▓▓    ▓▓▓▓▓▓▓▀    ▐▓▓▓▓▓▓    ▐▓▓▓▓▓   ▓▓▓▓▓▓     ▓▓▓▓▓   ▐▓▓▓▓▓▌   ▐▓▓▓▓▓▓
  ▓▓▓▓▓▓▄▄▓▓▓▓▓▓▓▀      ▐▓▓▓▓▓▓▄▄▄▄         ▓▓▓▓▓▓▄▄▄▄         ▐▓▓▓▓▓▌   ▐▓▓▓▓▓▓
  ▓▓▓▓▓▓▓▓▓▓▓▓▓▀        ▐▓▓▓▓▓▓▓▓▓▓         ▓▓▓▓▓▓▓▓▓▓▌        ▐▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
  ▓▓▓▓▓▓▀▀▓▓▓▓▓▓▄       ▐▓▓▓▓▓▓▀▀▀▀         ▓▓▓▓▓▓▀▀▀▀         ▐▓▓▓▓▓▓▓▓▓▓▓▓▓▓▀
  ▓▓▓▓▓▓   ▀▓▓▓▓▓▓▄     ▐▓▓▓▓▓▓     ▓▓▓▓▓   ▓▓▓▓▓▓     ▓▓▓▓▓   ▐▓▓▓▓▓▌
▓▓▓▓▓▓▓▓▓▓ █▓▓▓▓▓▓▓▓▓ ▐▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  ▓▓▓▓▓▓▓▓▓▓
▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓ ▐▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  ▓▓▓▓▓▓▓▓▓▓

Trust math, not hardware.

------------------------------------------------------------------------------------------------
| Keep Client Node                                                                             |
|                                                                                              |
| Operator: 0x6299496199d99941193Fdd2d717ef585F431eA05                                         |
|                                                                                              |
| Port: 3919                                                                                   |
| IPs : /ip4/127.0.0.1/tcp/3919/ipfs/16Uiu2HAmM2WfE4uwymj9n1EsaN5jL38cG79jCCSXLX49KoL18aZQ     |
|       /ip6/::1/tcp/3919/ipfs/16Uiu2HAmM2WfE4uwymj9n1EsaN5jL38cG79jCCSXLX49KoL18aZQ           |
|       /ip4/192.168.1.167/tcp/3919/ipfs/16Uiu2HAmM2WfE4uwymj9n1EsaN5jL38cG79jCCSXLX49KoL18aZQ |
|       /ip4/10.2.0.2/tcp/3919/ipfs/16Uiu2HAmM2WfE4uwymj9n1EsaN5jL38cG79jCCSXLX49KoL18aZQ      |
|                                                                                              |
| Contracts:                                                                                   |
| RandomBeacon   : 0x2bA82903B635a96154A515488d2952E86D6adc3A                                  |
| WalletRegistry : 0x2363cc10b7680000C02E4a7067A68d1788ffc86F                                  |
| TokenStaking   : 0x69f962a0fbA5635e84eC94131f9072108E2E4F24                                  |
------------------------------------------------------------------------------------------------
If you want to share your LibP2P address with others you can get it from the startup log. When sharing remember to substitute the /ipv4/ address with the public facing IP of your client if you’re running on a private machine, or replace the entire /ipv4/ segment with a DNS entry if you’re using a hostname.

Client Info

The client exposes metrics and diagnostics on a configurable port (default: 9601) under /metrics and /diagnostics resources.

The data can be consumed by Prometheus to monitor the state of a node.

Metrics

The client exposes the following metrics:

  • connected peers count,

  • connected bootstraps count,

  • Ethereum client connectivity status (if a simple read-only CALL can be executed).

Metrics are enabled once the client starts. It is possible to customize the port at which metrics endpoint is exposed as well as the frequency with which the metrics are collected.

Exposed metrics contain the value and timestamp at which they were collected.

Example metrics endpoint call result:

$ curl localhost:9601/metrics
# TYPE connected_peers_count gauge
connected_peers_count 108 1623235129569

# TYPE connected_bootstrap_count gauge
connected_bootstrap_count 10 1623235129569

# TYPE eth_connectivity gauge
eth_connectivity 1 1623235129789

Diagnostics

The client exposes the following diagnostics:

  • list of connected peers along with their network id and Ethereum operator address,

  • information about the client’s network id and Ethereum operator address.

Diagnostics are enabled once the client starts. It is possible to customize the port at which diagnostics endpoint is exposed.

Example diagnostics endpoint call result:

$ curl localhost:9601/diagnostics
{
  "client_info" {
   "ethereum_address":"0xDcd4199e22d09248cA2583cBDD2759b2acD22381",
   "network_id":"16Uiu2HAkzYFHsqbwt64ZztWWK1hyeLntRNqWMYFiZjaKu1PZgikN"
  },
  "connected_peers": [
    {"ethereum_address":"0x3712C6fED51CECA83cA953f6FF3458f2339436b4","network_id":"16Uiu2HAkyYtzNoWuF3ULaA7RMfVAxvfQQ9YRvRT3TK4tXmuZtaWi"},
    {"ethereum_address":"0x4bFa10B1538E8E765E995688D8EEc39C717B6797","network_id":"16Uiu2HAm9d4MG4LNrwkFmugD2pX7frm6ZmA4vE3EFAEjk7yaoeLd"},
    {"ethereum_address":"0x650A9eD18Df873cad98C88dcaC8170531cAD2399","network_id":"16Uiu2HAkvjVWogUk2gq6VTNLQdFoSHXYpobJdZyuAYeoWD66e8BD"},
    ...
  ]
}

Testnet

The Keep Network Testnet environment is running against Ethereum Görli Testnet. To run the client connected to the Testnet add --goerli flag to the start command.

A Docker image for Testnet is published under: us-docker.pkg.dev/keep-test-f3e0/public/keep-client:latest
See Docker section for details.

Bootstrap Nodes

A client running on testnet establishes connection to the following bootstrap nodes:

/dns4/bootstrap-0.test.keep.network/tcp/3919/ipfs/16Uiu2HAmCcfVpHwfBKNFbQuhvGuFXHVLQ65gB4sJm7HyrcZuLttH
/dns4/bootstrap-1.test.keep.network/tcp/3919/ipfs/16Uiu2HAm3eJtyFKAttzJ85NLMromHuRg4yyum3CREMf6CHBBV6KY
/dns4/bst-a01.test.keep.boar.network/tcp/4001/ipfs/16Uiu2HAmMosdpAuRSw1ahNhqFq8e3Y4d4c5WZkjW1FGQi5WJwWZ7
/dns4/bootstrap-alpha.test.threshold.p2p.org/tcp/3919/ipfs/16Uiu2HAky2Y4Tyq5vTA1CxikcDes6o5EH11i2qcg5dBV9W3Lks5c
/dns4/keep-validator-0.eks-ap-northeast-2-secure.staging.staked.cloud/tcp/3919/ipfs/16Uiu2HAm77eSvRq5ioD4J8VFPkq3bJHBEHkssCuiFkgAoABwjo2S

Contracts

To get adresses of contracts on Testnet please see the link.

Installation

Docker

To get the Docker image run one of the following commands:

  • Latest: us-docker.pkg.dev/keep-test-f3e0/public/keep-client

  • Tagged: us-docker.pkg.dev/keep-test-f3e0/public/keep-client:<tag-version>

Run

Docker

This is a sample run command for illustration purposes only:

ETHEREUM_WS_URL="<Ethereum API WS URL>"

OPERATOR_KEY_FILE_NAME="<Operator Key File Name>"
OPERATOR_KEY_FILE_PASSWORD="<Ethereum Key File Password>"

CONFIG_DIR=$(pwd)/config
STORAGE_DIR=$(pwd)/storage

docker run --detach \
    --volume $CONFIG_DIR:/mnt/keep/config \
    --volume $STORAGE_DIR:/mnt/keep/storage \
    --env KEEP_ETHEREUM_PASSWORD=$OPERATOR_KEY_FILE_PASSWORD \
    --env LOG_LEVEL=debug \
    --log-opt max-size=100m \
    --log-opt max-file=3 \
    -p 3919:3919 \
    -p 9601:9601 \
    us-docker.pkg.dev/keep-test-f3e0/public/keep-client:latest \
    start \
    --goerli \
    --ethereum.url $ETHEREUM_WS_URL \
    --ethereum.keyFile /mnt/keep/config/$OPERATOR_KEY_FILE_NAME \
    --storage.dir /mnt/keep/storage