MultiversX keys monitor

This tool allows the monitoring of BLS keys that can participate into the consensus, regardless the shard they are currently operating. This is done by continuously polling the /validator/statistics API endpoint route. It can monitor one or more networks so one instance of this tool is enough for all keys used on mainnet/testnet or devnet networks. The monitored BLS keys will be defined as lists. Also, there is the possibility to define just an owner address that staked the BLS keys and the application will automatically fetch the registered BLS keys on that identity.

Feature list

• Keys monitor
  • Monitor any number of networks (mainnet/testnet/devnet) with one instance
  • Monitor any number of BLS keys defined in separate files
  • Automatically fetch the BLS keys staked by an address
  • Threshold definition on each set for the allowed rating drop
  • Configurable polling time for each definition set
• Notification system
  • Integrated the Pushover service that allows easy mobile device notifications
  • Integrated the SMTP email service to notify thorough emails the events encountered
  • Integrated the Telegram bot notification service
• System self-check messages
  • Integrated a self-check system that can send messages periodical messages about the status of the app
• Scripts & installation support
  • Added scripts for easy setup & upgrade

Installation

Initial setup

Although is possible, it is not recommended to run the application as root. For that reason, a new user is required to be created. For example, the following script creates a new user called ubuntu. This script can be run as root.

# host update/upgrade
apt-get update
apt-get upgrade
apt autoremove

adduser ubuntu
# set a long password
usermod -aG sudo ubuntu
echo 'StrictHostKeyChecking=no' >> /etc/ssh/ssh_config

visudo   
# add this line:
ubuntu  ALL=(ALL) NOPASSWD:ALL
# save & exit

sudo su ubuntu
sudo visudo -f /etc/sudoers.d/myOverrides
# add this line:
ubuntu  ALL=(ALL) NOPASSWD:ALL
# save & exit

Repo clone & scripts init

cd
git clone https://github.com/iulianpascalau/mvx-keys-monitor
cd ~/mvx-keys-monitor/scripts
# the following init call will create ~/mvx-keys-monitor/scripts/config/local.cfg file
# and will copy the configs from ~/mvx-keys-monitor/cmd/monitor/config/example to ~/mvx-keys-monitor/cmd/monitor/config
# to avoid github pull problems
scripts.sh init
cd config
# edit the local.cfg file for the scripts setup
nano local.cfg

local.cfg configuration

The generated local.cfg file contains the following lines:

#!/bin/bash
set -e

CUSTOM_HOME=/home/ubuntu
CUSTOM_USER=ubuntu
GITHUBTOKEN=""
MONITOR_EXTRA_FLAGS=""

#Allow user to override the current version of the monitor
OVERRIDE_VER=""

The CUSTOM_HOME and CUSTOM_USER will need to be changed if the current user is not ubuntu. To easily figure out the current user, the bash command whoami can be used.

It is strongly recommended to use a GitHub access token because the scripts consumes the GitHub APIs and throttling might occur without the access token.

The MONITOR_EXTRA_FLAGS can contain extra flags to be called whenever the application is started. The complete list of the cli command can be found here

The OVERRIDE_VER can be used during testing to manually specify an override tag/branch that will be used when building the application. If left empty, the upgrade process will automatically fetch and use the latest release.

Install

After the local.cfg configuration step, the scripts can now install the application.

cd ~/mvx-keys-monitor/scripts
./script.sh install

Application config

After the application has been installed, it is now time to configure the application. For this, you should edit the config.toml and credentials.toml files and add files containing lists of BLS keys or addresses in the ~/mvx-keys-monitor/cmd/monitor/config directory.

The scripts init step already created some initial .toml and .list files to be ready to be used directly.

Configuring the config.toml file:

This file contains the general application configuration file.

[General]
    ApplicationName = "Keys monitoring app"
    # the application can send messages about the internal status at regular intervals
    [General.SystemSelfCheck]
        Enabled = true
        DayOfWeek = "every day" # can be "every day", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday" and "Sunday"
        Hour = 12 # valid interval 0-23
        Minute = 00 # valid interval 0-59
        PollingIntervalInSec = 30
    [General.Logs]
        LogFileLifeSpanInMB = 1024 # 1GB
        LogFileLifeSpanInSec = 86400 # 1 day


[OutputNotifiers]
    # Uses Pushover service that can notify Desktop, Android or iOS devices. Requires a valid subscription.
    # If you enable this notifier, remember to specify the credentials in credentials.toml file
    [OutputNotifiers.Pushover]
        Enabled = false
        URL = "https://api.pushover.net/1/messages.json"

    # SMTP (email) based notification
    # If you enable this notifier, remember to specify the credentials in credentials.toml file
    # If you are using gmail server, please make sure you activate the IMAP server and use App passwords instead of the account's password
    [OutputNotifiers.Smtp]
        Enabled = false
        To = "to@email.com"
        SmtpPort = 587
        SmtpHost = "smtp.gmail.com"

    # Uses Telegram service that can notify Desktop, Android or iOS devices. Requires a running bot and the chat ID for
    # the user that will be notified.
    # If you enable this notifier, remember to specify the credentials in credentials.toml file
    [OutputNotifiers.Telegram]
        Enabled = false
        URL = "https://api.telegram.org"

[[BLSKeysMonitoring]]
    AlarmDeltaRatingDrop = 1.0 # maximum Rating-TempRating value that will trigger an alarm, for the public testnet might use a higher value (2 or 3)
    Name = "network 1"
    ApiURL = "API URL 1"
    ExplorerURL = ""
    PollingIntervalInSeconds = 300  # 5 minutes
    ListFile = "./config/network1.list"

• The General section

  • The ApplicationName can be any string, it will be used in the notification messages.

  • The General.SystemSelfCheck section can enable & configure the self check monitoring system that will output a notification message once a day or week, depending on the configuration parameters.

  • The General.Logs will configure the internal logging system (for debugging purposes).

• The OutputNotifiers is the section containing the implemented notifiers. There are 3 types of notifiers implemented: Pushover, Smtp and Telegram, each with its configuration sections. The credentials for the notifiers are defined separately in the credentials.toml so that is the place where passwords or access tokens will be specified.

• The BLSKeysMonitoring defines the section used on one network.

  The config.toml file accepts any number of this type of sections.

  • The AlarmDeltaRatingDrop option will specify how large the difference between the "Epoch start rating" - "Current rating" is allowed before emitting an alert. Usually, a value of 1 will suffice in most cases. On the public testnet, this value might be increased to 2 or even 3 to reduce the number of false alarms due to the nature of the other nodes operating in the network.

  • The Name defines a string for the monitored network. Can be something like Mainnet, Testnet, Devnet or any kind of identification string.

  • The ApiURL defines the API url for that network. Examples here include https://api.multiversx.com for the mainnet, https://testnet-api.multiversx.com for the testnet, and https://devnet-api.multiversx.com for the devnet.

  • The ExplorerURL is used whenever a BLS key alert message is emitted, to automatically include the link to that BLS key page. If left empty, the message will still be emitted, but it will not contain any link. Examples here include https://explorer.multiversx.com for the mainnet, https://testnet-explorer.multiversx.com for the testnet, and https://devnet-explorer.multiversx.com for the devnet.

  • The PollingIntervalInSeconds represents the time in seconds between the calls on the API URL.

  • The ListFile will contain the name of the file containing BLS or identity keys. Refer to the example file called network1.list to check how keys/identities can be defined.

Application start

After editing the required config files, the application can be started.

cd ~/mvx-keys-monitor/scripts
./script.sh start

When the application starts, it automatically emits a notification message. This is valid also for the stop operation issued by the ./script.sh stop command.

Backup and upgrade

It is a good practice to save the .toml, .list and the local.cfg files somewhere else just in case the application is cleaned up accidentally. The upgrade call for the monitor app is done through this command:

cd ~/mvx-keys-monitor/scripts
./script.sh upgrade