Troubleshooting

Veriscope Status Checks ("Smoke Screen")

The Veriscope Status Checks ("Smoke Screen") was introduced in v4.2.0. The table below provides a brief description of the checks in place and provides an excellent overview of all of the components and services that make up Veriscope.

Instructions on how to troubleshoot and fix checks that have failed can be found below. You can use the side-menu on the right hand side to navigate to a particular check.

Check label Description

addressproofs

Is the Address Proofs module installed?

contract-addresses

Do all contract addresses exist on the network? Connects to RPC-client to check network and compares/matches values against local ABIs.

database

Is database can be connected?

disk-space

Is there sufficient storage on the file systems?

horizon

Is horizon up and running?

http-api

Is 8080 port enabled?

http-api-url

Is HTTP-API_URL set?

internal-webhook-secret

Does the internal webhook key match what’s in the database? (node/.env and dashboard/.env)

internal-webhook-url

Is the internal webhook reachable? Any webhook/server errors?

internet-connectivity

Does the node have internet access?

nethermind

Is the Nethermind url set? (in veriscope_ta_node/.env file)

nethermind-peer

Is the Shyft Network blockchain communicating with peers?

nethermind-sync

Is the Shyft Network blockchain at the tip of the chain?

queue

Is the Veriscope event queue up and running?

redis

Is Redis up and running?

slack-webhook-config-url

Is the #slack webhook working?

ssl

Is the SSL cert valid?

trust-anchor-balance

Does each trust-anchor account have a non-zero balance?

trust-anchor-count

Does the number of trust-anchor accounts in the dashboard database match that in the node .env file?

trust-anchor-extra-data-unique

Is the API_URL set in the Laravel database?

trust-anchor-key

Can each trust-anchor public key be derived from its private key?

trust-anchor-verified

Is each trust-anchor account verified? (in the Laravel database)

webhook-config-secret

Does the dashboard webhook have a secret/password? A password is required if the webhook value is !null.

webhook-config-url

Is the dashboard webook set?

Troubleshooting Guide

addressproofs

Please install addressproof in your Veriscope Server.

For more details please refer to our Address Proofs Guide

contract-addresses

Please contact Veriscope to discuss.

database

Please check your database info is corrected in dashboard/.env.

DB_CONNECTION=pgsql
DB_HOST=localhost
DB_PORT=5432
DB_DATABASE=yourDataBase
DB_USERNAME=yourUserName
DB_PASSWORD=yourPWD

disk-space

Provision disk space greater than 1GB.

horizon

SSH into your Veriscope Server and run:

sudo systemctl restart horizon

To learn more about Horizon, refer to WebHooks and Laravel Horizon Setup

http-api

In some cases after a new install, port 8080 is disabled which in turn impacts the Veriscope WebApp. To enable port 8080, SSH into your Veriscope Server web instance and run the following commands:

sudo apt install net-tools
netstat -tulpn # check active ports. Is port 8080 listed? If not, run the following
sudo systemctl restart ta-node-1 # wait 1-2 mins and run the command above again

If your 8080 port is still disabled, run the following command to log command the output for more information from your server:

sudo journalctl -f -u ta-node-1

http-api-url

The HTTP_API_URL parameter should be set to localhost:8080 in dashboard/.env.

internal-webhook-secret

Does the internal webhook key match what’s in the database? (Check node/.env and dashboard/.env.)

internal-webhook-url

The WEBHOOK parameter should be set to localhost:8000/webhook in node/.env.

internet-connectivity

Your Versicope Server must be connected to the internet to receive blockchain events and message from other VASPs. To check, SSH into your machine and run

ping www.google.com

nethermind

You can check your Node status at (Nodes in green are at the tip of the chain and are synchronized):

Alternatively, you can SSH into your server and run:

sudo systemctl status nethermind

If Nethermind is not running, restart:

sudo systemctl restart nethermind

To see the logs:

sudo journalctl -f -u nethermind

If the problem persists, check that the HTTP parameter is set to localhost:8545 in veriscope_ta_node/.env.

nethermind-peer

Number of peers should be greater than 3.

If there’s problem persists, please check that the HTTP parameter is set to localhost:8545 in veriscope_ta_node/.env.

nethermind-sync

During installation, Veriscope needs to download & install historical data. There are two elements to this:

  • Relay Node synchronization with the Shyft Network

  • "Event" synchronization with transactions on the Shyft Network and their corresponding emitted events.

These processes are sequential and the Relay (Nethermind) Node must be synchronized before the event data is downloaded and synced. The event data comes from the Node itself. The expected duration for both processes to complete is between 4-6 hours (assuming 2 CPU cores, 16 GB RAM)

If there’s problem persists, please check that the HTTP parameter is set to localhost:8545 in veriscope_ta_node/.env.

Relay Node (Nethermind) synchronization

Nodes in green are fully synchronized.

You can also check by running the following command:

sudo systemctl status nethermind

If you can’t see your Node at either of the domains above and you haven’t hidden it (i.e. set ethstats to false in your Nethermind config file), SSH into your machine and restart all services. Run the setup-vasp.sh script and select w to restart services.

Additionally, you can run the following command to check your Node is online and is syncing:

sudo systemctl status nethermind

To confirm your web-app, http-api and (synched) Nethermind Node are all connected, you should be able to fetch account balance in the Veriscope WebApp.

"Event" synchronization

Events in this context are:

  • Verified Trust Anchors,

  • Discovery Layer Key Value Pairs, and

  • Attestations

Downloaded events can be seen in your Veriscope application in the Backoffice section.

If you suspect some events are missing, SSH into your machine and restart all services. Run the setup-vasp.sh script and select w to restart services. This will initiate a re-sync.

Additionally, you can run the following commands to check the event sync (aka eth-sync) and are running as expected

sudo systemctl status ta-node-1
sudo journalctl -f -u ta-node-1

queue

First check your queue status.

sudo systemctl status ta-queue

If not running please use command below.

sudo systemctl restart ta-queue

If your queue is still disabled, run the following command to log command the output for more information from your server:

sudo journalctl -f -u ta-queue

redis

First check your redis status.

sudo systemctl status redis

If not running please use command below.

sudo systemctl restart redis

If your redis is still disabled, run the following command to log command the output for more information from your server:

sudo journalctl -f -u redis

slack-webhook-config-url

Please install Incoming Webhooks on Slack and copy the url into Slack & Webhook Url page.

Incoming Webhooks

Alt text

Setting Page

Alt text

ssl

Please check if your domain is registered with SSL.

trust-anchor-balance

Please make sure all your trust anchor accounts are verified and have the correct format in veriscope_ta_node/.env.

trust-anchor-count

Please make sure all your trust anchor accounts have the correct format in veriscope_ta_node/.env.

Each time your update the veriscope_ta_node/.env you need to SSH into your Veriscope Server web instance and run the following commands:

sudo systemctl restart ta-node-1

trust-anchor-extra-data-unique

Please make sure all your trust anchor accounts have the API_URL setting and have the correct format in veriscope_ta_node/.env.

Please input domain/kyc-template as the API_URL.

e.g. https://veriscope.com/kyc-template

API_URL setting

Alt text

trust-anchor-key

Please make sure all your trust anchor accounts have the correct privateKey in veriscope_ta_node/.env.

trust-anchor-verified

Please make sure all your trust anchor accounts are verified and have the correct format in veriscope_ta_node/.env.

webhook-config-secret

Please make sure your webhook secret have the correct setting in setting page.

Alt text

webhook-config-url

Please make sure your webhook URL have the correct setting in setting page.

Alt text

Troubleshooting Guide - FAQ

Transactions failing due to Out of gas

If you find some of your transactions are failing on mainnet, please check that you’re not posting duplicate attestations. For more on duplicate attestations, see Duplicate attestations.

Duplicate attestations

Each attestation must be unique; thus, if you have a single customer who wishes to withdraw a coin/token to the same destination adddress multiple times, each attestation must be different. It is recommended to add a transaction id or timestamp to the memo to achieve this.

Webhook events not getting triggered

The Veriscope state-machines expect to receive a 200 http response code whenever a message is sent over http to another VASP / Veriscope client. As such, it is necessary for each VASP to respond with a 200 code <i>before</i> taking further action e.g. responding with an updated kyc-template.

Postman collection isn’t working

Please check the Postman Collection checklist:

  • Log in to your Vericope web-application and create an API token

  • In Postman, import the Veriscope Postman collection

  • Create a sample Environment with the following variables:

  • baseUrl (with no trailing /)

  • token (i.e. the api token mentioned above)

  • address (your testnet trust anchor address)

  • Save the above!

  • Navigate back to the Collection, and select the environment you just created. Click save.

Why I can’t get any response?

You haven’t imported the environment file or you’ve imported it but haven’t selected it from the dropdown menu.

How can I debug a request or find the used URL?

Open the Postman’s console to find requests' parameters and URL.

OR_DATA_REQ webhook event is received twice

In this instance, it’s likely that your Nethermind node isn’t fully synchronized. To resolve this, just wait awhile longer :) You can check the status of your node via the Horizon dashboard that’s accessible through you Veriscope web-application.

How can I check Nethermind’s liveness?

In the Nethermind directory you can find config.cfg where there are a number of custom configurations you can set. For example Health checks.

 $ pwd
 /opt/nm
 $ tree -L 1
 .
 ├── Data
 ├── NLog.config
 ├── Nethermind.Cli
 ├── Nethermind.Launcher
 ├── Nethermind.Runner
 ├── VaspTestnet.json
 ├── config.cfg
 ├── git-hash
 ├── keystore
 ├── logs
 ├── nethermind_db
 ├── plugins
 └── static-nodes.json

Open the config.cfg and add the following:

  "HealthChecks": {
    "Enabled": true,
    "WebhooksEnabled": true,
    "UIEnabled": true,
    "Slug": "/api/health",
    "MaxIntervalWithoutProcessedBlock ": 15,
    "MaxIntervalWithoutProducedBlock": 45
  }
The Slug parameter /api/health

Restart Nethermind and try the following. It should show as status Healthy.

 $ curl localhost:8545/api/health

For more information, see Nethermind’s documentation on Node Health.

Logging

Both the web-app and NodeJS logs to files in the following directories:

webapp: /opt/veriscope/veriscope_ta_dashboard/storage/logs

pwd
/opt/veriscope/veriscope_ta_dashboard/storage/logs
ls
laravel.log

nodejs: /opt/veriscope/veriscope_ta_node/logs

pwd
/opt/veriscope/veriscope_ta_node/logs
ls
blockchain-data.combined.log http-api.error.log
blockchain-data.error.log shyft-template-helper.combined.log
http-api.combined.log shyft-template-helper.error.log

Where can I learn more about InterVASP Messaging Standard 101 (IVMS101)

IVMS101 (the interVASP Messaging Standard) is an internationally recognized standard that helps with language encodings, numeric identification systems, phonetic name pronunciations, and standardized country codes (ISO 3166). For general information about IVMS, please visit InterVASP.org.