Troubleshooting

Trust Anchor fails to load in Veriscope WebApp

In some cases after a new install, port 8080 is disabled which in turn impacts the Veriscope web application. To enable port 8080, SSH into your Veriscope Server web instance and run the following commands:

sudo apt install net-tools
netstat -tulpn // check active ports. Is port 8080 listed? If not, run the following
sudo systemctl restart ta-node-1 // wait 1-2 mins and run the command above again

If your 8080 port is still disabled, run the following command to log command to output more information from your server:

sudo journalctl -f -u ta-node-1

Is your Veriscope Node Synced?

During installation, Veriscope needs to download & install historical data. There are two elements to this:

  • Relay Node synchronization with the Shyft Network

  • "Event" synchronization with transactions on the Shyft Network and their corresponding emitted events.

These processes are sequential and the Relay (Nethermind) Node must be synchronized before the event data is downloaded and synced. The event data comes from the Node itself.

The expected duration for both processes to complete is between 4-6 hours (assuming 2 CPU cores, 16 GB RAM)

Relay Node (Nethermind) synchronization

Nodes in green are fully synchronized.

If you can't see your Node at either of the domains above and you haven't hidden it (i.e. set `ethstats` to false in your Nethermind config file), SSH into your machine and restart all services. Run the `setup-vasp.sh` script and select `w` to restart services.

Additionally, you can run the following command to check your Node is online and is syncing:

[source,bash]

systemctl status nethermind

=== "Event" synchronization

Events in this context are:

* Verified Trust Anchors,
* Discovery Layer Key Value Pairs, and
* Attestations

Downloaded events can be seen in your Veriscope application in the Backoffice section.

[NOTE]

If you suspect some events are missing, SSH into your machine and restart all services. Run the setup-vasp.sh script and select w to restart services. This will initiate a re-sync.

Additionally, you can run the following commands to check the event sync (aka eth-sync) and are running as expected

systemctl status ta-node-1
journalctl -f -u ta-node-1

Transactions failing due to Out of gas

If you find some of your transactions are failing on mainnet, please check that you’re not posting duplicate attestations. For more on duplicate attestations, see Duplicate attestations.

Duplicate attestations

Each attestation must be unique; thus, if you have a single customer who wishes to withdraw a coin/token to the same destination adddress multiple times, each attestation must be different. It is recommended to add a transaction id or timestamp to the memo to achieve this.

Webhook events not getting triggered

The Veriscope state-machines expect to receive a 200 http response code whenever a message is sent over http to another VASP / Veriscope client. As such, it is necessary for each VASP to respond with a 200 code <i>before</i> taking further action e.g. responding with an updated kyc-template.

Postman collection isn’t working

Please check the following

  • Log in to your Vericope web-application and create an API token

  • In Postman, import the Veriscope Postman collection

  • Create a sample Environment with the following variables:

    • baseUrl (with no trailing /)

    • token (i.e. the api token mentioned above)

    • address (your testnet trust anchor address)

    • Save the above!

  • Navigate back to the Collection, and select the environment you just created. Click save.

OR_DATA_REQ webhook event is received twice

In this instance, it’s likely that your Nethermind node isn’t fully synchronized. To resolve this, just wait awhile longer :) You can check the status of your node via the Horizon dashboard that’s accessible through you Veriscope web-application.

Can I run mutliple trust anchors on a single Veriscope instance?

No, not yet! We expect to deliver this feature in December 2022.

Node Installation

1) Has your trust anchor account been set in veriscope_ta_node/.env?

Trust anchor accounts are managed by the http-api.js script which loads it from .env. Ensure you have your trust anchor account (TRUST_ANCHOR_ACCOUNT) and private key (TRUST_ANCHOR_PK) set. If you plan to manage multiple accounts for testing purposes, you can swap these accounts in the .env file and restart ta-node-1 as follows:

sudo systemctl restart ta-node-1
Example 1. veriscope_ta_node/.env file
#From v3.6.0, Multiple Trust Anchors can be added as illustrated
#DO NOT INCLUDE "0x" prefix in TRUST_ANCHOR_PK
TRUST_ANCHOR_PREFNAME="Singapore VASP", "ta1", "ta2"
TRUST_ANCHOR_ACCOUNT=0x...15, 0x...25, 0x...35
TRUST_ANCHOR_PK=aa...10, ab...20, ac...30
WEBHOOK_CLIENT_SECRET=abcdefg*

HTTP="http://localhost:8545"
WS="ws://localhost:8545"
WEBHOOK="http://localhost:8000/webhook"
HTTP_API_PORT=8080
TEMPLATE_HELPER_PORT=8090

CONTRACTS=/opt/veriscope/veriscope_ta_node/artifacts/

#Veriscope Smart Contracts
TRUST_ANCHOR_MANAGER_CONTRACT_ADDRESS="0x........06"
TRUST_ANCHOR_STORAGE_CONTRACT_ADDRESS="0x........66"
TRUST_ANCHOR_EXTRA_DATA_GENERIC_CONTRACT_ADDRESS="0x.......4c"
TRUST_ANCHOR_EXTRA_DATA_UNIQUE_CONTRACT_ADDRESS="0x........a8"

#redis
REDIS_URI=redis://127.0.0.1:6379
LOG_LEVEL=info

SIGN_MESSAGE="VERISCOPE"
As mentioned in the example, do not include "0x" prefix in TRUST_ANCHOR_PK value(s)!

2) Is your trust anchor account loaded in the web-app the same as in veriscope_ta_node/.env?

When running Step 2 of the installation script (Nethermind), the trust anchor account in .env is overwritten. By refreshing the Load TA Account in the web-app, your trust anchor account in the veriscope_ta_node/.env will be loaded in the web-app.

3) Is Nethermind running? Is your Nethermind Node in Stats? And has it completed syncing?

In order to receive blockchain events or post transactions, your Nethermind client must be running. You can confirm this by viewing the relevant stats page to see if your Node is up and synchronized: * stats.veriscope.network (Testnet) * stats.shyft.network (Mainnet)

And by running the following command:

sudo systemctl status nethermind
 ● nethermind.service - Nethermind Ethereum Daemon
      Loaded: loaded (/etc/systemd/system/nethermind.service; disabled; vendor p>
      Active: active (running) since Thu 2021-11-25 21:25:19 UTC; 1 weeks 0 days>
    Main PID: 1419555 (Nethermind.Runn)
       Tasks: 72 (limit: 4631)
      Memory: 1.2G
      CGroup: /system.slice/nethermind.service
              └─1419555 /opt/nm/Nethermind.Runner -c /opt/nm/config.cfg

 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]: eth_blockNumber          >
 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]: eth_chainId              >
 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]: eth_getLogs              >
 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]: ------------------------->
 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]: TOTAL                    >
 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]: ------------------------->
 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]:
 Dec 03 19:59:50 pcf Nethermind.Runner[1419555]: 2021-12-03 19:59:50.0323|>
 Dec 03 19:59:54 pcf Nethermind.Runner[1419555]: 2021-12-03 19:59:54.0361|>
 Dec 03 19:59:54 pcf Nethermind.Runner[1419555]: 2021-12-03 19:59:54.0361|>
 Dec 03 19:59:58 pcf Nethermind.Runner[1419555]: 2021-12-03 19:59:58.0401|>
 Dec 03 19:59:58 pcf Nethermind.Runner[1419555]: 2021-

The service should show as active.

To confirm your web-app, http-api and (synched) Nethermind Node are all connected, you should be able to fetch account balance in the web-app.

4) How can I check Nethermind’s liveness?

In the Nethermind directory you can find config.cfg where there are a number of custom configurations you can set. For example Health checks.

 $ pwd
 /opt/nm
 $ tree -L 1
 .
 ├── Data
 ├── NLog.config
 ├── Nethermind.Cli
 ├── Nethermind.Launcher
 ├── Nethermind.Runner
 ├── VaspTestnet.json
 ├── config.cfg
 ├── git-hash
 ├── keystore
 ├── logs
 ├── nethermind_db
 ├── plugins
 └── static-nodes.json

Open the config.cfg and add the following:

  "HealthChecks": {
    "Enabled": true,
    "WebhooksEnabled": true,
    "UIEnabled": true,
    "Slug": "/api/health",
    "MaxIntervalWithoutProcessedBlock ": 15,
    "MaxIntervalWithoutProducedBlock": 45
  }
The Slug parameter /api/health

Restart Nethermind and try the following. It should show as status Healthy.

 $ curl localhost:8545/api/health
 {"status":"Healthy","totalDuration":"00:00:00.0034157","entries":{"node-health":{"data":{},"description":"The node is now fully synced with a network. Peers: 3.","duration":"00:00:00.0030673","status":"Healthy","tags":[]}}}

For more information, see Nethermind’s documentation on Node Health.

5) Logging

Both the web-app and NodeJS logs to files in the following directories:

  • webapp: /opt/veriscope/veriscope_ta_dashboard/storage/logs

$ pwd
/opt/veriscope/veriscope_ta_dashboard/storage/logs
$ ls
laravel.log

  • nodejs: /opt/veriscope/veriscope_ta_node/logs

$ pwd
/opt/veriscope/veriscope_ta_node/logs
$ ls
blockchain-data.combined.log  http-api.error.log
blockchain-data.error.log     shyft-template-helper.combined.log
http-api.combined.log         shyft-template-helper.error.log

6) 503 when loading blockchain data

If when loading any of the blockchain data like so

$ node -e 'require("./blockchain-data").getAllAttestations()'

…​you may notice 503 errors in the logs. 503 is returned when the WEBHOOK_CLIENT_SECRET in both .env files may not be set. If you need to recreate the secret you can do so via the Install Script, step 10.

10) Regenerate webhook secret

This will overwrite WEBHOOK_CLIENT_SECRET in both .env files for veriscope_ta_dashboard and veriscope_ta_node.

Postman Collection

Why I can’t get any response?

You haven’t imported the environment file or you’ve imported it but haven’t selected it from the dropdown menu.

How can I debug a request or find the used URL?

Open the Postman’s console to find requests' parameters and URL.

Where can I learn more about InterVASP Messaging Standard 101 (IVMS101)

IVMS101 (the interVASP Messaging Standard) is an internationally recognized standard that helps with language encodings, numeric identification systems, phonetic name pronunciations, and standardized country codes (ISO 3166). For general information about IVMS, please visit InterVASP.org.