Provision a Veriscope Server in AWS using Terraform

a) Create a backend.<env>.tfvars file for appropriate environment (test, dev, staging, prod)

From the root of the repository (~/veriscope), navigate to the following folder and create the appropriate backend.<env>.tfvars file with the values noted in the previous step.

cd infra/terraform
Sample backend.dev.tfvars file 
bucket         = "dev-app-us-east-1-terraform"
region         = "us-east-1"
dynamodb_table = "dev-app-us-east-1-terraform-locks"
encrypt        = true
If you are unsure about the values for the bucket, region, etc. contact your infrastructure administrator.

b) Create and configure a .tfvars file

On the controller machine (e.g. your local machine or Bastion):

  • Navigate into the Veriscope folder on your machine. E.g. cd source/ShyftNetwork/veriscope

  • Check-out the latest tag/version listed here or the branch you would like to install

  • cd into infra/terraform/instances

  • Make a copy of sample.tfvars and create a file named <your-github-handle>.tfvars

  • Edit that file and update (add/remove) your instances

Example 1. Corresponding commands
cd ~/opt/veriscope
git branch
git checkout <branch-name>
cd infra/terraform/instances
ll
cp sample.tfvars <your-github-handle>.tfvars

c) Initialise Terraform

Run the following command for each environment, replacing <env> with the environment in question: test, dev, staging, or prod.

terraform init -backend-config="./../backend.<env>.tfvars" -backend-config="key=instances.tfstate"

For example, for prod:

terraform init -backend-config="./../backend.prod.tfvars" -backend-config="key=instances.tfstate"
Sample output 
...
...
...
Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

d) Validate your Terraform module

Run the following command:

terraform validate

You should see:

Success! The configuration is valid.

e) Create or switch to your workspace

Run the following command/script:

./workspace.sh <your-github-handle>

This will either create or switch to your workspace (if it already exists). New terraform workspace’s are created based on your GitHub handle.

./workspace.sh <your-github-handle>
Sample output 
./workspace.sh art-cmyk
...
...
...
Workspace for art-cmyk already exists. Switching to it...
  default
* dev-art-cmyk
  dev-art-cmyk-1

If the output is all green, you’re good to go! If not, contact your infrastructure administrator and resolve the issue.

f) Set environment variables

If you don’t know your organizations AWS account number or IAM role name, contact your infrastructure administrator before proceeding.

Run the following commands:

export AWS_PROFILE=default (or whatever you use in your environment)
export AWS_REGION=(whatever aws region you are using e.g. us-east-1)
export TF_VAR_aws_account_number=<your-aws-account-number>
export TF_VAR_role_name=<iam-role-name-to-assume>
export TF_VAR_cloudflare_creds='{ api_token = "kjsdhfksjhfskdjfhksjdhf", account_id = "lkjdshflksjhdfjhkjhslkjdf" }'

  • TF_VAR_cloudflare_creds are not used but are needed due to minor bug in our Terraform module (which will be resolved soon).

g) Run Terraform plan

Run the following command:

terraform plan -var-file <your-github-handle>.tfvars -out=<your-github-handle>.tfplan

For example,

terraform plan -var-file art-cmyk.tfvars -out=art-cmyk.tfplan
Sample output 
...
...
...
Saved the plan to: art-cmyk.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "art-cmyk.tfplan"

h) Run Terraform apply

Review the plan and ensure terraform says it will create the appropriate infrastructure. If all looks good, run the apply command:

terraform apply "<your-github-handle>.tfplan"

For example,

terraform apply "art-cmyk.tfplan"
Sample output 
terraform apply "art-cmyk.tfplan"
...
...
...

Apply complete! Resources: 13 added, 0 changed, 0 destroyed.

Outputs:

...
...
...
If you encounter an error at this point (during the apply operation), you will need to run terraform plan followed by terraform apply again after the error/issue has been resolved.

The above step could take a few minutes. So, time for your favourite drink! At the end of the apply run, outputs will be printed. Some of them might be hidden due to being sensitive.

You can run the following command to see the hidden outputs:

terraform output -json

Included in the outputs are FQDN, public and private IP addresses of the node created, SSH key secret name, TA DB password secret name etc.

Sometimes an error might occur acquiring state lock like below. This could be for various reasons, including forcibly exiting a previous plan/apply/destroy actions.

Sample error 
terraform plan -var-file art-cmyk.tfvars -out=art-cmyk.tfplan
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        fa50f66d-489c-3782-7f2f-4571db2ed92d
│   Path:      dev-veriscope-us-east-1-terraform/instances.tfstate
│   Operation: OperationTypeApply
│   Who:       ravit@KRSNA-x360
│   Version:   1.3.3
│   Created:   2022-12-13 12:11:21.526195545 +0000 UTC
│   Info:
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

To unlock, run the following command:

terraform force-unlock -force fa50f66d-489c-3782-7f2f-4571db2ed92d

Sample output:

Terraform state has been successfully unlocked!

The state has been unlocked, and Terraform commands should now be able to obtain a new lock on the remote state.

When the Terraform apply procedure is completed, proceed to the next step.

Sample Terraform Output 
ta_db = <sensitive>
veriscope_nodes = {
  "nm_instances" = {
    "art-12" = {
      "az" = "us-east-1a"
      "instance_id" = "i-0bdd13161ca9a818a"
      "private_fqdn" = "art-12-nm.veriscope.org"
      "private_ip" = "172.118.1.8"
      "public_fqdn" = ""
      "public_ip" = "44.201.52.168"
      "ssh_priv_key_secret_name" = "/ravitej/dev/instances/dev-art-12-nm"
      "subnet" = "subnet-05eb2978611fa87b8"
      "tags" = tomap({
        "Component" = "Instances"
        "DeployedBy" = "Veriscope Automated Deployment"
        "Environment" = "Dev"
        "GIT_BRANCH" = ""
        "GIT_TAG" = "v1.0.0-infra"
        "Name" = "art-12-nm.veriscope.org"
        "Owner" = "Ravitej"
        "ServerType" = "nm"
        "TF_WORKSPACE" = "dev-art-cmyk"
      })
    }
    "art-15" = {
      "az" = "us-east-1b"
      "instance_id" = "i-051278de8ab21fd79"
      "private_fqdn" = "art-15-nm.veriscope.org"
      "private_ip" = "172.118.1.27"
      "public_fqdn" = ""
      "public_ip" = "54.227.11.83"
      "ssh_priv_key_secret_name" = "/ravitej/dev/instances/dev-art-15-nm"
      "subnet" = "subnet-02deb231cbe42bbda"
      "tags" = tomap({
        "Component" = "Instances"
        "DeployedBy" = "Veriscope Automated Deployment"
        "Environment" = "Dev"
        "GIT_BRANCH" = ""
        "GIT_TAG" = "v1.0.0-infra"
        "Name" = "art-15-nm.veriscope.org"
        "Owner" = "Ravitej"
        "ServerType" = "nm"
        "TF_WORKSPACE" = "dev-art-cmyk"
      })
    }
  }
  "web_instances" = {
    "art-12" = {
      "az" = "us-east-1a"
      "instance_id" = "i-006165166dc1fcd05"
      "private_fqdn" = "art-12.veriscope.org"
      "private_ip" = "172.118.1.10"
      "public_fqdn" = "art-12.veriscope.org"
      "public_ip" = "35.170.60.169"
      "ssh_priv_key_secret_name" = "/ravitej/dev/instances/dev-art-12-web"
      "subnet" = "subnet-05eb2978611fa87b8"
      "tags" = tomap({
        "Component" = "Instances"
        "DeployedBy" = "Veriscope Automated Deployment"
        "Environment" = "Dev"
        "GIT_BRANCH" = ""
        "GIT_TAG" = "v1.0.0-infra"
        "Name" = "art-12.veriscope.org"
        "Owner" = "Ravitej"
        "ServerType" = "web"
        "TF_WORKSPACE" = "dev-art-cmyk"
      })
    }
    "art-15" = {
      "az" = "us-east-1b"
      "instance_id" = "i-05587c7211e2049f7"
      "private_fqdn" = "art-15.veriscope.org"
      "private_ip" = "172.118.1.30"
      "public_fqdn" = "art-15.veriscope.org"
      "public_ip" = "35.171.20.156"
      "ssh_priv_key_secret_name" = "/ravitej/dev/instances/dev-art-15-web"
      "subnet" = "subnet-02deb231cbe42bbda"
      "tags" = tomap({
        "Component" = "Instances"
        "DeployedBy" = "Veriscope Automated Deployment"
        "Environment" = "Dev"
        "GIT_BRANCH" = ""
        "GIT_TAG" = "v1.0.0-infra"
        "Name" = "art-15.veriscope.org"
        "Owner" = "Ravitej"
        "ServerType" = "web"
        "TF_WORKSPACE" = "dev-art-cmyk"
      })
    }
  }
}