Automate - Deploy in AWS with FSx Lustre and Dynamic Scaling

The material in this document is for informational purposes only. This guide assumes that the most recent version of Rampiva Automate is in use unless otherwise noted in the prerequisites. The products it describes are subject to change without prior notice, due to the manufacturer’s continuous development program. Rampiva makes no representations or warranties with respect to this document or with respect to the products described herein. Rampiva shall not be liable for any damages, losses, costs or expenses, direct, indirect or incidental, consequential or special, arising out of, or related to the use of this material or the products described herein.

Introduction

This document outlines the process of deploying Rampiva Automate in AWS with the option to dynamically scale resources according to the Job queue, using distributed workers from multiple instances during worker-based operations and using a FSx Lustre backed file system.

Prerequisites

AWS Subscription
Familiarity with deploying and configuring Automate on-premise

Instructions

A. Create the Scheduler instance and the Engine AMIs and Launch Template

1	Create a new EC2 instance and deploy the Rampiva Scheduler component.	https://rampiva.atlassian.net/wiki/spaces/KB/pages/812843009
2	Allow inbound HTTPS (port TCP/443) in to the Scheduler instance.
3	Create a new EC2 instance and deploy the Rampiva Engine Server component.	https://rampiva.atlassian.net/wiki/spaces/KB/pages/812843009
4	Configure FSx Lustre on the Engine Server instance, and ensure the configuration gets applied when the instance reboots.	https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html
5	Reboot the Engine Server to ensure that the Rampiva services start and the FSx Lustre filesystem is mounted.
6	Add the Engine Server to Automate and take note of the certificate fingerprint. This will be used when configuring the Resource Pool.
7	Queue a test Job to confirm that the Engine Server is configured correctly.
8	To ensure Remote Workers can connect to the main Engine, allow all communication inside the VPC in which the Engine Server are deployed.
9	Create an image from the Engine Server instance.
10	Create a template for the Main Engine, from the instance, using the image created at the previous step and the settings of the Engine Server instance.
11	Create a template for the Remote workers from the instance, using the image created at the previous step, the settings of the Engine Server instance, and additionally the setting to start the instances On Demand.

B. Configure the Rampiva Resource Pool

1

Obtain an Access Key and Secret Key for the account that Rampiva Automate will use to connect to AWS.

https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys

2

Create a new Resource Pool in Rampiva Automate with the following options:

Settings - the keys and AWS region in which the AMIs were created:
Engine - The settings for the main Engine which will run the job:
- Nuix License Source: The License Source from which the main Engines will acquire licenses.
- Target Workers: The number of Nuix workers that the Main Engines will attempt to acquire a license for, if available.
- Min Workers: The minimum number of Nuix workers required to initialize the Main Engines.
- Instance Idle Action:
  Terminate / Delete - this will delete the instance on which the main Engine ran.
- Force Idle Actions Between Jobs: Checked - this will start a fresh instance for the Main Engine for each new Job.
- Virtual Machine Source: Launch Template ID
- Launch Template ID: The ID of the main Engine template created previously.
- Max Concurrent Instances: The maximum number of EC2 instances running at the same time with the main Engine role. This is equivalent to the maximum number of parallel Jobs.
Remote Workers - The settings for the Remote Workers which will participate to jobs:
- Use Remote Workers: Checked
- Nuix License Source: The License Source from which the Remote Workers will acquire licenses.
- Target Workers: The number of Nuix workers that the Remote Workers will attempt to acquire a license for, if available.
- Min Workers: The minimum number of Nuix workers required to initialize each Remote Worker instance.
- Instance Idle Action:
  Terminate / Delete - this will delete the instance on which the Remote Workers ran.
- Virtual Machine Source: Launch Template ID
- Launch Template ID: The ID of the Remote Worker template created previously.
- Max Concurrent Instances: The maximum number of EC2 instances running at the same time using the Remote Workers role.
Whitelisted Certificate Fingerprints - the certificate fingerprint recorded when setting up the Engine Server image.

3

Queue a test Job and monitor the Resource Pool log as well as the status of the Instances in the EC2 AWS console.