AWS Services
The below table lists mandatory and optional managed AWS services that are used by Usage Engine
Service | Documentation | Mandatory | Pricing | Backup Method | IaC |
|---|---|---|---|---|---|
Amazon RDS for PostgreSQL | Yes | Point in Time Recovery | Terraform | ||
Amazon Elastic File System | Yes | AWS Backup Snapshotting | Terraform | ||
Amazon Route 53 | Yes | IaC | Terraform | ||
Amazon Elastic Kubernetes Service | Yes | IaC | Eksctl | ||
AWS Key Management Service | Yes | IaC | Terraform | ||
Elastic Load Balancing | Yes | IaC | Terraform | ||
Amazon Memory DB for Redis | No | Multi-AZ transactional log | Terraform | ||
AWS Backup | No | - | |||
Amazon S3 | No | S3 | Terraform | ||
Amazon S3 Glacier | No | S3 | Terraform |
Assets
Stateful Data Assets - To Include in Backups
| Excerpt |
|---|
When running in AWS, Usage Egnine stores data persistently in RDS, EFS and optionally MemoryDB. All these services support encryption at rest. The default Terraform templates provided as part of the product are configured to use encryption at rest. MemoryDB is configured by the customer. It is recommended to configure it with encryption at rest if sensitive information will be stored in it. |
...
As workflows are executed in the system and process payload they persist intermittent data. This data is stored in various locations:
Transaction data - in system database in RDS
Aggregation session - on EFS or in MemoryDB
Duplicate detection caches - on EFS
Inter workflow persisted transmission buffers - on EFS
Error correction data - in system database in RDS (Data Veracity subsystem) or on EFS (older ECS subsystem)
Depending on the characteristics on the solution, this information can contain sensitive information about subscribers etc. Thus it is recommended to use encryption at rest for EFS and MemoryDB as soon as there is some data with personal integrity being processed in the workflows.
...
Solutions that use disk persisted data is storing this on a mounted EFS disk resource. Using the provided Terraform template, the EFS resource is given the name "[EKS-CLUSTER-ANMENAME]-efs-disk". It is mounted into the pods at the path "/opt/mz/persistent" which is the path where executing workflows can access the storage.
The following features use or can use disk based storage to persist data:
Aggregation Aggregation (4.
...
Duplicate UDR Detection Duplicate UDR (4.
...
Inter Workflow Inter Workflow (4.
...
RDS Managed database
The provided Terraform installation templates sets up a PostgreSQL database in RDS to be used as the system database. The RDS resource gets the name "[EKS-CLUSTER-ANMENAME]-db".
MemoryDB Redis database
Some features can use a Redis compatible database to store state. In AWS the MemoryDB for Redis service can be used for this. Access to this is configured in:
and can be consumed by:
Aggregation Profile Aggregation (4.
2Distributed Storage Profile Distributed Storage (4.
2
Stateless Assets - To Manage as Infrastructure as Code
The AWS infrastructure should be managed as IaC using Terraform, CloudFormation or similar tool. Usage Engine comes with templates written in Terraform that will setup a basic environment. The default configuration of these templates setup the following resources, which are sufficient to run a standard Usage Engine system with solutions that are not particularly demanding on resources.
EFS Mount Target
EFS Security Group
KMS Key
KMS Alias
RDS DB Instance (Postgres)
RDS DB Subnet Group
RDS DB Access Security Group
SSM RDS Parameters
DB User
DB Password
Route53 DNS Zone
Route53 DNS NS Record
ACM Certificate
Route53 Cert Validation Record
ACM Certificate Validation
EKS Cluster
VPC Cluster Endpoint
Node Group
Minimum three nodes m5.Large
CloudWatch cluster logging
MZ Platform Application Helm chart
MZ ECD Solution Helm charts