Before installing Usage Engine Private Edition, you must set up a Kubernetes cluster on Azure AKS (Azure’s managed Kubernetes service).

First, a basic Kubernetes cluster needs to be created. This can be done in two different ways:

Using the terraform tool.
Using the Azure management console.

In this guide, terraform will be used. Mainly because it will enable you to create the basic Kubernetes cluster in minutes with just a single command.

Once the basic Kubernetes cluster has been created, additional infrastructure must be added. For this terraform is also used.

Before proceeding, go to Release Information, and download the azure.tar.gz file for the Usage Engine Private Edition version that is being installed. Once downloaded, extract its content to a suitable location.

Assumptions

There are a few assumptions which have been made when using terraform to create cluster resources:

We assume you have an existing parent domain i.e. example.com hosted on the same account as the cluster that we are going to create in the coming section and you wish to access the cluster environment through the hostname. Terraform will create a subdomain in format <cluster_name>.<domain>.
1. cluster name: uepe-aks
2. domain: example.com
3. final domain: uepe-aks.example.com
In addition, we also assume terraform is allowed to add a NS (NameServer) record to the parent domain. This is to allow DNS delegation from the parent domain to subdomain.
Terraform needs to persist the state of your provisioned infrastructure, by default the state file is stored locally on the computer that terraform is executed from. However if you have multiple person working on the infrastructure then it is recommended to store the state file on remote persistent such as Azure Storage, see https://learn.microsoft.com/en-us/azure/developer/terraform/store-state-in-azure-storage?tabs=terraform for more information.
We use the Azure Files (SMB/NFS) as the default persistent storage for data needs to be persisted.
We use the Azure Database for PostgreSQL - Flexible Server service for Usage Engine Private Edition database.
We assume that you have the privileges to create user managed identity and grant role to the identity created by terraform within the same resource group. See https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp for more details.
To use the terraform template, preview feature has to be enabled in your Azure subscription. Refer to https://learn.microsoft.com/en-us/azure/aks/api-server-vnet-integration#install-the-aks-preview-azure-cli-extension for the commands to run. You only need to execute these sections:
1. Install the aks-preview Azure CLI extension
2. Register the 'EnableAPIServerVnetIntegrationPreview' feature flag

Create Basic Cluster and additional infrastructure

Follow these steps to create a basic Kubernetes cluster with public and private VPC:

Extract the downloaded azure.tar.gz file.
Go to <your local path>/azure/terraform folder and copy the terraform.tfvars.example to terraform.tfvars.
Edit the terraform.tfvars file.
Specify the desired cluster_name, Azure location and kubernetes_version (see Compatibility Matrix (4.2) to find out which Kubernetes versions that are compatible with this release of Usage Engine Private Edition). Also specify your Azure resource_group_name, domain as well as the desired number of nodes per cluster (user_node_count).
If you will be running with a database other than Derby also specify db_password and db_version.

terraform.tfvars	Where to get the value from?
`resource_group_name`	Name of the resource group. The resource group must exist in the Azure subscription.
`location`	Location of the cluster Use command `az account list-locations --query "[].displayName" -o tsv` to get all available locations.
`cluster_name`	Name of the cluster, it must be unique within the resource group.
`domain`	Your existing domain name. In the Azure management console, this is the DNS zones that is listed on page `Home \| DNS zones \| Name`. The DNS zone must already exist in Azure DNS zones or in other cloud provider DNS zone. The service created by Usage Engine Private Edition in the coming section will be accessible in format `<service_name>.<cluster_name>.<domain>` i.e. desktop-online.uepe-aks.example.com.
`kubernetes_version`	Kubernetes version.
`user_node_count`	Number of nodes in the user node pool.
`db_password`	Choose a secure password for the system database administrator. Minimum 10 characters.
`db_version`	PostgreSQL database version. Supported versions are 11, 12, 13, 14, 15 and 16.
`db_enabled`	Boolean flag to enable PostgreSQL database resource creation.
`auto_create_ns_record`	Boolean flag to enable subdomain NS record auto creation in parent domain. In case your parent domain is not under the same resource group or your parent domain is hosted in another cloud provider, then you must set it to false.
`storage_enabled`	Boolean flag to enable file storage resource creation.

Example:

#  ____  _____ _____   _____ _   _ _____ ____  _____
# / ___|| ____|_   _| |_   _| | | | ____/ ___|| ____|_
# \___ \|  _|   | |     | | | |_| |  _| \___ \|  _| (_)
#  ___) | |___  | |     | | |  _  | |___ ___) | |___ _
# |____/|_____| |_|     |_| |_| |_|_____|____/|_____(_)

# The below values must be set explicitly in order for the setup to work correctly.

# Name of the resource group
# The resource group must be existed in the Azure subscription.
resource_group_name = "PT_Stratus"

# Location of the cluster
# Use command `az account list-locations --query "[].displayName" -o tsv` to get all available locations.
# Or see https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/?products=kubernetes-service&regions=all
location = "Southeast Asia"

# Name of the cluster, it must be unique within the resource group.
cluster_name = "uepe-aks"

# Domain DNS name
# The DNS zone must already exist in Azure DNS or in other cloud provider DNS zone.
# We'll create a subdomain zone from parent domain, the final domain will be in format "<cluster_name>.<domain>".
# Please note that if this domain is hosted on another resource group or other cloud provider, then you must
# set auto_create_ns_record = false and manually add the subdomain NS record to the parent domain.
domain = "stratus.az.digitalroute.net"

# Admin user password to the database
db_password = "super_SeCrEt_db_pAsSwOrD_457!"

#  _______        _______    _    _  __    _    ____  _     _____
# |_   _\ \      / / ____|  / \  | |/ /   / \  | __ )| |   | ____|_
#   | |  \ \ /\ / /|  _|   / _ \ | ' /   / _ \ |  _ \| |   |  _| (_)
#   | |   \ V  V / | |___ / ___ \| . \  / ___ \| |_) | |___| |___ _
#   |_|    \_/\_/  |_____/_/   \_\_|\_\/_/   \_\____/|_____|_____(_)

# The below sections are the default values, tweak them to your needs.

#     _    _  ______
#    / \  | |/ / ___|
#   / _ \ | ' /\___ \
#  / ___ \| . \ ___) |
# /_/   \_\_|\_\____/

# The size of the Virtual Machine
# Use command `az vm list-sizes --location "West Europe" --output table` to get available VM sizes.
vm_size = "Standard_DS2_v2"
# Kubernetes version
kubernetes_version = "1.29"
# Number of nodes in the system node pool, must be at least 1 node.
system_node_count = 2
# Number of nodes in the user node pool. If zero, the user node pool will not be created.
user_node_count = 1
# DNS prefix specified when creating the managed cluster.
# It must begin and end with a letter or number, contain only letters, numbers, and hyphens and be between 1 and 54 characters in length.
dns_prefix = "aks"
# Operating system disk size in GB
os_disk_size_gb = 30
# The priority for Virtual Machines within the Virtual Machine Scale Set that powers the node pool.
# Valid values are Regular and Spot.
priority = "Regular"
# The eviction policy for Virtual Machines within the Virtual Machine Scale Set that powers the node pool.
eviction_policy = "Delete"
# The maximum price you are willing to pay for Spot VMs. The value must be -1 or a positive value with up to five decimal places.
# See https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms for more information.
spot_max_price = -1

..........

#  ____  _   _ ____
# |  _ \| \ | / ___|
# | | | |  \| \___ \
# | |_| | |\  |___) |
# |____/|_| \_|____/

# Specify if nameservers of the subdomain should automatically inserted as a NS record to the parent domain's zone to allow DNS delegation.
# By default we assume your parent domain zone is hosted on the same resource group, if this is not the case then you MUST set it to false.
auto_create_ns_record = true

#  ____ _____ ___  ____      _    ____ _____
# / ___|_   _/ _ \|  _ \    / \  / ___| ____|
# \___ \ | || | | | |_) |  / _ \| |  _|  _|
#  ___) || || |_| |  _ <  / ___ \ |_| | |___
# |____/ |_| \___/|_| \_\/_/   \_\____|_____|

# Persistent storage
storage_enabled = true
# Kind of account to create. Valid options are Storage, StorageV2, BlobStorage, BlockBlobStorage or FileStorage.
account_kind = "FileStorage"
# Tier to use for the storage account. Either Standard or Premium.
# For BlockBlobStorage and FileStorage accounts only Premium is valid.
account_tier = "Premium"
# Storage redundancy type
# See https://learn.microsoft.com/en-us/azure/storage/files/files-redundancy for available options.
account_replication_type = "LRS"
# Protocol used for the share, either SMB or NFS.
# See https://learn.microsoft.com/en-us/azure/storage/files/storage-files-planning#available-protocols for the comparison.
enabled_protocol = "SMB"
# Storage capacity in GB
storage_capacity = 100

#  ____    _  _____  _    ____    _    ____  _____
# |  _ \  / \|_   _|/ \  | __ )  / \  / ___|| ____|
# | | | |/ _ \ | | / _ \ |  _ \ / _ \ \___ \|  _|
# | |_| / ___ \| |/ ___ \| |_) / ___ \ ___) | |___
# |____/_/   \_\_/_/   \_\____/_/   \_\____/|_____|

# PostgreSQL database
db_enabled = true
# SKU Name for PostgreSQL, it follows the tier + name pattern.
# Available tiers are Basic (B), GeneralPurpose (GP), MemoryOptimized (MO).
# Use command `az postgres flexible-server list-skus --location "West Europe" --output table` to find out available SKU names.
# For example, tier "GeneralPurpose" and name "Standard_D2s_v3", the SKU name is "GP_Standard_D2s_v3".
sku_name = "GP_Standard_D2s_v3"
# Storage performance tier for IOPS. See https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/postgresql_flexible_server#storage_tier-defaults-based-on-storage_mb for more information
db_tier = "P10"
# DB version. Supported versions are 11, 12, 13, 14, 15 and 16.
db_version = "16"
# DB storage in MB. See https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/postgresql_flexible_server#storage_tier-defaults-based-on-storage_mb for available storage sizes.
db_allocated_storage = 131072
# Specify which availability zone the instance should be located in.
zone = 1

Note!

If your parent domain zone is not under the same Resource Group:

You must set auto_create_ns_record = false to disable subdomain NS record auto creation in the parent domain.
Perform terraform apply.
When terraform apply is finished, copy the name servers value from terraform output name_servers and manually add them to parent domain as NS record. If you are not using Azure DNS as the parent domain, consult your Domain Registrar documentation on how to add NS record.

Authenticate your computer with Azure by running the following command:
```
az login --scope https://management.core.windows.net//.default
```

Run the following commands:

terraform init
terraform plan
terraform apply

Wait for the terraform commands to finish and you will see an output similar to:

Apply complete! Resources: 40 added, 0 changed, 0 destroyed.

Outputs:

aks_domain_zone_name = "uepe-aks.stratus.az.digitalroute.net"
appgw_ingress_controller_client_id = "1aee4369-d6c2-4912-ab5a-45fe5d3e3526"
appgw_ingress_controller_namespace = "uepe"
appgw_ingress_controller_service_account = "ingress-azure"
appgw_name = "uepe-aks-application-gateway"
azure_resource_group = "PT_Stratus"
azure_subscription_id = "e6d5f094-a3a1-4ae1-b660-4aa689593bf7"
azure_tenant_id = "bc3dbf22-18e0-4f00-b10d-9595a47abf83"
cert_manager_client_id = "b21cf932-fa5a-4d81-8230-05cfa85245ac"
cert_manager_namespace = "cert-manager"
cert_manager_service_account = "cert-manager"
cluster_location = "Southeast Asia"
db_endpoint = "uepe-aks-db.postgres.database.azure.com"
db_password = "super_SeCrEt_db_pAsSwOrD_457!"
db_username = "postgres"
external_dns_client_id = "c97fad12-109e-4fb4-971e-9446b0964f3c"
external_dns_namespace = "uepe"
external_dns_service_account = "external-dns"
kube_config = "./kubeconfig"
kubernetes_cluster_name = "uepe-aks"
name_servers = toset([
  "ns1-38.azure-dns.com.",
  "ns2-38.azure-dns.net.",
  "ns3-38.azure-dns.org.",
  "ns4-38.azure-dns.info.",
])
storage_account_key = "trfSDOsyDS2SFCp59Q3x4t3RA8a2KZGtNUJ2lLJ7xS0L6pDm8BZ2AFkoCoMHmtUizQSSLOA+8nQA+AStvjbRaw=="
storage_account_name = "uepeaks"
storage_capacity_gb = 100
storage_persistence_yaml = "./manifests/storage_persistence.yaml"
storage_share_name = "uepe-aks-storage-share"

Ensure that you save the output from terraform above since it will be used as input throughout the remainder of this installation guide.

A basic Kubernetes cluster has now been set up successfully.

Note!

It is important to keep the Terraform state file safe after the cluster creation is completed, as it will be needed if you wish to destroy the cluster later.

In this example, an PostgreSQL flexible servers database instance up and running on private subnet VNET with default listening port 5432. The default database postgres is accessible within the cluster at end point uepe-aks-db.postgres.database.azure.com with admin username postgres.

You can check the status of the cluster, db and the other resources in the Azure dashboard.

Configure Cluster Access

The previous terraform command will generate a ./kubeconfig file containing information on how to connect to your newly created cluster. Make sure to set the KUBECONFIG environment variable to point to that file on the terraform’s working directory:

export KUBECONFIG=<full path to ./kubeconfig>

This will ensure that tools like kubectl and helm will connect to your newly created cluster.

Alternatively, you can use the following command to bind your local kubectl to the created cluster:

az aks get-credentials --name uepe-aks --resource-group PT_Stratus

You can check the status of the cluster nodes with the following command:

kubectl get nodes

For this example cluster the output will look something like this:

NAME                               STATUS   ROLES    AGE    VERSION
aks-default-41718565-vmss000000    Ready    <none>   6d2h   v1.29.5
aks-default-41718565-vmss000001    Ready    <none>   6d1h   v1.29.5
aks-internal-23883603-vmss000000   Ready    <none>   6d2h   v1.29.5

Namespace

Create a namespace called uepe:

kubectl create namespace uepe

Unless explicitly stated, this is the namespace that is used throughout the remainder of this installation guide.

Hint!

You can also create and use a namespace with another name.

This command shows all namespaces that currently exist in your cluster:

kubectl get namespaces

Persistent Volume and Persistent Volume Claim

Note!

The persistent volume setup is an optional step. Skip this section if you do not intend to have persistent file storage.

Persistent Volume (PV) and Persistent Volume Claim (PVC) must be setup before Usage Engine Private Edition Helm Chart installation. The PV and PVC yaml files have already been generated in <terraform script directory>/manifests/storage_persistence.yaml.

Kubernetes needs credentials to access the Azure storage that was created by the terraform command in the previous section. Create the storage account secret with this command:

kubectl create secret generic <kubernetes_cluster_name from terraform output>-persistence-secret \
--from-literal=azurestorageaccountname=<storage_account_name from terraform output> \
--from-literal=azurestorageaccountkey=<storage_account_key from terraform output> \
-n uepe

Change directory to <terraform script directory>/manifests.

To setup Persistent Volume and Persistent Volume Claim run the following command:

kubectl apply -f storage_persistence.yaml -n uepe

To ensure PVC bounded to allocated Persistent Volume run the following command:

kubectl get pvc -n uepe

The output should display PVC status is Bound, similar to:

NAME           STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
uepe-aks-pvc   Bound    uepe-aks-pv   100Gi      RWX            azurefile-csi   <unset>                 6s

This section is now complete and you can proceed to the Kubernetes Cluster Add-ons - Azure (4.2) section.

Set Up Kubernetes Cluster - Azure (4.3)

Create Basic Cluster and additional infrastructure

Configure Cluster Access

Namespace

Persistent Volume and Persistent Volume Claim