This section describes the standard upgrade procedure for upgrading Usage Engine on Microsoft Azure. This procedure should be used to ensure that configurations, persisted data (in the file system or database), and system properties are properly migrated. Workflows will be interrupted shortly at the restart of the ECDs. These instructions will also describe how the timing of the restart of ECDs running non-scalable real-time workflows can be controlled by enabling manual upgrade.

Preparing ECDs

Note!

The instructions for backup and upgrade of the database below are only relevant if you are using Azure Database for PostgreSQL - Flexible Server as platform database. If the platform database used is Derby, the backup of the Azure Files Storage covers the database as well (assuming persistent storage of the platform is enabled).

For database backup, please refer to https://learn.microsoft.com/en-us/azure/backup/backup-azure-database-postgresql-flex for guidance.

The next step is to do a backup of the file system used.

Note!

If there are standalone ECs that are still running and writing their logs to the same file storage , whatever happens after the backup has been initiated will not be included in the backup.

To create an Azure File share backup, see https://learn.microsoft.com/en-us/azure/backup/backup-azure-files?tabs=backup-center or https://learn.microsoft.com/en-us/azure/backup/backup-afs-cli for instructions.

The section below contains an example of how to create a backup vault, followed by enabling an Azure File share backup protection and performing an on-demand backup via the command line.

export RESOURCE_GROUP=PT_Stratus
export LOCATION="Southeast Asia"
export STORAGE_ACCOUNT_NAME=uepeaks
export STORAGE_ACCOUNT_KEY=$(az storage account keys list --account-name $STORAGE_ACCOUNT_NAME --query "[0].value")
export STORAGE_ACCOUNT_ID=$(az storage account show --resource-group $RESOURCE_GROUP --name $STORAGE_ACCOUNT_NAME --query "id")
export SUBSCRIPTION_ID=$(az account subscription list --query "[0].subscriptionId" | tr -d '"')
export FILE_SHARE=$(az storage share list --account-name $STORAGE_ACCOUNT_NAME --account-key $STORAGE_ACCOUNT_KEY --query "[0].name" | tr -d '"')
export FILE_BACKUP_VAULT=azurefilesvault
export FILE_BACKUP_POLICY=FileBackupPolicy

# Create new file backup vault
az backup vault create --resource-group $RESOURCE_GROUP --name $FILE_BACKUP_VAULT --location $LOCATION --output table
az backup vault list --query "[].{Name:name}"

# Create new file backup policy
# References:
# https://learn.microsoft.com/en-us/azure/backup/manage-afs-backup-cli#create-policy
# https://learn.microsoft.com/en-us/azure/templates/microsoft.recoveryservices/vaults/backuppolicies?pivots=deployment-language-bicep#property-values
cat <<-EOF > $FILE_BACKUP_POLICY.json
{
  "eTag": null,
  "id": "/Subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.RecoveryServices/vaults/$FILE_BACKUP_VAULT/backupPolicies/$FILE_BACKUP_POLICY",
  "location": null,
  "name": "$FILE_BACKUP_POLICY",
  "properties": {
    "backupManagementType": "AzureStorage",
    "protectedItemsCount": 0,
    "retentionPolicy": {
      "dailySchedule": {
        "retentionDuration": {
          "count": 30,
          "durationType": "Days"
        },
        "retentionTimes": [
          "2024-07-19T03:00:00+00:00"
        ]
      },
      "monthlySchedule": null,
      "retentionPolicyType": "LongTermRetentionPolicy",
      "weeklySchedule": null,
      "yearlySchedule": null
    },
    "schedulePolicy": {
      "schedulePolicyType": "SimpleSchedulePolicy",
      "scheduleRunDays": null,
      "scheduleRunFrequency": "Daily",
      "scheduleRunTimes": [
        "2024-07-19T03:00:00+00:00"
      ],
      "scheduleWeeklyFrequency": 0
    },
    "timeZone": "UTC",
    "workLoadType": "AzureFileShare"
  },
  "resourceGroup": "$RESOURCE_GROUP",
  "tags": null,
  "type": "Microsoft.RecoveryServices/vaults/backupPolicies"
}
EOF

az backup policy list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --query "[].{Name:name}"
az backup policy create --policy $FILE_BACKUP_POLICY.json --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --name $FILE_BACKUP_POLICY --backup-management-type AzureStorage
az backup policy show --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --name $FILE_BACKUP_POLICY

# Enable Azure File share backup protection
az backup protection enable-for-azurefileshare --vault-name $FILE_BACKUP_VAULT --resource-group $RESOURCE_GROUP --policy-name $FILE_BACKUP_POLICY --storage-account $STORAGE_ACCOUNT_NAME --azure-file-share $FILE_SHARE  --output table

# Command output as below:
# Name                                  ResourceGroup
# ------------------------------------  ---------------
# 2b85d01d-9a27-4a5a-aa9d-cbdad082cac2  PT_Stratus

# Track job status
az backup job show --name 2b85d01d-9a27-4a5a-aa9d-cbdad082cac2 --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT

# Retrieve container registered to the Recovery services vault and export as env variable
export CONTAINER_NAME=$(az backup container list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --backup-management-type AzureStorage --query "[0].name" | tr -d '"')

# Retrieve backed up item and export as env variable
export ITEM_NAME=$(az backup item list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --query "[0].name" | tr -d '"')

# Perform on-demand backup
az backup protection backup-now --vault-name $FILE_BACKUP_VAULT --resource-group $RESOURCE_GROUP --container-name $CONTAINER_NAME --item-name $ITEM_NAME --retain-until 20-01-2025 --output table

# Command output as below:
# Name                                  Operation    Status      Item Name               Backup Management Type    Start Time UTC                    Duration
# ------------------------------------  -----------  ----------  ----------------------  ------------------------  --------------------------------  --------------
# 23300e34-b1e0-409c-804e-c247d4587f8f  Backup       InProgress  uepe-aks-storage-share  AzureStorage              2024-07-19T11:01:07.436164+00:00  0:00:02.178697

Restore Database Backup

You can restore a database backup into Azure Blob Storage and use PostgreSQL native tool pg_restore to restore data to a new PostgreSQL flexible server database, see https://learn.microsoft.com/en-us/azure/backup/restore-azure-database-postgresql-flex for detailed steps.

Note!

The restored PostgreSQL flexible server is a new database instance and is not managed by Terraform. If you plan to destroy the cluster later, ensure that the new database instance is deleted first.

Restore File System Snapshot

To restore an Azure File share, follow the instructions from https://learn.microsoft.com/en-us/azure/backup/restore-afs?tabs=full-share-recovery or https://learn.microsoft.com/en-us/azure/backup/restore-afs-cli.

The section below contains an example of how to restore an Azure File backup using the command line. In this example the backup is restored to the existing File share. If you wish to restore to a new File share instance, you need to adjust accordingly.

export RESOURCE_GROUP=PT_Stratus
export LOCATION="Southeast Asia"
export STORAGE_ACCOUNT_NAME=uepeaks
export STORAGE_ACCOUNT_KEY=$(az storage account keys list --account-name $STORAGE_ACCOUNT_NAME --query "[0].value")
export STORAGE_ACCOUNT_ID=$(az storage account show --resource-group $RESOURCE_GROUP --name $STORAGE_ACCOUNT_NAME --query "id")
export SUBSCRIPTION_ID=$(az account subscription list --query "[0].subscriptionId" | tr -d '"')
export FILE_SHARE=$(az storage share list --account-name $STORAGE_ACCOUNT_NAME --account-key $STORAGE_ACCOUNT_KEY --query "[0].name" | tr -d '"')
export FILE_BACKUP_VAULT=azurefilesvault
export FILE_BACKUP_POLICY=FileBackupPolicy
export CONTAINER_NAME=$(az backup container list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --backup-management-type AzureStorage --query "[0].name" | tr -d '"')
export ITEM_NAME=$(az backup item list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --query "[0].name" | tr -d '"')

# Fetch recovery points
az backup recoverypoint list --vault-name $FILE_BACKUP_VAULT --resource-group $RESOURCE_GROUP --container-name $CONTAINER_NAME --backup-management-type azurestorage --item-name $ITEM_NAME --workload-type azurefileshare --out table

# Command output as below:
# Name            Time                       Consistency
# --------------  -------------------------  --------------------
# 68988215529834  2024-07-19T11:01:09+00:00  FileSystemConsistent

# Full restore snapshot to existing file share
az backup restore restore-azurefileshare --vault-name $FILE_BACKUP_VAULT --resource-group $RESOURCE_GROUP --rp-name 68988215529834 --container-name $CONTAINER_NAME --item-name $ITEM_NAME --restore-mode originallocation --resolve-conflict overwrite --out table

# Track job status
az backup job show --name 249c1bbb-da9f-4b3b-b612-f9917ea2cecd --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT

Usage Engine Private Edition 4 Documentation

Upgrade Instructions - Azure (4.3)

Preparing ECDs

Restore Database Backup

Restore File System Snapshot