Before doing anything to the running installation, the config file for the new installation should be prepared by following these steps: Retrieve the Where Extract the values manually from the output. Copy the lines below “USER-SUPPLIED VALUES:” and stop at the blank line. Save the copied content to the config file Update helm repository to get the latest helm chart versions by running the following command. Retrieve the new version from the repository by running the following command. Refer to Release Information for the Helm Chart version. For example: Next, check the file CHANGELOG.md inside the created folder to find out what may have changed in the new version when it comes to the values-file. means that in the values file they should be entered as: Each part of the key does not necessarily follow directly after the previous one, but always before any other “parent” on the same level. So in this example of a an example of a key could be Make any necessary updates based on changed field you may be using in the Take note of any fields that have been deprecated or removed since the last version so any configuration of those fields can be replaced. When you have updated the Preparations
values.yaml
file that you have used previously, or if you want to start from scratch, you extract it from the installation by running these commands:helm -n <namespace> get all <helm name>
E.g:
helm -n uepe get all uepe
uepe
is the helm name you have selected for your installation. You will see list similar to the one below.helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
external-dns uepe 1 2024-05-08 15:27:48.258978207 +0200 CEST deployed external-dns-7.2.0 0.14.1
ingress-nginx uepe 1 2024-05-08 16:18:43.919980224 +0200 CEST deployed ingress-nginx-4.10.0 1.10.0
uepe uepe 3 2024-05-10 14:16:17.724426589 +0200 CEST deployed usage-engine-private-edition-4.0.0 4.0.0
valuesFromSystem.yaml
.helm repo list
helm repo update
helm fetch <repo name>/usage-engine-private-edition --version <version> --untar
helm fetch digitalroute/usage-engine-private-edition --version 4.0.0 --untar
If you are uncertain about how to interpret the content of the file, see below for some examples of keys and how to interpret them:The following values have been removed:
* ```mzOperator.clusterWide```
* ```mzOperator.experimental.performPeriodicWorkflowCleanup```
* ```jmx.remote```
* ```platform.debug.jmx```
mzOperator:
clusterWide:
experimental:
performPeriodicWorkflowCleanup
jmx:
remote:
platform:
debug:
jmx:
values.yaml
file:debug:
script:
enabled: false
log:
level:
codeserver: info
jetty: 'off'
others: warn
debug.log.level.jetty
.valuesFromSystem.yaml
file you got from the existing installation so it matches the new version.valuesFromSystem.yaml
file you can test it by running this command:helm upgrade --install uepe digitalroute/usage-engine-private-edition --atomic --cleanup-on-fail --version 4.0.0 -n uepe -f valuesFromSystem.yaml --dry-run=server
When all the running batch workflows have stopped you should make a backup so that the system can be restored in case of any issues during the upgrade. Note! Before proceeding with the backup you must shut down the platform. This is very important since otherwise the backup of the database may become corrupt. The platform can be shut down in various ways, see examples below. Examples - Shutting Down the Platform Option 1 Reduce the number of replicas (under “spec”) to 0 by running the following command: where uepe is the namespace used. Option 2 Run this command: and then this command: And ensure that the pod platform-0 is no longer presentBackup and Database Upgrade
kubectl edit statefulset platform -n uepe
kubectl scale --replicas=0 sts/platform -n uepe
kubectl get pods -n uepe
Note!
The instructions for backup and upgrade of the database below are only relevant if you are using Azure Database as platform database. If the platform database used is derby, the backup of the Azure Storage covers the database as well (assuming persistent storage of the platform is enabled).
For database backup, please refer to https://learn.microsoft.com/en-us/azure/backup/backup-azure-database-postgresql-flex for guidance.
It is now time to do a backup of the file system used.
Note!
If there are standalone ECs that are still running and writing their logs to the same EFS, whatever happens after the backup has been initiated will not be included in the backup.
To create an Azure File share backup using the console, see https://learn.microsoft.com/en-us/azure/backup/backup-azure-files?tabs=backup-center for instructions. Alternatively, you can also refer to Azure CLI version guide https://learn.microsoft.com/en-us/azure/backup/backup-afs-cli.
The section below contains an example of how to run an on-demand backup job using the command line. The snapshot will in this case be stored under the default backup vault.
export RESOURCE_GROUP=PT_Stratus export LOCATION="Southeast Asia" export STORAGE_ACCOUNT_NAME=uepeaks export STORAGE_ACCOUNT_KEY=$(az storage account keys list --account-name $STORAGE_ACCOUNT_NAME --query "[].{Value:value}" | jq -rc ".[0].Value") export STORAGE_ACCOUNT_ID=$(az storage account show --resource-group $RESOURCE_GROUP --name $STORAGE_ACCOUNT_NAME | jq -rc ".id") export SUBSCRIPTION_ID=$(az account subscription list | jq -rc ".[0].subscriptionId") export FILE_SHARE=$(az storage share list --account-name $STORAGE_ACCOUNT_NAME --account-key $STORAGE_ACCOUNT_KEY --query "[].{Name:name}" | jq -rc ".[0].Name") export FILE_BACKUP_VAULT=azurefilesvault export FILE_BACKUP_POLICY=MyBackupPolicy # Create new file backup vault az backup vault create --resource-group $RESOURCE_GROUP --name $FILE_BACKUP_VAULT --location $LOCATION --output table az backup vault list --query "[].{Name:name}" # Create new file backup policy for scheduled backup # https://learn.microsoft.com/en-us/azure/backup/manage-afs-backup-cli#create-policy # https://learn.microsoft.com/en-us/azure/templates/microsoft.recoveryservices/vaults/backuppolicies?pivots=deployment-language-bicep#property-values cat <<-EOF > /path/to/$FILE_BACKUP_POLICY.json { "eTag": null, "id": "/Subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.RecoveryServices/vaults/$FILE_BACKUP_VAULT/backupPolicies/$FILE_BACKUP_POLICY", "location": null, "name": "$FILE_BACKUP_POLICY", "properties": { "backupManagementType": "AzureStorage", "protectedItemsCount": 0, "retentionPolicy": { "dailySchedule": { "retentionDuration": { "count": 30, "durationType": "Days" }, "retentionTimes": [ "2024-07-19T03:00:00+00:00" ] }, "monthlySchedule": null, "retentionPolicyType": "LongTermRetentionPolicy", "weeklySchedule": null, "yearlySchedule": null }, "schedulePolicy": { "schedulePolicyType": "SimpleSchedulePolicy", "scheduleRunDays": null, "scheduleRunFrequency": "Daily", "scheduleRunTimes": [ "2024-07-19T03:00:00+00:00" ], "scheduleWeeklyFrequency": 0 }, "timeZone": "UTC", "workLoadType": "AzureFileShare" }, "resourceGroup": "$RESOURCE_GROUP", "tags": null, "type": "Microsoft.RecoveryServices/vaults/backupPolicies" } EOF az backup policy list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --query "[].{Name:name}" az backup policy create --policy $FILE_BACKUP_POLICY.json --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --name $FILE_BACKUP_POLICY --backup-management-type AzureStorage az backup policy show --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --name $FILE_BACKUP_POLICY # Enable Azure File share backup protection az backup protection enable-for-azurefileshare --vault-name $FILE_BACKUP_VAULT --resource-group $RESOURCE_GROUP --policy-name $FILE_BACKUP_POLICY --storage-account $STORAGE_ACCOUNT_NAME --azure-file-share $FILE_SHARE --output table # Result output as in following: # Name ResourceGroup # ------------------------------------ --------------- # 2b85d01d-9a27-4a5a-aa9d-cbdad082cac2 PT_Stratus # Track job status az backup job show --name 2b85d01d-9a27-4a5a-aa9d-cbdad082cac2 --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT # Retrieve container registered to the Recovery services vault and export as env variable export CONTAINER_NAME=$(az backup container list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT --backup-management-type AzureStorage | jq -rc ".[].name") # Retrieve backed up item and export as env variable export ITEM_NAME=$(az backup item list --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT | jq -rc ".[].name") # Perform on-demand backup az backup protection backup-now --vault-name $FILE_BACKUP_VAULT --resource-group $RESOURCE_GROUP --container-name $CONTAINER_NAME --item-name $ITEM_NAME --retain-until 20-01-2025 --output table # Result output as in following: # Name Operation Status Item Name Backup Management Type Start Time UTC Duration # ------------------------------------ ----------- ---------- ---------------------- ------------------------ -------------------------------- -------------- # 23300e34-b1e0-409c-804e-c247d4587f8f Backup InProgress uepe-aks-storage-share AzureStorage 2024-07-19T11:01:07.436164+00:00 0:00:02.178697 # Track job status az backup job show --name 23300e34-b1e0-409c-804e-c247d4587f8f --resource-group $RESOURCE_GROUP --vault-name $FILE_BACKUP_VAULT
Restoring from Backup
If restoring becomes necessary, you can restore the recovery point into Azure Blob Storage and use DB native tools pg_restore to restore data as a new PostgreSQL flexible server, see Azure guide https://learn.microsoft.com/en-us/azure/backup/restore-azure-database-postgresql-flex for more information.
To restore EFS, follow the instructions in https://docs.aws.amazon.com/aws-backup/latest/devguide/restore-resource.html and https://repost.aws/knowledge-center/aws-backup-restore-efs-file-system-cli.
If you want to restore the backup into a new file system, the EFS mount target needs to be manually re-configured to allow access from the cluster, see https://docs.aws.amazon.com/efs/latest/ug/manage-fs-access.html#manage-fs-access-create-delete-mount-targets for more information. If you are using access points, you need to configure access point for the new file system after the restore is done.
The section below contains an example of how to restore the EFS backup using the command line. In this example the volume mount is using access point path /uepe
, and the snapshot is stored under default vault, and then the backup is restored as a new file system. If this is not how you have set it up, or if you wish to restore backup to the existing EFS instance, you need to adjust accordingly.
#################### Retrieve backup ARN id #################### aws backup list-recovery-points-by-backup-vault --backup-vault-name $VAULT_NAME # NOTE: Record the RecoveryPointArn that you wish to recover from # e.g. arn:aws:backup:ap-southeast-1:027763730008:recovery-point:0a82d94c-3d56-481d-98e3-b810d3df363b # To view the recovery point restore metadata aws backup get-recovery-point-restore-metadata \ --backup-vault-name $VAULT_NAME \ --recovery-point-arn <RECOVERY_POINT_ARN> #################### Restore from the backup #################### # Prerequisites: # 1) Generate an UUID, "uuidgen" (Mac) or "uuid -r" (Linux) # 2) Create a metadata json file, properties details are mentioned in # https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-efs.html#efs-restore-cli # NOTE: If newFileSystem=true, file-system-id parameter will be ignored. # 3) Substitute "CreationToken" value with the generated UUID. # 4) If existing file system is encrypted, you may use the existing KMS key. # # Example metadata json: # cat <<-EOF > /path/to/metadata_json_file # { # "file-system-id": "fs-6a1dcba2", # "Encrypted": "true", # "KmsKeyId": "arn:aws:kms:ap-southeast-1:027763730008:key/4859a845-3ef2-464d-80d2-16c1b2c58ff4", # "PerformanceMode": "generalPurpose", # "CreationToken": "944713C9-C6BB-42A4-AF91-E7DB5761FDBD", # "newFileSystem": "true" # } # EOF aws backup start-restore-job --recovery-point-arn <RECOVERY_POINT_ARN> --iam-role-arn "$BACKUP_ROLE_ARN" --metadata file:///path/to/metadata_json_file watch aws backup list-restore-jobs --by-resource-type EFS #################### Export new file system id #################### # If you recover as new file system (newFileSystem=true), please use command `aws efs describe-file-systems` to find out the new file system id. # After that export the new file system id env variable. export NEW_EFS_FILE_SYSTEM_ID="fs-xxxxxxxxxxxxxxxxx"; #################### Create mount targets for new file system #################### # Retrieve mount targets from existing file system and create the same to new file system. for mountTarget in $(jq -c '.[]' <<< $(aws efs describe-mount-targets --file-system-id $EFS_FILE_SYSTEM_ID --query "MountTargets[?MountTargetId!=null]")); do zoneName=$(jq -r '.AvailabilityZoneName' <<< $mountTarget); mountTargetId=$(jq -r '.MountTargetId' <<< $mountTarget); subnetId=$(jq -r '.SubnetId' <<< $mountTarget); securityGroup=$(aws efs describe-mount-target-security-groups --mount-target-id $mountTargetId --query "SecurityGroups" --output text) echo "Creating mount target for file system id $NEW_EFS_FILE_SYSTEM_ID on zone $zoneName." aws efs create-mount-target \ --file-system-id $NEW_EFS_FILE_SYSTEM_ID \ --subnet-id $subnetId \ --security-groups $securityGroup \ --no-cli-pager done #################### Create root path access point to manage recovered data #################### aws efs create-access-point \ --file-system-id $NEW_EFS_FILE_SYSTEM_ID \ --posix-user Uid=6000,Gid=6000 \ --root-directory 'Path="/",CreationInfo={OwnerUid=6000,OwnerGid=6000,Permissions="0755"}' #################### Create access point for application access #################### aws efs create-access-point \ --file-system-id $NEW_EFS_FILE_SYSTEM_ID \ --posix-user Uid=6000,Gid=6000 \ --root-directory 'Path="/uepe",CreationInfo={OwnerUid=6000,OwnerGid=6000,Permissions="0755"}' aws efs describe-access-points #################### Create a static persistent yaml #################### # NOTE: Update volumeHandle to your file system id and access points accordingly. # Example below are using two sets of PV and PVC that each of them corresponded to root path ("/") and application path ("/uepe"). # Use command `aws efs describe-access-points` to find out access point ids. cat > efs_uepe_persistent.yaml << EOF apiVersion: v1 kind: PersistentVolume metadata: name: root-persistent spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain storageClassName: aws-efs csi: driver: efs.csi.aws.com volumeHandle: fs-0faa7c3cdc681af41::fsap-08232180e9af33cab --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: root-persistent spec: volumeName: root-persistent storageClassName: aws-efs accessModes: - ReadWriteMany resources: requests: storage: 5Gi --- apiVersion: v1 kind: PersistentVolume metadata: name: platform-persistent spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain storageClassName: aws-efs csi: driver: efs.csi.aws.com volumeHandle: fs-0faa7c3cdc681af41::fsap-06ee3201e68a278cd --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: platform-persistent spec: volumeName: platform-persistent storageClassName: aws-efs accessModes: - ReadWriteMany resources: requests: storage: 5Gi EOF #################### Kubectl apply persistence yaml #################### kubectl apply -f efs_uepe_persistent.yaml #################### Create temporary pods to manage volume #################### cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: root-pv-pod spec: volumes: - name: root-persistent persistentVolumeClaim: claimName: root-persistent containers: - name: root-pv-container image: nginx volumeMounts: - mountPath: /root name: root-persistent --- apiVersion: v1 kind: Pod metadata: name: platform-pv-pod spec: volumes: - name: platform-persistent persistentVolumeClaim: claimName: platform-persistent containers: - name: platform-pv-container image: nginx volumeMounts: - mountPath: /uepe name: platform-persistent EOF #################### Locate and move up the application backup directory (uepe) #################### # Purpose of the below steps is to lift up restored folder to the root path, this is for allowing data to be accessible by the application access point. kubectl exec -ti root-pv-pod -- ls -al /root/ kubectl exec -ti root-pv-pod -- ls -al /root/aws-backup-restore_2024-06-17T07-36-15-412650687Z kubectl exec -ti root-pv-pod -- ls -al /root/aws-backup-restore_2024-06-17T07-36-15-412650687Z/uepe kubectl exec -ti root-pv-pod -- cp -rf /root/aws-backup-restore_2024-06-17T07-36-15-412650687Z/uepe /root/ kubectl exec -ti root-pv-pod -- ls -al /root/uepe #################### Verify restored data is visible by application mount point #################### kubectl exec -ti platform-pv-pod -- ls -al /uepe #################### Clean up unused pod, pv and pvc #################### kubectl delete pod root-pv-pod kubectl delete pod platform-pv-pod kubectl delete pvc root-persistent kubectl delete pv root-persistent #################### Helm install PE with existing claim #################### # The persistent volume has now been restored, you can install PE with the existing claim "platform-persistent".