/
Upgrade Instructions - AWS (4.2)

Upgrade Instructions - AWS (4.2)

Note!

The instructions for backup and upgrade of the database below are only relevant if you are using RDS as platform database. If the platform database used is derby, the backup of the EFS covers the database as well (assuming persistent storage of the platform is enabled).

  1. List the databases and locate the one used for Usage Engine with this command:

    aws rds describe-db-instances --query 'DBInstances[].DBInstanceIdentifier[]'

  2. Perform a backup of the RDS database with this command:

    aws rds create-db-snapshot --db-snapshot-identifier <database backup name> --db-instance-identifier <database instance name>

    for example:

    aws rds create-db-snapshot --db-snapshot-identifier uepe-eks-db-postgresql-backup --db-instance-identifier uepe-eks-db-postgresql

  3. Check if the backup was created successfully by running this command:

    aws rds describe-db-snapshots --snapshot-type manual --db-snapshot-identifier <database backup name>

It is now time to do a backup of the file system used.

Note!

If there are standalone ECs that are still running and writing their logs to the same EFS, whatever happens after the backup has been initiated will not be included in the backup.

To create an EFS backup using the console, see https://docs.aws.amazon.com/aws-backup/latest/devguide/recov-point-create-on-demand-backup.html for instructions.

The section below contains an example of how to run an on-demand backup job using the command line. The snapshot will in this case be stored under the default backup vault.

export EFS_NAME=uepe-eks-efs-disk export EFS_FILE_SYSTEM_ID=$(aws efs describe-file-systems --query "FileSystems[?Name==\`$EFS_NAME\`].FileSystemId" --output text) export EFS_ARN=$(aws efs describe-file-systems --query "FileSystems[?Name==\`$EFS_NAME\`].FileSystemArn" --output text) export VAULT_NAME=Default export BACKUP_ROLE_ARN=$(aws iam get-role --role-name AWSBackupDefaultServiceRole --query "Role.Arn" --output text) # Run on demand backup job aws backup start-backup-job \ --backup-vault-name $VAULT_NAME \ --resource-arn $EFS_ARN \ --iam-role-arn $BACKUP_ROLE_ARN # View backup job status aws backup list-backup-jobs --by-resource-type EFS

 

Restore database backup

If restoring becomes necessary, you can restore the DB instance from a snapshot backup, see the AWS guide https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html for more information.

You can also restore a new DB instance using the commands below:

export EXISTING_DB=uepe-eks-db-postgresql export NEW_DB=uepe-eks-db-postgresql-2 export SNAPSHOT=uepe-eks-db-postgresql-backup export INSTANCE_CLASS=db.t3.small export SUBNET_GROUP_NAME=$(aws rds describe-db-instances --query "DBInstances[?DBInstanceIdentifier==\`$EXISTING_DB\`].DBSubnetGroup.DBSubnetGroupName" --output text) export SECURITY_GROUP_ID=$(aws rds describe-db-instances --query "DBInstances[?DBInstanceIdentifier==\`$EXISTING_DB\`].VpcSecurityGroups[].VpcSecurityGroupId" --output text) # Restore snapshot to a new database aws rds restore-db-instance-from-db-snapshot \ --db-instance-identifier $NEW_DB \ --db-snapshot-identifier $SNAPSHOT \ --db-instance-class $INSTANCE_CLASS \ --db-subnet-group-name $SUBNET_GROUP_NAME \ --vpc-security-group-ids=$SECURITY_GROUP_ID # Rename existing DB instance to other name aws rds modify-db-instance \ --db-instance-identifier $EXISTING_DB \ --new-db-instance-identifier $EXISTING_DB-old \ --apply-immediately # Rename the new DB instance to use existing identifier name aws rds modify-db-instance \ --db-instance-identifier $NEW_DB \ --new-db-instance-identifier $EXISTING_DB \ --apply-immediately

If you are using the console to do the RDS restore, remember to include the existing database security group so that it can be accessible by the cluster.

image-20240530-190733.png

Note!

The restored RDS instance is a new database instance and is not managed by Terraform. If you plan to destroy the cluster later, ensure that the new database instance is deleted first. This is necessary because the database instance may still reference to the RDS subnet group.

Restore file system snapshot

To restore EFS, follow the instructions in https://docs.aws.amazon.com/aws-backup/latest/devguide/restore-resource.html and https://repost.aws/knowledge-center/aws-backup-restore-efs-file-system-cli.

The section below contains an example of how to restore the EFS backup using the command line. In this example the volume mount is using access point path /uepe, and the snapshot is stored under default vault, and then the backup is restored to the existing file system.

export EFS_NAME=uepe-eks-efs-disk export EFS_FILE_SYSTEM_ID=$(aws efs describe-file-systems --query "FileSystems[?Name==\`$EFS_NAME\`].FileSystemId" --output text) export EFS_ARN=$(aws efs describe-file-systems --query "FileSystems[?Name==\`$EFS_NAME\`].FileSystemArn" --output text) export VAULT_NAME=Default export BACKUP_ROLE_ARN=$(aws iam get-role --role-name AWSBackupDefaultServiceRole --query "Role.Arn" --output text) #################### Retrieve backup ARN id #################### aws backup list-recovery-points-by-backup-vault --backup-vault-name $VAULT_NAME # NOTE: Record the RecoveryPointArn that you wish to recover from # e.g. arn:aws:backup:ap-southeast-1:027763730008:recovery-point:0a82d94c-3d56-481d-98e3-b810d3df363b # To view the recovery point restore metadata aws backup get-recovery-point-restore-metadata \ --backup-vault-name $VAULT_NAME \ --recovery-point-arn <RECOVERY_POINT_ARN> #################### Restore from the backup #################### # Prerequisites: # 1) Generate an UUID, "uuidgen" (Mac) or "uuid -r" (Linux) # 2) Create a metadata json file, properties details are mentioned in # https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-efs.html#efs-restore-cli # NOTE: If newFileSystem=true, file-system-id parameter will be ignored. # 3) Substitute "CreationToken" value with the generated UUID. # 4) If existing file system is encrypted, you may use the existing KMS key. # # Example metadata json: cat <<-EOF > /path/to/metadata_json_file { "file-system-id": "$EFS_FILE_SYSTEM_ID", "Encrypted": "true", "KmsKeyId": "arn:aws:kms:ap-southeast-1:027763730008:key/4859a845-3ef2-464d-80d2-16c1b2c58ff4", "PerformanceMode": "generalPurpose", "CreationToken": "FEC83B16-F43A-4D5A-A678-2D27FC6C7DBD", "newFileSystem": "false" } EOF aws backup start-restore-job --recovery-point-arn <RECOVERY_POINT_ARN> --iam-role-arn "$BACKUP_ROLE_ARN" --metadata file:///path/to/metadata_json_file watch aws backup list-restore-jobs --by-resource-type EFS #################### Run a pod with command prompt #################### kubectl run nfscli --rm --tty -i --restart='Never' --namespace uepe --image oraclelinux:8 --privileged=true --command -- bash #################### Install NFS client #################### [root@nfscli /]# yum -y install nfs-utils #################### Make a folder for mounting purpose #################### [root@nfscli /]# mkdir -p /mnt/efs #################### Mount EFS volume root path #################### # EFS DNS name in format <file-system-id>.efs.<aws-region>.amazonaws.com [root@nfscli /]# mount -o nolock fs-0a3a60103ae00a5a1.efs.ap-southeast-1.amazonaws.com:/ /mnt/efs #################### Locate the restored directory #################### # Go to the mounted directory [root@nfscli /]# cd /mnt/efs/ # List folders # NOTE: Existing platform volume mount folder is 'uepe' folder [root@nfscli efs]# ls -al total 16 drwxr-xr-x 5 root root 6144 Aug 13 06:35 . drwxr-xr-x 1 root root 18 Aug 14 10:37 .. drwxr-xr-x 5 root root 6144 Aug 13 06:35 aws-backup-restore_2024-08-13T17-58-42-978741167Z drwxr-xr-x 9 6000 6000 6144 Aug 13 18:47 uepe # The restored data folder which is also called 'uepe', it is located under aws-backup-restore_<timestamp> folder. [root@nfscli efs]# ls -al aws-backup-restore_2024-08-13T17-58-42-978741167Z/ total 20 drwxr-xr-x 5 root root 6144 Aug 13 06:35 . drwxr-xr-x 5 root root 6144 Aug 13 06:35 .. drw--w---- 2 root root 6144 Aug 13 17:58 aws-backup-lost+found_2024-08-13T17-58-13-086602146Z drwxr-xr-x 2 6000 6000 6144 Aug 13 18:57 uepe #################### Cleanup existing platform volume mount folder #################### [root@nfscli efs]# rm -rf uepe/* #################### Copy restored data to platform volume mount folder #################### # NOTE: Specify '-p' flag in the cp commmand to preserve file permissions and timestamp. [root@nfscli efs]# cp -rfp aws-backup-restore_2024-08-13T17-58-42-978741167Z/uepe/* uepe/ # Check if all datas are copied [root@nfscli efs]# ls -al uepe/ total 48 drwxr-xr-x 9 6000 6000 6144 Aug 13 18:47 . drwxr-xr-x 5 root root 6144 Aug 13 06:35 .. drwxr-xr-x 2 6000 6000 6144 Aug 13 06:37 3pp drwxr-xr-x 2 6000 6000 6144 Aug 13 06:37 backup drwxr-xr-x 2 6000 6000 6144 Aug 13 06:37 jni drwxr-xr-x 2 6000 6000 6144 Aug 13 06:37 keys drwxr-xr-x 5 6000 6000 6144 Aug 13 17:13 log drwxr-xr-x 3 6000 6000 6144 Aug 13 06:37 pico-cache drwxr-xr-x 2 6000 6000 6144 Aug 13 06:37 storage #################### Clean up the redundant restored data #################### [root@nfscli efs]# rm -rf aws-backup-restore_2024-08-13T17-58-42-978741167Z/uepe/* #################### Unmount volume and exit pod #################### [root@nfscli efs]# umount /mnt/efs/ [root@nfscli efs]# exit #################### Restore completed #################### # Backup data has been restored, proceed to the next section to rollback UEPE.