Before installing Usage Engine Private Edition, you need to set up a Kubernetes cluster on OCI OKE (Oracle’s managed Kubernetes service).

First a basic Kubernetes cluster needs to be created. This can be done in two different ways:

Using the terraform tool.
Using the OCI management console.

In this guide, terraform will be used. Mainly because it will enable you to create the basic Kubernetes cluster in minutes with just a single command.

Once the basic Kubernetes cluster has been created, additional infrastructure needs to be added. For this terraform is also used.

Before proceeding, go to Release Information, and download the oci.tar.gz file for the Usage Engine Private Edition version that is being installed. Once downloaded, extract its content to a suitable location.

Assumptions

There are a few assumptions been made when using terraform to create cluster resources:

We assume you have an existing parent domain i.e. example.com hosted on the same account as the cluster that we going to create in the coming section and you wish to access the cluster environment through the hostname. Terraform will create a subdomain in format <cluster_name>.<domain>.
1. cluster name: uepe-oke
2. domain: example.com
3. final domain: uepe-oke.example.com
In addition, we also assume terraform is allowed to add a NS (NameServer) record to the parent domain. This is to allow DNS delegation from the parent domain to subdomain.
Terraform needs to persist the state of your provisioned infrastructure, by default the state file is stored locally on the computer that terraform is executed from. However if you have multiple person working on the infrastructure then it is recommended to store the state file on remote persistent such as Object Storage, see https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformUsingObjectStore.htm for more information.
We use the OCI File System service (NFS) as the default persistent storage for data needs to be persisted.
We use the OCI Managed PostgreSQL service for Usage Engine Private Edition database.
User Principle is used through out the entire installation. User must get ready with the private key file locally. User may create and download the private key via OCI console, through Profile | My Profile | API keys | Add API key.

Create Basic Cluster and additional infrastructure

The following steps explains how to create a basic Kubernetes cluster with public and private VPC:

Go to <the location where you extracted the oci.tar.gz file>/oci/terraform and copy theterraform.tfvars.example to terraform.tfvars.
Edit the terraform.tfvars file.
Specify the desired cluster name, OCI region and kubernetes_version (please refer to the Compatibility Matrix (4.1) to find out which Kubernetes versions that are compatible with this release of Usage Engine Private Edition). Also specify your OCI tenancy_ocid, user_ocid, fingerprint, compartment_ocid and private_key_path (which can be found on the OCI dashboard’s Profile page), as well as the desired number of nodes per cluster (oke_num_nodes).
If you will be running with a database other than Derby also specify db_password, db_version and db_username.

terraform.tfvars	Where to get the value from?
`tenancy_ocid`	In the OCI management console, this is listed on `Profile \| Tenancy: <tenant-name> \| Tenancy Details`.
`fingerprint`	Fingerprint only available after user created the API keys. Refer to `private_key_path` row in the same table. In the OCI management console, this is listed on `Profile \| My Profile \| Resources \| API keys` after API keys being created.
`user_ocid`	In the OCI management console, this is listed on `Profile \| My Profile`
`private_key_path`	The full path to your private key file’s filename. To create and download your private key, go to `Profile \| My Profile \| Resources \| API keys`, create your API key and click download
`region`	The region in which you will install your cluster. (for example `"eu-frankfurt-1"`)
`cluster_name`	A name for your cluster. Cluster names must start with a lowercase letter followed by up to 39 lowercase letters, numbers or hyphens. They can't end with a hyphen. The cluster name must be unique in the project.
`domain`	Your existing domain name. In the OCI management console, this is the DNS name that is listed on page `Networking \|DNS management \| Zones`. The service hostname that created by Usage Engine Private Edition will be accessible in format `<service_name>.<cluster_name>.<domain>` i.e. desktop-online.uepe-oke.example.com.
`kubernetes_version`	version for kubernetes in alpha numeric string (for example “`v1.29.1`").
`oke_num_nodes`	Number of cluster nodes in numeric (for example “`3`”).
`oke_availability_domain`	Availability domain name for the cluster. (for example `"Vafx:EU-FRANKFURT-1-AD-1"`)
`db_password`	Choose a secure password for the system database administrator. Minimum 10 characters.
`db_version`	Database version in numeric string (for example “`14`“)
`oke_image_id`	OCID of the image to be used for worker node instance creation. To find out available image under your compartment, use command `oci ce node-pool-options get --node-pool-option-id all --compartment-id <your compartment ocid>` .
`db_enabled`	Boolean flag to enable cloud SQL database resource creation.
`fss_enabled`	Boolean flag to enable file storage resource creation. It is false by default. Set to true if persistent file storage is needed.
`auto_create_ns_record`	Boolean flag to enable subdomain NS record auto creation in parent domain. In case your parent domain is not under the same compartment or your parent domain is hosted in another cloud provider, then you must set it to false.

Example:

#  ____  _____ _____   _____ _   _ _____ ____  _____
# / ___|| ____|_   _| |_   _| | | | ____/ ___|| ____|_
# \___ \|  _|   | |     | | | |_| |  _| \___ \|  _| (_)
#  ___) | |___  | |     | | |  _  | |___ ___) | |___ _
# |____/|_____| |_|     |_| |_| |_|_____|____/|_____(_)

# The below values must be set explicitly in order for the setup to work correctly.

tenancy_ocid     = "ocid1.tenancy.oc1..aaaaaaaamnl7f7t2yrlas2si7b5hpo6t23dqi6mjo3eot6ijl2nqcog5h6ha"
fingerprint      = "7d:67:b3:9d:a3:8f:6d:37:f3:e9:7d:e5:45:ec:df:56"
user_ocid        = "ocid1.user.oc1..aaaaaaaauhk3uhiryg7sw2xjmvf45zasduqwr2cium53gmdxwipe4iqdrfuq"
private_key_path = "/Users/kamheng.choy/Downloads/kamheng.choy@digitalroute.com_2024-04-07T10_07_56.490Z.pem"

# Deployment compartment
compartment_ocid = "ocid1.compartment.oc1..aaaaaaaa56wmblidgvvicamsqkf7sqcqu5yxdhvu3wlvomzgonhflcrv6kcq"

# region
region = "eu-frankfurt-1"

# Name of the cluster, it must be unique in the project.
cluster_name = "test-uepe-cluster-1"

# Domain DNS name
# We'll create a subdomain zone from parent domain, the final domain will be in format "<cluster_name>.<domain>".
# Please note that if this domain is hosted on another OCI project or other cloud provider, then you must
# set auto_create_ns_record = false and manually add the subdomain NS record to the parent domain.
# auto_create_ns_record = false
domain = "stratus.oci.digitalroute.net"

# Admin user password to the database
db_password = "super_SeCrEt_db_pAsSwOrD_457!"

#  _______        _______    _    _  __    _    ____  _     _____
# |_   _\ \      / / ____|  / \  | |/ /   / \  | __ )| |   | ____|_
#   | |  \ \ /\ / /|  _|   / _ \ | ' /   / _ \ |  _ \| |   |  _| (_)
#   | |   \ V  V / | |___ / ___ \| . \  / ___ \| |_) | |___| |___ _
#   |_|    \_/\_/  |_____/_/   \_\_|\_\/_/   \_\____/|_____|_____(_)

# The below sections are the default values, tweak them to your needs.

# Kubernetes version
kubernetes_version = "v1.29.1"

# Number of nodes per cluster
oke_num_nodes = 3
# Worker node machine type
node_pool_shape = "VM.Standard.E4.Flex"
oke_availability_domain = "Vafx:EU-FRANKFURT-1-AD-1"

oke_image_id = "ocid1.image.oc1.eu-frankfurt-1.aaaaaaaapwbqurbd2hpmj2at354r3dkok4o4644am4hwgdagoekpcaon7shq"

# IP CIDR range allocate to the control plane
vcn_cidr_blocks = "10.0.0.0/16"

# Network file system (NFS) persistent storage
fss_enabled = true
filestore_availability_domain = "Vafx:EU-FRANKFURT-1-AD-1"

# Cloud SQL database
db_enabled = true
# DB instance type
db_instance_shape = "PostgreSQL.VM.Standard.E4.Flex.4.64GB"
# DB version
db_version = "14"

Important notes if your parent domain zone is not under the same project:

You need to set auto_create_ns_record = false to disable subdomain NS record auto creation in the parent domain.
Perform terraform apply.
After terraform apply is finished, copy the name servers value from terraform output and manually add them to parent domain as NS record. If you are not using OCI DNS as the parent domain, please refer to your Domain Registrar documentation on how to add NS record.

Run the following commands

terraform init
terraform plan
terraform apply

Wait for the terraform commands to finish.

Apply complete! Resources: 35 added, 0 changed, 0 destroyed.

Outputs:

backend_nsg = "ocid1.networksecuritygroup.oc1.eu-frankfurt-1.aaaaaaaacreo4kf5kd2n7nk4fn2kcsuv6kye2noowhpjypcmrqmms32gpg3a"
cluster_dns_zone_name = "test-uepe-cluster-1.stratus.oci.digitalroute.net"
cluster_dns_zone_name_servers = [
  "ns1.p201.dns.oraclecloud.net.",
  "ns2.p201.dns.oraclecloud.net.",
  "ns3.p201.dns.oraclecloud.net.",
  "ns4.p201.dns.oraclecloud.net.",
]
cluster_dns_zone_ocid = "ocid1.dns-zone.oc1..aaaaaaaacd5nsfzmir3efo5e2pcuga4t622vcxcqkc3ezizl64e5gofo7dza"
cluster_name = "test-uepe-cluster-1"
cluster_ocid = "ocid1.cluster.oc1.eu-frankfurt-1.aaaaaaaaerg6ctgepnuaipifispmuweqi5nvfhswxpu3luuctcvitslu3fea"
compartment_ocid = "ocid1.compartment.oc1..aaaaaaaa56wmblidgvvicamsqkf7sqcqu5yxdhvu3wlvomzgonhflcrv6kcq"
db_admin_user = "postgres"
db_endpoint = "db5j5pt3qwjqmmjgfremgugr7cxtsq-dbinstance-70c946d1330e.postgresql.eu-frankfurt-1.oc1.oraclecloud.com"
db_port = 5432
filesystem_mount_path = "/uepe"
filesystem_ocid = "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaais2zcnmzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa"
kms_key_ocid = ""
loadbalancer_ocid = "ocid1.loadbalancer.oc1.eu-frankfurt-1.aaaaaaaanmx4u2yllufrjetacqt5bsgiyznkg7fif3bjfl36xoduyngesvra"
loadbalancer_subnet_ocid = "ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaapyqsowgik7gak3wkihsm3jtronnc5klbf46jerjnudrqsnlbco5q"
mount_target_IP_address = "10.0.4.212"
mount_target_subnet_ocid = "ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaaoh36ywx4rki7qtre33f53amjy2zylm6mnqeix6cydn5ul4shfqja"
region = "eu-frankfurt-1"
tenancy_ocid = "ocid1.tenancy.oc1..aaaaaaaamnl7f7t2yrlas2si7b5hpo6t23dqi6mjo3eot6ijl2nqcog5h6ha"

Make sure to save the output from terraform above. Reason being that it is used as input throughout the remainder of this installation guide.

A basic Kubernetes cluster has now been set up successfully.

A RDS PostgreSQL database instance up and running on private subnet VPC with default listening port 5432. The default database postgres is accessible within the cluster at end point db5j5pt3qwjqmmjgfremgugr7cxtsq-dbinstance-70c946d1330e.postgresql.eu-frankfurt-1.oc1.oraclecloud.com with admin username postgres.

You can check the status of the cluster, db and the other resources in the OCI dashboard.

Configure Cluster Access

oci ce cluster create-kubeconfig --cluster-id <cluster ocid> --file ./kubeconfig.yaml --region eu-frankfurt-1 --token-version 2.0.0  --kube-endpoint PUBLIC_ENDPOINT

The above oci command will generate a ./kubeconfig.yaml file containing information on how to connect to your newly created cluster. Make sure to set the KUBECONFIG environment variable to point to that file:

export KUBECONFIG=<full path to ./kubeconfig.yaml>

This will ensure that tools like kubectl and helm will connect to your newly created cluster.

You can check the status of the cluster nodes like this:

kubectl get nodes

For this example cluster the output will looks something like this:

NAME         STATUS   ROLES   AGE   VERSION
10.0.2.111   Ready    node    27h   v1.29.1
10.0.2.158   Ready    node    27h   v1.29.1
10.0.2.230   Ready    node    27h   v1.29.1

Namespace

Create a namespace called uepe:

kubectl create namespace uepe

Unless explicitly stated, this is the namespace that is used throughout the remainder of this installation guide.

Hint!

You can also create and use a namespace with another name.

This command shows all namespaces that currently exist in your cluster:

kubectl get namespaces

Now proceed to the Kubernetes Cluster Add-ons - OCI (4.2) section.

Set Up Kubernetes Cluster - OCI (4.2)

Assumptions

Create Basic Cluster and additional infrastructure

Configure Cluster Access

Namespace