Having When you have completed the preparations , it is now time you can proceed to install Usage Engine Private Edition.

Main Installation Example

In this main installation example, it is assumed that the following optional resources have been added while preparing for the installation (see Kubernetes Cluster Add-ons - OCI (4.2)):

• ingress-nginx-controller

• cert-manager

Example Certificate

Since cert-manager is being used to provide TLS to the Usage Engine Private Edition installation in this example, you need to create an issuer in order to generate the required certificate.

Here In this example, we are going to use an ACME issuer type that is configured to match the Kubernetes cluster that was set up previously in the Preparations - OCI (4.2) chapter:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: example-issuer
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    # You email: <yourmust replace this email address with ofyour choice>own.
    # NameLet's ofEncrypt awill secretuse usedthis to storecontact theyou ACME account about expiring
    # certificates, and issues related to your account.
    email: <your valid email address>
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
      - dns01:
          webhook:
            groupName: acme.d-n.be
            solverName: oci
            config:
              ociProfileSecretName: oci-profile

...

apiVersion: v1
kind: Secret
metadata:
  name: oci-profile
type: Opaque
stringData:
  tenancy: "your tenancy ocid"<tenancy_ocid from terraform output>
  user: "your user ocid" <user_ocid configured in terraform.tfvars>
  region: "your region" <region from terraform output>
  fingerprint: <fingerprint "yourconfigured key fingerprint"in terraform.tfvars>
  privateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    ...KEY DATA HERE...
    -----END RSA PRIVATE KEY-----
  privateKeyPassphrase: "private keys passphrase or empty string if none"
 

Create the secret prior to ClusterIssuer creation. To install secret oci-profile to cert-manager namespace run the following command:

kubectl apply -f oci-profile.yaml -n cert-manager

Assuming that the issuer spec above has been saved into a file called example-issuer.yaml, it can be created like thisyou can create it by running the following command:

kubectl apply -f example-issuer.yaml

Install Helm Chart

Although the number of helm value combinations to set is virtually endless, some values should more or less always be set.

So let’s start by creating a file called uepe-values.yaml, and in that file, specify a minimal set of values that will serve as a good starting point:

...

...

Load Balancer TLS Certificate

With ClusterIssuer setup properly, we can proceed to generate TLS Certificate and import into OCI Certificates Service.

To generate certificate, create a yaml file named certificate.yaml with the following contents:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lb-cert
spec:
  commonName: <cluster_dns_zone_name from terraform output>
  dnsNames:
    - <cluster_dns_zone_name from terraform output>
    - desktop-online.<cluster_dns_zone_name from terraform output>
    - platform.<cluster_dns_zone_name from terraform output>
    - ingress.<cluster_dns_zone_name from terraform output>
    - grafana.<cluster_dns_zone_name from terraform output>
  issuerRef:
    kind: ClusterIssuer
    name: example-issuer
  secretName: lb-cert

1. Execute the yaml file by running the following command:

kubectl apply -f certificate.yaml -n uepe

2. Wait for a while and confirm that the certificate has been generated successfully by running the following command:

kubectl get certificate -n uepe 

3. The output will show that the certificate named lbcert’s status is ready

NAME                domain: example-cluster.stratus.digitalroute.net       READY   SECRET kind: ClusterIssuer           name: example-issuer                 AGE
lb-cert           enabled: true     postgres:   adminUsername: dbadminTrue   host: example-cluster-db.c70g0ggo8m66.eu-west-1.rds.amazonaws.comlb-cert    port: 5432

Here follows information on how you can determine the values to set in your particular installation:

...

Value

...

Comment

...

aws.acm_certificate

...

This value should be set to match the certificate_arn listed in the terraform output produced in the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091598/Set+Up+Kubernetes+Cluster+-+OCI#Setup-Additional-Infrastructure-Resources-on-AWS section.

...

aws.ingress.serviceName

...

This is the name of the Kubernetes Service that was created adding the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091624/Kubernetes+Cluster+Add-ons+-+OCI#Ingress-NGINX-Controller.

...

global.domain

...

This value should be set to match the eks_domain_zone_name listed in the terraform output produced in the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091598/Set+Up+Kubernetes+Cluster+-+OCI#Setup-Additional-Infrastructure-Resources-on-AWS section.

...

global.imagePullSecrets

...

This is referencing an image pull secret containing the credentials required in order to pull container images from the Digital Route AWS ECR registry. If you are hosting the container images in your own container registry, depending on how that is configured, another image pull secret is probably needed. See https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/161481567/Common+Usage+Engine+Private+Edition+Preparations#Container-Images for additional information.

...

licenseKey

...

The license key that can be found in the licenseKey file that you have previously received (see the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/161481605/General+Pre-requisites#License section). 

...

log.format

...

If you need to use dedicated log collection and monitoring tools like Fluent-bit, Elasticsearch, Kibana or AWS CloudWatch for Usage Engine Private Edition, make sure the log format is configured to json. See https://infozone.atlassian.net/wiki/x/Q4BDD for additional information.

...

platform.tls.*

...

These values are set to use the example issuer created at the beginning of this chapter. This should only be seen as an example and the values should be adjusted according to the real world situation.

...

platform.tls.certManager.public.issuer.domain

...

Should be set to match the eks_domain_zone_name listed in the terraform output produced in the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091598/Set+Up+Kubernetes+Cluster+-+OCI#Setup-Additional-Infrastructure-Resources-on-AWS section.

...

platform.db.type

...

Set to match the RDS PostgreSQL service that was created in the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091598/Set+Up+Kubernetes+Cluster+-+OCI#Setup-Additional-Infrastructure-Resources-on-AWS section. If another database service is being used, the value must be adjusted accordingly.

...

postgres.adminUsername

...

Value is taken from the db_user listed in the terraform output produced in the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091598/Set+Up+Kubernetes+Cluster+-+OCI#Setup-Additional-Infrastructure-Resources-on-AWS section.

...

postgres.host

...

Value is taken from the first part of the db_endpoint listed in the terraform output produced in the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091598/Set+Up+Kubernetes+Cluster+-+OCI#Setup-Additional-Infrastructure-Resources-on-AWS section.

...

postgres.port

...

Value is taken from the second part of the db_endpoint listed in the terraform output produced in the https://infozone.atlassian.net/wiki/spaces/UEPE4D/pages/211091598/Set+Up+Kubernetes+Cluster+-+OCI#Setup-Additional-Infrastructure-Resources-on-AWS section.

The command below can be used to install Usage Engine Private Edition:

helm install uepe digitalroute/usage-engine-private-edition --version <version> -f uepe-values.yaml -n uepe

Where <version> is the version of Usage Engine Private Edition to install. For example 4.0.0.

Check that all pods are running and that all pod containers become ready (this may take a little while):

kubectl get pods -w                  
NAME                                                READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-8657757b7f-7dqgs       1/1     Running   0          7d13h
aws-load-balancer-controller-8657757b7f-h2b2m       1/1     Running   0          7d13h
desktop-online-7c54755c99-hd5zw                         46h

4. Extract the server certificate and CA certificate from secret lbcert by running the following command:

kubectl get secrets lb-cert -n uepe -o yaml | yq '.data' | grep "tls.crt" | awk -F : '{print $2}'| tr -d " "|base64 -d > tls.crt

5. Separate server certificate and CA certificate into two files by running the following command:

csplit tls.crt '/^-----BEGIN CERTIFICATE-----$/' 

6. Rename first generated file as server certificate file by running the following command:

mv xx00 tls.crt

7. Rename second generated file as CA certificate file by running the following command:

mv xx01 ca.crt

8. Extract the private key from secret lbcert by running the following command:

kubectl get secrets lb-cert -n uepe -o yaml | yq '.data' | grep "tls.key" | awk -F : '{print $2}'| tr -d " "|base64 -d > tls.key

The server certificate, CA certificate and private key are now stored in tls.crt, ca.crt and tls.key respectively. The next step is to import them into the OCI Certificates Service.

Import into OCI Certificates Service

Go to the OCI console management and search for Certificates service. On the Certificates service page, click Create Certificate and follow these steps:

1. Select Certificate Type Imported and key in a unique name.

2. Click Next to go to Certificate Configuration page.

3. Upload the tls.crt, ca.crt and tls.key files according to table below:

4. Click Next and proceed to Create Certificate.

5. Wait for the certificate to be created.

6. Copy and save the certificate’s ocid. This ocid will be set to the oci.certificates.id property in the helm chart value file in the next section.

TLS Backendset Secret

The SSL configuration between the load balancer and the backend servers (worker nodes) in the backend set is known as backend SSL. In this case, the backend set is referring to the Platform Pod on the worker nodes. To implement backend SSL, store the SSL certificates and private key in the form of a Kubernetes secret.

The CA certificate and private key generated from the previous section can be reused to generate the Kubernetes secret needed by the backend set.

To store the certificate and the private key as a secret in Kubernetes, run the following command:

kubectl create secret generic ca-ser-secret -n uepe --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=ca.crt=ca.crt

The backend set secret named ca-ser-secret has now been created in the namespace uepe.

Install Helm Chart

Although the number of helm value combinations to set is virtually endless, there are some values should practically always be set.

Start by creating a file called uepe-values.yaml, and in that file, specify a minimal set of values that will serve as a good starting point:

oci:
  certificates:
    enabled: false
  backendNSG: <backend_nsg from terraform output>
  healthcheck:
    desktoponline:
      port: 9001
    ingressnginx:
      port: 443  
environment: oci
global:
  domain: <cluster_dns_zone_name from terraform output>
  ingressController:
    serviceName: ingress-nginx-controller
  imagePullSecrets:
  - name: ecr-cred  
licenseKey: <insert-your-license-key-string-here>
log:
  format: json
platform:
  db:
    type: postgresql
  tls:
    cert:
      public: certManager
    certManager:
      public:
        issuer:
          kind: ClusterIssuer
        1/1  name: example-issuer
 Running   0enabled: true    
postgres:
  adminUsername: 60s
efs-csi-controller-77c44b5fc7-6cjqtpostgres
  host: <db_endpoint from terraform output>
  port: <db_port from terraform output>
persistence:
  3/3enabled: true
   Running   0          7d13h
efs-csi-controller-77c44b5fc7-qjqx8                 3/3     Running   0          7d13h
efs-csi-node-5tcmt                   existingClaim: fss-pvc

Below you can find information on how you can determine the values to set in your particular installation:

You can use the following command to install Usage Engine Private Edition:

helm install uepe digitalroute/usage-engine-private-edition --version <version> -f uepe-values.yaml -n uepe

Where <version> is the version of Usage Engine Private Edition to install, for example 4.0.0.

Check that all pods are running and that all pod containers become ready by running the following command:

kubectl get pods -w -n uepe               3/3   
NAME Running   0          7d13h efs-csi-node-c9kfm                                 READY 3/3   STATUS  Running  RESTARTS 0          7d13h
efs-csi-node-zbwzc          AGE
desktop-online-5fdd4df85b-5hc6z                        3/31/1     Running   0          7d13h97m
external-dns-78d56d8b7454fb5cb46b-r257g4lfld                       1/1     Running   0          7d13h27h
ingress-nginx-controller-7c5cb64567477648b4c-2gjmj sz2nw           1/1     Running   0          5h37m
platform-0                                27h
oci-native-ingress-controller-6cd8cf8d79-dz8zp          1/1     Running   0          60s29h
uepe-operator-controller-manager-86b758f558-2t94rplatform-0   2/2     Running   0          60s uepe-operator-controller-manager-86b758f558-c92s7   2/2     Running   0          60s

To get the Desktop Online web user interface hostname:

kubectl get ingress -n uepe

The output shows FQDN hostname, IP address and port to access desktop online web user interface.

NAMESPACE   NAME  1/1     Running   0        CLASS   HOSTS97m
uepe-operator-controller-manager-69c4b499c8-h9l8w   2/2     Running   0          97m
uepe-operator-controller-manager-69c4b499c8-hxdcb   2/2     Running   0          97m

This may take a little while

To get the Desktop Online web user interface hostname run the following command:

kubectl get ingress -n uepe

The output shows the FQDN hostname, IP address and port to access the desktop online user interface.

NAME                       CLASS                      HOSTS          ADDRESS                                                   ADDRESS           PORTS   AGE
uepedesktop-online        ingress-alb   alb  native-ic-ingress-class   desktop-online.example-cluster.stratus.oci.digitalroute.net,   130.162.252.220   80      99m
ingress-nginx-controller   native-ic-ingress-class   ingress.example-cluster.stratus.oci.digitalroute.net          k8s-uepe-ingressa-bc9e668f78-186509862.eu-west-1.elb.amazonaws.com130.162.252.220   80      14d

...

99m

You should now be able to access the Desktop Online user interface should now be accessible at:
https://desktop-online.example-cluster.stratus.oci.digitalroute.net/
Note that it It may take a little while before the DNS record gets registered..

Other Common Installation Configurations

Here follows Bellow are a few common installation configurations for the Usage Engine Private Edition helm chart.

They should be seen as variations to the main installation example outlined above.

Persistent File Storage

If you have chosen selected to prepare for persistent file storage, by installing the efs-csi-controller resource in the Kubernetes Cluster Add-ons - OCI chapter, there are two different ways of configuring your Usage Engine Private Edition installation to use it.

Use Bundled

...

OCI Specific PVC

Specifically for AWSOCI, the Usage Engine Private Edition helm chart contains a bundled persistent volume claim. This persistent volume claim is using the awsfss-dyn-efsstorage storage class. To enable it, simply set the following helm values:

...

Where the persistence.bundledClaim.storageRequest value is used to control the size of the requested storage (default is 1Gi).

Use a command like this You can use the following command to inspect the persistent volume claim that gets created as a result of setting the above helm values:

kubectl get persistentvolumeclaims mz-bundled-pvc -o yaml

Reference Arbitrary PVC

Usage Engine Private Edition can be configured to reference an arbitrary persistent volume claim by setting the following helm values:

...

In this example, my-pvc is an arbitrary persistent volume claim that you have created beforehandbefore hand.