Kubernetes Cluster Add-ons - OCI

OCI Add-ons

The following OCI specific resources should be added:

• oci-file-service-storage

• oci-native-ingress-controller

oci-file-service-storage

The OCI File Storage service provides a durable, scalable, distributed, enterprise-grade network file system.

A persistent volume claim (PVC) is a request for persistent file storage. The OCI File Storage service file systems are mounted inside containers running on clusters created by Container Engine for Kubernetes using a CSI (Container Storage Interface) volume plugin deployed on the clusters.

To enable the CSI volume plugin to create and manage File Storage resources, appropriate IAM policies must be installed:

1. Policy to create and/or manage file systems, mount targets, and export paths:

ALLOW any-user to manage file-family in compartment <compartment-name> where request.principal.type = 'cluster'

2. Policy to use VNICs, private IPs, private DNS zones, and subnets:

ALLOW any-user to use virtual-network-family in compartment <compartment-name> where request.principal.type = 'cluster'

3. Policy to enable the CSI volume plugin to access that master encryption key:

Allow service FssOc1Prod to use keys in compartment <compartment-name> where target.key.id = '<key_OCID>'

Allow any-user to use key-delegates in compartment <compartment-name> where ALL {request.principal.type = 'cluster', target.key.id = '<key_OCID>'}

Where <compartment-name> and <key_OCID> can be retrieved from the console

Dynamic Provisioning

These steps describe how to create a dynamically provisioned volume created through OCI File Storage access points and a corresponding persistent volume claim (PVC).

1. Prepare a storageclass.yaml file with StorageClass manifest for OCI File Storage:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fss-dyn-storage
provisioner: fss.csi.oraclecloud.com
parameters:
  availabilityDomain: <availability_Domain>
  mountTargetSubnetOcid: <mountTarget_Subnet_Ocid>
  kmsKeyOcid: <key_Ocid>

2. Deploy the storage class

kubectl apply -f storageclass.yaml

For more information, please refer to the dynamic provisioning documentation.

oci-native-ingress-controller

The OCI native ingress controller implements the rules and configuration options defined in a Kubernetes ingress resource to load balance and route incoming traffic to service pods running on worker nodes in a cluster. The OCI native ingress controller creates an OCI flexible load balancer to handle requests, and configures the OCI load balancer to route requests according to the rules defined in the ingress resource.

The OCI Native Ingress controller creates the following OCI load balancer resources:

• A load balancer for each IngressClass resource where you have specified the OCI native ingress controller as the controller.

• A load balancer backend set for each unique Kubernetes service name and port number combination that you include in routing rules in Ingress resources in the cluster.

• A routing policy that reflect the rules defined in the ingress resource, that is used to route traffic to backend set.

• A load balancer listener for each unique port that you include in routing rules in Ingress resources in the cluster

To install OCI Native Ingress Controller, follow these steps:

1. Create a config file named user-auth-config.yaml, containing credential information, in the following format:

auth:
  region: <region-identifier>
  user: <user-ocid>
  fingerprint: <fingerprint>
  tenancy: <tenancy-ocid>

2. Create a Kubernetes secret resource in the cluster by entering:

kubectl create secret generic <secret-name> \
--from-file=config=user-auth-config.yaml \
--from-file=private-key=<private-key-file-path>.pem \
--namespace uepe

3. Grant permission to the OCI Native Ingress Controller to access resources created by other OCI services, such as the Load Balancer service and the Certificates service. Hence, these IAM policies must be installed.

Allow group <group-name> to manage load-balancers in compartment <compartment-name>
Allow group <group-name> to use virtual-network-family in compartment <compartment-name>
Allow group <group-name> to manage cabundles in compartment <compartment-name>
Allow group <group-name> to manage cabundle-associations in compartment <compartment-name>
Allow group <group-name> to manage leaf-certificates in compartment <compartment-name>
Allow group <group-name> to read leaf-certificate-bundles in compartment <compartment-name>
Allow group <group-name> to manage certificate-associations in compartment <compartment-name>
Allow group <group-name> to read certificate-authorities in compartment <compartment-name>
Allow group <group-name> to manage certificate-authority-associations in compartment <compartment-name>
Allow group <group-name> to read certificate-authority-bundles in compartment <compartment-name>
Allow group <group-name> to read cluster-family in compartment <compartment-name>

ALLOW any-user to manage network-security-groups in <compartment-name> Team-Stratus where request.principal.type = 'cluster'
ALLOW any-user to manage vcns in compartment <compartment-name> where request.principal.type = 'cluster'
ALLOW any-user to manage virtual-network-family in compartment <compartment-name> where request.principal.type = 'cluster'

Allow group <group-name> to inspect certificate-authority-family in compartment <compartment-name>
Allow group <group-name> to use certificate-authority-delegate in compartment <compartment-name>
Allow group <group-name> to manage leaf-certificate-family in compartment <compartment-name>
Allow group <group-name> to use leaf-certificate-family in compartment <compartment-name>
Allow group <group-name> to use certificate-authority-delegate in compartment <compartment-name>
Allow group <group-name> to manage certificate-associations in compartment <compartment-name>
Allow group <group-name> to inspect certificate-authority-associations in compartment <compartment-name>
Allow group <group-name> to manage cabundle-associations in compartment <compartment-name>

4. Clone the OCI native ingress controller repository from GitHub

git clone https://github.com/oracle/oci-native-ingress-controller

5. In the local Git repository, navigate to the oci-native-ingress-controller directory and create a config file named oci-native-ingress-controller-values.yaml with the following content:

compartment_id: <ocid of compartment>
subnet_id: <ocid of load balancer's subnet>
cluster_id: <ocid of the cluster>
authType: user
deploymentNamespace: uepe

6. Generate the manifest .yaml files for the required resources

helm template --include-crds oci-native-ingress-controller helm/oci-native-ingress-controller -f oci-native-ingress-controller-values.yaml --output-dir deploy/manifests

7. Deploy the required resources using the manifest .yaml files

kubectl apply -f deploy/manifests/oci-native-ingress-controller/crds

kubectl apply -f deploy/manifests/oci-native-ingress-controller/templates

8. Confirm that OCI native ingress controller has been installed successfully

kubectl get pods -n uepe

Kubernetes Add-ons

The following general Kubernetes resources should be added:

• external-dns

• ingress-nginx-controller

external-dns

ExternalDNS is a Kubernetes add-on that configures public DNS servers with information about exposed Kubernetes services to make them discoverable.

To install ExternalDNS, follow these steps:

1. Add the bitnami helm repository:

   helm repo add bitnami https://charts.bitnami.com/bitnami

2. Update the helm repository to get the latest software:

   helm repo update

3. Create a file called external-dns-values.yaml and populate it with the following helm values:

   aws:
     zoneType: public
   domainFilters:
     - <eks_domain_zone_name from terraform output>
   policy: sync
   provider: aws
   txtOwnerId: <eks_domain_zone_id from terraform output>
   serviceAccount:
     create: false
     name: external-dns

4. Install the ExternalDNS helm chart:

   helm install external-dns bitnami/external-dns -n uepe \
   --version <helm chart version> -f external-dns-values.yaml

   Where <helm chart version> is a compatible version listed in the Compatibility Matrix (4.1).

ingress-nginx-controller

The Ingress NGINX Controller is an ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer.

To install the Ingress NGINX Controller, follow these steps:

1. Add the ingress-nginx helm repository:

   helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

2. Update the helm repository to get the latest software:

   helm repo update

3. Create a file called ingress-nginx-values.yaml and populate it with the following helm values:

   controller:
     scope:
       enabled: true
     admissionWebhooks:
       enabled: false
     metrics:
       enabled: false
       serviceMonitor:
         enabled: false
     ingressClassResource:
       name: nginx
       enabled: true
       default: false
       controllerValue: "k8s.io/ingress-nginx"
     watchIngressWithoutClass: false
     service:
       targetPorts:
         http: 80
         https: 443
       type: NodePort
     extraArgs:
       v: 1
   serviceAccount:
     create: false

4. Install the ingress-nginx-controller helm chart:

   helm install ingress-nginx ingress-nginx/ingress-nginx --version <helm chart version> -f ingress-nginx-values.yaml -n uepe

   Where <helm chart version> is a compatible version listed in the Compatibility Matrix (4.1).

Executing helm list should show all add-ons added in this section. Example:

NAME                        	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                             	APP VERSION
aws-efs-csi-driver          	uepe     	1       	2024-02-06 14:00:36.817518 +0800 +08	deployed	aws-efs-csi-driver-2.5.4          	1.7.4      
aws-load-balancer-controller	uepe     	1       	2024-02-06 14:09:22.86071 +0800 +08 	deployed	aws-load-balancer-controller-1.7.0	v2.7.0     
external-dns                	uepe     	1       	2024-02-06 14:06:28.705309 +0800 +08	deployed	external-dns-6.31.5               	0.14.0     
ingress-nginx-controller        uepe     	1       	2024-02-22 11:44:54.18561 +0800 +08 	deployed	ingress-nginx-4.9.1               	1.9.6

This section is now complete. Now proceed to the Usage Engine Private Edition Preparations - OCI section.