Install AWS EBS CSI Driver

This page explains steps by step instruction to install the AWS EBS CSI driver on AWS EKS cluster.

A CSI driver with snapshotting capability is a requirement for T4K to function properly. It has been discovered that in the Fall of 2022, the snapshot controller has been removed from the AWS EBS CSI driver install process. In addition, CSI is not installed by default with the creation of an EKS cluster so a customer will have to add this CSI support manually after the creation of an EKS cluster.

The instructions are validated for AWS EKS v 1.21, 1.22, and 1.23.

Getting Started with AWS CLI and EKS Cluster deployment:

1. Getting started with AWS EKS

2. Once you have an existing EKS cluster user with eks-admin privileges then fetch the kubeconfig file of the cluster

   aws eks update-kubeconfig --region us-east-1 --name sa-eks-cluster-1

Add CSI driver:

1. Install AWS EBS CSI Driver

2. Install the AWS EBS CSI Driver through Amazon EKS Add-on. You can follow the below step-by-step instruction with commands:

3. Get OIDC endpoint:

   aws eks describe-cluster \
     --name sa-eks-cluster-1 \
     --query "cluster.identity.oidc.issuer" \
     --output text

4. The output will show the OIDC endpoint:

   sa-eks-cluster-1
   https://oidc.eks.us-east-1.amazonaws.com/id/A357D6680CACC3FE811997EAE0BEDDCD

To create an IAM OIDC identity provider for your cluster with the AWS Management Console

1. Open the Amazon EKS console at https://console.aws.amazon.com/eks/home#/clusters.
2. In the left pane, select Clusters, and then select the name of your cluster on the Clusters page.

3. In the Details section on the Overview tab, note the value of the OpenID Connect provider URL.

4. Open the IAM console at https://console.aws.amazon.com/iam/
5. In the left navigation pane, choose Identity Providers under Access management. If a Provider is listed that matches the URL for your cluster, then you already have a provider for your cluster. If a provider isn't listed that matches the URL for your cluster, then you must create one.

6. To create a provider, choose Add provider.

7. For Provider type, select OpenID Connect.

8. For Provider URL, enter the OIDC provider URL for your cluster, and then choose Get thumbprint.

9. For Audience, enter sts.amazonaws.com and choose Add provider.

Create role with policy:

1. Get OIDC provider for cluster:

   user@demo-server:~# aws eks describe-cluster \
     --name sa-eks-cluster-1 \
     --query "cluster.identity.oidc.issuer" \
     --output text

2. The output will show the OIDC endpoint with region name:

   sa-eks-cluster-1
   https://oidc.eks.us-east-1.amazonaws.com/id/A357D6680CACC3FE811997EAE0BEDDCD

3. Create role JSON file to be applied with AWS CLI

4. Change the AWS account number, OIDC provider URL and region code as per the EKS cluster deployment.

   % cat aws-ebs-csi-driver-trust-policy-eks1.json 
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::195495575045:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/A357D6680CACC3FE811997EAE0BEDDCD"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "oidc.eks.us-east-1.amazonaws.com/id/A357D6680CACC3FE811997EAE0BEDDCD:aud": "sts.amazonaws.com",
             "oidc.eks.us-east-1.amazonaws.com/id/A357D6680CACC3FE811997EAE0BEDDCD:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
           }
         }
       }
     ]
   }

5. Create Role by applying role ISON file with AWS CLI:

   aws iam create-role \
     --role-name AmazonEKS_EBS_CSI_DriverRole-eks1 \
     --assume-role-policy-document file://"aws-ebs-csi-driver-trust-policy-eks1.json"

6. Attach above created role to Amazon EBS CSI policy

   % aws iam attach-role-policy \
     --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
     --role-name AmazonEKS_EBS_CSI_DriverRole-eks1

7. Adding the Amazon EBS CSI add-on:

   % aws eks create-addon \
     --cluster-name sa-eks-cluster-1 \
     --addon-name aws-ebs-csi-driver \
     --service-account-role-arn arn:aws:iam::195495575045:role/AmazonEKS_EBS_CSI_DriverRole-eks1

8. Add ebs-sc storage class, example pod and PVC using a CSI volume:

   % kubectl get sc
   NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  63d
   % git clone https://github.com/kubernetes-sigs/aws-ebs-csi-driver.git
   Cloning into 'aws-ebs-csi-driver'...
   remote: Enumerating objects: 23572, done.
   remote: Counting objects: 100% (90/90), done.
   remote: Compressing objects: 100% (62/62), done.
   remote: Total 23572 (delta 25), reused 70 (delta 19), pack-reused 23482
   Receiving objects: 100% (23572/23572), 24.90 MiB | 36.01 MiB/s, done.
   Resolving deltas: 100% (12207/12207), done.
   % cd aws-ebs-csi-driver/examples/kubernetes/dynamic-provisioning/
   % ls
   README.md	manifests
   % kubectl apply -f manifests/
   persistentvolumeclaim/ebs-claim created
   pod/app created
   storageclass.storage.k8s.io/ebs-sc created
   % kubectl get csidrivers     
   NAME              ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES        AGE
   ebs.csi.aws.com   true             false            false             <unset>         false               Persistent   5m14s
   efs.csi.aws.com   false            false            false             <unset>         false               Persistent   17m
   % kubectl get sc        
   NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   ebs-sc          ebs.csi.aws.com         Delete          WaitForFirstConsumer   false                  17s
   gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  17m
   % kubectl get pvc
   NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   ebs-claim   Bound    pvc-ed4dbc13-10f6-4176-96e1-3e0d2146f345   4Gi        RWO            ebs-sc         22s
   % kubectl get pod
   NAME   READY   STATUS    RESTARTS   AGE
   app    1/1     Running   0          26s

9. Change default StorageClass to ebs-sc:

   % kubectl get csidrivers     
   NAME              ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES        AGE
   ebs.csi.aws.com   true             false            false             <unset>         false               Persistent   5m14s
   efs.csi.aws.com   false            false            false             <unset>         false               Persistent   17m
   % kubectl get sc        
   NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   ebs-sc          ebs.csi.aws.com         Delete          WaitForFirstConsumer   false                  17s
   gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  17m
   
   kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
   kubectl patch storageclass ebs-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
   
   % kubectl get sc
   NAME               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   ebs-sc (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   false                  4m33s
   gp2                kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  22m

10. Add VolumeSnapshot CRDs and Snapshot Controller:

    # Change to the latest supported snapshotter version
    # https://kubernetes-csi.github.io/docs/external-snapshotter.html
    $ SNAPSHOTTER_VERSION=v5.0.1
    
    # Apply VolumeSnapshot CRDs
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
    
    # Create snapshot controller
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

11. With all above steps, EBS CSI driver with VolumeSnapshot capability is installed.

Validate the EBS driver installation

If T4K Preflight plugin is not already installed, follow the instructions here

Install and run the T4K preflight plugin:

% kubectl tvk-preflight run --storage-class ebs-sc
INFO[0000] Created log file with name - preflight-2022-11-4T16-44-27.log 
INFO[0000] Setting log level as info                    
INFO[0001] ====PREFLIGHT RUN OPTIONS====                
INFO[0001] LOG-LEVEL="INFO"                             
INFO[0001] KUBECONFIG-PATH=""                           
INFO[0001] NAMESPACE="default"                          
INFO[0001] INCLUSTER="false"                            
INFO[0001] STORAGE-CLASS="ebs-sc"     
...
INFO[0134] ✔ restored pod from volume snapshot of unmounted pv has expected data 
INFO[0134] ✔ Preflight check for volume snapshot and restore is successful 
INFO[0134] All preflight checks succeeded!              
INFO[0138] All preflight resources cleaned