Restricted Network installation for TVK on Rancher (Upstream)

TVK installation guide for restricted Network on Rancher (upstream)
  1. Setup Local Registry - Ref Deploy a registry server

    1. Set up local registry on Ubuntu 20.04 deployed on AWS EC2

    2. Install docker

    3. Start docker registry (using one of the options below)

      1. Unsecured registry

      2. docker run -d -p 5000:5000 --restart=always --name registry registry:2
      3. A secured registry using TLS authentication (Valid certificates are required in “certs” dir)

      4. docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/skregistry.kulkarnisachin.ml.crt -e REGISTRY_HTTP_TLS_KEY=/certs/skregistry.kulkarnisachin.ml.key -p 443:443 registry:2
  2. By default, the registry stores its data on the local filesystem, whether you use a bind mount or a volume. You can store the registry data in an Amazon S3 bucket, Google Cloud Platform, or on another storage back-end by using storage drivers.

  3. Download TVK charts (k8s-triliovault-operator and k8s-triliovault) locally

    1. Using curl

    2. curl http://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator/charts/k8s-triliovault-operator-v2.1.0.tgz --output k8s-triliovault-operator-v2.1.0.tgz
      curl http://charts.k8strilio.net/trilio-stable/k8s-triliovault/charts/k8s-triliovault-v2.1.0.tgz --output k8s-triliovault-v2.1.0.tgz
    3. Using Helm pull

    4. helm repo add triliovault-operator http://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator
      helm repo add triliovault http://charts.k8strilio.net/trilio-stable/k8s-triliovault
      helm repo update
      helm pull triliovault-operator/k8s-triliovault-operator
      helm pull triliovault/k8s-triliovault
  4. Enable Helm registry - Ref Helm | Registries

    1. export HELM_EXPERIMENTAL_OCI=1
  5. Login to local helm registry and save the TVK charts in the local registry

    1. helm registry login --insecure http://skregistry.kulkarnisachin.ml
      helm chart save k8s-triliovault-operator/ skregistry.kulkarnisachin.ml/tvk-op
      helm chart save k8s-triliovault/ skregistry.kulkarnisachin.ml/tvk
    2. You should see list of charts as below

    3. $ helm chart list
      REF NAME VERSION DIGEST SIZE CREATED
      skregistry.kulkarnisachin.ml/tvk-op:v2.0.5 k8s-triliovault-operator v2.1.0 85ec8d2 8.7 KiB 38 seconds
      skregistry.kulkarnisachin.ml/tvk:v2.1.0 k8s-triliovault v2.1.0 85ec8d2 39.2 KiB 10 seconds
      $
  6. Now, these charts can be used on any cluster which has access to this local registry.

  7. Create secret, daemonset to push certificates for all new pods/containers. This step is required when registry needs TLS certificates. These certificates will be needed on every new pod/container to authenticate with the registry to access the charts, images.

    1. Copy the image “busybox” in the local registry.

    2. $ kubectl create secret generic registry-ca --namespace kube-system --from-file=registry-ca=./rootCA.crt
      secret/registry-ca created
      $
      $ kubectl get secrets -A | grep registry
      kube-system registry-ca Opaque 1 11s
      $
      $ cat registry-ca-ds.yaml
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
      name: registry-ca
      namespace: kube-system
      labels:
      k8s-app: registry-ca
      spec:
      selector:
      matchLabels:
      name: registry-ca
      template:
      metadata:
      labels:
      name: registry-ca
      spec:
      containers:
      - name: registry-ca
      image: busybox
      command: [ 'sh' ]
      args: [ '-c', 'cp /home/core/registry-ca /etc/docker/certs.d/skregistry.kulkarnisachin.ml/ca.crt && exec tail -f /dev/null' ]
      volumeMounts:
      - name: etc-docker
      mountPath: /etc/docker/certs.d/skregistry.kulkarnisachin.ml
      - name: ca-cert
      mountPath: /home/core
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etc-docker
      hostPath:
      path: /etc/docker/certs.d/skregistry.kulkarnisachin.ml
      - name: ca-cert
      secret:
      secretName: registry-ca
      $ kubectl create -f registry-ca-ds.yaml
      daemonset.apps/registry-ca created
      $
      $ kubectl get ds -A
      NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
      cattle-system cattle-node-agent 6 6 6 6 6 <none> 26m
      cattle-system kube-api-auth 3 3 3 3 3 <none> 26m
      ingress-nginx nginx-ingress-controller 3 3 3 3 3 <none> 26m
      kube-system canal 6 6 6 6 6 kubernetes.io/os=linux 27m
      kube-system registry-ca 3 3 3 3 3 <none> 22s
      $
  8. Push images to local registry

    1. Pull TVK operator image from Google Cloud Platform

    2. docker pull eu.gcr.io/amazing-chalice-243510/operator-webhook-init:v2.0.5
    3. Tag the image

    4. docker tag eu.gcr.io/amazing-chalice-243510/operator-webhook-init:v2.0.5 skregistry.kulkarnisachin.ml/operator-webhook-init:v2.0.5
    5. Push the image

    6. docker push skregistry.kulkarnisachin.ml/operator-webhook-init:v2.0.5
    7. Repeat the same for all the images required for TVK (listed below)

      1. TVK operator images

        1. k8s-triliovault-operator
          operator-webhook-init
      2. TVK app images

        1. backup-scheduler
          control-plane
          datamover
          datastore-attacher
          metamover
          exporter
          trilio-webhook-init
          trilio-admission-webhook
          backup-cleaner
          resource-cleaner
          web
          web-backend
          backup-retention
          target-browser
          hook-executor
          analyzer
          ingress-controller
    8. The bash script below can be used to download and push images to local registry. Please check the list of images and tags required for given TVK version.

    9. #!/bin/bash
      ## Update below details as required for the specific TVK version
      ## TVK pkg list for v2.1.0
      pkg_list="k8s-triliovault-operator operator-webhook-init backup-scheduler control-plane datamover datastore-attacher metamover exporter trilio-webhook-init trilio-admission-webhook backup-cleaner resource-cleaner web web-backend backup-retention target-browser hook-executor analyzer ingress-controller"
      gcr_link="eu.gcr.io/amazing-chalice-243510"
      local_reg="skregistry.kulkarnisachin.ml"
      for pkg in ${pkg_list}
      do
      echo "Adding $pkg"
      docker pull ${gcr_link}/${pkg}:v2.1.0
      docker tag ${gcr_link}/${pkg}:v2.1.0 ${local_reg}/${pkg}:v2.1.0
      docker push ${local_reg}/${pkg}:v2.1.0
      done
      ## List all the pkgs
      docker image ls
  9. Install TVK operator by adding local registry in the k8s-triliovault-operator/values.yaml. Here the chart can be directly installed from local registry or downloaded from it

  10. $ head -10 ./k8s-triliovault-operator/values.yaml
    ## TrilioVault Operator
    registry: "skregistry.kulkarnisachin.ml"
    operator-webhook-init:
    repository: operator-webhook-init
    k8s-triliovault-operator:
    repository: k8s-triliovault-operator
    tag: "v2.1.0"
    $
    $ helm install local-tvk-op ./k8s-triliovault-operator
    $ helm list
    NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
    local-tvk-op default 1 2021-05-12 12:09:40.543186572 +0000 UTC deployed k8s-triliovault-operator-v2.1.0 v2.1.0
    $
  11. Install TriliovaultManager using the manifest for helm

  12. $ cat triliovault-manager.yaml
    apiVersion: triliovault.trilio.io/v1
    kind: TrilioVaultManager
    metadata:
    labels:
    triliovault: triliovault
    name: triliovault-manager
    namespace: default
    spec:
    trilioVaultAppVersion: v2.1.0
    tvkHelmRepo: skregistry.kulkarnisachin.ml
    helmVersion:
    version: v3
    applicationScope: Cluster
    $
    $ kubectl create -f triliovault-manager.yaml
    triliovaultmanager.triliovault.trilio.io/triliovault-manager created
    $ kubectl get triliovaultmanager
    NAME TRILIOVAULT-VERSION SCOPE STATUS RESTORE-NAMESPACES
    triliovault-manager v2.1.0 Cluster
    $
  13. TVK installation is complete. Continue using it following Getting Started Guide.