Search…
Restricted Network installation for TVK on Upstream Kubernetes
This page describes the instructions for deploying TVK in a network-restricted environment for Upstream or any Upstream compatible Kubernetes distribution.
    1.
    Setup Local Registry - Ref
    Deploy a registry server
      1.
      Set up local registry on Ubuntu 20.04 deployed on AWS EC2
      2.
      Install docker
      3.
      Start docker registry (using one of the options below)
        1.
        Unsecured registry
        2.
        1
        docker run -d -p 5000:5000 --restart=always --name registry registry:2
        Copied!
        3.
        A secured registry using TLS authentication (Valid certificates are required in “certs” dir)
        4.
        1
        docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/skregistry.kulkarnisachin.ml.crt -e REGISTRY_HTTP_TLS_KEY=/certs/skregistry.kulkarnisachin.ml.key -p 443:443 registry:2
        Copied!
    2.
    By default, the registry stores its data on the local filesystem, whether you use a bind mount or a volume. You can store the registry data in an Amazon S3 bucket, Google Cloud Platform, or on another storage back-end by using storage drivers.
    3.
    Download TVK charts (k8s-triliovault-operator and k8s-triliovault) locally
      1.
      Using curl
      2.
      1
      curl http://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator/charts/k8s-triliovault-operator-2.5.0.tgz --output k8s-triliovault-operator-2.5.0.tgz
      2
      curl http://charts.k8strilio.net/trilio-stable/k8s-triliovault/charts/k8s-triliovault-2.5.0.tgz --output k8s-triliovault-2.5.0.tgz
      Copied!
      3.
      Using Helm pull
      4.
      1
      helm repo add triliovault-operator http://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator
      2
      helm repo add triliovault http://charts.k8strilio.net/trilio-stable/k8s-triliovault
      3
      helm repo update
      4
      helm pull triliovault-operator/k8s-triliovault-operator
      5
      helm pull triliovault/k8s-triliovault
      Copied!
    4.
    Enable Helm registry - Ref
    Helm | Registries
      1.
      1
      export HELM_EXPERIMENTAL_OCI=1
      Copied!
    5.
    Login to local helm registry and save the TVK charts in the local registry
      1.
      1
      helm registry login --insecure http://skregistry.kulkarnisachin.ml
      2
      helm chart save k8s-triliovault-operator/ skregistry.kulkarnisachin.ml/tvk-op
      3
      helm chart save k8s-triliovault/ skregistry.kulkarnisachin.ml/tvk
      Copied!
      2.
      You should see list of charts as below
      3.
      1
      $ helm chart list
      2
      REF NAME VERSION DIGEST SIZE CREATED
      3
      skregistry.kulkarnisachin.ml/tvk-op:v2.5.0 k8s-triliovault-operator v2.5.0 85ec8d2 8.7 KiB 38 seconds
      4
      skregistry.kulkarnisachin.ml/tvk:v2.5.0 k8s-triliovault v2.5.0 85ec8d2 39.2 KiB 10 seconds
      5
      $
      Copied!
    6.
    Now, these charts can be used on any cluster which has access to this local registry.
    7.
    Create secret, daemonset to push certificates for all new pods/containers. This step is required when registry needs TLS certificates. These certificates will be needed on every new pod/container to authenticate with the registry to access the charts, images.
      1.
      Copy the image “busybox” in the local registry.
      2.
      1
      $ kubectl create secret generic registry-ca --namespace kube-system --from-file=registry-ca=./rootCA.crt
      2
      secret/registry-ca created
      3
      $
      4
      $ kubectl get secrets -A | grep registry
      5
      kube-system registry-ca Opaque 1 11s
      6
      $
      7
      $ cat registry-ca-ds.yaml
      8
      apiVersion: apps/v1
      9
      kind: DaemonSet
      10
      metadata:
      11
      name: registry-ca
      12
      namespace: kube-system
      13
      labels:
      14
      k8s-app: registry-ca
      15
      spec:
      16
      selector:
      17
      matchLabels:
      18
      name: registry-ca
      19
      template:
      20
      metadata:
      21
      labels:
      22
      name: registry-ca
      23
      spec:
      24
      containers:
      25
      - name: registry-ca
      26
      image: busybox
      27
      command: [ 'sh' ]
      28
      args: [ '-c', 'cp /home/core/registry-ca /etc/docker/certs.d/skregistry.kulkarnisachin.ml/ca.crt && exec tail -f /dev/null' ]
      29
      volumeMounts:
      30
      - name: etc-docker
      31
      mountPath: /etc/docker/certs.d/skregistry.kulkarnisachin.ml
      32
      - name: ca-cert
      33
      mountPath: /home/core
      34
      terminationGracePeriodSeconds: 30
      35
      volumes:
      36
      - name: etc-docker
      37
      hostPath:
      38
      path: /etc/docker/certs.d/skregistry.kulkarnisachin.ml
      39
      - name: ca-cert
      40
      secret:
      41
      secretName: registry-ca
      42
      $ kubectl create -f registry-ca-ds.yaml
      43
      daemonset.apps/registry-ca created
      44
      $
      45
      $ kubectl get ds -A
      46
      NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
      47
      cattle-system cattle-node-agent 6 6 6 6 6 <none> 26m
      48
      cattle-system kube-api-auth 3 3 3 3 3 <none> 26m
      49
      ingress-nginx nginx-ingress-controller 3 3 3 3 3 <none> 26m
      50
      kube-system canal 6 6 6 6 6 kubernetes.io/os=linux 27m
      51
      kube-system registry-ca 3 3 3 3 3 <none> 22s
      52
      $
      Copied!
    8.
    Push images to local registry
      1.
      Pull TVK operator image from
      Google Cloud Platform
      2.
      1
      docker pull eu.gcr.io/amazing-chalice-243510/operator-webhook-init:v2.5.0
      Copied!
      3.
      Tag the image
      4.
      1
      docker tag eu.gcr.io/amazing-chalice-243510/operator-webhook-init:v2.5.0 skregistry.kulkarnisachin.ml/operator-webhook-init:v2.5.0
      Copied!
      5.
      Push the image
      6.
      1
      docker push skregistry.kulkarnisachin.ml/operator-webhook-init:v2.5.0
      Copied!
      7.
      Repeat the same for all the images required for TVK (listed below)
        1.
        TVK operator images
          1.
          1
          k8s-triliovault-operator
          2
          operator-webhook-init
          Copied!
        2.
        TVK app images
          1.
          1
          backup-scheduler
          2
          control-plane
          3
          datamover
          4
          datastore-attacher
          5
          metamover
          6
          exporter
          7
          dex:v2.28.1
          8
          trilio-init
          9
          trilio-admission-webhook
          10
          backup-cleaner
          11
          resource-cleaner
          12
          web
          13
          web-backend
          14
          backup-retention
          15
          target-browser
          16
          hook-executor
          17
          analyzer
          18
          ingress-controller
          Copied!
      8.
      The bash script below can be used to download and push images to local registry. Please check the list of images and tags required for given TVK version.
      9.
      1
      #!/bin/bash
      2
      3
      ## Update below details as required for the specific TVK version
      4
      ## TVK images list for v2.5.0
      5
      img_list="k8s-triliovault-operator operator-webhook-init backup-scheduler control-plane datamover datastore-attacher metamover exporter trilio-init trilio-admission-webhook backup-cleaner resource-cleaner web web-backend backup-retention target-browser hook-executor analyzer ingress-controller"
      6
      gcr_link="eu.gcr.io/amazing-chalice-243510"
      7
      local_reg="skregistry.kulkarnisachin.ml"
      8
      9
      for img in ${img_list}
      10
      do
      11
      echo "Adding $img"
      12
      docker pull ${gcr_link}/${img}:v2.5.0
      13
      docker tag ${gcr_link}/${img}:v2.5.0 ${local_reg}/${img}:v2.5.0
      14
      docker push ${local_reg}/${img}:v2.5.0
      15
      done
      16
      17
      ## Pushing dex:v2.28.1 image as the version is different
      18
      docker pull ${gcr_link}/dex:v2.28.1
      19
      docker tag ${gcr_link}/dex:v2.28.1 ${local_reg}/dex:v2.28.1
      20
      docker push ${local_reg}/dex:v2.28.1
      21
      22
      ## List all the pkgs
      23
      docker image ls
      Copied!
    9.
    Install TVK operator by adding local registry in the k8s-triliovault-operator/values.yaml. Here the chart can be directly installed from local registry or downloaded from it
    10.
    1
    $ head -10 ./k8s-triliovault-operator/values.yaml
    2
    ## TrilioVault Operator
    3
    registry: "skregistry.kulkarnisachin.ml"
    4
    5
    operator-webhook-init:
    6
    repository: operator-webhook-init
    7
    8
    k8s-triliovault-operator:
    9
    repository: k8s-triliovault-operator
    10
    11
    tag: "v2.5.0"
    12
    $
    13
    $ helm install local-tvk-op ./k8s-triliovault-operator
    14
    $ helm list
    15
    NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
    16
    local-tvk-op default 1 2021-05-12 12:09:40.543186572 +0000 UTC deployed k8s-triliovault-operator-v2.5.0 v2.5.0
    17
    $
    Copied!
    11.
    Install TriliovaultManager using the manifest for helm
    12.
    1
    $ cat triliovault-manager.yaml
    2
    apiVersion: triliovault.trilio.io/v1
    3
    kind: TrilioVaultManager
    4
    metadata:
    5
    labels:
    6
    triliovault: triliovault
    7
    name: triliovault-manager
    8
    namespace: default
    9
    spec:
    10
    trilioVaultAppVersion: v2.5.0
    11
    tvkHelmRepo: skregistry.kulkarnisachin.ml
    12
    helmVersion:
    13
    version: v3
    14
    applicationScope: Cluster
    15
    $
    16
    $ kubectl create -f triliovault-manager.yaml
    17
    triliovaultmanager.triliovault.trilio.io/triliovault-manager created
    18
    $ kubectl get triliovaultmanager
    19
    NAME TRILIOVAULT-VERSION SCOPE STATUS RESTORE-NAMESPACES
    20
    triliovault-manager v2.5.0 Cluster
    21
    $
    Copied!
    13.
    TVK installation is complete. Continue using it following Getting Started Guide.
Last modified 27d ago
Copy link