Search…
Air-Gapped Install - Upstream Kubernetes
This page describes the instructions for deploying TVK in a network-restricted environment for Upstream or any Upstream compatible Kubernetes distribution.
If a local registry is available, then skip to step 2 directly.
The instructions provided here for setting up local registry are just for reference. It is recommended to follow proper official instructions for setting up local registry from docker or other similar products.
IMPORTANT - Please read the value of TVK_VERSION in this doc as "2.9.1" for the current release

Steps to install TVK in air-gapped upstream Kubernetes environment

There are 3 simple steps that a user needs to perform:
  1. 1.
    Step 1 - Setup Local Registry - Ref
    Deploy a registry server
  2. 2.
    Step 2 - Download TVK charts (k8s-triliovault-operator and k8s-triliovault) locally
  3. 3.
    Step 3 - Install TVK using one of the methods below
    • Directly install k8s-triliovault by referencing the helm chart downloaded locally
    • Use Helm Registry to store local charts and install the charts referring to this helm registry

Step 1 - Setup Local Registry

Setup Local Registry - Ref
Deploy a registry server
  1. 1.
    Set up local registry on Ubuntu 20.04
  2. 2.
    Install docker
  3. 3.
    Start docker registry (using one of the options below)
    • A secured registry using TLS authentication (Valid certificates are required in “certs” dir) - Recommended
    docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/skregistry.kulkarnisachin.ml.crt -e REGISTRY_HTTP_TLS_KEY=/certs/skregistry.kulkarnisachin.ml.key -p 443:443 registry:2
    • Insecure registry - Not Recommended
    docker run -d -p 5000:5000 --restart=always --name registry registry:2
  4. 4.
    By default, the registry stores its data on the local filesystem, whether you use a bind mount or a volume. You can store the registry data in an Amazon S3 bucket, Google Cloud Platform, or on another storage back-end by using storage drivers.

Step 2 - Download TVK charts (k8s-triliovault-operator and k8s-triliovault) locally

Use one of the methods below

Using curl commands

1
curl http://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator/charts/k8s-triliovault-operator-TVK_VERSION.tgz --output k8s-triliovault-operator-TVK_VERSION.tgz
2
curl http://charts.k8strilio.net/trilio-stable/k8s-triliovault/charts/k8s-triliovault-TVK_VERSION.tgz --output k8s-triliovault-TVK_VERSION.tgz
Copied!

Using Helm pull

1
helm repo add triliovault-operator http://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator
2
helm repo add triliovault http://charts.k8strilio.net/trilio-stable/k8s-triliovault
3
helm repo update
4
helm pull triliovault-operator/k8s-triliovault-operator
5
helm pull triliovault/k8s-triliovault
Copied!

Step 3 - Install TVK

Upload TVK images to local registry

Use the script below to download and push images to local registry. Please edit the TVK_VERSION, list of images and tags required for given TVK version.
1
#!/bin/bash
2
3
## TVK version
4
tvk_version="<TVK_VERSION>"
5
6
## Update below details as required for the specific TVK version
7
## TVK images list for specific version TVK_VERSION
8
img_list="k8s-triliovault-operator operator-webhook-init backup-scheduler control-plane datamover datastore-attacher metamover exporter trilio-init trilio-admission-webhook backup-cleaner resource-cleaner web web-backend backup-retention target-browser hook-executor analyzer"
9
gcr_link="eu.gcr.io/amazing-chalice-243510"
10
## Edit the value of local_reg below to point to local registry
11
## e.g. local_reg="skregistry.kulkarnisachin.ml"
12
local_reg="<local registry url>"
13
14
for img in ${img_list}
15
do
16
echo "Adding $img"
17
docker pull ${gcr_link}/${img}:${tvk_version}
18
docker tag ${gcr_link}/${img}:${tvk_version} ${local_reg}/${img}:${tvk_version}
19
docker push ${local_reg}/${img}:${tvk_version}
20
done
21
22
## Pushing dex:v2.28.1, kube-certgen:v1.1.1, ingress-controller:v1.1.1 images as the versions are different
23
docker pull ${gcr_link}/dex:v2.28.1
24
docker tag ${gcr_link}/dex:v2.28.1 ${local_reg}/dex:v2.28.1
25
docker push ${local_reg}/dex:v2.28.1
26
docker pull ${gcr_link}/kube-certgen:v1.1.1
27
docker tag ${gcr_link}/kube-certgen:v1.1.1 ${local_reg}/kube-certgen:v1.1.1
28
docker push ${local_reg}/kube-certgen:v1.1.1
29
docker pull ${gcr_link}/ingress-controller:v1.1.1
30
docker tag ${gcr_link}/ingress-controller:v1.1.1 ${local_reg}/ingress-controller:v1.1.1
31
docker push ${local_reg}/ingress-controller:v1.1.1
32
33
## List all the pkgs
34
docker image ls
Copied!
Individual commands to push the TVK images to local registry
  1. 1.
    Pull TVK operator image from
    Google Cloud Platform
  2. 2.
    1
    docker pull eu.gcr.io/amazing-chalice-243510/operator-webhook-init:TVK_VERSION
    Copied!
  3. 3.
    Tag the image
  4. 4.
    1
    docker tag eu.gcr.io/amazing-chalice-243510/operator-webhook-init:TVK_VERSION skregistry.kulkarnisachin.ml/operator-webhook-init:TVK_VERSION
    Copied!
  5. 5.
    Push the image
  6. 6.
    1
    docker push skregistry.kulkarnisachin.ml/operator-webhook-init:TVK_VERSION
    Copied!
  7. 7.
    Repeat the same for all the images required for TVK (listed below)
    1. 1.
      TVK operator images
      1. 1.
        1
        k8s-triliovault-operator
        2
        operator-webhook-init
        Copied!
    2. 2.
      TVK app images
      1. 1.
        1
        backup-scheduler
        2
        control-plane
        3
        datamover
        4
        datastore-attacher
        5
        metamover
        6
        exporter
        7
        dex:v2.28.1
        8
        trilio-init
        9
        trilio-admission-webhook
        10
        backup-cleaner
        11
        resource-cleaner
        12
        web
        13
        web-backend
        14
        backup-retention
        15
        target-browser
        16
        hook-executor
        17
        analyzer
        18
        ingress-controller:v1.1.1
        19
        kube-certgen:v1.1.1
        Copied!

Directly install k8s-triliovault chart locally

Install TriliovaultManager by adding below values in k8s-triliovault/values.yaml or using "--set". Here the chart can be directly installed. However, future chart updates need to be managed manually.
1
## TrilioVault Manager values - Update relevant values and replace "TVK_VERSION" with actual version
2
registry: "skregistry.kulkarnisachin.ml"
3
applicationScope: "Clustered"
4
tag: "TVK_VERSION"
5
6
$ helm install local-tvk ./k8s-triliovault \
7
--set registry="skregistry.kulkarnisachin.ml" \
8
--set applicationScope="Clustered" \
9
--set tag="TVK_VERSION"
10
$ helm list
11
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
12
local-tvk default 1 2021-12-07 12:09:40.543186572 +0000 UTC deployed k8s-triliovault-TVK_VERSION TVK_VERSION
13
$
14
$ kubectl get triliovaultmanager
15
NAME TRILIOVAULT-VERSION SCOPE STATUS RESTORE-NAMESPACES
16
triliovault-manager TVK_VERSION Cluster
17
$
Copied!

Using local helm registry, it provides a way to access it on multiple systems by referring to local helm registry. Please note this is an alternative to method described in above step.

Enable Helm registry - Ref
Helm | Registries
1
export HELM_EXPERIMENTAL_OCI=1
Copied!
Login to local helm registry and save the TVK charts in the local registry
1
helm registry login --insecure http://skregistry.kulkarnisachin.ml
2
helm chart save k8s-triliovault-operator/ skregistry.kulkarnisachin.ml/tvk-op
3
helm chart save k8s-triliovault/ skregistry.kulkarnisachin.ml/tvk
Copied!
You should see list of charts as below
1
$ helm chart list
2
REF NAME VERSION DIGEST SIZE CREATED
3
skregistry.kulkarnisachin.ml/tvk-op:TVK_VERSION k8s-triliovault-operator TVK_VERSION 85ec8d2 8.7 KiB 38 seconds
4
skregistry.kulkarnisachin.ml/tvk:TVK_VERSION k8s-triliovault TVK_VERSION 85ec8d2 39.2 KiB 10 seconds
5
$
Copied!
Now, these charts can be used on any cluster which has access to this local registry.
Note - For a secured registry with TLS authentication, follow the additional steps below.
Create secret, daemonset to push certificates for all new pods/containers. This step is required when registry needs TLS certificates. These certificates will be needed on every new pod/container to authenticate with the registry to access the charts, images.
Copy the image “busybox” in the local registry.
1
$ kubectl create secret generic registry-ca --namespace kube-system --from-file=registry-ca=./rootCA.crt
2
secret/registry-ca created
3
$
4
$ kubectl get secrets -A | grep registry
5
kube-system registry-ca Opaque 1 11s
6
$
7
$ cat registry-ca-ds.yaml
8
apiVersion: apps/v1
9
kind: DaemonSet
10
metadata:
11
name: registry-ca
12
namespace: kube-system
13
labels:
14
k8s-app: registry-ca
15
spec:
16
selector:
17
matchLabels:
18
name: registry-ca
19
template:
20
metadata:
21
labels:
22
name: registry-ca
23
spec:
24
containers:
25
- name: registry-ca
26
image: busybox
27
command: [ 'sh' ]
28
args: [ '-c', 'cp /home/core/registry-ca /etc/docker/certs.d/skregistry.kulkarnisachin.ml/ca.crt && exec tail -f /dev/null' ]
29
volumeMounts:
30
- name: etc-docker
31
mountPath: /etc/docker/certs.d/skregistry.kulkarnisachin.ml
32
- name: ca-cert
33
mountPath: /home/core
34
terminationGracePeriodSeconds: 30
35
volumes:
36
- name: etc-docker
37
hostPath:
38
path: /etc/docker/certs.d/skregistry.kulkarnisachin.ml
39
- name: ca-cert
40
secret:
41
secretName: registry-ca
42
$ kubectl create -f registry-ca-ds.yaml
43
daemonset.apps/registry-ca created
44
$
45
$ kubectl get ds -A
46
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
47
cattle-system cattle-node-agent 6 6 6 6 6 <none> 26m
48
cattle-system kube-api-auth 3 3 3 3 3 <none> 26m
49
ingress-nginx nginx-ingress-controller 3 3 3 3 3 <none> 26m
50
kube-system canal 6 6 6 6 6 kubernetes.io/os=linux 27m
51
kube-system registry-ca 3 3 3 3 3 <none> 22s
52
$
Copied!
For an insecure registry, the additional steps below may be needed. This is not a recommended way as it is not secured and also needs restarting docker daemon on master/ worker nodes.
1
# Edit /etc/docker/daemon.json file with following settings
2
{ "insecure-registries":["host:port"] }
3
example: { "insecure-registries":["skregistry.kulkarnisachin.ml:5000"] }
4
5
# May need to restart docker daemon on master & worker nodes using appropriate commands.
Copied!

One-Click Installation

In one click install for upstream operator, a cluster scope TVM custom resource triliovault-manager is created. Install TVK operator by adding local registry in the k8s-triliovault-operator/values.yaml or using "--set". Here the chart can be directly installed from local registry or downloaded from it.
1
## TrilioVault Operator - Update relevant values and replace "TVK_VERSION" with actual version
2
registry: "skregistry.kulkarnisachin.ml"
3
tag: "TVK_VERSION"
4
tvkHelmRepo: skregistry.kulkarnisachin.ml
5
6
$ helm install local-tvk-op ./k8s-triliovault-operator \
7
--set registry="skregistry.kulkarnisachin.ml" \
8
--set tag="TVK_VERSION" \
9
--set tvkHelmRepo="skregistry.kulkarnisachin.ml"
10
$ helm list
11
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
12
local-tvk-op default 1 2021-12-07 12:09:40.543186572 +0000 UTC deployed k8s-triliovault-operator-TVK_VERSION TVK_VERSION
13
$
Copied!
One-Click Configuration - The following table lists the configuration parameter of the upstream operator one click install feature and their default values.
Parameter
Description
Default
installTVK.enabled
1 click install feature is enabled
true
installTVK.applicationScope
scope of TVK application created
Cluster
installTVK.ingressConfig.host
host of the ingress resource created
""
installTVK.ingressConfig.tlsSecretName
tls secret name which contains ingress certs
""
installTVK.ingressConfig.annotations
annotations to be added on ingress resource
""
installTVK.ingressConfig.ingressClass
ingress class name for the ingress resource
""
installTVK.ComponentConfiguration.ingressController.enabled
TVK ingress controller should be deployed
true
installTVK.ComponentConfiguration.ingressController.service.type
TVK ingress controller service type
"LoadBalancer"
Check the TVM CR configuration by running following command:
1
kubectl get triliovaultmanagers.triliovault.trilio.io triliovault-manager -o yaml
Copied!
Once the operator pod is in running state, the TVK pods getting spawned. Confirm that the TVK pods are up.

Manual Installation

For manual install of TVK operator, add local registry in the k8s-triliovault-operator/values.yaml or using "--set". Here the chart can be directly installed from local registry or downloaded from it.
1
## TrilioVault Operator - Update relevant values and replace "TVK_VERSION" with actual version
2
registry: "skregistry.kulkarnisachin.ml"
3
tag: "TVK_VERSION"
4
tvkHelmRepo: skregistry.kulkarnisachin.ml
5
installTVK.enabled: false
6
7
$ helm install local-tvk-op ./k8s-triliovault-operator \
8
--set registry="skregistry.kulkarnisachin.ml" \
9
--set tag="TVK_VERSION" \
10
--set tvkHelmRepo="skregistry.kulkarnisachin.ml" \
11
--set installTVK.enabled=false
12
$ helm list
13
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
14
local-tvk-op default 1 2021-12-07 12:09:40.543186572 +0000 UTC deployed k8s-triliovault-operator-TVK_VERSION TVK_VERSION
15
$
Copied!
Now, create a TrilioVaultManager CR to install the TrilioVault for Kubernetes. You can provide the custom configurations for the TVK resources as follows:
1
apiVersion: triliovault.trilio.io/v1
2
kind: TrilioVaultManager
3
metadata:
4
labels:
5
triliovault: k8s
6
name: tvk
7
spec:
8
trilioVaultAppVersion: latest
9
applicationScope: Cluster
10
# User can configure the ingress hosts, annotations and TLS secret through the ingressConfig section
11
ingressConfig:
12
host: "trilio.co.in"
13
tlsSecretName: "secret-name"
14
# TVK components configuration, currently supports control-plane, web, exporter, web-backend, ingress-controller, admission-webhook.
15
# User can configure resources for all componentes and can configure service type and host for the ingress-controller
16
componentConfiguration:
17
web-backend:
18
resources:
19
requests:
20
memory: "400Mi"
21
cpu: "200m"
22
limits:
23
memory: "2584Mi"
24
cpu: "1000m"
25
ingress-controller:
26
enabled: true
27
service:
28
type: LoadBalancer
Copied!
Apply TVM.yaml:
1
kubectl create -f TVM.yaml
Copied!
Check if the installation is successful using check-tvk-install.
TVK installation is complete. Continue using it following Getting Started Guide.