YAML Examples

YAML examples for Trilio Custom Resource Definitions

TrilioVaultManager

TrilioVaultManager CRD, owned by Trilio Operator for Upstream/Non-Operator Lifecycle Manager (OLM) based environments, manages the lifecycle of Trilio for Kubernetes Application.

The following snippet provides an example on how to use the TrilioVaultManager CRD.

The version number provided below is only for illustrative purposes. Please refer to the Compatibility Matrix to find the latest version to use.

apiVersion: triliovault.trilio.io/v1
kind: TrilioVaultManager
metadata:
  labels:
    triliovault: triliovault
  name: trilio-app
spec:
  trilioVaultAppVersion: 0.2.5
  helmVersion:
    version: v3
    tillerNamespace: "kube-system"
  applicationScope: Namespaced
  restoreNamespaces: ["kube-system", "default", "restoreinme", "restore-namespace", "restore"]
  resources:
    requests:
      memory: 400Mi

In the above CRD restoreNamespaces is optional. Application scope of Namespaced, will enable additonal namespaces that can be restored into. Application scope of clustered will restrict restores to those specific namespaces only.

Provide Resource Limits for Trilio Pods

apiVersion: triliovault.trilio.io/v1
kind: TrilioVaultManager
metadata:
  labels:
    triliovault: k8s
  name: tvk
spec:
  trilioVaultAppVersion: 2.5.0
  applicationScope: Cluster
  # T4K components configuration, currently supports control-plane, web, exporter, web-backend, ingress-controller, admission-webhook.
  # User can configure resources for all componentes and can configure service type and host for the ingress-controller
  componentConfiguration:
    web-backend:
      resources:
        requests:
          memory: "400Mi"
          cpu: "200m"
        limits:
          memory: "2584Mi"
          cpu: "1000m"
    ingress-controller:
      service:
        type: LoadBalancer
      host: "trilio.co.in"

Backup Target

The Backup Target CRD specifies the backup storage media. Trilio supports either AWS S3 compatible object storage or NFS. A user can configure multiple backup targets and choose the target when an Application CR is created by providing target name and the name of the namespace where it resides. The target credentials can be saved as a secret and refer to the target CR for better security reasons. All backups that are created for that Application will be saved on the backup target specified in Application CR spec. Once a backup target is chosen for an Application, it cannot be changed.

Example 1 - S3 Target: AWS

Sample YAML file for AWS based S3 bucket and credentialSecret

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
type: Opaque
stringData:
  accessKey: AKIAS5B35DGFSTY7T55D
  secretKey: xWBupfGvkgkhaH8ansJU1wRhFoGoWFPmhXD6/vVD

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  name: demo-s3-target
spec:
  type: ObjectStore
  vendor: AWS
  objectStoreCredentials:
    region: us-east-1
    bucketName: trilio-browser-test
    credentialSecret:
      name: sample-secret
      namespace: TARGET_NAMESPACE
  thresholdCapacity: 5Gi

Example 2 - S3 Target: Non-AWS

Sample YAML file for non-AWS S3 compatible bucket. The only difference between this spec and AWS S3 spec is the explicit specification on URL.

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  name: sample-target
spec:
  type: ObjectStore
  vendor: Ceph
  objectStoreCredentials:
    url: "http://bucket.s3.dualstack.region.acme.com/key"
    accessKey: "XXXXXXXXXXXXXXXXXXXX"
    secretKey: "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"
    bucketName: "datahub-dev"
    region: "us-west-1"

Example 3 - S3 Target with SSL Certificate

The following example details how a target can be created when the S3 target has SSL.

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
type: Opaque
data:
  accessKey: VFJJTElPTUlOSU8=
  ca-bundle.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURpekNDQW5PZ0F3SUJBZ0lKQUk4aGM0ZEMrVE0vTUEwR0NTcUdTSWIzRFFFQkN3VUFNRnd4Q3pBSkJnTlYKQkFZVEFrRlZNUk13RVFZRFZRUUlEQXBUYjIxbExWTjBZWFJsTVNFd0h3WURWUVFLREJoSmJuUmxjbTVsZENCWAphV1JuYVhSeklGQjBlU0JNZEdReEZUQVRCZ05WQkFNTURETTBMalkyTGpFM0xqRTJNREFlRncweU1UQTJNamt3Ck9EUXdORE5hRncweU1qQTJNamt3T0RRd05ETmFNRnd4Q3pBSkJnTlZCQVlUQWtGVk1STXdFUVlEVlFRSURBcFQKYjIxbExWTjBZWFJsTVNFd0h3WURWUVFLREJoSmJuUmxjbTVsZENCWGFXUm5hWFJ6SUZCMGVTQk1kR1F4RlRBVApCZ05WQkFNTURETTBMalkyTGpFM0xqRTJNRENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DCmdnRUJBS1lpblBOS3NqUzRYM09xcTJKc3NycndLREREUDI4dzZ3MmRRQnRROW1xMHIrOXpRRVBEUk5qWEo3UEcKaDNDT3V6UEtPeUw0WnFSZHdHbmZnTXRiMGdVTWtHVVZwTUxHV1FaRXJvS1pDSGcwOGhHRzRpTWxubWthK0NrUQp2OGNYYnNCOEpZcEd6SXJLdytGaG41NzdYeFRtZzRlK2cxLzZyeE1OMnlhUE9XMUNPdGpZTmpDNkNVeGFHWUt3CnVOQzg2NDJHbjhkR2IxbFZuVUF0SDVXajgvQUlvd1Z0cHB5ZXI0K0liWWJwRVlvN0l2WVd0NGd5OXZhWjJaaHkKS2h1akhpWmFIWVhpYmdMclZLU2VxTWFoOUNRWEVtRHltVHBkRCtieVc1ZnlLdnVoRXlmaWROelNMQjhDclhicwoxODR3MmFPVTUveVp2dndrV1dRK016bnZwL01DQXdFQUFhTlFNRTR3SFFZRFZSME9CQllFRkc0NmJPMklrQW9jCjc2d0J0L3RTTGRwb2NQWXJNQjhHQTFVZEl3UVlNQmFBRkc0NmJPMklrQW9jNzZ3QnQvdFNMZHBvY1BZck1Bd0cKQTFVZEV3UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFBeWlxVis4ZkJmanE4cGFuSmJTNlFnTwpKcDlRZTZpYWloQ2xwaHZCWVJnSVdnUkNrVkJjRmpCcGhEanlxajI2ZFZBSnhJMWFFYUswTXVqSVpaRHp5MXVLCmJRTXA4c0ZJcnUzbU5oZSt5bTBaVFl1eXRVRzVOMHJnQ0hmNWFlbk91T1krem5IdE1PY1BOQlFENGVnZEhlSXkKdVI4bG9FNGRWWkxkSGRLb3RGSjFHQlA2Qy9GWkxiK3NjSzhEM0UxeWhCZWZOUC9pMWh3RVB6Rk5WU2U3VzFOego4cSs3UWNudDRMajJsR2hIdE9KTmlabEdIYW5BbENnSVNoaXB4OU5RK1hUcjc4VHozOFBPL3hOVFN4V0hYTXV0CmEzS2p1UGRtSHNpd0FWenNoOFVSdUtyOHpFQ043Mk5NM25sWkVzdXNPMFo4NHg0dVdxVUhNNFR0YzFLM3ppZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  secretKey: SEFBTUFBTFVNSEFJQ0hBTE5B

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  name: demo-s3-target
spec:
  type: ObjectStore
  vendor: AWS
  objectStoreCredentials:
    region: us-east-1
    bucketName: trilio-browser-test
    credentialSecret:
      name: sample-secret
      namespace: TARGET_NAMESPACE
  thresholdCapacity: 5Gi

Example 4 - OVH Object Storage as Target using Swift S3 API

Sample YAML for OVH Object Storage. This is S3 compatible storage can be accessed using Swift S3 APIs and can be configured as a Target.

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  name: demo-ovh-s3-target
spec:
  type: ObjectStore
  vendor: other
  objectStoreCredentials:
    url: "https://s3.bhs.cloud.ovh.net"
    bucketName: demo-ovh-s3-bucket
    region: bhs
    credentialSecret:
      name: sample-ovh-s3-secret
      namespace: TARGET_NAMESPACE
  thresholdCapacity: 5Gi

To configure the OVH S3 Object Storage to use as a Target and to create access key, secret key follow the Configure OVH Object Storage as a Target section.

Example 5 - NFS Target Example

Sample YAML for NFS share.

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  name: sample-target
spec:
  type: NFS
  vendor: Other
  nfsCredentials:
    nfsExport: 192.168.1.1:/src/nfs/kubedata
    nfsOptions: nfsvers=4
  thresholdCapacity: 1000Gi

Additional values for the nfsOptions field can be found here

Example 6 - S3 Based Event Target Example

Any backup target can be designated as an event target. An event target is used to share state information between clusters with the shared event target. Two or more clusters can share the same event target.

Sample YAML file for S3-based event target is given below

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  annotations:
    trilio.io/event-target: "true"
  name: s3-event-target
  namespace: default
spec:
  objectStoreCredentials:
    bucketName: aj-test-s3
    credentialSecret:
      name: s3-cred-secret
      namespace: default
    region: us-east-1
  type: ObjectStore
  vendor: AWS

Example 7 - NFS Based Event Target Example

Sample YAML file for NFS-based event target is given below

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  annotations:
    trilio.io/event-target: "true"
  name: nfs-event-target
  namespace: default
spec:
  nfsCredentials:
    nfsExport: 34.66.17.160:/src/nfs/aj
    nfsOptions: nfsvers=4
  thresholdCapacity: 100Gi
  type: NFS
  vendor: Other

Policy

Trilio provides a Policy Custom Resource Definition through which scheduling, retention, and cleanup policies can be created.

Scheduling Policy

A scheduling policy can be created to automate the capture of applications within a Kubernetes system on a periodic basis. T4K enables users to create a scheduling policy with multiple cron strings defined within it. Each Cron string creates an associated cron job within the Kubernetes system and as a result daily, weekly, yearly, etc. policies can be created independently of each other within the same policy CR.

Below is an example of a scheduling policy CR:

kind: "Policy"
apiVersion: "triliovault.trilio.io/v1"
metadata:
  name: "test-all-sch"
spec:
  type: "Schedule"
  scheduleConfig:
    schedule:
      - "0 0 * * *"
      - "0 */1 * * *"
      - "0 0 * * 0"
      - "0 0 1 * *"
      - "0 0 1 1 *"

Retention Policy

The retention policy enables users to define the number of backups to retain and the cadence to delete backups as per compliance requirements. The retention policy CR provides a simple YAML specification to define the number of backups to retain in terms of days, weeks, months, years, latest etc.

Retention Polices are referenced within a BackupPlan and can be added to any of the example YAML files provided below

Example - Retention Policy

apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
  name: sample-policy
spec:
  type: Retention
  default: false
  retentionConfig:
    latest: 2
    weekly: 1
    dayOfWeek: Wednesday
    monthly: 1
    dateOfMonth: 15
    monthOfYear: March
    yearly: 1

Cleanup Policy

By default, Trilio does not delete failed backup job backup images or backup CRs. Users must manually delete failed backups. However, Trilio provides a cleanup policy to clean up failed backups. This policy only cleans up the backup files on the target associated with the failed backup job, not the backup CR itself. Users must use the kubectl command to delete any failed backup CRs from the cluster. If the spec.default is true and the policy is created in the T4K installed namespace, the policy applies to the entire cluster. If the spec.default is false, then the policy applies to the namespace where the policy was created. UI automatically sets the spec.default to false. A cron job that runs every 30 mins and deletes failed backups data based on the value specified by spec.cleanupConfig.backupdays. The policy does not delete any volume snapshots left behind by the failed backup jobs. Since backup CRs have an ownerRef on the volume snapshots, they will be automatically deleted when the failed backup CRs are deleted

An example cleanup policy that cleans up failed backups after five days is provided below.

apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
  name: sample-cleanup-policy
spec:
  type: Cleanup
  default: true
  cleanupConfig:
    backupDays: 5

Continuous Restore Policy

Continuous Restore policy defines the number of consistent sets on a remote site.

apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
  name: continuous-restore-policy
  namespace: default
spec:
  continuousRestoreConfig:
    consistentSets: 3
  default: false
  type: ContinuousRestore

Immutability

Trilio provides the ability to create immutable backups at the application level. Once the backup is taken and stored on an immutable target, it can not be altered (overwritten/deleted) until the retention period set through T4K is up.

Immutable Target

To create immutable backups, user needs to create an immutable target as shown in the sample below. The immutable targets work only on object storage targets.

kind: Target
apiVersion: triliovault.trilio.io/v1
metadata:
  name: sample-immutable-target
spec:
  type: ObjectStore
  vendor: AWS
  objectStoreCredentials:
    # immutable targets works only on object store targets
    region: us-east-1
    bucketName: trilio-browser-test
    credentialSecret:
      name: sample-secret
      namespace: TARGET_NAMESPACE    
  objectLockingEnabled: true
  thresholdCapacity: 5Gi

Retention Policy for immutable backup

After immutable target, user needs to create a retention policy. This sets up the retention period for the backup. Refer the sample Example - Retention Policy.

Immutable Backup

After immutable target and retention policy are set, user needs to create a backup. Once the backup is taken and stored on an immutable target, it can not be altered (overwritten/deleted) until the retention period set through T4K is up. Refer the BackupPlan and Backup samples below.

Example - BackupPlan with Retention Period

Sample for BackupPlan with Retention period

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-immutable-backupplan
spec:
  backupConfig:
    target:
      name: sample-immutable-target
      namespace: TARGET_NAMESPACE
    retentionPolicy:
      name: sample-retention-policy
      namespace: POLICY_NAMESPACE
    schedulePolicy:
      fullBackupPolicy:
        name: sample-schedule-policy
        namespace: POLICY_NAMESPACE
  backupPlanComponents:
    customSelector:
      selectResources:
        labelSelector:
          - matchLabels:
              app: gcp-compute-persistent-disk-csi-driver

Example - Immutable Backup

apiVersion: triliovault.trilio.io/v1
kind: Backup
metadata:
  name: sample-immutable-backup
spec:
  type: Full
  backupPlan:
    name: sample-immutable-backupplan
    namespace: BACKUP_NAMESPACE

Hooks

Hooks enable injecting commands into pods/containers before and after a backup via pre/post commands. Hooks enable taking application consistent backups and extending backup workflows.

Note: Hook should be created in the same namespace as that of BackupPlan referencing it resides.

Hook for MySQL

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: mysql-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "mysql --user=root --password=$MYSQL_ROOT_PASSWORD -Bse 'FLUSH TABLES WITH READ LOCK;system ${WAIT_CMD};'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "mysql --user=root --password=$MYSQL_ROOT_PASSWORD -Bse 'FLUSH LOGS; UNLOCK TABLES;'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating MySQL Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: mysql-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: mysql-hook
       podSelector:
         labels:
           - matchLabels:
               app: mysql-qa
         regex: mysql-qa*
       containerRegex: mysql-qa*

Hook for Cassandra

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: cassandra-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "nodetool flush -- demodb;"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "nodetool verify -- demodb;"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating Cassandra Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: cassandra-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: cassandra-hook
       podSelector:
         labels:
           - matchLabels:
               app: my-release-cassandra
         regex: my-release-cassandra*
       containerRegex: my-release-cassandra*

Hook for MongoDB

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: mongo-hook
spec:
  pre:
    execAction:
      command:
        - "/bin/bash"
        - "-c"
        - "/opt/bitnami/mongodb/bin/mongo --eval 'printjson(db.fsyncLock())' --host 'mongotest-mongodb' --authenticationDatabase admin -u root  -p $MONGODB_ROOT_PASSWORD"
    ignoreFailure: false
    maxRetryCount: 2
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "/bin/bash"
        - "-c"
        - "/opt/bitnami/mongodb/bin/mongo --eval 'printjson(db.fsyncUnlock())' --host 'mongotest-mongodb' --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD"
    ignoreFailure: false
    maxRetryCount: 2
    timeoutSeconds: 10

BackupPlan illustrating MongoDB Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: mongo-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: mongo-hook
       podSelector:
         regex: mongotest-mongodb*
       containerRegex: mongodb*

Hook for MariaDB

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: mariadb-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "mysql --user=root --password=$MARIADB_ROOT_PASSWORD -Bse 'FLUSH TABLES WITH READ LOCK;system ${WAIT_CMD};'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "mysql --user=root --password=$MARIADB_ROOT_PASSWORD -Bse 'FLUSH LOGS; UNLOCK TABLES;'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating MariaDB Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: mariadb-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: mariadb-hook
       podSelector:
         labels:
           - matchLabels:
               app.kubernetes.io/name: mariadb
         regex: mariadb*
       containerRegex: mariadb*

Hook for Redis

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: redis-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "echo 'SAVE' | redis-cli -h my-release-redis-master -a $REDIS_PASSWORD --no-auth-warning"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "echo 'post hook action completed'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating Redis Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: redis-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: redis-hook
       podSelector:
         labels:
           - matchLabels:
               app: redis
               role: master
         regex: my-release-redis-master*

Hook for PostgreSQL

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: postgres-hook
spec:
  pre:
    execAction:
      command:
        - "PGPASSWORD=$POSTGRES_PASSWORD; psql -U '$POSTGRES_USER' -c 'CHECKPOINT';"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "echo 'post hook action completed'"
    Ignore Failure:   false
    Max Retry Count:  1
    Timeout Seconds:  10

BackupPlan illustrating PostgreSQL Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: postgres-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: postgres-hook
       podSelector:
         labels:
           - matchLabels:
               app.kubernetes.io/name: postgresql
         regex: postgres-postgresql*
       containerRegex: postgres-postgresql*

Hook for InfluxDB

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: influxdb-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "bkpfile=/tmp/snap`date +%Y%m%dT%H%M`; influxd backup $bkpfile"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "echo 'post hook action completed'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating InfluxDB Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: influxdb-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 backupPlanComponents:
   custom:
     - matchLabels:
         app: influxdb
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: influxdb-hook
       podSelector:
         labels:
           - matchLabels:
               app: influxdb
         regex: influxdb*

Hook for Elasticsearch

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: es-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "curl -XPOST 'http://localhost:9200/test/_flush?pretty=true'; curl -H'Content-Type: application/json' -XPUT localhost:9200/test/_settings?pretty -d'{\"index\": {\"blocks.read_only\": true} }'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "curl -H'Content-Type: application/json' -XPUT localhost:9200/test/_settings?pretty -d'{\"index\": {\"blocks.read_only\": false} }'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating Elasticsearch Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: es-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: es-hook
       podSelector:
         labels:
           - matchLabels:
               app: master
               app.kubernetes.io/name: elasticsearch
         regex: my-release-elasticsearch-master*
       containerRegex: elasticsearch*

Hook for Kafka

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: kafka-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "/opt/bitnami/kafka/bin/kafka-server-stop.sh -daemon /opt/bitnami/kafka/config/server.properties"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "/opt/bitnami/kafka/bin/kafka-server-start.sh -daemon /opt/bitnami/kafka/config/server.properties"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating Kafka Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: kafka-backupplan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: kafka-hook
       podSelector:
         labels:
           - matchLabels:
               app.kubernetes.io/name: kafka
         regex: my-release-kafka*
       containerRegex: kafka*

Hook for CockroachDB

apiVersion: triliovault.trilio.io/v1
kind: Hook
metadata:
  name: cockroachdb-hook
spec:
  pre:
    execAction:
      command:
        - "bash"
        - "-c"
        - "bkpfile=/tmp/snap`date +%Y%m%dT%H%M`.sql;cockroach dump --dump-all --insecure > $bkpfile"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10
  post:
    execAction:
      command:
        - "bash"
        - "-c"
        - "echo 'post hook action completed'"
    ignoreFailure: false
    maxRetryCount: 1
    timeoutSeconds: 10

BackupPlan illustrating CockroachDB Hook

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
 name: cockroach-helm-backup-plan
spec:
 backupConfig:
   target:
     namespace: default
     name: demo-s3-target
   retentionPolicy:
     name: sample-policy
 hookConfig:
   mode: Sequential
   hooks:
     - hook:
         name: cockroachdb-hook
       podSelector:
         labels:
           - matchLabels:
               app.kubernetes.io/name: cockroachdb
               app.kubernetes.io/component: cockroachdb
         regex: my-release-cockroachdb-0*

BackupPlan

The BackupPlan CRD specifies the backup job. The specification includes the backup schedule, backup target and the resources to backup. Trilio supports three types of resources to backup and an BackupPlan CR may include combination of these resources.

This BackupPlan CR defines a set of resources to backup. Resources can be defined in the form of Helm release, Operators or just bare k8s api resources.

In this release, Trilio supports backup of the following:

1. Helm releases

2. Operator-based application instances

3. Label-based selection of resources

4. Namespaces

Type: Custom Label - Example 1

The following sample BackupPlan CR specifies a Label-based selection of resources. In this example any resource has a label app with valuegcp-compute-persistent-disk-csi-driver will be backed up.

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: sample-target
    schedulePolicy:
      fullBackupCron:
        schedule: "* 0 1 * *"
      incrementalCron:
        schedule: "* 0 * * *"
  backupPlanComponents:
    custom:
      - matchLabels:
          app: gcp-compute-persistent-disk-csi-driver

Type: Custom Label - Example 2 - Multiple Labels

The example below explains how a backupPlan CR can be used to protect data using multiple labels.

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: sample-target
  backupPlanComponents:
    custom:
      - matchLabels:
          app: frontend
      - matchLabels:
          app: backend

Type: Helm - Example 1 - Single Helm Release

The following sample BackupPlan CR that specifies Helm release based resources to backup. Trilio automatically discovers the resources that belong to the release and backups the helm chart as whole. The release that this example backups is mysql-releasename.

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
    schedulePolicy:
      fullBackupCron:
        schedule: "* 0 1 * *"
      incrementalCron:
        schedule: "* 0 * * *"
  backupPlanComponents:
    helmReleases:
      - mysql-releasename

Type: Helm - Example 2 - Multiple Helm Releases

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    helmReleases:
      - sample-release
      - sample-release1

Type Operator - Example 1

Another type of resource that Trilio supports is an Operator-based applications and the following YAML snippet describes the BackupPlan CR that includes the operator based application.

Example 1a

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    operators:
      - operatorId: sample-mysqlcluster
        customResources:
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlCluster"
            objects:
            - sample-mysqlcluster
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlBackup"
        operatorResourceSelector:
          - matchLabels:
              app: mysql-operator
              release: sample-release
        applicationResourceSelector:
          - matchLabels:
              app.kubernetes.io/managed-by: mysql.presslabs.org
              app.kubernetes.io/name: mysql

Example 1b

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    operators:
      - operatorId: etcd-cluster
        customResources:
          - groupVersionKind:
              group: "etcd.database.coreos.com"
              version: "v1beta2"
              kind: "EtcdCluster"
            objects:
              - etcd-cluster
        operatorResourceSelector: # sa, clusterrole, clusterrolebinding, deployment
          - matchLabels:
              app: etcd-operator
              release: sample-release
        applicationResourceSelector: # svc: etcd-cluster, etcd-cluster-client
          - matchLabels:
              app: etcd
              etcd_cluster: etcd-cluster

Type Operator - Example 2 - Multiple Operators

The YAML definition below provides an example of how multiple operators can be protected through the BackupPlan CR.

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    operators:
      - operatorId: sample-mysqlcluster
        customResources:
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlCluster"
            objects:
              - sample-mysqlcluster
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlBackup"
        operatorResourceSelector:
          - matchLabels:
              app: mysql-operator
              release: sample-release
        applicationResourceSelector:
          - matchLabels:
              app.kubernetes.io/managed-by: mysql.presslabs.org
              app.kubernetes.io/name: mysql
      - operatorId: sample-fluxcd-helm-release
        customResources:
          - groupVersionKind:
              group: "helm.fluxcd.io"
              kind: "HelmRelease"
              version: "v1"
            objects:
              - redis
        operatorResourceSelector:
          - matchLabels:
              app: helm-operator
              release: helm-operator
        applicationResourceSelector:
          - matchLabels:
              app: redis
              release: redis

Type: Operator - Example 3 - Helm based operator

Operator based applications can also be protected by providing the helm release name for the Operator resources.

Example 3a

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    operators:
      - operatorId: sample-mysqlcluster
        customResources:
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlCluster"
            objects:
            - sample-mysqlcluster
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlBackup"
        helmRelease: sample-release
        applicationResourceSelector:
          - matchLabels:
              app.kubernetes.io/managed-by: mysql.presslabs.org
              app.kubernetes.io/name: mysql

Example 3b

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    operators:
      - operatorId: sample-fluxcd-helm-release
        customResources:
          - groupVersionKind:
              group: "helm.fluxcd.io"
              kind: "HelmRelease"
              version: "v1"
            objects:
              - redis
        helmRelease: helm-operator
        applicationResourceSelector:
          - matchLabels:
              app: redis
              release: redis

Type: Operator - Example 4 - Helm based Operator without Operator Custom Resources

Operator Example where the Operator resources are defined via the helm release. No Operator custom resources are being backed up in this example.

Example 4a

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    operators:
      - operatorId: sample-mysqlcluster
        helmRelease: sample-release
        applicationResourceSelector:
          - matchLabels:
              app.kubernetes.io/managed-by: mysql.presslabs.org
              app.kubernetes.io/name: mysql

Example 4b

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    operators:
      - operatorId: sample-fluxcd-helm-release
        helmRelease: helm-operator
        applicationResourceSelector:
          - matchLabels:
              app: redis
              release: redis

Type: Operator - Example 5 - OLM operators

OLM or Openshift Operators can also be protected by providing the subscription name and namespace in operator resources. Example 5a. backups CockroachDB operator subscription named cockroachdb-certified along with its custom resource named crdb-tls-example of the kind CrdbCluster

Example 5a

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      name: demo-s3-target
      namespace: default
  backupPlanComponents:
    operators:
    - customResources:
      - groupVersionKind:
          group: crdb.cockroachlabs.com
          kind: CrdbCluster
          version: v1alpha1
        objects:
        - crdb-tls-example
      operatorId: crdb
      subscription:
        name: cockroachdb-certified
        namespace: openshift-operators

Example 5b. backups Openshift Pipelines operator subscription named openshift-pipelines-operator-rh along with its custom resources apply-manifests and update-deployment of the kind Task , and build-and-deploy of kind Pipeline

Example 5b

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
  namespace: pipelines-tutorial
spec:
  backupConfig:
    target:
      name: sample-target
      namespace: pipelines-tutorial
  backupPlanComponents:
    operators:
    - customResources:
      - groupVersionKind:
          group: tekton.dev
          kind: Task
          version: v1
        objects:
        - apply-manifests
        - update-deployment
      - groupVersionKind:
          group: tekton.dev
          kind: Pipeline
          version: v1
        objects:
        - build-and-deploy
      operatorId: pipelines
      subscription:
        name: openshift-pipelines-operator-rh
        namespace: openshift-operators

Type: All - Example with all 3 Application Types -1

The BackupPlan CRD is an extremely flexible CRD within which multiple application components can be specified.

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
  backupPlanComponents:
    custom:
      - matchLabels:
          triliobackupall: all
    helmReleases:
      - sample-release
    operators:
      - operatorId: sample-mysqlcluster
        customResources:
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlCluster"
            objects:
            - sample-mysqlcluster
          - groupVersionKind:
              group: "mysql.presslabs.org"
              version: "v1alpha1"
              kind: "MysqlBackup"
        operatorResourceSelector:
          - matchLabels:
              app: mysql-operator
              release: sample-release
        applicationResourceSelector:
          - matchLabels:
              app.kubernetes.io/managed-by: mysql.presslabs.org
              app.kubernetes.io/name: mysql

Type: Namespace - Example 1

For namespace level backup, BackupPlan components are optional

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: ns-backupplan-1
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target
      

Type: ClusterBackupPlan - Example 1

ClusterBackupPlan is used to protect multiple namespaces in the cluster. User can specify multiple namespaces to be backed up.

apiVersion: triliovault.trilio.io/v1
kind: ClusterBackupPlan
metadata:
  name: sample-clusterbplan
spec:
  backupConfig:
    target:
      name: sample-target
      namespace: TARGET_NAMESPACE
  retentionPolicy:
    name: sample-retention-policy
    namespace: POLICY_NAMESPACE
  schedulePolicy:
    fullBackupPolicy:
      name: sample-schedule-policy
      namespace: POLICY_NAMESPACE
  backupComponents:
    - namespace: NS_1
    - namespace: NS_2

Type: Inclusion/Exclusion

User can provide specific resource which needs to be included or excluded while doing backup. Resources can be specified either by GVKO or by the Kind directly.

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-backupplan
spec:
  backupConfig:
    target:
      name: sample-target
      namespace: TARGET_NAMESPACE
  includeResources:
    labelSelector:
      - matchLabels:
          triliobackupall: trilio
  excludeResources:
    labelSelector:
      - matchLabels:
          triliobackupall: trilio

Type: Encryption

Setup Master Encryption key using this

Create a secret for encrypting the backups.

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
type: Opaque
data:
  encryptKey: bXllbmNyeXB0aW9ua2V5

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: sample-application
spec:
  encryption:
    encryptionSecret:
      name: sample-secret
      namespace: BACKUPPLAN_NAMESPACE
  backupConfig:
    target:
      name: sample-target
    retentionPolicy:
      name: sample-retention-policy
  backupPlanComponents:
    helmReleases:
      - mysql

Backup

The Backup CRD takes a backup of the resources specified in the BackupPlan spec. It takes either a Full backup or an Incremental Backup. The first backup of the Application will always be a Full backup, even if the user specifies their backup type as Incremental.

Note: BackupPlan and Backup CR should be created in the same namespace.

The incremental backup includes all the YAML files and delta changes to PV data. Incremental backups are not complete by themselves and rely on all the previous incremental backups and the full backup.

Example 1 - Full Backup

The following sample Backup YAML for Full backup

apiVersion: triliovault.trilio.io/v1
kind: Backup
metadata:
  name: sample-backup
spec:
  type: Full
  backupPlan:
    name: sample-application
    namespace: default

Example 2 - Incremental Backup

Sample Backup YAML for incremental backup

apiVersion: triliovault.trilio.io/v1
kind: Backup
metadata:
  name: sample-backup
spec:
  type: Incremental
  backupPlan:
    name: sample-application
    namespace: default

The scheduleType the field is to specify if the backup should follow the backup schedule provided in the backupPlan or if it should be a one-time backup only.

Example 3 - ClusterBackup

Sample Backup YAML for ClusterBackup. This CR is used to back up multiple namespaces and should be used with ClusterBackupPlan. It takes either a Full backup or an Incremental Backup. The first backup will always be a Full backup, even if the user specifies their backup type as Incremental.

apiVersion: triliovault.trilio.io/v1
kind: ClusterBackup
metadata:
  name: sample-clusterbackup
spec:
  clusterBackupPlan:
    name: sample-clusterbplan
  type: Full

Cancel In-Progress Backup or ClusterBackup:

To cancel any in-progress backup or cluster backup, user can add triliovault.trilio.io/mark-for-cancel: "true" annotation to the backup yaml:

apiVersion: triliovault.trilio.io/v1
kind: Backup
metadata:
  annotations:
    triliovault.trilio.io/mark-for-cancel: "true"
  name: mysql-backup-1
  namespace: default
spec:
  backupPlan:
    apiVersion: triliovault.trilio.io/v1
    kind: BackupPlan
    name: mysql-bplan
    namespace: default
    resourceVersion: "4473"
    uid: eb33a8cc-b9e6-40f8-a6a2-60e62e2472aa
  type: Full

User can only cancel in-progress backup and canceling completed or available backups will be blocked.

Restore

The Restore CRD specifies the backup that resources need to be restored from. Resources can be restored to the same namespace or a different namespace. If a backup target with existing backups are created to a different cluster, those backups can be restored to the cluster. A migration or disaster recovery use case can be implemented using this functionality.

Example 1 - Restore from a specific backup

Sample YAML file to restore from a specific backup. The backup is identified by name sample-backup.

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default

Example 2 - Restore last successful backup

Sample YAML that restores latest backup of a BackupPlan.

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: BackupPlan
    backupPlan:
      name: sample-application
      namespace: default

Example 3 - Skip Restore

Trilio provides the capability to Skip the restoration of objects if they already exist in the namespace. This is achieved by using the skipIfAlreadyExists field within the Restore custom resource

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default
  skipIfAlreadyExists: true 

Example 4 - Patch Restore

Trilio provides the capability to Patch resources if they already existing in the namespace during the restore

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default
  patchIfAlreadyExists: true

Example 5 -Restore From Specific Location - Migration Scenario

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Location
    location: e90ce943-a0ce-11ea-93b2-42010a8e0038/66777359-a0d1-11ea-93b2-42010a8e0038
    target:
      name: sample-target
      namespace: default

Example 6 - Restore with Transformations (StorageClass)

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default
  transformComponents:
    helm:
      - release: mysql
        transformName: t1
        set:
          - key: persistence.storageClass
            value: insert.storage.class.here
    custom:
      - transformName: t3
        resources:
          groupVersionKind:
            group: ""
            version: v1
            kind: PersistentVolumeClaim
          objects:
            - insert-pvc1-to-be-replaced
            - insert-pvc2-to-be-replaced
        jsonPatches:
          - op: replace
            path: "/spec/storageClassName"
            value: insert.updated.storage.class.here

Example 7 - Restore with Transformations (NodePort)

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default
  transformComponents:
    custom:
    - transformName: t1
      resources:
        groupVersionKind:
          group: ""
          version: v1
          kind: Service
        objects:
          - insert-svc-name-to-be-updated
      jsonPatches:
        - op: replace
          path: "/spec/ports/0/nodePort"
          value: 31366

Example 8 - Restore with Exclusions

Exclude the specific resources from the backup by defining them with a labelSelector or gvkSelector in the excludeResourceSelector field.

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default
  excludeResourceSelector:
    labelSelector:
      - matchLabels:
          run: "nginx"
    gvkSelector:
      - groupVersionKind:
          group: ""
          version: "v1"
          kind: "ConfigMap"

Example 9 - Restore with Inclusions

Select the resources for restoration using the resourceSelector field. This allows the user to restore only the specified resources, rather than the entire backup.

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default
  resourceSelector:
    labelSelector:
      - matchLabels:
          app: "nginx"
    gvkSelector:
      - groupVersionKind:
          version: "v1"
          kind: "ServiceAccount"
        objects:
          - "default"

Example 10 - Restore from BackupPlan

Restore from BackupPlan enables users to restore the last successful backup based on a BackupPlan by providing only the name of the BackupPlan.

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: BackupPlan
    backupPlan:
      name: sample-backupplan-helm3
      namespace: default

Example 11 - ClusterRestore

ClusterRestore enables users to restore the last successful ClusterBackup based on a ClusterBackupPlan. It is used to restore multiple namespaces protected by the ClusterBackup. The ClusterBackup is identified by name sample-clusterbackup. It also provides a flag to cleanup in case of a failure.

apiVersion: triliovault.trilio.io/v1
kind: ClusterRestore
metadata:
  name: sample-clusterrestore
spec:
  source:
    type: ClusterBackup
    clusterBackup:
      name: sample-clusterbackup
  globalConfig:
    restoreFlags:
      skipIfAlreadyExists: true
  components:
    - backupNamespace: NS_1
      restoreNamespace: NS_RES_1
    - backupNamespace: NS_2
      restoreNamespace: NS_RES_2
  cleanupOnFailure: true

Example 12 - Encryption - Restore From Specific Location with encryption

Restore with encryption key is used to decrypt the encrypted data as shown in Type: ClusterBackupPlan - Example 1.

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Location
    location: e90ce943-a0ce-11ea-93b2-42010a8e0038/66777359-a0d1-11ea-93b2-42010a8e0038
    target:
      name: sample-target
  encryption:
    encryptionSecret:
      name: sample-secret
      namespace: SECRET_NAMESPACE