LogoLogo
4.0.X
4.0.X
  • About Trilio for Kubernetes
    • Welcome to Trilio For Kubernetes
    • Version 4.0.X Release Highlights
    • Compatibility Matrix
    • Marketplace Support
    • Features
    • Use Cases
  • Getting Started
    • Getting Started with Trilio on Red Hat OpenShift (OCP)
    • Getting Started with Trilio for Upstream Kubernetes (K8S)
    • Getting Started with Trilio for AWS Elastic Kubernetes Service (EKS)
    • Getting Started with Trilio on Google Kubernetes Engine (GKE)
    • Getting Started with Trilio on VMware Tanzu Kubernetes Grid (TKG)
    • More Trilio Supported Kubernetes Distributions
      • General Installation Prerequisites
      • Rancher Deployments
      • Azure Cloud AKS
      • Digital Ocean Cloud
      • Mirantis Kubernetes Engine
      • IBM Cloud
    • Licensing
    • Using Trilio
      • Overview
      • Post-Install Configuration
      • Management Console
        • About the UI
        • Navigating the UI
          • UI Login
          • Cluster Management (Home)
          • Backup & Recovery
            • Namespaces
              • Namespaces - Actions
              • Namespaces - Bulk Actions
            • Applications
              • Applications - Actions
              • Applications - Bulk Actions
            • Backup Plans
              • Create Backup Plans
              • Backup Plans - Actions
            • Targets
              • Create New Target
              • Targets - Actions
          • Trilio Monitoring
          • Resource Management
          • Guided Tours
        • UI How-to Guides
          • Multi-Cluster Management
          • Creating Backups
            • Cleanup Failed Backups
          • Restoring Backups
            • Cross-Cluster Restores
          • Monitoring Details
          • Disaster Recovery Plan
          • Continuous Restore
      • Command-Line Interface
        • YAML Examples
        • Trilio Helm Operator Values
    • Upgrade
    • Air-Gapped Installations
    • Uninstall
  • Reference Guides
    • T4K Pod/Job Capabilities
      • Resource Quotas
    • Trilio Operator API Specifications
    • Custom Resource Definition - Application
  • Advanced Configuration
    • AWS S3 Target Permissions
    • Management Console
      • KubeConfig Authenticaton
      • Authentication Methods Via Dex
      • UI Authentication
      • RBAC Authentication
      • Configuring the UI
    • Resource Request Requirements
      • Fine Tuning Resource Requests and Limits
    • Observability
      • Observability of Trilio with Prometheus and Grafana
      • Exported Prometheus Metrics
      • T4K Integration with Observability Stack
    • Modifying Default T4K Configuration
  • T4K Concepts
    • Supported Application Types
    • Support for Helm Releases
    • Support for OpenShift Operators
    • T4K Components
    • Backup and Restore Details
      • Immutable Backups
      • Application Centric Backups
    • Backup Retention Process
      • Retention Use Case
    • Continuous Restore
      • Architecture and Concepts
  • Performance
    • S3 as Backup Target
      • T4K S3 Fuse Plugin performance
    • Measuring Backup Performance
  • Ecosystem
    • T4K Integration with Slack using BotKube
    • Monitoring T4K Logs using ELK Stack
    • Rancher Navigation Links for Trilio Management Console
    • Optimize T4K Backups with StormForge
    • T4K GitHub Runner
    • AWS RDS snapshots using T4K hooks
    • Deploying Trilio For Kubernetes with Openshift ACM Policies
  • Krew Plugins
    • T4K QuickStart Plugin
    • Trilio for Kubernetes Preflight Checks Plugin
    • T4K Log Collector Plugin
    • T4K Cleanup Plugin
    • OCP ETCD Plugin
    • RKE ETCD Plugin
  • Support
    • Troubleshooting Guide
    • Known Issues and Workarounds
    • Contacting Support
  • Appendix
    • Ignored Resources
    • OpenSource Software Disclosure
    • CSI Drivers
      • Installing VolumeSnapshot CRDs
      • Install AWS EBS CSI Driver
    • T4K Product Quickview
    • OpenShift OperatorHub Custom CatalogSource
      • Custom CatalogSource in a restricted environment
    • Configure OVH Object Storage as a Target
    • Connect T4K UI hosted with HTTPS to another cluster hosted with HTTP or vice versa
    • Fetch DigitalOcean Kubernetes Cluster kubeconfig for T4K UI Authentication
    • Force Update T4K Operator in Rancher Marketplace
    • Backup and Restore Virtual Machines running on OpenShift
    • T4K For Volumes with Generic Storage
Powered by GitBook
On this page
  • Introduction to ETCD
  • RKE Cluster Backup & Disaster Recovery (DR)
  • ETCD backup and restore using rke-etcd-backup-restore
  • Important Notes & Plugin Prerequisites
  • Installation, Upgrade, Removal of Plugins
  • Linux/macOS
  • Usage
  • Arguments/Flags:
  • Arguments Details:
  • Examples:
  • Important Additional Information
  1. Krew Plugins

RKE ETCD Plugin

PreviousOCP ETCD PluginNextTroubleshooting Guide

Introduction to ETCD

Etcd is the persistent data store for Kubernetes. It is a distributed key-value store that records the state of all resources in a Kubernetes cluster. Etcd is a distributed reliable key-value store which is simple, fast and secure. It acts like a backend service discovery and database, runs on different servers in Kubernetes clusters at the same time to monitor changes in clusters and to store state/configuration data that should to be accessed by a Kubernetes master or clusters.

RKE Cluster Backup & Disaster Recovery (DR)

RKE clusters can be configured to take snapshots of etcd. In a disaster scenario, you can restore these snapshots. This snapshots can be shared outside cluster like s3 storage so that in case if we loose server, we will have backups to restore.

ETCD backup and restore using rke-etcd-backup-restore

The plugin helps user to perform ETCD backup and restore of RKE1 clusters and enables the user to store the snapshot on s3 storage created using T4K target.

Important Notes & Plugin Prerequisites

Important notes for the plugin
  1. Please do not switch of any node in cluster while restore is in progress and do not abort restore task in between, else you may loose cluster accessibility**

  2. Restore functionality will only work on same cluster from where the backup was taken**

  3. Restore will only work if cluster is accessible and one of the etcd nodes in the cluster should be up and running.

  4. Plugin is supported on RKE1 cluster(Local cluster created on Rancher server is not supported)

  5. Plugin supports ETCD backup and restore of downstream cluster created on Rancher server and NOT imported clusters.

  6. Backup T4K Target URL should have DNS name not IP.

Prerequisites
  1. krew - kubectl-plugin manager. Install from .

  2. kubectl - Kubernetes command-line tool. Install from .

  3. Trilio for Kubernetes and backup target. Install from .

ETCD backup restore use cases
  1. An etcd backup ensures that the cluster can be restored if an upgrade failure occurs

  2. You have deleted something critical in the cluster by mistake.

  3. You have lost the majority of your control plane hosts, leading to ETCD quorum loss.

Installation, Upgrade, Removal of Plugins

  • Add T4K custom plugin index of krew:

    kubectl krew index add tvk-interop-plugin https://github.com/trilioData/tvk-interop-plugins.git
  • Installation:

    kubectl krew install tvk-interop-plugin/rke-etcd-backup-restore
  • Upgrade:

    kubectl krew upgrade rke-etcd-backup-restore
  • Removal:

    kubectl krew uninstall rke-etcd-backup-restore
  1. Choose a version of preflight plugin to install and check if release assets have preflight plugin's package[rke-etcd-backup-restore-${OS}.tar.gz]

  2. To check OS & Architecture, execute command uname -a on linux/macOS

  3. Set env variable TVK_RKE_ETCD_BACKUP_RESTORE_VERSION=v1.x.x [update with your desired version]. If TVK_RKE_ETCD_BACKUP_RESTORE_VERSION is not exported, latest tagged version will be considered.

Linux/macOS

Bash or ZSH shells:

(
  set -ex; cd "$(mktemp -d)" &&
  OS="$(uname)" &&
  if [[ -z ${TVK_RKE_ETCD_BACKUP_RESTORE_VERSION} ]]; then version=$(curl -s https://api.github.com/repos/trilioData/tvk-interop-plugins/releases/ | grep -oP '"tag_name": "\K(.*)(?=")'); fi &&
  echo "Installing version=${TVK_RKE_ETCD_BACKUP_RESTORE_VERSION}" &&
  package_name="rke-etcd-backup-restore-${OS}.tar.gz" &&
  curl -fsSLO "https://github.com/trilioData/tvk-interop-plugins/releases/download/"${TVK_RKE_ETCD_BACKUP_RESTORE_VERSION}"/${package_name}" &&
  tar zxvf ${package_name} && sudo mv rke-etcd-backup-restore /usr/local/bin/kubectl-rke_etcd_backup_restore
)

Verify installation with: kubectl rke-etcd-backup-restore --help

Usage

ETCD Backup and restore on Rancher cluster. Available flags: -backup -restore.
   [-h] [-backup] [-restore] [--target-name TARGET_NAME]
   [--target-namespace TARGET_NAMESPACE] --rancher-url RANCHER_URL
   --bearer-token BEARER_TOKEN --cluster-name CLUSTER_NAME
   [--log-location LOG_LOC]

Arguments/Flags:

Parameter
Description

-backup

Flag to notify backup is to be taken.

-restore

Falg to notify restore is to be performed.

--target-name

The name of a single datastore on which etcd backup needs to be stored i.e. T4K target name.

--target-namespace

Namespace name in which T4K target is created.

--rancher-url

Rancher server URL

--bearer-token

--cluster-name

Cluster name to perform Backup/Restore on.

--log-location

Log file name along with path where the logs should be save default - /tmp/etcd-ocp-backup.log

Arguments Details:

  • -backup: Flag to notify the plugin to perform backup.

  • -restore: Flag to notify the plugin to perform restore.

  • --target-name: T4K target name.The target should be created and in available state. Currently S3 target type is supported. This target should be the target where the backups should be stored. This argument is mandatory if -backup flag is provided.

  • --target-namespace: Namespace name in which T4K target resides. This argument is mandatory if -backup flag is provided.

  • --rancher-url: This is the rancher server URL through which rancher can be accessed. should be given in the below form: "https://<rancher server ip>/" This is the URL to access rancher server. This is mandatory argument.

  • --cluster-name: Rancher server hosts many RKE cluster, so specify the one cluster name for which ETCD backup is to be taken. This is mandatory argument.

  • --log-location: specify the log file location. default: /tmp/etcd-ocp-backup.log

Examples:

A user may specify more than one option with each command execution. For example, to create a backup with a configured target name and associated namespace, and to set the cluster API URL with the associated bearer token, execute the following single command:

kubectl rke-etcd-backup-restore -backup --target-name <target-name> --target-namespace <target-namespace> --rancher-url <https://rancher server ip/> --bearer-token <bearer_token> --cluster-name <cluster_name>

Then, to restore from the same cluster API URL with the associated bearer token, execute the following single command:

kubectl rke-etcd-backup-restore -restore --rancher-url <https://rancher server ip/> --bearer-token <bearer_token> --cluster-name <cluster_name>

Restoring to a previous cluster state can be destructive and destabilizing action to take on a running cluster.

Important Additional Information

  1. If restoring the backup which is a different T4K version than the one you are currently using, the operation fails and cluster accessibility is lost. The workaround is to delete the current T4K and then re-try restoring.

  2. Supported GLIBC version ≥ 2.27.

  3. Plugin is tested with RKE1 with kubernetes version v1.21.9.

List of available releases:

Token to access rancher server

--bearer-token: This is the token provided by rancher server to access its cluster/apis without using password. More info about how to get bearer-token can be found at The scope of API key should be "No scope" as to access API's, plugin needs access to complete scope of Rancher server This is mandatory argument.

⚠️
here
here
here
https://github.com/trilioData/tvk-interop-plugins/releases
https://rancher.com/docs/rancher/v2.5/en/user-settings/api-keys/
More info here