1 of 96

T4O-4.1

About Trilio for OpenStack

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media
Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button
Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.
Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

Trilio for OpenStack Architecture

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.
Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.
Trilio Datamover is a python module that is installed on every OpenStack compute nodes

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Trilio 4.1 HF1 Release Notes

Release Versions

Packages

Deployment Guide

Requirements

Trilio has four main software components:

Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.
Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes
Trilio Datamover is a python module that is installed on every OpenStack compute nodes
Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance is not supported as an instance inside Openstack.

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

Software

Supported

The recommended size of the VM for the Trilio Appliance is:

When running Trilio in production, a 3-node cluster of the Trilio appliance is recommended for high availability and load balancing.

Ressource

Value

The qcow2 image itself defines the 40GB disk size of the VM.

In the rare case of the Trilio Appliance database or log files getting larger than 40GB disk, contact or open a ticket with Trilio Customer Success to attach another drive to the Trilio Appliance.

2.3. Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Each Openstack distribution comes with a set of supported operating systems. Please check the to see, which Openstack Distribution is supported with which Operating System.

Additional it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

Trilio network considerations

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

Preparing the installation

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

Spinning up the Trilio VM

For Canonical Openstack it is not necessary to spin up the Trilio VM.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Installing Trilio Components

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the installation process can begin. This process contains the following steps:

Install the Trilio dm-api service on the control plane.
Install the Trilio datamover service on the compute plane.
Install the Trilio Horizon plugin into the Horizon service.

How these steps look in detail is dependent on the Openstack distribution that Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio integrates into these deployment tools to provide a native integration from the beginning to the end.

Installing on Canonical Openstack

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

Trilio is not providing the JuJu Charms to deploy Trilio 4.1 in Canonical Openstack. These are developed and maintained by Canonical.

Installing on Kolla Victoria

The installation of Trilio for Openstack on Kolla Victoria with Trilio 4.1 is following this procedure:

Deplo T4O-4.1 GA Appliance
Upgrade to 4.1 HF5 or higher on the appliance
Deploy Trilio components of 4.1 HF5 or higher on the Kolla Openstack Victoria
Configure the Trilio appliance

Deplo T4O-4.1 GA Appliance

Please follow to spin up the base Trilio 4.1GA appliance.

Upgrade the T4O appliance to the latest 4.1 HF

Trilio supports Kolla Victoria from 4.1HF5 onwards, so it is recommended to upgrade to the latest available hotfix on 4.1 to make deployment successful. Please follow to upgrade the appliance to the latest 4.1 Hotfix.

Deploy Trilio components of 4.1 HF11 or higher

Run the deployment of the components following using the following values:

Variable

Value

Configure the Trilio Appliance

Please follow to configure the upgraded Trilio 4.1 appliance.

Installing on Ansible Openstack Victoria

The installation of Trilio for Openstack on Kolla Victoria with Trilio 4.1 is following this procedure:

Deplo T4O-4.1 GA Appliance
Upgrade to 4.1 HF5 packages on the appliance

Apply the Trilio license

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

Advanced Ceph configuration

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

Additions for multiple CEPH configurations

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

Additions for multiple Ceph users

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to adopt the keyring extension in the tvault-contego.conf inside the Ceph block.

The following example will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none


vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:[email protected]:5672,openstack:[email protected]:5672,openstack:[email protected]:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi


[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .keyring
ceph_dir = /etc/ceph/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper


[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

Uninstall Trilio

The uninstallation of Trilio is depending on the Openstack Distribution it is installed in. The high-level process is the same for all Distributions.

Uninstall the Horizon Plugin or the Trilio Horizon container
Uninstall the datamover-api container

Uninstalling from Canonical OpenStack

Trilio is not providing the JuJu Charms to deploy Trilio 4.1 in Canonical Openstack. At the time of release are the JuJu Charms not yet updated to Trilio 4.1. We will update this page once the Charms are available.

Uninstalling from Kolla OpenStack

Clean triliovault_datamover_api container

The container needs to be cleaned on all nodes where the triliovault_datamover_api container is running. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the triliovault_datamover_api container:

Stop the triliovault_datamover_api container.

Remove the triliovault_datamover_api container.

Clean /etc/kolla/triliovault-datamover-api directory.

Clean log directory of triliovault_datamover_api container.

Clean triliovault_datamover container

The container needs to be cleaned on all nodes where the triliovault_datamover container is running. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the triliovault_datamover container:

Stop the triliovault_datamover container.

Remove the triliovault_datamover container.

Clean /etc/kolla/triliovault-datamover directory.

Clean log directory of triliovault_datamover container.

Clean haproxy of Trilio Datamover API

The Trilio Datamover API entries need to be cleaned on all haproxy nodes. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the haproxy container:

Clean Kolla Ansible deployment procedure

Delete all Trilio related entries from:

To cross-verify the uninstallation undo all steps done in and .

Trilio entries can be found in:

/usr/local/share/kolla-ansible/ansible/roles/ ➡️ There is a role triliovault
/etc/kolla/globals.yml➡️ Trilio entries had been appended at the end of the file

Revert to original Horizon container

Run deploy command to replace the Trilio Horizon container with original Kolla Ansible Horizon container.

Clean Keystone resources

Trilio created a dmapi service with dmapi user.

Clean Trilio database resources

Trilio Datamover API service has its own database in the Openstack database.

Delete dmapi database and user.

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

Destroy the Trilio VMs

Undefine the Trilio VMs

Delete the TrlioVault VM disk from KVM Host storage

Upgrade Trilio

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

Old

New

4.0 GA (4.0.92)

4.0 SP1 (4.0.115)

The upgrade process contains upgrading the Trilio appliance and the Openstack components and is dependent on the underlying operating system.

The Upgrade of Trilio for Canonical Openstack is managed through the charms.

Switch NFS Backing file

Trilio is using a base64 hash for the mount point of NFS Backup targets. This hash makes sure, that multiple NFS Shares can be used with the same Trilio installation.

This base64 hash is part of the Trilio incremental backups as an absolute path of the backing files. This requires the usage of mount bind during a DR scenario or quick migration scenario.

In the case that there is time for a thorough migration there is another option to change the backing file and make the Trilio backups available on a different NFS Share. This option is updating the backing file to the new NFS Share mount point.

Trilio Appliance Administration Guide

Set Trilio GUI login banner

To configure the banner shown upon accessing the Trilio Appliance GUI do the following.

Login into Trilio Appliance console
edit the banner.yaml at /etc/tvault-config/banner.yaml
restart tvault-config service

The content of the banner.yaml looks as follows and can be edited as required:

Trilio Appliance Dashboard

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. This dashboard is accessible through the virtual IP.

If service status panels on the dashboard page are not visible then access the virtual IP on port 3001 (https://<TVO-VIP>:3001/) and accept the SSL exception, and then refresh the dashboard page.

It shows for each Trilio Appliance the Status of the following Trilio services:

wlm-workloads
wlm-scheduler
wlm-api
wlm-cron

The wlm-cron service runs on only one Trilio appliance at all times. That they are shown inactive on other nodes is not an error

To give administrators an overview of the HA status, does the dashboard also show the service status for:

Pacemaker
RabbitMQ
MySQL Galera Cluster \

Set network accessibility of Trilio GUI

By default is the Trilio GUI available on all NICs on port 443.

To limit this to only one IP the following steps need to be applied.

Network Setup

The Trilio Appliance provides by default the possibility of 4 VIPs.

Reconfigure the Trilio Cluster

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the Configuring Trilio guide afterwards.

Once the Trilio configurator has started, it needs to run through successfully to continue to use Trilio.

The cluster will not roll back to its last working state in case of any errors.

When the reconfiguration is required to switch to an external database it is necessary to and configure it from scratch.

Change the Trilio GUI password

Change Trilio Dashboard password

To change the Trilio GUI password do:

Reset the Trilio GUI password

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

The dashboard login will be reset to:

Reinitialize Trilio

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

Login into the Trilio Dashboard
Click on "admin" in the upper right corner to open the submenu

Download Trilio logs

It is possible to download the Trilio logs directly through the Trilio web gui.

To download logs throught the Trilio web gui:

Change Certificates used by Trilio

The following Trilio services are providing certificates for secured access to the Trilio solution.

Service

Port used

Description

User Guide

Schedulers

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

E-Mail Notifications

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

Both requirements need to be set or configured by the Openstack Administrator. Please contact your Openstack Administrator to verify the requirements.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

Login to Horizon
Navigate to the Backups
Navigate to Settings

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Admin Guide

Managing Trusts

Trilio is using the Openstack Keystone Trust system which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Openstack Administrators should never have the need to directly work with the trusts created.

The cloud-trust is created during the Trilio configuration and further trusts are created as necessary upon creating or modifying a workload.

Trusts can only be worked with via CLI

List all trusts

Show a trust

<trust_id> ➡️ ID of the trust to show

Create a trust

<role_name> ➡️Name of the role that trust is created for
--is_cloud_trust {True,False} ➡️ Set to true if creating cloud admin trust. While creating cloud trust use same user and tenant which used to configure Trilio and keep the role admin.

Delete a trust

<trust_id> ➡️ ID of the trust to be deleted

Disaster Recovery

Trilio Workloads are designed to allow a Desaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

Troubleshooting

General Troubleshooting Tips

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Using the workloadmgr CLI tool on the Trilio Appliance

To use the workloadmgr CLI tool on the Trilio appliance it is only necessary to activate the virtual environment of the workloadmgr

An rc-file to authenticate against Openstack is required.

Important log files

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log

API GUIDE

Trilio 4.1 Release Notes

Trilio release 4.1 introduces new features and capabilities including:

Openstack Ussuri support
New File Recovery Process
Increased Openstack independency
Installation Optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform
Support for External MySQL/MariaDB databases
Incremental Backup for nova booted instances
Support of Openstack User Groups
New Quota: Snapshots
S3 Support for Kolla Ansible Openstack
UI enhancement for selective Restore

Trilio 4.1.94 is the GA release of Trilio 4.1

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Name

Tag

Openstack Ussuri support

Trilio 4.1 continues to enable Openstack versions and Distributions, allowing Trilio customers to stay up to date with Openstack releases.

Trilio 4.1 introduces full support for Openstack Ussuri for Kolla Ansible Openstack, Ansible Openstack, and Canonical Openstack. In addition, Trilio 4.1 does of course support the active long-term releases from Red Hat and Canonical. Openstack users of those releases can continue to use the latest Trilio functionalities.

New File Recovery Manager Process

Since Trilio was released for the first time was the File Recovery Manager instance in tandem with Trilio. The File Recovery Manager instance helped customers to easily and quickly fetch and restore files and folders directly from the backups.

Over the years did more and more customers request to have the File Recovery Manager on their own images or on a specific Linux distribution. We looked into the possibility to create multiple versions of the File Recovery Manager which was identified as not getting to the point of flexibility our customers need.

The result was a revamp of the File Recovery Manager which allows the installation on any CentOS-based or Ubuntu-based instance.

Increased Openstack independency

Trilio has the goal to become an integrated part of Openstack since its first draft on a whiteboard. Back then were the Trilio services defined as sub-services of Nova and Trilio tied deeply into the already existing nova services and became an integral part of it.

Over the years did the nova service change and possibilities that were once available have been taken out from nova. This plus the required lengthy qualification of all Openstack Versions and Distributions made it clear that a change is required.

Trilio 4.1 is now a complete stand-alone service, which is communicating with nova and other Openstack services through APIs only. The goal of this independence is to speed up future qualification and requalification cycles to provide a broader support matrix again.

Installation optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform

Trilio's integration into Openstack already starts with the installation process. This process required manual installation steps in the past, which were hard to automize and scale on bigger environments.

Trilio 4.1 is therefore introducing Ansible Playbooks for Kolla-Ansible and Ansible Openstack, while the integration with Red Hat Director has been deepened. Further, the Trilio configurator does now allow to set the backup target on the Trilio Appliance directly based on Distribution.

Support for external MySQL/MariaDB

The Trilio Appliance provides its own database. A database that suits the needs of 99% of Trilio's customers. Some customers do have higher requirements. Be it performance, security, or just the general design of the Openstack itself.

For these customers and everyone who wants to use it is Trilio 4.1 introducing the possibility to configure Trilio with an external database.

Incremental Backups for nova booted instance root volumes

Openstack provides the possibility to start an instance from a Cinder Volume or directly using the Glance Image and a nova volume.

Trilio always provided incremental forever backups for all cinder volumes. Root-Volumes from nova-booted instances were always taken as a full backup of the Glance image and the actual VM root volume.

Trilio 4.1 does introduce incremental backups for nova booted instance root volumes. This allows Trilio 4.1 to provide incremental backups to any type of root volume.

Support for Openstack User Groups

Trilio is using the Openstack Keystone service to authenticate any user and to verify that the right permissions are set. Openstack Keystone allows to group users and set the permissions to the group.

These Openstack User Groups are now fully supported.

New Quota: Snapshots

Trilio 4.0 introduced the Quota functionality, which allowed to set quotas for the number of workloads, number of VMs and amount of storage used by a single tenant.

Trilio 4.1 extends this feature by the number of Trilio Snapshots that a Tenant is allowed to have.

S3 Support for Kolla Ansible Openstack

S3 is becoming the standard to transfer data to and from storage solutions. Trilio introduced S3 already in Version 3.0 but had to take it out for Kolla Ansible since Kolla Ansible has been added to the Support Matrix.

Trilio 4.1 is now closing that gap to other Openstack Distributions and provides full S3 support for Kolla Ansible Openstack Ussuri.

UI enhancement for the Selective Restore

The Selective Restore is the most powerful and complex restore Trilio has to offer. The UI needs to be easy to understand and help the user to fulfill its task.

Several points have been identified to improve this requirement of usability. The selective restore now allows to select or deselect all VMs at once. Further are the VMs now provided in an easy to overview list and the sub-controls can be expanded and collapsed as necessary.

Known Issues

This release contains the following known issues which are tracked for a future update.

Snapshots getting stuck in uploading stage on RHOSP13 with ISCSI based Volumes

Observation:

Login into the iscsi device is getting rejected for the Trilio service
Result is the Snapshot not moving forward until timing out

Workloadmgr Quota feature does not support RHOSP13 & Canonical Queens Horizon

The workloadmgr Quota feature is still fully supported through CLI.

[Openstack Ansible] snapshot mount failed with error 'Permission denied'

Observation:

It has been observed that on non-kolla, non-rhosp setups, such as openstack ansible, nova user id is not same as we consider as default(162).
Another observation is that, the id which is assigned to nova, was conflicting with id of system user in TVM, this created a situation where we had to redeploy Openstack.

Workaround:

Update permissions of /var/triliovault-mounts to 755

[Volume Backup Exclude] Excluded Ceph Volume after restore not mountable or formattable

Observation:

VM Volumes stored on Ceph are successfully excluded from backup if desired
Restore does create empty Ceph Volume
created empty Ceph Volume is not attachable or formattable

Restored VMs have blank metadata config_drive attached

Observation:

For every restore will the metadata config_drive be set as blank value
No impact on restored VMs known

Workaround

delete metadata config_drive
or set desired value

TVM reconfig fails when adding new Trilio VM to the cluster

Observation:

TVault re-configuration while adding nodes to existing TVM cluster fails at "Configuring Trilio Cluster"
Reason is that the prev mysql password was not working and mysql root access has be reset.

Workaround:

remove /root/.my.cnf file on already configured TVM and reconfigure it

Database does not sync after Trilio cluster gets new nodes

Observation:

After TVault re-configuration post addition of 2 more nodes to existing TVM cluster ("import workloads" was not seleted), the databases do not sync against already existing TVM.
It is expected that while adding the 2 new nodes, the db on node1 should get synced up with 2 new nodes and the existing workloads should be available post the reconfig on the new 3 node TVM cluster.

Workaround:

Run workload import from CLI

[exclude_boot_disk] Data on boot disk gets backed up despite exclusion

Observation:

VM was set with metadata exclude_boot_disk_from_backup set to true
Restored instance showed, that data was backed up and restored

After reinitialize and import Openstack certificates are missing

Observation:

Reinitialize does not keep the already uploaded Openstack Certificates used to communicate with Openstack.

Workaround:

Upload Certificates again

CLI import changes scheduler trust value to disabled

Observation:

When the is used via CLI is the scheduler trust changed from enabled to disabled.

Workaround:

Configure/re-configure T4O with import option from UI after reinitialize.

Unable to get node details after reinitializing the Trilio Appliance

Observation:

After reinitializing was neither the UI nor CLI showing Node information

Workaround:

Restart wlm-workloads and wlm-cron services on Trilio nodes
systemctl restart wlm-workloads
systemctl restart wlm-cron

Snapshots fails with "object is not subscriptable" for many workload jobs at the exact same time

Oberservation:

Running more than 25 workloads at the exact same time leads to error
dmapi service is not responding
Snapshots fail with "object is not subscriptable"

Workaround:

Contact Trilio Support to implement a known workaround.

[Kolla Ansible] dmapi container stuck in restarting if storage backend changes

Observation:

Just changing the backup target in the Kolla Ansible configuration files and redeploying leads to dmapi container stuck in restart

Workaround:

Follow guide to

No operation is permitted in insecure way for ssl enabled Keystone URL

Observation:

SSL enabled Openstack
Backup and Restore jobs fail with with missing TLS CA certificate bundle error

Workaround:

Configure the Trilio appliance with Openstack CA provided
OR Provide Openstack CA to /etc/workloadmgr/ca-chain.pem

Snapshot Mount

Definition

Trilio allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.

Create a File Recovery Manager Instance

It is recommended to do these steps once to the chosen cloud-Image and then upload the modified cloud image to Glance.

Create an Openstack image using a Linux based cloud-image like Ubuntu, CentOS or RHEL with the following metadata parameters.

Spin up an instance from that image It is recommended to have at least 8GB RAM for the mount operation. Bigger Snapshots can require more RAM.

Steps to apply on CentOS and RHEL cloud-images

install and activate qemu-guest-agent
Edit /etc/sysconfig/qemu-ga and remove the following from BLACKLIST_RPC section

Disable SELINUX in /etc/sysconfig/selinux

Install python3 and lvm2

Reboot the Instance

Steps to apply on Ubuntu cloud-images

install and activate qemu-guest-agent

Verify the loaded path of qemu-guest-agent

Loaded path init.d (Ubuntu 18.04)

Follow this path when systemctl returns the following loaded path

Edit /etc/init.d/qemu-guest-agent and add Freeze-Hook file path in daemon args

Loaded path systemd (Ubuntu 20.04)

Follow this path when systemctl returns the following loaded path

Edit qemu-guest-agent systemd file

Add the following lines

Finalize the FRM on Ubuntu

Restart qemu-guest-agent service

Install Python3

Reboot the VM

Mounting a Snapshot

Mounting a Snapshot to a File Recovery Manager provides read access to all data that is located on the in the mounted Snapshot.

It is possible to run the mounting process against any Openstack instance. During this process will the instance be rebooted.

Always mount Snapshots to File Recovery Manager instances only.

To be able to successfully mount Windows (NTFS) Snapshots the ntfs filesystem support is required on the File Recovery Manager instance.

Unmount any mounted Snapshot once there is no further need to keep it mounted. Mounted Snapshots will not be purged by the Retention policy.

Using Horizon

There are 2 possibilities to mount a Snapshot in Horizon.

Through the Snapshot list

To mount a Snapshot through the Snapshot list follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads

Should all instances of the project be listed and there is a File Recovery Manager instance existing verify together with the administrator that the File Recovery Manager image has the following property set:

tvault_recovery_manager=yes

Through the File Search results

To mount a Snapshot through the File Search results follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads

tvault_recovery_manager=yes

Using CLI

<snapshot_id> ➡️ ID of the Snapshot to be mounted
<mount_vm_id> ➡️ ID of the File Recovery Manager instance to mount the Snapshot to.

Accessing the File Recovery Manager

The File Recovery Manager is a normal Linux based Openstack instance.

It can be accessed via SSH or SSH based tools like FileZila or WinSCP.

SSH login is often disabled by default in cloud-images. Enable SSH login if necessary.

The mounted Snapshot can be found at the following path:

/home/ubuntu/tvault-mounts/mounts/

Each VM in the Snapshot has its own directory using the VM_ID as the identifier.

Identifying mounted Snapshots

Sometimes a Snapshot is mounted for a longer time and it needs to be identified, which Snapshots are mounted.

Using Horizon

There are 2 possibilities to identify mounted Snapshots inside Horizon.

From the File Recovery Manager instance Metadata

Login to Horizon
Navigate to Compute
Navigate to Instances

The mounted_snapshot_url contains the Snapshot ID of the Snapshot that has been mounted last.

This value only gets updated, when a new Snapshot is mounted.

From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

--workloadid <workloadid> ➡️ Restrict the list to snapshots in the provided workload

Unmounting a Snapshot

Once a mounted Snapshot is no longer needed it is possible and recommended to unmount the snapshot.

Unmounting a Snapshot frees the File Recovery Manager instance to mount the next Snapshot and allows Trilio retention policy to purge the former mounted Snapshot.

Deleting the File Recovery Manager instance will not update the Trilio appliance. The Snapshot will be considered mounted until an unmount command has been received.

Using Horizon

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using the CLI

<snapshot_id> ➡️ ID of the snapshot to unmount.

Restart Trilio Services

In complex environments it is sometimes necessary to restart a single service or the complete solution. Rarely is restarting the complete node, where a service is running possible or even the ideal solution.

This page describes the services running by Trilio and how to restart those.

Trilio Appliance Services

The Trilio Appliance is the controller of Trilio. Most services on the Appliance are running in a High Availability mode on a 3-node cluster.

wlm-api

The wlm-api service takes the API calls against the Trilio Appliance. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-api service run on each Trilio node:

wlm-scheduler

The wlm-scheduler service is taking job requests and identifies which Trilio node should take the request. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-scheduler service run on each Trilio node:

wlm-workloads

The wlm-workloads service is the task worker of Trilio executing all jobs given to the Trilio node. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-workloads service run on each Trilio node:

wlm-cron

The wlm-cron service is responsible for starting scheduled Backups according to the configurtation of Tenant Workloads. It is running in active-passive mode and controlled by the pacemaker cluster.

To restart the wlm-workloads service run on the Trilio node with VIP assigned:

VIP resources

The Trilio appliance is running 1 to 4 virtual IPs on the Trilio cluster. These are controlled by the pacemaker cluster and provided through NGINX.

To restart these resources the pacemaker NGINX resource is getting restarted:

RabbitMQ

The Trilio cluster is using RabbitMQ as messaging service. It is running in active-active mode on all nodes of the Trilio cluster.

RabbitMQ is a complex system in itself. This guide will only provide the basic commands to do a restart of a node and check the health of the cluster afterward. For complete documentation of how to restart RabbitMQ, please follow the .

To restart a RabbitMQ node run on each Trilio node:

It is recommended to wait for the node to rejoin and sync with the cluster before restarting another RabbitMQ node.

When the complete cluster is getting stopped and restarted it is important to keep the order of nodes in mind. The last node to be stopped needs to be the first node to be started.

Galera Cluster (MariaDB)

The Galera Cluster is managing the Trilio MariaDB database. It is running in active-active mode on all nodes of the Trilio cluster.

Galera Cluster is a complex system in itself. This guide will only provide the basic commands to do a restart of a node and check the health of the cluster afterward. For complete documentation of how to restart Galera clusters, please follow the .

When restarting Galera two different use-cases need to be considered:

Restarting a single node
Restarting the whole cluster

Restarting a single node

A single node can be restarted without any issues. It will automatically rejoin the cluster and sync against the remaining nodes.

The following commands will gracefully stop and restart the mysqld service.

After a restart will the cluster start the syncing process. Don't restart node after node to reach a complete cluster restart.

Check the cluster health after the restart.

Restarting the complete cluster

Restarting a complete cluster requires some additional steps as the Galera cluster is basically destroyed once all nodes have been shut down. It needs to be rebuild afterwards.

First gracefully shutdown the Galera cluster on all nodes:

The second step is to identify the Galera node with the latest dataset. This can be achieved by reading the grastate.dat file on the Trilio nodes.

When this documentation is followed the last mysqld service that got shut down will be the one with the latest dataset.

The value to check for are the seqno.

The node with the highest seqno is the node that contains the latest data. This node will also contain safe_to_bootstrap: 1 to indicate that the Galera cluster can be rebuild from this node.

On the identified node the new cluster is getting generated with the following command:

Running galera_new_cluster on the wrong node will lead to data loss as this command will set the node the command is issued on as the first node of the cluster. All nodes which join afterward will sync against the data of this first node.

After the command has been issued is the mysqld service running on this node. Now the other nodes can be restarted one by one. The started nodes will automatically rejoin the cluster and sync against the master node. Once a synced status has been reached is each node a primary node in the cluster.

Check the Cluster health after all services are up again.

Verify Health of the Galera Cluster

Verify the cluster health by running the following commands inside each Trilio MariaDB. The values returned from these statements have to be the same for each node.

Canonical workloadmgr container services

Canonical Openstack is not using the Trilio Appliance. In Canonical environments is the Trilio controller unit part of the JuJu deployment as workloadmgr container.

To restart the services inside this container the following commands are to be issued.

Single Node deployment

HA deployment

On all nodes:

On a single node:

Trilio dmapi service

The Trilio dmapi service is running on the Openstack controller nodes. Depending on the Openstack Distribution Trilio is installed on different commands are issued to restart the dmapi service.

RHOSP13

RHOSP13 is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

RHOSP16

RHOSP16 is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Canonical

Canonical is running the Trilio services in JuJu controlled LXD containers. The dmapi service can be restarted by issuing the following command from the MASS node.

Kolla-Ansible Openstack

Kolla-Ansible Openstack is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Ansible Openstack

Ansible Openstack is running the Trilio services as LXD containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Trilio datamover service (tvault-contego)

The Trilio datamover service is running on the Openstack compute nodes. Depending on the Openstack Distribution Trilio is installed on different commands are issued to restart the datamover service.

RHOSP13

RHOSP13 is running the Trilio services as docker containers. The datamover service can be restarted by issuing the following command on the compute node.

RHOSP16

RHOSP16 is running the Trilio services as docker containers. The datamover service can be restarted by issuing the following command on the compute node.

Canonical

Canonical is running the Trilio services in JuJu controlled LXD containers. The datamover service can be restarted by issuing the following command from the MASS node.

Kolla-Ansible Openstack

Kolla-Ansible Openstack is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Ansible Openstack

Ansible Openstack is running the Trilio datamover service directly on the compute node. The datamover service can be restarted by issuing the following command on.

Workloads

Definition

A workload is a backup job that protects one or more Virtual Machines according to a configured policy. There can be as many workloads as needed. But each VM can only be part of one Workload.

List of Workloads

Using Horizon

To view all available workloads of a project inside Horizon do:

Login to Horizon
Navigate to Backups
Navigate to Workloads

The overview in Horizon lists all workloads with the following additional information:

Creation time
Workload Name
Workload description

Using CLI

--all {True,False}➡️List all workloads of all projects (valid for admin user only)
--nfsshare <nfsshare>➡️List all workloads of nfsshare (valid for admin user only)

Workload Create

Using Horizon

To create a workload inside Horizon do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads

The created Workload will be available after a few seconds and starts to take backups according to the provided schedule and policy.

Using CLI

--display-name➡️Optional workload name. (Default=None)
--display-description➡️Optional workload description. (Default=None)

Workload Overview

A workload contains many information, which can be seen in the workload overview.

Using Horizon

To enter the workload overview inside Horizon do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads

Details Tab

The Workload Details tab provides you with the general most important information about the workload:

Name
Description
Availability Zone

The status of the qemu-guest-agent just shows, whether the necessary Openstack configuration has been done for this VM to provide qemu guest agent integration. It does not check, whether the qemu guest agent is installed and configured on the VM.

It is possible to navigate to the protected VM directly from the list of protected VMs.

Snapshots Tab

The Workload Snapshots Tab shows the list of all available Snapshots in the chosen Workload.

From here it is possible to work with the Snapshots, create Snapshots on demand and start Restores.

Please refer to the and User Guide to learn more about those.

Policy Tab

The Workload Policy Tab gives an overview of the current configured scheduler and retention policy. The following elements are shown:

Scheduler Enabled / Disabled
Start Date / Time
End Date / Time

Filesearch Tab

The Workload Filesearch Tab provides access to the powerful search engine, which allows to find files and folders on Snapshots without the need of a restore.

Please refer to the File Search User Guide to learn more about this feature.

Misc. Tab

The Workload Miscellaneous Tab shows the remaining metadata of the Workload. The following information are provided:

Creation time
last update time
Workload ID

Using CLI

<workload_id> ➡️ ID/name of the workload to show
--verbose➡️option to show additional information about the workload

Edit a Workload

Workloads can be modified in all components to match changing needs.

Editing a Workload will set the User, who edits the Workload, as the new owner.

Using Horizon

To edit a workload in Horizon do the following steps:

Login to the Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

--display-name ➡️ Optional workload name. (Default=None)
--display-description➡️Optional workload description. (Default=None)

Delete a Workload

Once a workload is no longer needed it can be safely deleted.

All Snapshots need to be deleted before the workload gets deleted. Please refer to the User Guide to learn how to delete Snapshots.

Using Horizon

To delete a workload do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads

Using CLI

<workload_id> ➡️ ID/name of the workload to delete
--database_only <True/False>➡️Keep True if want to delete from database only.(Default=False)

Unlock a Workload

Workloads that are actively taking backups or restores are locked for further tasks. It is possible to unlock a workload by force if necessary.

It is highly recommend to use this feature only as last resort in case of backups/restores being stuck without failing or a restore is required while a backup is running.

Using Horizon

Login to the Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

<workload_id> ➡️ ID of the workload to unlock

Reset a Workload

In rare cases it might be necessary to start a backup chain all over again, to ensure the quality of the created backups. To not recreate a Workload in such cases is it possible to reset a Workload.

The Workload reset will:

Cancel all ongoing tasks
Delete all existing Openstack Trilio Snapshots from the protected VMs
recalculate the next Snapshot time

Using Horizon

To reset a Workload do the following steps:

Login to the Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

<workload_id> ➡️ ID/name of the workload to reset

Online upgrade Trilio Appliance

This describes the upgrade process from Trilio 4.0 or Trilio 4.0SP1 to Trilio 4.1 GA or its hotfix releases.

Kolla Ansible Openstack only: The mount point for the Trilio Backup Target has changed in Trilio 4.1. A reconfiguration after the upgrade is required.

Generic Pre-requisites

The prerequisites should already be fulfilled from upgrading the Trilio components on the Controller and Compute nodes.

Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of TVM.
The mentioned Gemfury repository should be accessible from TVault VM.
Please ensure the following points before starting the upgrade process:

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it has been completely shut down.

Verify if the service is shut down with the below set of commands and expected output:

Backup old configuration data

Take a backup of the conf files on all TVM nodes.

Setup Python3.8 virtual environment

Check if Python 3.8 virtual environment exists on the T4O nodes

If the virtual environment does not exist, perform the below steps on the T4O nodes

Setup Python3.6 virtual environment

Activate the Python3.6 virtual environment on all T4O nodes for wlm services upgrade

[T4O 4.0 to T4O 4.1 only] uninstall Ansible

Ansible doesn't support the upgrade from previous versions to the latest one (2.10.4) and needs to be uninstalled for that reason

Upgrade pip package

Run the following command on all TVM nodes to upgrade the pip package

Set pip package repository env variable

Upgrade s3fuse/tvault-object-store

Major Upgrade

Run the following commands on all TVM nodes to upgrade s3fuse and its dependent packages.

Hotfix Upgrade

Run the following commands on all TVM nodes to upgrade s3fuse packages only.

Upgrade tvault-configurator

Post upgrade, the password for T4O configurator will be reset to the default one i.e. 'password' for user 'admin'. Reset T4O configurator password after the upgrade.

Make sure the correct virtual environment(myansible_3.8) has been activated

Major Upgrade

Run the following command on all TVM nodes to upgrade tvault-configurator and its dependent packages.

Hotfix Upgrade

Run the following command on all TVM nodes to upgrade tvault-configurator packages only.

During the update of the tvault-configurator the following error might be shown:

This error can be ignored.

Upgrade workloadmgr

Major Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr and its dependent packages.

Hotfix Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr packages only.

Upgrade workloadmgrclient

Major Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr and its dependent packages.

Hotfix Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr packages only.

Upgrade contegoclient

Major Upgrade

Run the upgrade command on all TVM nodes to upgrade contegoclient and its dependent packages.

Hotfix Upgrade

Run the upgrade command on all TVM nodes to upgrade contegoclient packages only.

Set oslo.messaging version

Using the latest available oslo.messaging version can lead to stuck RPC and API calls.

It is therefore required to fix the oslo.messaging version on the TVM.

Post Upgrade Steps

Restore the backed-up config files

[Major Upgrade 4.0 to 4.1 only] Delete wlm-scheduler pcs resource

Delete the wlm-scheduler pcs resource because in 4.1 it is not a part of pcs

Restart services

Restart the following services on all node(s) using respective commands\

tvault-object-store restart required only if Trilio is configured with S3 backend storage

Enable Global Job Scheduler ****Restart pcs resources only on the primary node

Verify the status of the services

tvault-object-store will run only if TVault configured with S3 backend storage

Additional check for wlm-cron on the primary node

The above command should show only 2 processes running: sample below:

Check the mount point using “df -h” command

[Upgrade to HF1 and higher only] Reconfigure the Trilio Appliance

Trilio for Openstack 4.1 HF1 is introducing several new config parameters, which will be automatically set upon reconfiguration.

[RHOSP and Kolla only] Reconfigure the Trilio Appliance

Trilio for Openstack 4.1 is changing the Trilio mount point as follows:

RHOSP 13 & 16.0 & 16.1: /var/lib/nova/triliovault-mounts Kolla Ansible Ussuri: /var/trilio/triliovault-mounts

Reconfiguring the Trilio Appliance will automatically handle this change.

[RHOSP and Kolla only] Create the mount bind to the old Trilio Mountpoint

Trilio for Openstack 4.1 is changing the Trilio mount point as follows:

RHOSP 13 & 16.0 & 16.1: /var/lib/nova/triliovault-mounts Kolla Ansible Ussuri: /var/trilio/triliovault-mounts

After reconfiguration of the Trilio Appliance, it is necessary to create a mount bind between the old and new mount points to provide full access to the old Trilio backups.

For RHOSP:

For Kolla:

To have this change persistent it is recommended to change the fstab accordingly:

For RHOSP:

For Kolla:

[RHOSP and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436,

In case of the UID/GID is changed back to 162:162 follow these steps to set it back to 42436:42436.

Download the shell script that will change the user id
Assign executable permissions
Execute the script

Upgrading on RHOSP

0] Pre-requisites

Please ensure the following points are met before starting the upgrade process:

No Snapshot or Restore is running
Global job scheduler is disabled
wlm-cron is disabled on the Trilio Appliance

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

1.] [On Undercloud node] Clone latest Trilio repository and upload Trilio puppet module

All commands need to be run as user 'stack' on undercloud node

The Trilio appliance connected to this installation needs to be of version 4.1 HF10

1.1] Clone Trilio cfg scripts repository

Separate directories are created as per Redhat OpenStack release under 'triliovault-cfg-scripts/redhat-director-scripts/' directory. Use all scripts/templates from respective directory. For ex, if your RHOSP release is 13, then use scripts/templates from 'triliovault-cfg-scripts/redhat-director-scripts/rhosp13' directory only.

Available RHOSP_RELEASE___DIRECTORY values are:

rhosp13 rhosp16.1 rhosp16.2

RHOSP 16.0 is not supported anymore as RedHat has officially stopped supporting it. However, Trilio maintained it for some time and stopped the support from 4.1HF11 onwards. The latest hotfix available for RHOSP16.0 is 41.HF10. Reach out to the Support team for any help.

1.2] If backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into the puppet directory of the right release.

2] Upload Trilio puppet module

3] Update overcloud roles data file to include Trilio services

Trilio has two services as explained below. You need to add these two services to your roles_data.yaml. If you do not have customized roles_data file, you can find your default roles_data.yaml file at /usr/share/openstack-tripleo-heat-templates/roles_data.yaml on undercloud.

You need to find that role_data file and edit it to add the following Trilio services.

i) Trilio Datamover Api Service:

Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamoverApi

This service needs to be co-located with database and keystone services. That said, you need to add this service on the same role as of keystone and database service.

Typically this service should be deployed on controller nodes where keystone and database runs. If you are using RHOSP's pre-defined roles, you need to addOS::TripleO::Services::TrilioDatamoverApiservice to Controller role.

ii) Trilio Datamover Service: Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamover This service should be deployed on role where nova-compute service is running.

If you are using RHOSP's pre-defined roles, you need to add our OS::TripleO::Services::TrilioDatamover service to Compute role.

If you have defined your custom roles, then you need to identify the role name where in 'nova-compute' service is running and then you need to add 'OS::TripleO::Services::TrilioDatamover' service to that role.

4] Prepare latest Trilio container images

All commands need to be run as user 'stack'

Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL is ''. The Trilio container URLs are as follows:

4.1] available container images

RHOSP 13

RHOSP 16.1

RHOSP 16.2

There are three registry methods available in RedHat OpenStack Platform.

Remote Registry
Local Registry
Satellite Server

4.2] Remote Registry

Please refer to the to see which containers are available.

Follow this section when 'Remote Registry' is used.

For this method, it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

Populate the trilio_env.yaml with container URLs for:

Trilio Datamover container
Trilio Datamover api container
Trilio Horizon Plugin

trilio_env.yaml will be available in __triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments

Example

4.2] Local Registry

Please refer to the to see which containers are available.

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and updates the trilio_env.yaml.

RHOSP13

Verify the changes

RHOSP16.1

Verify the changes:

RHOSP16.2

Verify the changes

The changes can be verified using the following commands.

4.3] Red Hat Satellite Server

Please refer to the to see which containers are available.

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given

Populate the trilio_env.yaml with container urls.

RHOSP 13

RHOSP16.1

RHOSP16.2

5. Verify Trilio environment details

It is recommended to re-populate the backup target details in the freshly downloaded trilio_env.yaml file. This will ensure that parameters that have been added since the last update/installation of Trilio are available and will be filled out too.

Locations of the trilio_env.yaml:

For more details about the trilio_env.yaml please check .

6] Update Overcloud Trilio components

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use correct Trilio endpoint map file as per available Keystone endpoint configuration

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

7] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

7.1] On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes. When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.

7.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

7.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

Upgrading on Kolla OpenStack

Upgrading from Trilio 4.0 to Trilio 4.1

Due to the new installation method of Trilio for Kolla OpenStack, it is required to reinstall the Trilio components running on the Kolla Openstack nodes when upgrading from Trilio 4.0.

The Trilio appliance can be upgraded as documented.

Upgrading from Trilio 4.1 to a higher version

Trilio 4.1 can be upgraded without reinstallation to a higher version of T4O if available.

Refer to the below-mentioned acceptable values for the placeholders in this document as per the Openstack environment: kolla_base_distro : ubuntu / centos triliovault_tag : 4.1.94-hotfix-13-ussuri / 4.1.94-hotfix-12-victoria

Pre-requisites

Please ensure the following points are met before starting the upgrade process:

Either 4.1 GA OR any hotfix patch against 4.1 should be already deployed
No Snapshot OR Restore is running
Global job scheduler should be disabled

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

Clone latest configuration scripts

Before the latest configuration script is loaded it is recommended to take a backup of the existing config scripts' folder & Trilio ansible roles. The following command can be used for this purpose:

Clone the latest configuration scripts of the required branch and access the deployment script directory for Kolla Ansible Openstack. Available branches to upgrade T4O 4.1 are:

Copy the downloaded Trilio ansible role into the Kolla-Ansible roles directory.

Append Trilio variables

Clean old Trilio variables and append new Trilio variables

This step is not always required. It is recommended to comparetriliovault_globals.ymlwith the Trilio entries in the/etc/kolla/globals.ymlfile.

In case of no changes, this step can be skipped.

This is required, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_globals.yml they need to be updated in /etc/kolla/globals.yml file.

Clean old Trilio passwords and append new Trilio password variables

This step is not always required. It is recommended to comparetriliovault_passwords.ymlwith the Trilio entries in the/etc/kolla/passwords.ymlfile.

In case of no changes, this step can be skipped.

This step is required, when some password variable names have been added, changed, or removed in the latest triliovault_passwords.yml. In this case, the /etc/kolla/passwords.yml needs to be updated.

Append triliovault_site.yml content to kolla ansible's site.yml

This step is not always required. It is recommended to comparetriliovault_site.ymlwith the Trilio entries in the/usr/local/share/kolla-ansible/ansible/site.ymlfile.

In case of no changes, this step can be skipped.

This is required because, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_site.yml they need to be updated in /usr/local/share/kolla-ansible/ansible/site.yml file.

Append triliovault_inventory.txt to the kolla-ansible inventory file

This step is not always required. It is recommended to comparetriliovault_inventory.yml ith the Trilio entries in the/root/multinodefile.

In case of no changes, this step can be skipped.

By default, the triliovault-datamover-api service gets installed on ‘control' hosts and the trilio-datamover service gets installed on 'compute’ hosts. You can edit the T4O groups in the inventory file as per your cloud architecture.

T4O group names are ‘triliovault-datamover-api’ and ‘triliovault-datamover’

Edit globals.yml to set T4O parameters

Edit '/etc/kolla/globals.yml' file to fill triliovault backup target and build details. You will find the triliovault related parameters at the end of globals.yml file. User needs to fill in details like triliovault build version, backup target type, backup target details, etc.

Following is the list of parameters that the user needs to edit.

Parameter

Defaults/choices

comments

Enable T4O Snapshot mount feature

This step is already part of the 4.1 GA installation procedure and should only be verified.

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

Edit /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults/main.yml and find nova_libvirt_default_volumes variable. Append the Trilio mount bind /var/trilio:/var/trilio:shared to the list of already existing volumes.

For a default Kolla installation, will the variable look as follows afterward:

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

In case of using Ironic compute nodes one more entry need to be adjusted in the same file. Find the variable nova_compute_ironic_default_volumes and append trilio mount /var/trilio:/var/trilio:shared to the list.

After the changes the variable will looks like the following:

Pull containers in case of private repository

In case, the user doesn’t want to use the docker hub registry for triliovault containers during cloud deployment, then the user can pull triliovault images before starting cloud deployment and push them to other preferred registries.

Following are the triliovault container image URLs. Replace kolla_base_distro and triliovault_tag variables with their values

Pull T4O container images

Run the below command from the directory with the multinode file tull pull the required images.

Run Kolla-Ansible upgrade command

Run the below command from the directory with the multinode file to start the upgrade process.

Verify Trilio deployment

Verify on the nodes that are supposed to run the Trilio containers, that those are available and healthy.

Advance settings/configuration for Trilio services

Customize HAproxy configuration parameters for Trilio datamover api service

Following are the default haproxy conf parameters set against triliovault datamover api service.

These values work best for triliovault dmapi service. It’s not recommended to change these parameter values. However, in some exceptional cases, If needed to change any of the above parameter values then same can be done on kolla-ansible server in the following file.

After editing, run kolla-ansible deploy command again to push these changes to openstack cloud.

Post kolla-ansible deploy, to verify the changes, please check following file, available on all controller/haproxy nodes.

Restores

Definition

A Restore is the workflow to bring back the backed up VMs from a Trilio Snapshot.

List of Restores

Using Horizon

To reach the list of Restores for a Snapshot follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

--snapshot_id <snapshot_id> ➡️ ID of the Snapshot to show the restores of

Restores overview

Using Horizon

To reach the detailed Restore overview follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads

Details Tab

The Restore Details Tab shows the most important information about the Restore.

Name
Description
Restore Type

The Restore Options are the restore.json provided to Trilio.

List of VMs restored
- restored VM Name
- restored VM Status

Misc Tab

The Misc tab provides additional Metadata information.

Creation Time
Restore ID
Snapshot ID containing the Restore

Using CLI

<restore_id> ➡️ ID of the restore to be shown
--output <output> ➡️ Option to get additional restore details, Specify --output metadata for restore metadata,--output networks --output subnets --output routers --output flavors

Delete a Restore

Once a Restore is no longer needed, it can be safely deleted from a Workload.

Deleting a Restore will only delete the Trilio information about this Restore. No Openstack resources are getting deleted.

Using Horizon

There are 2 possibilities to delete a Restore.

Possibility 1: Single Restore deletion through the submenu

To delete a single Restore through the submenu follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads

Possibility 2: Multiple Restore deletion through a checkbox in Snapshot overview

To delete one or more Restores through the Restore list do the following:

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

<restore_id> ➡️ ID of the restore to be deleted

Cancel a Restore

Ongoing Restores can be canceled.

Using Horizon

To cancel a Restore in Horizon follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

<restore_id> ➡️ ID of the restore to be deleted

One Click Restore

The One Click Restore will bring back all VMs from the Snapshot in the same state as they were backed up. They will:

be located in the same cluster in the same datacenter
use the same storage domain
connect to the same network

The user can't change any Metadata.

The One Click Restore requires, that the original VM's that have been backed up are deleted or otherwise lost. If even one VM is still existing, will the One Click Restore fail.

The One Click Restore will automatically update the Workload to protect the restored VMs.

Using Horizon

There are 2 possibilities to start a One Click Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

<snapshot_id> ➡️ ID of the snapshot to restore.
--display-name <display-name> ➡️ Optional name for the restore.

Selective Restore

The Selective Restore is the most complex restore Trilio has to offer. It allows to adapt the restored VMs to the exact needs of the User.

With the selective restore the following things can be changed:

Which VMs are getting restored
Name of the restored VMs
Which networks to connect with

The Selective Restore is always available and doesn't have any prerequirements.

The Selective Restore will automatically update the Workload to protect the created instance in the case that the original instance is no longer existing.

Using Horizon

There are 2 possibilities to start a Selective Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

<snapshot_id> ➡️ ID of the snapshot to restore.
--display-name <display-name> ➡️ Optional name for the restore.

Inplace Restore

The Inplace Restore covers those use cases, where the VM and its Volumes are still available, but the data got corrupted or needs to a rollback for other reasons.

It allows the user to restore only the data of a selected Volume, which is part of a backup.

The Inplace Restore only works when the original VM and the original Volume are still available and connected. Trilio is checking this by the saved Object-ID.

The Inplace Restore will not create any new RHV resources. Please use one of the other restore options if new Volumes or VMs are required.

Using Horizon

There are 2 possibilities to start an Inplace Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads

Using CLI

<snapshot_id> ➡️ ID of the snapshot to restore.
--display-name <display-name> ➡️ Optional name for the restore.

required restore.json for CLI

The workloadmgr client CLI is using a restore.json file to define the restore parameters for the selective and the inplace restore.

An example for a selective restore of this restore.json is shown below. A detailed analysis and explanation is given afterwards.

The restore.json requires many information about the backed up resources. All required information can be gathered in the .

General required information

Before the exact details of the restore are to be provided it is necessary to provide the general metadata for the restore.

name➡️the name of the restore
description➡️the description of the restore

Selective Restore required information

The Selective Restore requires a lot of information to be able to execute the restore as desired.

Those information are divided into 3 components:

instances
restore_topology
networks_mapping

Information required in instances

This part contains all information about all instances that are part of the Snapshot to restore and how they are to be restored.

Even when VMs are not to be restored are they required inside the restore.json to allow a clean execution of the restore.

Each instance requires the following information

id ➡️ original id of the instance
include <True/False> ➡️ Set True when the instance shall be restored

All further information are only required, when the instance is part of the restore.

name ➡️ new name of the instance
availability_zone ➡️ Nova Availability Zone the instance shall be restored into. Leave empty for "Any Availability Zone"

To use the next free IP available in the set Nics to an empty list [ ]

Using an empty list for Nics combined with the Network Topology Restore, will the restore automatically restore the original IP address of the instance.

vdisks ➡️ List of all Volumes that are part of the instance. Each Volume requires the following information:
- id ➡️ Original ID of the Volume

The root disk needs to be at least as big as the root disk of the backed up instance was.

The following example describes a single instance with all values.

Information required in network topology restore or network mapping

Do not mix network topology restore together with network mapping.

To activate a network topology restore set:

To activate network mapping set:

When the network mapping is activated it is used, it is necessary to provide the mapping details, which are part of the networks_mapping block:

networks ➡️ list of snapshot_network and target_network pairs
- snapshot_network ➡️ the network backed up in the snapshot, contains the following:

Full selective restore example

Inplace Restore required information

The Inplace Restore requires less information thana selective restore. It only requires the base file with some information about the Instances and Volumes to be restored.

Information required in instances

id ➡️ ID of the instance inside the Snapshot
restore_boot_disk ➡️ Set to True if the boot disk of that VM shall be restored.

When the boot disk is at the same time a Cinder Disk, both values need to be set true.

include ➡️ Set to True if at least one Volume from this instance shall be restored
vdisks ➡️ List of disks, that are connected to the instance. Each disk contains:

Network mapping information required

There are no network information required, but the field have to exist as empty value for the restore to work.

Full Inplace restore example

Example runbook for Disaster Recovery using NFS

This runbook will demonstrate how to set up Disaster Recovery with Trilio for a given scenario.

The chosen scenario is following an actively used Trilio customer environment.

Scenario

There are two Openstack clouds available "Openstack Cloud A" and Openstack Cloud B". "Openstack Cloud B" is the Disaster Recovery restore point of "Openstack Cloud A" and vice versa. Both clouds have an independent Trilio installation integrated. These Trilio installations are writing their Backups to NFS targets. "Trilio A" is writing to "NFS A1" and "Trilio B" is writing to "NFS B1". The NFS Volumes used are getting synced against another NFS Volume on the other side. "NFS A1" is syncing with "NFS B2" and "NFS B1" is syncing with "NFS A2". The syncing process is set up independently from Trilio and will always favor the newer dataset.

This scenario will cover the Disaster Recovery of a single Workload and a complete Cloud. All processes are done be the Openstack administrator.

Prerequisites for the Disaster Recovery process

This runbook will assume that the following is true:

"Openstack Cloud A" and "Openstack Cloud B" both have an active Trilio installation with a valid license
"Openstack Cloud A" and "Openstack Cloud B" have free resources to host additional VMs
"Openstack Cloud A" and "Openstack Cloud B" have Tenants/Projects available that are the designated restore points for Tenant/Projects of the other side

For ease of writing will this runbook assume from here on, that "Openstack Cloud A" is down and the Workloads are getting restored into "Openstack Cloud B".

In the case of the usage of shared Tenant networks, beyond the floating IP, the following additional requirement is needed: All Tenant Networks, Routers, Ports, Floating IPs, and DNS Zones are created

Disaster Recovery of a single Workload

A single Workload can do a Disaster Recovery in this Scenario, while both Clouds are still active. To do so the following high-level process needs to be followed:

Copy the Workload directories to the configured NFS Volume
Make the right Mount-Paths available
Reassign the Workload

Copy the Workload directories to the configured NFS Volume

This process only shows how to get a Workload from "Openstack Cloud A" to "Openstack Cloud B". The vice versa process is similar.

As only a single Workload is to be recovered it is more efficient to copy the data of that single Workload over to the "NFS B1" Volume, which is used by "Trilio B".

Mount "NFS B2" Volume to a Trilio VM

It is recommended to use the Trilio VM as a connector between both NFS Volumes, as the nova user is available on the Trilio VM.

Identify the Workload on the "NFS B2" Volume

Trilio Workloads are identified by their ID und which they are stored on the Backup Target. See below example:

In the case that the Workload ID is not known can available Metadata inside the Workload directories be used to identify the correct Workload.

Copy the Workload

The identified workload needs to be copied with all subdirectories and files. Afterward, it is necessary to adjust the ownership to nova:nova with the right permissions.

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

This hash value is calculated based on the provided NFS-Share path: <NFS_IP>/<path> If even one character in the NFS-Share path is different between the provided NFS-Share paths a completely different hash value is generated.

Workloads, that have been moved between NFS-Shares, require that their incremental backups can follow the same path as on their original Source Cloud. To achieve this it is necessary to create the mount path on all compute nodes of the Target Cloud.

Afterwards a mount bind is used to make the workloads data accessible over the old and the new mount path. The following example shows the process of how to successfully identify the necessary mount points and create the mount bind.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by Openstack administrators.

Add admin-user to required domains and projects

To fulfill the required tasks an admin role user is used. This user will be used until the workload has been restored. Therefore, it is necessary to provide this user access to the desired Target Project on the Target Cloud.

Discover orphaned Workloads from NFS-Storage of Target Cloud

Each Trilio installation maintains a database of workloads that are known to the Trilio installation. Workloads that are not maintained by a specific Trilio installation, are from the perspective of that installation, orphaned workloads. An orphaned workload is a workload accessible on the NFS-Share, that is not assigned to any existing project in the Cloud the Trilio installation is protecting.

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described .

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

Get Snapshot Details with network details for the desired snapshot

Get Snapshot Details with disk details for the desired Snapshot

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

Run the selective restore

To do the actual restore use the following command:

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

Clean up

After the Desaster Recovery Process has been successfully completed it is recommended to bring the TVM installation back into its original state to be ready for the next DR process.

Delete the workload

Delete the workload that got restored.

Remove the database entry

The Trilio database is following the Openstack standard of not deleting any database entries upon deletion of the cloud object. Any Workload, Snapshot or Restore, which gets deleted, is marked as deleted only.

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here:

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

Disaster Recovery of a complete cloud

This Scenario will cover the Disaster Recovery of a full cloud. It is assumed that the source cloud is down or lost completly. To do the disaster recovery the following high-level process needs to be followed:

Reconfigure the Target Trilio installation
Make the right Mount-Paths available
Reassign the Workload

Reconfigure the Target Trilio installation

Before the Desaster Recovery Process can start is it necessary to make the backups to be restored available for the Trilio installation. The following steps need to be done to completely reconfigure the Trilio installation.

During the reconfiguration process will all backups of the Target Region be on hold and it is not recommended to create new Backup Jobs until the Desaster Recovery Process has finished and the original Trilio configuration has been restored.

Add NFS B2 to the Trilio Appliance Cluster

To add the NFS-Vol2 to the Trilio Appliance cluster the Trilio can either be to use both NFS Volumes or it is possible to edit the configuration file and then restart all services. This procedure describes how to edit the conf file and restart the services. This needs to be repeated on every Trilio Appliance.

Edit the workloadmgr.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the workloadmgr.conf

Restart the wlm-workloads service

Add NFS B2 to the Trilio Datamovers

Trilio is integrating natively into the Openstack deployment tools. When using the Red Hat director or JuJu charms it is recommended to adapt the environment files for these orchestrators and update the Datamovers through them.

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the tvault-contego.conf

Restart the tvault-contego service

Make the Mount-Paths available

Trilio backups are using qcow2 backing files, which make every incremental backup a full synthetic backup. These backing files can be made visible using the qemu-img tool.

The MTAuMTAuMi4yMDovdXBzdHJlYW0= part of the backing file path is the base64 hash value, which will be calculated upon the configuration of a Trilio installation for each provided NFS-Share.

Identify the base64 hash values

The used hash values can be calculated using the base64 tool in any Linux distribution.

Create and bind the paths

Based on the identified base64 hash values the following paths are required on each Compute node.

/var/triliovault-mounts/MTAuMTAuMi4yMDovdXBzdHJlYW1fc291cmNl

and

/var/triliovault-mounts/MTAuMjAuMy4yMjovdXBzdHJlYW1fdGFyZ2V0

In the scenario of this runbook is the workload coming from the NFS_A1 NFS-Share, which means the mount path of that NFS-Share needs to be created and bound to the Target Cloud.

To keep the desired mount past a reboot it is recommended to edit the fstab of all compute nodes accordingly.

Reassign the workload

Trilio workloads have clear ownership. When a workload is moved to a different cloud it is necessary to change the ownership. The ownership can only be changed by Openstack administrators.

Add admin-user to required domains and projects

Discover orphaned Workloads from NFS-Storage of Target Cloud

List available projects on Target Cloud in the Target Domain

The identified orphaned workloads need to be assigned to their new projects. The following provides the list of all available projects viewable by the used admin-user in the target_domain.

List available users on the Target Cloud in the Target Project that have the right backup trustee role

To allow project owners to work with the workloads as well will they get assigned to a user with the backup trustee role that is existing in the target project.

Reassign the workload to the target project

Now that all informations have been gathered the workload can be reassigned to the target project.

Verify the workload is available at the desired target_project

After the workload has been assigned to the new project it is recommended to verify the workload is managed by the Target Trilio and is assigned to the right project and user.

Restore the workload

The reassigned workload can be restored using Horizon following the procedure described .

This runbook will continue on the CLI only path.

Prepare the selective restore by getting the snapshot information

To be able to do the necessary selective restore a few pieces of information about the snapshot to be restored are required. The following process will provide all necessary information.

List all Snapshots of the workload to restore to identify the snapshot to restore

Get Snapshot Details with network details for the desired snapshot

Get Snapshot Details with disk details for the desired Snapshot

Prepare the selective restore by creating the restore.json file

The selective restore is using a restore.json file for the CLI command. This restore.json file needs to be adjusted according to the desired restore.

Run the selective restore

To do the actual restore use the following command:

Verify the restore

To verify the success of the restore from a Trilio perspective the restore status is checked.

Reconfigure the Target Trilio installation back to the original one

After the Desaster Recovery Process has finished it is necessary to return the Trilio installation to its original configuration. The following steps need to be done to completely reconfigure the Trilio installation.

Delete NFS B2 to the Trilio Appliance Cluster

Edit the workloadmgr.conf

Look for the line defining the NFS mounts

Delete NFS B2 from the comma-seperated list

Write and close the workloadmgr.conf

Restart the wlm-workloads service

Delete NFS B2 to the Trilio Datamovers

To add the NFS B2 to the Trilio Datamovers manually the tvault-contego.conf file needs to be edited and the service restarted.

Edit the tvault-contego.conf

Look for the line defining the NFS mounts

Add NFS B2 to that as comma-seperated list. Space is not necessary, but can be set.

Write and close the tvault-contego.conf

Restart the tvault-contego service

Clean up

After the Desaster Recovery Process has been successfully completed and the Trilio installation reconfigured to its original state, it is recommended to do the following additional steps to be ready for the next Disaster Recovery process.

Remove the database entry

To allow the Trilio installation to be ready for another disaster recovery it is necessary to completely delete the entries of the Workloads, which have been restored.

Trilio does provide and maintain a script to safely delete workload entries and all connected entities from the Trilio database.

This script can be found here:

Remove the admin user from the project

After all restores for the target project have been achieved it is recommended to remove the used admin user from the project again.

T4O-4.1

About Trilio for OpenStack

Trilio for OpenStack Architecture

hashtagBackup-as-a-Service

hashtagMain Components

hashtagService Endpoints

hashtagNetwork Topology

Trilio 4.1 HF1 Release Notes

hashtagRelease Versions

hashtagPackages

Deployment Guide

Requirements

hashtagSystem requirements Trilio Appliance

hashtag2.3. Software Requirements

Trilio network considerations

hashtagExisting endpoints in Openstack

Preparing the installation

hashtagTenant Quotas

Spinning up the Trilio VM

Installing Trilio Components

Installing on Canonical Openstack

Installing on Kolla Victoria

hashtagDeplo T4O-4.1 GA Appliance

hashtagUpgrade the T4O appliance to the latest 4.1 HF

hashtagDeploy Trilio components of 4.1 HF11 or higher

hashtagConfigure the Trilio Appliance

Installing on Ansible Openstack Victoria

Apply the Trilio license

hashtagApply license through Horizon

Advanced Ceph configuration

Additions for multiple CEPH configurations

Additions for multiple Ceph users

Uninstall Trilio

Uninstalling from Canonical OpenStack

Uninstalling from Kolla OpenStack

hashtagClean triliovault_datamover_api container

hashtagClean triliovault_datamover container

hashtagClean haproxy of Trilio Datamover API

hashtagClean Kolla Ansible deployment procedure

hashtagRevert to original Horizon container

hashtagClean Keystone resources

hashtagClean Trilio database resources

hashtagDestroy the Trilio VM Cluster

Upgrade Trilio

Switch NFS Backing file

hashtag

Trilio Appliance Administration Guide

Set Trilio GUI login banner

Trilio Appliance Dashboard

Set network accessibility of Trilio GUI

hashtagNetwork Setup

Reconfigure the Trilio Cluster

Change the Trilio GUI password

hashtagChange Trilio Dashboard password

Reset the Trilio GUI password

Reinitialize Trilio

Download Trilio logs

hashtagDownload Trilio logs

Change Certificates used by Trilio

User Guide

Schedulers

hashtagDefinition

E-Mail Notifications

hashtagDefinition

hashtagRequirements to activate E-Mail Notifications

hashtagUser E-Mail assigned

hashtagTrilio E-Mail Server configured

hashtagActivate/Deactivate the E-Mail Notifications

hashtagExample E-Mails

Admin Guide

Managing Trusts

hashtagList all trusts

hashtagShow a trust

hashtagCreate a trust

hashtagDelete a trust

Disaster Recovery

hashtagDisaster Recovery Process

Troubleshooting

General Troubleshooting Tips

hashtagWhat is happening where

Backup-as-a-Service

Main Components

Service Endpoints

Network Topology

Release Versions

Packages

System requirements Trilio Appliance

2.3. Software Requirements

Existing endpoints in Openstack

Tenant Quotas

Deplo T4O-4.1 GA Appliance

Upgrade the T4O appliance to the latest 4.1 HF

Deploy Trilio components of 4.1 HF11 or higher

Configure the Trilio Appliance

Apply license through Horizon

Clean triliovault_datamover_api container

Clean triliovault_datamover container

Clean haproxy of Trilio Datamover API

Clean Kolla Ansible deployment procedure

Revert to original Horizon container

Clean Keystone resources

Clean Trilio database resources

Destroy the Trilio VM Cluster

Network Setup

Change Trilio Dashboard password

Download Trilio logs

Definition

Definition

Requirements to activate E-Mail Notifications

User E-Mail assigned

Trilio E-Mail Server configured

Activate/Deactivate the E-Mail Notifications

Example E-Mails

List all trusts

Show a trust

Create a trust

Delete a trust

Disaster Recovery Process

What is happening where

On the Trilio Nodes

Backup-as-a-Service

Main Components

Service Endpoints

Network Topology

Apply license through Horizon

Tenant Quotas

Change Trilio Dashboard password

Download Trilio logs

AWS S3 eventual consistency¶

Trilio Cluster¶

Apply license through CLI

System requirements Trilio Appliance

2.3. Software Requirements

Deplo T4O-4.1 GA Appliance

Upgrade the T4O appliance to the latest 4.1 HF

Deploy Trilio components of 4.1 HF11 or higher

Configure the Trilio Appliance

List all trusts

Show a trust

Create a trust

Delete a trust

Definition

Requirements to activate E-Mail Notifications

User E-Mail assigned

Trilio E-Mail Server configured

Activate/Deactivate the E-Mail Notifications

Example E-Mails

Disaster Recovery Process

What is happening where

On the Trilio Nodes

Deplo T4O-4.1 GA Appliance

Upgrade to the latest 4.1 HF Appliance

Deploy Trilio components on Openstack

Update Trilio components