Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

The uninstallation of Trilio is depending on the Openstack Distribution it is installed in. The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover

4. Delete the Trilio Appliance Cluster

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button

3. Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.

4. Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

AWS S3 eventual consistency¶

AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of them describes how an object will reach its consistent state after an object is created/updated or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though Trilio employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reach its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest to use these regions when creating s3 buckets for Trilio. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to restarted.

Trilio Cluster¶

Trilio can be deployed as a single node or a three node cluster. It is highly recommended that Trilio is deployed as three node cluster for fault tolerance and load balancing. Starting with 3.0 release, Trilio requires additional IP for cluster and is required for both single node and three node deployments. Cluster ip a.k.a virtual ip is used for managing cluster and is used to register Trilio service endpoint in the keystone sevice catalog.

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

• Test text

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none


vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:[email protected]:5672,openstack:[email protected]:5672,openstack:[email protected]:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi


[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .volumes.keyring
ceph_dir = /etc/ceph/directory1/,/etc/ceph/directory2/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper


[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to adopt the keyring extension in the tvault-contego.conf inside the Ceph block.

The following example will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none


vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:[email protected]:5672,openstack:[email protected]:5672,openstack:[email protected]:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi


[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .keyring
ceph_dir = /etc/ceph/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper


[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

The recommended size of the VM for the Trilio Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Each Openstack distribution comes with a set of supported operating systems. Please check the to see, which Openstack Distribution is supported with which Operating System.

Additionally, it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

OpenStack Barbican for encryption

Since Trilio for OpenStack 4.2 is T4O capable of providing encrypted backups

The Trilio for OpenStack solution is leveraging the OpenStack Barbican service to provide encryption capabilities for its backups.

To be precise, T4O is using secrets provided by Barbican to encrypt and decrypt backups. Barbican is therefore required to utilize this feature.

Clean triliovault_datamover_api container

The container needs to be cleaned on all nodes where the triliovault_datamover_api container is running. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the triliovault_datamover_api container:

Stop the triliovault_datamover_api container.

Remove the triliovault_datamover_api container.

Clean /etc/kolla/triliovault-datamover-api directory.

Clean log directory of triliovault_datamover_api container.

Clean triliovault_datamover container

The container needs to be cleaned on all nodes where the triliovault_datamover container is running. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the triliovault_datamover container:

Stop the triliovault_datamover container.

Remove the triliovault_datamover container.

Clean /etc/kolla/triliovault-datamover directory.

Clean log directory of triliovault_datamover container.

Clean haproxy of Trilio Datamover API

The Trilio Datamover API entries need to be cleaned on all haproxy nodes. The Kolla Openstack inventory file helps to identify the nodes with the service.

Following steps need to be done to clean the haproxy container:

Clean Kolla Ansible deployment procedure

Delete all Trilio related entries from:

Trilio entries can be found in:

• /usr/local/share/kolla-ansible/ansible/roles/ ➡️ There is a role triliovault

• /etc/kolla/globals.yml➡️ Trilio entries had been appended at the end of the file

• /etc/kolla/passwords.yml➡️

Revert to original Horizon container

Run deploy command to replace the Trilio Horizon container with original Kolla Ansible Horizon container.

Clean Keystone resources

Trilio created a dmapi service with dmapi user.

Clean Trilio database resources

Trilio Datamover API service has its own database in the Openstack database.

Login to Openstack database as root user or user with similar priviliges.

Delete dmapi database and user.

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

Destroy the Trilio VMs

Undefine the Trilio VMs

Delete the TrlioVault VM disk from KVM Host storage

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

Unmount old mount point

The first step is to remove the datamover container and to unmount the old mounts. This is necessary to make sure, that the new datamover container with the new backend target is not getting any interference from the old backup target.

Change Trilio Dashboard password

To change the Trilio GUI password do:

• Login into the Trilio Dashboard

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the guide afterwards.

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

To use the workloadmgr CLI tool on the Trilio appliance it is only necessary to activate the virtual environment of the workloadmgr

Download Trilio logs

It is possible to download the Trilio logs directly through the Trilio web gui.

To download logs throught the Trilio web gui:

• Login into the Trilio web gui

• Go to "Logs"

• Choose the log to be downloaded

  • Each log for every Trilio Appliance can be downloaded seperatly

  • or a zip of all logfiles can be created and downloaded

Uninstall Trilio Services

The Trilio Ansible OpenStack playbook can be run to uninstall the Trilio services.

Destroy Trilio Datamover API container

To cleanly remove the Trilio Datamover API container run the following Ansible playbook.

Clean openstack_user_config.yml

Remove the tvault-dmapi_hosts and tvault_compute_hosts entries from /etc/openstack_deploy/openstack_user_config.yml

Remove Trilio haproxy settings in user_variables.yml

Remove Trilio Datamover API settings from /etc/openstack_deploy/user_variables.yml

Remove Trilio Datamover API inventory file

Remove Trilio Datamover API service endpoints

Delete Trilio Datamover API database and user

• Go inside galera container.

• Login as root user in mysql database engine.

• Drop dmapi database.

• Drop dmapi user

Remove dmapi rabbitmq user from rabbitmq container

• Go inside rabbitmq container.

• Delete dmapi user.

• Delete dmapi vhost.

Clean haproxy

Remove /etc/haproxy/conf.d/datamover_service file.

Remove HAproxy configuration entry from /etc/haproxy/haproxy.cfg file.

Restart the HAproxy service.

Remove certificates from Compute nodes

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

Destroy the Trilio VMs

Undefine the Trilio VMs

Delete the TrlioVault VM disk from KVM Host storage

Trilio for OpenStack is generating a base64 hash value for every NFS backup target connected to the T4O solution. This enables T4O to mount multiple NFS backup targets to the same T4O installation.

The mountpoints are generated utilizing a hash value inside the mountpoint, providing a unique mount for every NFS backup target.

This mountpoint is then used inside the incremental backups to point to the qcow2 backing files. The backing file path is required as a full path and can not be set as a relative path. This is a limitation of the qcow2 format.

T4O 4.2 has changed how the hash value gets calculated. T4O 4.1 and prior calculated the hash value out of the complete NFS path provided as shown in the example below.

T4O 4.2 is now only considering the NFS directory part for the hash value as shown below.

It is therefore necessary to make older backups taken by T4O 4.1 or prior available for TVO4.2 to restore.

This can be done by one of two methods:

1. Rebase all backups and change the backing file path

2. Make the old mount point available again and point it to the new one using mount bind

Rebase all backups

Trilio is providing a script, that is taking care of the rebase procedure. This script can be downloaded from the following location.

Copy and use the script from the Trilio appliance as the nova user.

The script is requiring the complete path to the workload directory on the backup target.

This needs to be repeated until all workloads have been rebased.

Use mount bind with the old mount point

This method is generating a second mount point based on the old hash value calculation and then mounting the new mount point to the old one. By doing this mount bind both mount points will be available and point to the same backup target.

To use this method the following information needs to be available:

• old mount path

• new mount path

Examples of how to calculate the base64 hash value are shown below.

Old mountpoint hash value:

New mountpoint hash value:

The mount bind needs to be done for all Trilio appliances and Datamover services.

Trilio Appliance

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

An example is given below.

RHOSP/TripleO Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

Kolla Ansible OpenStack Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

Ansible OpenStack Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

Canonical Openstack WLM & Datamover containers

Canonical OpenStack does the creation of the mountpoint and the mount bind through JuJu using the following commands.

To create the mountpoint, if it doesn't already exist:

To create the mount bind

Introduction

Filename and location:

This file is used only when the user wants to configure 'multiple IP/endpoints based NFS share' as a backup target for Trilio. In all other cases like single IP NFS, S3 this file does not get used. Follow regular install doc.

If the user is using multiple-IP/endpoints based NFS share as a backup target for triliovault then triliovault mounts any one IP/endpoint on a given compute node for a given NFS share. Users can distribute NFS share IPs/endpoints across compute nodes.

'triliovault_nfs_map_input.yml' file is a tool that users can use to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Examples

Using hostnames

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

The remaining 2 node hostnames do not follow any format/pattern.

Now the mapping file will look like this

Using IPs

Compute node IP range used here: 172.30.3.11-40 and 172.30.4.40, 172.30.4.50 Total of 32 compute nodes

Getting the correct compute hostnames/IPs

RHOSP or TripleO

Use the following command to get compute hostnames. Check the ‘Name' column. Use these exact hostnames in 'triliovault_nfs_map_input.yml' file.

In the following command output, ‘overcloudtrain1-novacompute-0' and ‘overcloudtrain1-novacompute-1' are correct hostnames.

Kolla-Ansible OpenStack

Compute hostnames/IPs should match in kolla-ansible inventory file and triliovault_nfs_map_input.yml.

If IP addresses are sued in kolla-ansible inventory file then the same IP addresses should be used in ‘triliovault_nfs_map_input.yml' file too. If the kolla-ansible deploy command looks like the following,

kolla-ansible -i multinode deploy

then, the inventory file is 'multinode'. Generally, it is available at “/root/multinode“.

OpenStack Ansible

Compute hostnames/IPs should match in the Openstack Ansible inventory file and triliovault_nfs_map_input.yml

If IP addresses have been used in the Openstack-ansible inventory file then you should use the same IP addresses in the ‘triliovault_nfs_map_input.yml' file too.

Generally inventory file available at /etc/openstack_deploy/openstack_user_config.yml

Openstack Ansible deploy command looks like following,

openstack-ansible os-tvault-install.yml

Openstack Ansible automatically picks up values that the user has set in the file openstack_user_config.yml

To configure the banner shown upon accessing the Trilio Appliance GUI do the following.

1. Login into Trilio Appliance console

2. edit the banner.yaml at /etc/tvault-config/banner.yaml

3. restart tvault-config service

The content of the banner.yaml looks as follows and can be edited as required:

The Trilio solution is using the qcow2 backing file to provide full synthetic backups.

Especially when the NFS backup target is used, there are scenarios at which this backing file needs to be updated to a new mount path.

To make this process easier and streamlined Trilio is providing the following rebase tool.

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. The dashboard can be accessed using the virtual IP.

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. This dashboard is accessible through the virtual IP.

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

The dashboard login will be reset to:

The Trilio database is following an older OpenStack database schema. This schema is not forgetting anything and is instead setting the deleted flag for any data that is no longer valid/deleted.

This leads to ever-growing databases, which can lead to performance degradation.

To counter this a database clean-up and optimization tool has been made available.

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio cluster

The Trilio Cluster is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio cluster also responsible to gather the metadata about the backed up VMs and networks from the Openstack environment. It is sending API calls towards the Openstack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received does the Trilio Cluster write it as json files on the Backup Target.

The Trilio cluster is also sending the Cinder Snapshot command.

During a restore process

During restore process is the Trilio cluster reading the VM metadata from its Database and is using the metadata to create the Shell for the restore. It is sending API calls to the Openstack environment to create the necessary resources.

dmapi

The dmapi service is the connector between the Trilio cluster and the datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so is the dmapi service connecting to the nova api database requesting the compute hose of a provided VM.

Once the compute host has been identified is the dmapi forwarding the command from the Trilio Cluster to the datamover running on the identified compute host.

datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover is controlling the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Input/Output Error with Cohesity NFS

On Cohesity NFS if an Input/Output error is observed, then increase the timeo and retrans parameter values in your NFS options.

Timeout receiving packet error in multipath iscsi environment

Logging inside all datamover containers and add uxsock_timeout with value as 60000 which is equals to 60 sec inside /etc/multipath.conf.Restart datamover container

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during creation of Workloads, backups or restores.

Openstack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The Openstack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

Trilio Configurator

Once Trilio is configured use virtual IP to access its dashboard. If service status panels on the dashboard page are not visible then access the virtual IP on port 3001 (https://<T4O-VIP>:3001/) and accept the SSL exception, and then refresh the dashboard page.

\

Trilio Workloads are designed to allow a Desaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

1. Install and Trilio for the target cloud

4. Notify users to of Workloads being available

Mount-paths

Trilio incremental Snapshots involve a backing file to the prior backup taken, which makes every Trilio incremental backup a synthetic full backup.

Trilio is using qcow2 backing files for this feature:

As can be seen in the example is the backing file an absolute path, which makes it necessary, that this path exists so the backing files can be accessed.

Trilio is using the base64 hashing algorithm for the NFS mount-paths, to allow the configuration of multiple NFS Volumes at the same time. The hash value is calculated using the provided NFS path.

When the path of the backing file is not available on the Trilio appliance and Compute nodes, will the restores of incremental backups fail.

The tested and recommended method to make the backing files available is creating the required directory path and using mount --bind to make the path available for the backups.

Running the mount --bind command will make the necessary path available until the next reboot. If it is required to have access to the path beyond a reboot is it necessary to edit the fstab.

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

4. Check/Uncheck the box for "Enable Email Alerts"

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Introduction

Integrating Trilio for OpenStack 4.2 with Barbican facilitates encryption support for the qcow2-data segment of Trilio backups. However, the JSON files housing the backed-up OpenStack metadata remain unencrypted.

This capability necessitates the presence of the OpenStack Barbican service. Absence of the Barbican service will result in the omission of encryption options for Workloads within the Horizon interface.

Trilio for OpenStack (T4O) 4.2 exclusively retrieves secrets from Barbican, without generating, modifying, or deleting any secrets within the Barbican platform.

Barbican secrets are indispensable for executing backups or restorations in encryption-enabled Workloads. The onus lies on the OpenStack project user to supply these secrets and guarantee their accurate availability.

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

Trilio is using the which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Trusts can only be worked with via CLI

1. Can Trilio for OpenStack restore instance UUIDs?

Answer: NO

Trilio for OpenStack does not restore Instance UUIDs (also known as Instance IDs). The only scenario where we do not modify the Instance UUID is during an Inplace Restore, where we only recover the data without creating new instances.

When Trilio for OpenStack restores virtual machines (VMs), it effectively creates new instances. This means that new Virtual Machine Instance UUIDs are generated for the restored VMs. We achieve this by orchestrating a call to Nova, which creates new VMs with new UUIDs.

Same cloud - different owner

Migration within the same cloud to a different owner Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project A — User B Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project B — User B Cloud A — Domain A — Project A — User A =>Cloud A — Domain B — Project B — User B

Steps used:

Trilio is using a base64 hash for the mount point of NFS Backup targets. This hash makes sure, that multiple NFS Shares can be used with the same Trilio installation.

This base64 hash is part of the Trilio incremental backups as an absolute path of the backing files. This requires the usage of mount bind during a DR scenario or quick migration scenario.

In the case that there is time for a thorough migration there is another option to change the backing file and make the Trilio backups available on a different NFS Share. This option is updating the backing file to the new NFS Share mount point.

Backing file change script

Trilio is providing a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Release Versions

Packages

Containers and Gitbranch

Changelog

Fixed issues and bugs

Snapshot getting hung in upload phase for multipath environments

An issue has been fixed, which lead to Snapshots being stuck in the upload phase in the case of using an FC multipath connected Cinder Storage.

Backups fail due to an old Trilio Cinder Snapshot not being in available state

An issue has been fixed, which lead to Trilio-created Cinder Snapshots, which are not in the state available, blocking the creation of new Cinder Snapshots.

Latin characters in restore names lead to failed restores

The support for Latin characters has been enhanced to no longer impact restores.

Clean Trilio Datamover API service

The following steps need to be run on all nodes, which have the Trilio Datamover API service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamoverApi.

Once the role that runs the Trilio Datamover API service has been identified will the following commands clean the nodes from the service.

Stop trilio_dmapi container.

Remove trilio_dmapi container.

Clean Trilio Datamover API service conf directory.

Clean Trilio Datamover API service log directory.

Clean Trilio Datamover Service

The following steps need to be run on all nodes, which have the Trilio Datamover service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamover.

Once the role that runs the Trilio Datamover service has been identified will the following commands clean the nodes from the service.

Stop trilio_datamover container.

Remove trilio_datamover container.

Unmount Trilio Backup Target on compute host.

Clean Trilio Datamover service conf directory.

Clean log directory of Trilio Datamover service.

Clean Trilio haproxy resources

The following steps need to be run on all nodes, which have the haproxy service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::HAproxy.

Once the role that runs the haproxy service has been identified will the following commands clean the nodes from all Trilio resources.

Edit the following file inside the haproxy container and remove all Trilio entries.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

An example of these entries is given below.

Restart the haproxy container once all edits have been done.

Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

Clean Trilio database resources

Trilio creates a database for the dmapi service. This database needs to be cleaned.

Login into the database cluster

Run the following SQL statements to clean the database.

Revert overcloud deploy command

Remove the following entries from roles_data.yaml used in the overcloud deploy command.

• OS::TripleO::Services::TrilioDatamoverApi

• OS::TripleO::Services::TrilioDatamover

Follow these steps to clean the overcloud deploy command from all Trilio entries.

1. Remove trilio_env.yaml entry

2. Remove trilio endpoint map file Replace with original map file if existing

Revert back to original RHOSP Horizon container

Run the cleaned overcloud deploy command.

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

Destroy the Trilio VMs

Undefine the Trilio VMs

Delete the TrlioVault VM disk from KVM Host storage

Generic Prerequisites

• Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of Trilio.

• The mentioned Gemfury repository should be accessible from Trilio VM.

• Either 4.2 GA OR any hotfix patch against 4.2 should be already deployed for performing upgrades mentioned in the current document.

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

Steps to upgrade Trilio having Internet access

1. Download the hf_upgrade.sh script on all T4O nodes where the upgrade is to be done.

You can check the usage of this script .

1.1 Upgrade to the latest Hotfix/Maintenance release

1.2 Upgrade to the intermediate Hotfix/Maintenance release

a. Download the hf_upgrade.sh of desired Hotfix/Maintenance release

b. Edit downloaded hf_upgrade.sh and set the BRANCH_NAME as triliodata-hotfix-4-2 for any Hotfix/Maintenance release and OFFLINE_PKG_NAME must be set to particular hotfix package name

2. Run the following command to download and Install the upgraded packages

3. Run the following command to enable the wlm-cron service

Steps to upgrade Trilio having No Internet access

Here the packages need to be downloaded on a separate host which has internet access. The script ./hf_upgrade.sh can be used with the below-mentioned option to download the required package

1. Download the hf_upgrade.sh script on a separate host which has internet access

Use this script to download the upgraded packages.

You can check the usage of this script .

1.1 Upgrade to the latest Hotfix/Maintenance release

1.2 Upgrade to the intermediate Hotfix/Maintenance release.

a. Download the hf_upgrade.sh of desired Hotfix/Maintenance release

b. Edit downloaded hf_upgrade.sh and set the BRANCH_NAME as triliodata-hotfix-4-2 for any Hotfix/Maintenance release and OFFLINE_PKG_NAME must be set to particular hotfix package name and then run the script.

2. Copy the downloaded 4.2-offlinePkgs.tar.gz package and hf_upgrade.sh script to all the Trilio nodes

3. Now access the appliance VMs and install the copied offline packages on each of the Trilio appliances by running the below-mentioned command:

4. Once the installation is done, run the following command to enable the wlm-cron service

Post upgrade steps

1. Verify all wlm services on all Trilio nodes

2. Make sure all wlm services are up and running from python version 3.8.x

3. Check the mount point using “df -h” command

4. Grafana dashboard shows the correct wlm service status on all T4O nodes.

Additional check for wlm-cron on the primary node

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436, to do so follow the below document provided here:

Release Versions

Packages

Containers and Gitbranch

Changelog

New Qualifications

This Hotfix extends the Support Matrix of T4O 4.2 as follows:

• Canonical Openstack Wallaby based on Focal (20.04) Support

• Kolla Ansible Openstack Wallaby on Ubuntu 20.04 and CentOS Stream

• Openstack Ansible Wallaby on Ubuntu 20.04 and CentOS Stream

Fixed Bugs and issues

Rebase permission error

An issue has been fixed which prevented the correct rebase of T4O incremental backups in the case that root didn't have the required permissions on the backup target.

The following Trilio services are providing certificates for secured access to the Trilio solution.

Changing the certificate of TVault-Config and Nginx for Grafana Service

The TVault-Config service and the Nginx Resource for the Grafana Dashboard are using the same certificate.

The certificate used is a symlink to a host-specific certificate. Each Trilio VM has its own self-signed certificate by default which is getting recreated every time the TVault-Config service is restarted.

When the certificate for the TVault-Config and Nginx (Grafana) is to be changed to a customer chosen certificate it is required to deactivate the recreation of the certificates upon service restart.

1. Login into the Trilio VM via SSH

2. Edit the following file: /home/stack/myansible/lib/python3.6/site-packages/tvault_configurator/tvault_config_bottle.py

3. Look for create_ssl_certificates() in the main function

The resulting main function will look like this:

Afterward, the certificates can be replaced manually by overwriting the files.

Once the certificates have been replaced by the desired ones restart the TVault-Config service and the Nginx pcs resource.

Changing the certificate used by Nginx for wlm-api service

The certificate provided by the Nginx for the wlm-api service is set during configuration when HTTPS endpoints are configured for the Trilio appliance. This certificate is provided to the end-user or Openstack every time an API call to the Trilio solution is sent.

The certificate and its related private key can be changed through the OS API certificate tab.

In this tab is the section to Upload Server certificate | Private key. Use this section to update the wlm-api certificate as required.

To gracefully shutdown/restart the Trilio cluster the following steps are recommended.

Verify no snapshots or restores are running

It is recommended to verify that no snapshots or restores are running on the Trilio Cluster.

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

• Status (Enabled/Disabled)

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log

The next important log is the workloadmgr-api.log, which contains all logs about API calls received by the Trilio Cluster. It can be found at:

Each Trilio Workload has a dedicated owner. The ownership of a Workload is defined by:

• Openstack User - The Openstack User-ID is assigned to a Workload

• Openstack Project - The Openstack Project-ID is assigned to a Workload

• Openstack Cloud - The Trilio Serviceuser-ID is assigned to a Workload

This ownership secures, that only the owners of a Workload are able to work with it.

Openstack Administrators can reassign Workloads or reimport Workloads from older Trilio installations.

Import workloads

Workload import allows to import Workloads existing on the Backup Target into the Trilio database.

To get a list of importable Workloads use the following CLI command:

• --project_id <project_id> ➡️ List workloads belongs to given project only.

To import Workloads into the Trilio database use the following CLI command:

• --workloadids <workloadid> ➡️ Specify workload ids to import only specified workloads. Repeat option for multiple workloads.

Orphaned Workloads

The definition of an orphaned Workload is from the perspective of a specific Trilio installation. Any workload that is located on the Backup Target Storage, but not known to the TrilioVualt installation is considered orphaned.

Further is to divide between Workloads that were previously owned by Projects/Users in the same cloud or are migrated from a different cloud.

The following CLI command provides the list of orphaned workloads:

• --migrate_cloud {True,False} ➡️ Set to True if you want to list workloads from other clouds as well. Default is False.

• --generate_yaml {True,False} ➡️ Set to True if want to generate output file in yaml format, which would be further used as input for workload reassign API.

Reassigning Workloads

Openstack administrators are able to reassign a Workload to a new owner. This involves the possibility to migrate a Workload from one cloud to another or between projects.

Use the following CLI command to reassign a Workload:

• --old_tenant_ids <old_tenant_id>➡️ Specify old tenant ids from which workloads need to reassign to new tenant. Specify multiple times to choose Workloads from multiple tenants.

• --new_tenant_id <new_tenant_id> ➡️ Specify new tenant id to which workloads need to reassign from old tenant. Only one target tenant can be specified.

• --workload_ids <workload_id>

A sample mapping file with explanations is shown below:

By default is the Trilio GUI available on all NICs on port 443.

To limit this to only one IP the following steps need to be applied.

Network Setup

The Trilio Appliance provides by default the possibility of 4 VIPs.

• A general VIP which can be used for everything

• A public VIP for the public endpoint

• An internal VIP for the internal endpoint

• An admin VIP for the admin endpoint

Should an additional VIP be required to restrict the access of the Trilio Dashboard to this VIP the new VIP needs to be created as a new resource inside the PCS cluster.

Nginx setup

When the new dashboard_ip has been created or decided, then the next step is to set up the proxy forwarding inside Nginx, which will make the Trilio GUI available through port 8000.

1. Create new conf file at /etc/nginx/conf.d/tvault-dashboard.conf. Replace variables dashboard_ip and virtual_ip as configured or decided.

2. edit /etc/nginx/nginx.conf and uncomment line #include /etc/nginx/conf.d/*.conf;

Limit the access of the Dashboard

The configured dashboard_ip will always end on the nginx service on port 8000 and will then be forwarded to the local dashboard service on port 443.

This configuration limits the required access to the local dashboard service to the Trilio appliance cluster itself. All other connections on port 443 can be dropped.

The following commands will set the required iptable rules.

Verify the accessibility as required

At this point is the Trilio GUI only reachable on the dashboard_ip on port 8000. Accessing the Trilio GUI through any other IP or on port 443 is not allowed.

Release Versions

Packages

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab in Horizon