1 of 96

T4O-4.1

About Trilio for OpenStack

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media
Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button
Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.
Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

Trilio for OpenStack Architecture

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.
Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.
Trilio Datamover is a python module that is installed on every OpenStack compute nodes
Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Deployment Guide

Requirements

Trilio has four main software components:

Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.
Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes
Trilio Datamover is a python module that is installed on every OpenStack compute nodes
Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance is not supported as an instance inside Openstack.

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

Software

Supported

The recommended size of the VM for the Trilio Appliance is:

When running Trilio in production, a 3-node cluster of the Trilio appliance is recommended for high availability and load balancing.

Ressource

Value

The qcow2 image itself defines the 40GB disk size of the VM.

In the rare case of the Trilio Appliance database or log files getting larger than 40GB disk, contact or open a ticket with Trilio Customer Success to attach another drive to the Trilio Appliance.

2.3. Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Each Openstack distribution comes with a set of supported operating systems. Please check the to see, which Openstack Distribution is supported with which Operating System.

Additional it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

Trilio network considerations

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

Preparing the installation

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

Spinning up the Trilio VM

For Canonical Openstack it is not necessary to spin up the Trilio VM.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

This guide shows the tested way to spin up the Trilio Appliance on a RHV Cluster. Please contact a RHV Administrator and Trilio Customer Success Agent in case of incompatibility with company standards.

Installing Trilio Components

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the installation process can begin. This process contains the following steps:

Install the Trilio dm-api service on the control plane.
Install the Trilio datamover service on the compute plane.
Install the Trilio Horizon plugin into the Horizon service.

Installing on Canonical Openstack

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

Trilio is not providing the JuJu Charms to deploy Trilio 4.1 in Canonical Openstack. These are developed and maintained by Canonical.

Installing on Kolla Victoria

The installation of Trilio for Openstack on Kolla Victoria with Trilio 4.1 is following this procedure:

Deplo T4O-4.1 GA Appliance
Upgrade to 4.1 HF5 or higher on the appliance
Deploy Trilio components of 4.1 HF5 or higher on the Kolla Openstack Victoria
Configure the Trilio appliance

Deplo T4O-4.1 GA Appliance

Please follow to spin up the base Trilio 4.1GA appliance.

Upgrade the T4O appliance to the latest 4.1 HF

Trilio supports Kolla Victoria from 4.1HF5 onwards, so it is recommended to upgrade to the latest available hotfix on 4.1 to make deployment successful. Please follow to upgrade the appliance to the latest 4.1 Hotfix.

Deploy Trilio components of 4.1 HF11 or higher

Run the deployment of the components following using the following values:

Variable

Value

Configure the Trilio Appliance

Please follow to configure the upgraded Trilio 4.1 appliance.

Installing on Ansible Openstack Victoria

The installation of Trilio for Openstack on Kolla Victoria with Trilio 4.1 is following this procedure:

Deplo T4O-4.1 GA Appliance
Upgrade to 4.1 HF5 packages on the appliance
Deploy Trilio components on Openstack Victoria

Apply the Trilio license

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

Login to Horizon using admin user.
Click on Admin Tab.
Navigate to Backups-Admin
Navigate to Trilio
Navigate to License
Click "Update License"
Click "Choose File"
choose license-file on client system
click "Apply"

Apply license through CLI

<license_file> ➡️ path to the license file

Advanced Ceph configuration

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

Additions for multiple CEPH configurations

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

Additions for multiple Ceph users

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to adopt the keyring extension in the tvault-contego.conf inside the Ceph block.

The following example will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none


vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:[email protected]:5672,openstack:[email protected]:5672,openstack:[email protected]:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi


[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .keyring
ceph_dir = /etc/ceph/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper


[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

Uninstall Trilio

The uninstallation of Trilio is depending on the Openstack Distribution it is installed in. The high-level process is the same for all Distributions.

Uninstall the Horizon Plugin or the Trilio Horizon container
Uninstall the datamover-api container
Uninstall the datamover
Delete the Trilio Appliance Cluster

Uninstalling from Canonical OpenStack

Trilio is not providing the JuJu Charms to deploy Trilio 4.1 in Canonical Openstack. At the time of release are the JuJu Charms not yet updated to Trilio 4.1. We will update this page once the Charms are available.

Upgrade Trilio

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

Old

New

4.0 GA (4.0.92)

4.0 SP1 (4.0.115)

4.0 GA (4.0.92)

4.1 GA (4.1.94)

4.1 HF1 (4.1.94-hotfix1)

The upgrade process contains upgrading the Trilio appliance and the Openstack components and is dependent on the underlying operating system.

The Upgrade of Trilio for Canonical Openstack is managed through the charms.

Switch NFS Backing file

Trilio is using a base64 hash for the mount point of NFS Backup targets. This hash makes sure, that multiple NFS Shares can be used with the same Trilio installation.

This base64 hash is part of the Trilio incremental backups as an absolute path of the backing files. This requires the usage of mount bind during a DR scenario or quick migration scenario.

In the case that there is time for a thorough migration there is another option to change the backing file and make the Trilio backups available on a different NFS Share. This option is updating the backing file to the new NFS Share mount point.

Backing file change script

Trilio is providing a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Trilio Appliance Administration Guide

Set Trilio GUI login banner

To configure the banner shown upon accessing the Trilio Appliance GUI do the following.

Login into Trilio Appliance console
edit the banner.yaml at /etc/tvault-config/banner.yaml
restart tvault-config service

The content of the banner.yaml looks as follows and can be edited as required:

Trilio Appliance Dashboard

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. This dashboard is accessible through the virtual IP.

If service status panels on the dashboard page are not visible then access the virtual IP on port 3001 (https://<TVO-VIP>:3001/) and accept the SSL exception, and then refresh the dashboard page.

It shows for each Trilio Appliance the Status of the following Trilio services:

wlm-workloads

Set network accessibility of Trilio GUI

By default is the Trilio GUI available on all NICs on port 443.

To limit this to only one IP the following steps need to be applied.

Network Setup

The Trilio Appliance provides by default the possibility of 4 VIPs.

Reconfigure the Trilio Cluster

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the guide afterwards.

Once the Trilio configurator has started, it needs to run through successfully to continue to use Trilio.

Change the Trilio GUI password

Change Trilio Dashboard password

To change the Trilio GUI password do:

Login into the Trilio Dashboard
Click on "admin" in the upper right corner to open the submenu
Choose "Reset Password"
Set the new Trilio password

Reset the Trilio GUI password

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

The dashboard login will be reset to:

Reinitialize Trilio

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

Login into the Trilio Dashboard
Click on "admin" in the upper right corner to open the submenu
Choose "Reinitialize"
Verify that you want to reinitialize the Trilio

Download Trilio logs

It is possible to download the Trilio logs directly through the Trilio web gui.

To download logs throught the Trilio web gui:

User Guide

E-Mail Notifications

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

Both requirements need to be set or configured by the Openstack Administrator. Please contact your Openstack Administrator to verify the requirements.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

Login to Horizon
Navigate to the Backups
Navigate to Settings
Check/Uncheck the box for "Enable Email Alerts"

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Admin Guide

Managing Trusts

Trilio is using the Openstack Keystone Trust system which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Openstack Administrators should never have the need to directly work with the trusts created.

The cloud-trust is created during the Trilio configuration and further trusts are created as necessary upon creating or modifying a workload.

Trusts can only be worked with via CLI

List all trusts

Show a trust

<trust_id> ➡️ ID of the trust to show

Create a trust

<role_name> ➡️Name of the role that trust is created for
--is_cloud_trust {True,False} ➡️ Set to true if creating cloud admin trust. While creating cloud trust use same user and tenant which used to configure Trilio and keep the role admin.

Delete a trust

<trust_id> ➡️ ID of the trust to be deleted

Disaster Recovery

Trilio Workloads are designed to allow a Desaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

Troubleshooting

General Troubleshooting Tips

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Using the workloadmgr CLI tool on the Trilio Appliance

To use the workloadmgr CLI tool on the Trilio appliance it is only necessary to activate the virtual environment of the workloadmgr

An rc-file to authenticate against Openstack is required.

Important log files

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log

The next important log is the workloadmgr-api.log, which contains all logs about API calls received by the Trilio Cluster. It can be found at:

/var/log/workloadmgr/workloadmgr-api.log

The log for the third service is the workloadmgr-scheduler.log, which contains all logs about the internal job scheduling between Trilio nodes in the Trilio Cluster.

/var/log/workloadmgr/workloadmgr-scheduler.log

The last but not least service running on the Trilio Nodes is the wlm-cron service, which is controlling the scheduled automated backups.

/var/log/workloadmgr/workloadmgr-workloads.log

In the case of using S3 as a backup target is there also a log file that keeps track of the S3-Fuse plugin used to connect with the S3 storage.

/var/log/workloadmgr/s3vaultfuse.py.log

Canonical Openstack is having these logs inside the workloadmgr container.

Trilio Datamover service logs on RHOSP

Datamover API log

The log for the Trilio Datamover API service is located on the nodes, typically controller, where the Trilio Datamover API container is running under:

/var/log/containers/trilio-datamover-api/dmapi.log

Datamover log

The log for the Trilio Datamover service is located on the nodes, typically compute, where the Trilio Datamover container is running under:

/var/log/containers/trilio-datamover/tvault-contego.log

In case of S3 being used in the log for the S3 Fuse plugin located on the same nodes under:

/var/log/containers/trilio-datamover/tvault-object-store.log

Trilio Datamover service logs on Kolla Openstack

Datamover API log

The log for the Trilio Datamover API service is located on the nodes, typically controller, where the Trilio Datamover API container is running under:

/var/log/kolla/trilio-datamover-api/dmapi.log

Datamover log

The log for the Trilio Datamover service is located on the nodes, typically compute, where the Trilio Datamover container is running under:

/var/log/kolla/triliovault-datamover/tvault-contego.log

In case of S3 being used in the log for the S3 Fuse plugin located on the same nodes under:

/var/log/kolla/trilio-datamover/tvault-object-store.log

Trilio Datamover service logs on Ansible Openstack

Datamover API log

The log for the Trilio Datamover API service is located on the nodes, typically controller, where the Trilio Datamover API container is running. Log into the dmapi container using lxc-attach command (example below).

lxc-attach -n controller_dmapi_container-a11984bf

The log file is then located under:

/var/log/dmapi/dmapi.log

Datamover log

The log for the Trilio Datamover service is typically located on the compute nodes and the logs can be found here:

/var/log/tvault-contego/tvault-contego.log

In case of S3 being used in the log for the S3 Fuse plugin located on the same nodes under:

/var/log/tvault-object-store/tvault-object-store.log

API GUIDE

Restart Trilio Services

In complex environments it is sometimes necessary to restart a single service or the complete solution. Rarely is restarting the complete node, where a service is running possible or even the ideal solution.

This page describes the services running by Trilio and how to restart those.

Trilio Appliance Services

The Trilio Appliance is the controller of Trilio. Most services on the Appliance are running in a High Availability mode on a 3-node cluster.

wlm-api

The wlm-api service takes the API calls against the Trilio Appliance. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-api service run on each Trilio node:

wlm-scheduler

The wlm-scheduler service is taking job requests and identifies which Trilio node should take the request. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-scheduler service run on each Trilio node:

wlm-workloads

The wlm-workloads service is the task worker of Trilio executing all jobs given to the Trilio node. It is running in active-active mode on all nodes of the Trilio cluster.

To restart the wlm-workloads service run on each Trilio node:

wlm-cron

The wlm-cron service is responsible for starting scheduled Backups according to the configurtation of Tenant Workloads. It is running in active-passive mode and controlled by the pacemaker cluster.

To restart the wlm-workloads service run on the Trilio node with VIP assigned:

VIP resources

The Trilio appliance is running 1 to 4 virtual IPs on the Trilio cluster. These are controlled by the pacemaker cluster and provided through NGINX.

To restart these resources the pacemaker NGINX resource is getting restarted:

RabbitMQ

The Trilio cluster is using RabbitMQ as messaging service. It is running in active-active mode on all nodes of the Trilio cluster.

RabbitMQ is a complex system in itself. This guide will only provide the basic commands to do a restart of a node and check the health of the cluster afterward. For complete documentation of how to restart RabbitMQ, please follow the .

To restart a RabbitMQ node run on each Trilio node:

It is recommended to wait for the node to rejoin and sync with the cluster before restarting another RabbitMQ node.

When the complete cluster is getting stopped and restarted it is important to keep the order of nodes in mind. The last node to be stopped needs to be the first node to be started.

Galera Cluster (MariaDB)

The Galera Cluster is managing the Trilio MariaDB database. It is running in active-active mode on all nodes of the Trilio cluster.

Galera Cluster is a complex system in itself. This guide will only provide the basic commands to do a restart of a node and check the health of the cluster afterward. For complete documentation of how to restart Galera clusters, please follow the .

When restarting Galera two different use-cases need to be considered:

Restarting a single node
Restarting the whole cluster

Restarting a single node

A single node can be restarted without any issues. It will automatically rejoin the cluster and sync against the remaining nodes.

The following commands will gracefully stop and restart the mysqld service.

After a restart will the cluster start the syncing process. Don't restart node after node to reach a complete cluster restart.

Check the cluster health after the restart.

Restarting the complete cluster

Restarting a complete cluster requires some additional steps as the Galera cluster is basically destroyed once all nodes have been shut down. It needs to be rebuild afterwards.

First gracefully shutdown the Galera cluster on all nodes:

The second step is to identify the Galera node with the latest dataset. This can be achieved by reading the grastate.dat file on the Trilio nodes.

When this documentation is followed the last mysqld service that got shut down will be the one with the latest dataset.

The value to check for are the seqno.

The node with the highest seqno is the node that contains the latest data. This node will also contain safe_to_bootstrap: 1 to indicate that the Galera cluster can be rebuild from this node.

On the identified node the new cluster is getting generated with the following command:

Running galera_new_cluster on the wrong node will lead to data loss as this command will set the node the command is issued on as the first node of the cluster. All nodes which join afterward will sync against the data of this first node.

After the command has been issued is the mysqld service running on this node. Now the other nodes can be restarted one by one. The started nodes will automatically rejoin the cluster and sync against the master node. Once a synced status has been reached is each node a primary node in the cluster.

Check the Cluster health after all services are up again.

Verify Health of the Galera Cluster

Verify the cluster health by running the following commands inside each Trilio MariaDB. The values returned from these statements have to be the same for each node.

Canonical workloadmgr container services

Canonical Openstack is not using the Trilio Appliance. In Canonical environments is the Trilio controller unit part of the JuJu deployment as workloadmgr container.

To restart the services inside this container the following commands are to be issued.

Single Node deployment

HA deployment

On all nodes:

On a single node:

Trilio dmapi service

The Trilio dmapi service is running on the Openstack controller nodes. Depending on the Openstack Distribution Trilio is installed on different commands are issued to restart the dmapi service.

RHOSP13

RHOSP13 is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

RHOSP16

RHOSP16 is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Canonical

Canonical is running the Trilio services in JuJu controlled LXD containers. The dmapi service can be restarted by issuing the following command from the MASS node.

Kolla-Ansible Openstack

Kolla-Ansible Openstack is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Ansible Openstack

Ansible Openstack is running the Trilio services as LXD containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Trilio datamover service (tvault-contego)

The Trilio datamover service is running on the Openstack compute nodes. Depending on the Openstack Distribution Trilio is installed on different commands are issued to restart the datamover service.

RHOSP13

RHOSP13 is running the Trilio services as docker containers. The datamover service can be restarted by issuing the following command on the compute node.

RHOSP16

RHOSP16 is running the Trilio services as docker containers. The datamover service can be restarted by issuing the following command on the compute node.

Canonical

Canonical is running the Trilio services in JuJu controlled LXD containers. The datamover service can be restarted by issuing the following command from the MASS node.

Kolla-Ansible Openstack

Kolla-Ansible Openstack is running the Trilio services as docker containers. The dmapi service can be restarted by issuing the following command on the host running the dmapi service.

Ansible Openstack

Ansible Openstack is running the Trilio datamover service directly on the compute node. The datamover service can be restarted by issuing the following command on.

Online upgrade Trilio Appliance

This describes the upgrade process from Trilio 4.0 or Trilio 4.0SP1 to Trilio 4.1 GA or its hotfix releases.

Kolla Ansible Openstack only: The mount point for the Trilio Backup Target has changed in Trilio 4.1. A reconfiguration after the upgrade is required.

Generic Pre-requisites

The prerequisites should already be fulfilled from upgrading the Trilio components on the Controller and Compute nodes.

Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of TVM.
The mentioned Gemfury repository should be accessible from TVault VM.
Please ensure the following points before starting the upgrade process:

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it has been completely shut down.

Verify if the service is shut down with the below set of commands and expected output:

Backup old configuration data

Take a backup of the conf files on all TVM nodes.

Setup Python3.8 virtual environment

Check if Python 3.8 virtual environment exists on the T4O nodes

If the virtual environment does not exist, perform the below steps on the T4O nodes

Setup Python3.6 virtual environment

Activate the Python3.6 virtual environment on all T4O nodes for wlm services upgrade

[T4O 4.0 to T4O 4.1 only] uninstall Ansible

Ansible doesn't support the upgrade from previous versions to the latest one (2.10.4) and needs to be uninstalled for that reason

Upgrade pip package

Run the following command on all TVM nodes to upgrade the pip package

Set pip package repository env variable

Upgrade s3fuse/tvault-object-store

Major Upgrade

Run the following commands on all TVM nodes to upgrade s3fuse and its dependent packages.

Hotfix Upgrade

Run the following commands on all TVM nodes to upgrade s3fuse packages only.

Upgrade tvault-configurator

Post upgrade, the password for T4O configurator will be reset to the default one i.e. 'password' for user 'admin'. Reset T4O configurator password after the upgrade.

Make sure the correct virtual environment(myansible_3.8) has been activated

Major Upgrade

Run the following command on all TVM nodes to upgrade tvault-configurator and its dependent packages.

Hotfix Upgrade

Run the following command on all TVM nodes to upgrade tvault-configurator packages only.

During the update of the tvault-configurator the following error might be shown:

This error can be ignored.

Upgrade workloadmgr

Major Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr and its dependent packages.

Hotfix Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr packages only.

Upgrade workloadmgrclient

Major Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr and its dependent packages.

Hotfix Upgrade

Run the upgrade command on all TVM nodes to upgrade workloadmgr packages only.

Upgrade contegoclient

Major Upgrade

Run the upgrade command on all TVM nodes to upgrade contegoclient and its dependent packages.

Hotfix Upgrade

Run the upgrade command on all TVM nodes to upgrade contegoclient packages only.

Set oslo.messaging version

Using the latest available oslo.messaging version can lead to stuck RPC and API calls.

It is therefore required to fix the oslo.messaging version on the TVM.

Post Upgrade Steps

Restore the backed-up config files

[Major Upgrade 4.0 to 4.1 only] Delete wlm-scheduler pcs resource

Delete the wlm-scheduler pcs resource because in 4.1 it is not a part of pcs

Restart services

Restart the following services on all node(s) using respective commands\

tvault-object-store restart required only if Trilio is configured with S3 backend storage

Enable Global Job Scheduler ****Restart pcs resources only on the primary node

Verify the status of the services

tvault-object-store will run only if TVault configured with S3 backend storage

Additional check for wlm-cron on the primary node

The above command should show only 2 processes running: sample below:

Check the mount point using “df -h” command

[Upgrade to HF1 and higher only] Reconfigure the Trilio Appliance

Trilio for Openstack 4.1 HF1 is introducing several new config parameters, which will be automatically set upon reconfiguration.

[RHOSP and Kolla only] Reconfigure the Trilio Appliance

Trilio for Openstack 4.1 is changing the Trilio mount point as follows:

RHOSP 13 & 16.0 & 16.1: /var/lib/nova/triliovault-mounts Kolla Ansible Ussuri: /var/trilio/triliovault-mounts

Reconfiguring the Trilio Appliance will automatically handle this change.

[RHOSP and Kolla only] Create the mount bind to the old Trilio Mountpoint

Trilio for Openstack 4.1 is changing the Trilio mount point as follows:

RHOSP 13 & 16.0 & 16.1: /var/lib/nova/triliovault-mounts Kolla Ansible Ussuri: /var/trilio/triliovault-mounts

After reconfiguration of the Trilio Appliance, it is necessary to create a mount bind between the old and new mount points to provide full access to the old Trilio backups.

For RHOSP:

For Kolla:

To have this change persistent it is recommended to change the fstab accordingly:

For RHOSP:

For Kolla:

[RHOSP and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436,

In case of the UID/GID is changed back to 162:162 follow these steps to set it back to 42436:42436.

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group ids have changed to '42436'

Trilio 4.1 HF1 Release Notes

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Name

Tag

Changelog

Enhancements

Support for passwordless SMTP servers

Trilio for Openstack supports sending notification emails upon succeeded or failed backup/restore jobs. The required SMTP server configuration enforced the usage of a password for the SMTP user.

A password is no longer necessary when the SMTP server doesn’t need it.

Support for Multi-Attach Volumes

Cinder supports a Volume Type, which allows attaching the same Volume to multiple instances simultaneously. Backups and Restore for this Volume Type failed. Trilio for Openstack is now providing base support for this Volume Type.

Cinder Boot Volumes with Multi-Attach activated are not yet supported.

This only allows the backup and restoration of Multi-Attach Volumes. Trilio will handle the Volume like any single attached Volume for now. For example, a multi-attach volume connected to 2 VMs will get backed up and restored twice.

Increased the timeout for data transfer

Trilio for Openstack is tracking the progress of backups and restore using a tracking file. If this file is not getting updated within a defined timeframe, the Trilio data transfer fails. This timeframe got extended from 10 minutes to 20 minutes.

This value will become configurable in T4O 4.1 SP1

Misleading and expected errors are no longer shown in logs by default

Trilio for Openstack was logging many system error messages, which were actually expected and handled internally without impacting the actual functions of the solution.

These error messages were misleading in the normal troubleshooting process. These error messages aren’t logged anymore by default. They can be reactivated using the debug mode for logging.

Image upload timeout window has been increased and made configurable

The upload of Trilio backups is limited in time to prevent stalling workloads with a stuck upload process.

This timeout window has been increased from 10 hours to 48h by default and is configurable in the workloadmgr config file.

Restart wlm-workloads after setting this value.

Grace time to retrigger a Snapshot in case of deactivated Global Job Scheduler

When the Global Job Scheduler is deactivated, no backups are triggered. The Global Job Scheduler contains a grace time for missed Snapshots. All Snapshots that were supposed to be triggered within this grace time before activation of the Global Job Scheduler are retriggered.

This grace period is now configurable in the workloadmgr config file.

After setting this value, restart the wlm-cron service.

Increased the amount of dmapi workers and made it configurable

In highly used environments, the dmapi worker got identified as a potential bottleneck. The amount of default workers used by a dmapi service has been increased to 16 and is configurable in the dmapi config file.

Restart the dmapi service after setting the configuration manually.

The upgrade process of RHOSP and Kolla Ansible will automatically set this value.

Added new settings in haproxy configuration for dmapi

Response times in highly used environments might be slow, leading to the dmapi service timing out in the haproxy connection. The default values of haproxy are not always suitable in that case.

The haproxy configuration for the dmapi service has been extended to the following values.

Restart the haproxy service after setting the configuration manually.

The upgrade process of RHOSP and Kolla Ansible will automatically set these values.

Added retries for Neutron API calls

The Openstack Neutron service is highly used up to the point that API calls are timing out. Trilio backups and restores failed when any Neutron API call timed out.

Trilio will now retry Neutron API calls three times before failing a backup or restore.

Added retries and rescan for mounting temporary Volumes using Cinder Storages with multipathing activated

It was observed in multipathing environments that sometimes backups failed due to errors with the temporary Cinder Volumes during the following actions:

Create Cinder Volume out of Cinder Snapshot
Mount Cinder Volume to Compute Node
Unmount Cinder Volume from Compute Node
Delete Cinder Volume

During these operations in multipath environments, errors are now handled by rescanning the connected devices and retrying the internal commands.

The amount of retries is configurable in the tvault-contego config file.

Restart the tvault-contego service after manually setting the value.

The upgrade process of RHOSP and Kolla Ansible will automatically set these values.

Enhanced logging of Trilio for Openstack GUI

The Trilio for Openstack GUI uses the admin account to secure access to the features and functionalities located on the Trilio appliance. The following events are now getting logged by the Trilio Appliance:

Login attempts
Logout events
Password changes for the admin user

The Trilio for Openstack login page can now is extendable to contain a text banner. This text banner is configurable on the Trilio appliance by editing the banner yaml file located under:

/etc/tvault-config/banner.yaml

The content of the file looks as follows:

Restart tvault-config after changing the banner to activate it.

Fixed Bugs and issues

Local job scheduler stays disabled after workload creation with enabled job scheduler

An issue got fixed for rare occasions in which the status of the local job scheduler of a single workload was disabled, despite the workload created with an enabled job scheduler.

Global Job Scheduler is shown as active even when wlm-cron service is down

An issue got fixed for the Global Job Scheduler returning enabled or disabled even when the wlm- cron service is deactivated. The status returned in this scenario is now an error message showing the wlm-cron service status.

Wrong documentation link in the Trilio Appliance

The documentation link available inside the Trilio Appliance was still pointing to the old outdated documentation webpage. The link has been updated to point towards the correct documentation.

Restore VMs into different Availability Zone failed

An issue got fixed, which prevented the restoration of VMs into different Availability Zones in the case of the original Availability Zone no longer being available.

Stopping workloadmgr service and all ongoing worker tasks

An issue got fixed, which lead to stall service jobs being left behind upon restart of workloadmgr services.

Mounting of LVM configured disks into the FRM failed

An issue got fixed, which prevented the correct mounting and access of Volumes partitioned and configured by LVM.

Upload of Backups failed intermittently using S3

An issue got fixed, which lead to a race condition between upload threads in the S3 fuse plugin, which led to backups failing during the upload phase.

Multipathing not enabled by default in the Data-Mover container used in RHOSP and Kolla Ansible Openstack

An issue got fixed, which lead to multipathing not being enabled in the Data-Mover container used by RHOSP and Kolla Ansible.

Upgrading to 4.1 HF1 will automatically activate multipathing where feasible.

An inaccessible S3 mount point got created when the S3 endpoint is not available during deployment or configuration

An issue got fixed, which lead to the creation of a Trilio mount point, even when the provided S3 backup target is not reachable during deployment or configuration.

The deployment will still succeed, but the tvault-object-store service will be in a failed state.

Trustee role not correctly inherited from user groups to users

An issue got fixed, which prevented the detection of the Trilio trustee role for a user, who had this role inherited from a user group.

The configurator always trying to use Keystone internal endpoint

An issue got fixed, during which the chosen endpoint type did not get honored for Keystone and the configurator always reached out to the Keystone internal endpoint.

The following value can be set in the api-paste ini file located under:

/etc/workloadmgr/api-paste.ini

Afterward the wlm-workloads service needs to be restarted.

It is recommended to reconfigure the appliance to activate the fix

Disk integrity check failing with a false positive

An issue got identified, which leads to the disk integrity check failing, although there is no data loss.

Snapshots with a failed disk integrity check are currently no longer failing and instead show a warning in the log files about the failed disk integrity check.

A complete fix of the root cause is planned for 4.1 SP1.

Workload creation and Backup creation failed due to Latin characters like á

An issue got identified in which Latin characters like á did lead to a Workload not being created or a backup not succeeding.

This hotfix implements support of Latin characters for the following:

Calendar shown and used during workload creation for the job scheduler
Name and description of security groups

Full support for Latin characters comes in 4.1 SP1

Backups for multipath environments using the FC protocol failed

An issue got fixed, which prevented successful backups in environments using multipathing with the FC storage protocol.

Trilio 4.1 Release Notes

Trilio release 4.1 introduces new features and capabilities including:

Openstack Ussuri support
New File Recovery Process
Increased Openstack independency
Installation Optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform
Support for External MySQL/MariaDB databases
Incremental Backup for nova booted instances
Support of Openstack User Groups
New Quota: Snapshots
S3 Support for Kolla Ansible Openstack
UI enhancement for selective Restore

Trilio 4.1.94 is the GA release of Trilio 4.1

Release Versions

Packages

Name

Type

Version

Containers and Gitbranch

Name

Tag

Openstack Ussuri support

Trilio 4.1 continues to enable Openstack versions and Distributions, allowing Trilio customers to stay up to date with Openstack releases.

Trilio 4.1 introduces full support for Openstack Ussuri for Kolla Ansible Openstack, Ansible Openstack, and Canonical Openstack. In addition, Trilio 4.1 does of course support the active long-term releases from Red Hat and Canonical. Openstack users of those releases can continue to use the latest Trilio functionalities.

New File Recovery Manager Process

Since Trilio was released for the first time was the File Recovery Manager instance in tandem with Trilio. The File Recovery Manager instance helped customers to easily and quickly fetch and restore files and folders directly from the backups.

Over the years did more and more customers request to have the File Recovery Manager on their own images or on a specific Linux distribution. We looked into the possibility to create multiple versions of the File Recovery Manager which was identified as not getting to the point of flexibility our customers need.

The result was a revamp of the File Recovery Manager which allows the installation on any CentOS-based or Ubuntu-based instance.

Increased Openstack independency

Trilio has the goal to become an integrated part of Openstack since its first draft on a whiteboard. Back then were the Trilio services defined as sub-services of Nova and Trilio tied deeply into the already existing nova services and became an integral part of it.

Over the years did the nova service change and possibilities that were once available have been taken out from nova. This plus the required lengthy qualification of all Openstack Versions and Distributions made it clear that a change is required.

Trilio 4.1 is now a complete stand-alone service, which is communicating with nova and other Openstack services through APIs only. The goal of this independence is to speed up future qualification and requalification cycles to provide a broader support matrix again.

Installation optimization for Kolla Ansible Openstack, Ansible Openstack, and Red Hat Openstack Platform

Trilio's integration into Openstack already starts with the installation process. This process required manual installation steps in the past, which were hard to automize and scale on bigger environments.

Trilio 4.1 is therefore introducing Ansible Playbooks for Kolla-Ansible and Ansible Openstack, while the integration with Red Hat Director has been deepened. Further, the Trilio configurator does now allow to set the backup target on the Trilio Appliance directly based on Distribution.

Support for external MySQL/MariaDB

The Trilio Appliance provides its own database. A database that suits the needs of 99% of Trilio's customers. Some customers do have higher requirements. Be it performance, security, or just the general design of the Openstack itself.

For these customers and everyone who wants to use it is Trilio 4.1 introducing the possibility to configure Trilio with an external database.

Incremental Backups for nova booted instance root volumes

Openstack provides the possibility to start an instance from a Cinder Volume or directly using the Glance Image and a nova volume.

Trilio always provided incremental forever backups for all cinder volumes. Root-Volumes from nova-booted instances were always taken as a full backup of the Glance image and the actual VM root volume.

Trilio 4.1 does introduce incremental backups for nova booted instance root volumes. This allows Trilio 4.1 to provide incremental backups to any type of root volume.

Support for Openstack User Groups

Trilio is using the Openstack Keystone service to authenticate any user and to verify that the right permissions are set. Openstack Keystone allows to group users and set the permissions to the group.

These Openstack User Groups are now fully supported.

New Quota: Snapshots

Trilio 4.0 introduced the Quota functionality, which allowed to set quotas for the number of workloads, number of VMs and amount of storage used by a single tenant.

Trilio 4.1 extends this feature by the number of Trilio Snapshots that a Tenant is allowed to have.

S3 Support for Kolla Ansible Openstack

S3 is becoming the standard to transfer data to and from storage solutions. Trilio introduced S3 already in Version 3.0 but had to take it out for Kolla Ansible since Kolla Ansible has been added to the Support Matrix.

Trilio 4.1 is now closing that gap to other Openstack Distributions and provides full S3 support for Kolla Ansible Openstack Ussuri.

UI enhancement for the Selective Restore

The Selective Restore is the most powerful and complex restore Trilio has to offer. The UI needs to be easy to understand and help the user to fulfill its task.

Several points have been identified to improve this requirement of usability. The selective restore now allows to select or deselect all VMs at once. Further are the VMs now provided in an easy to overview list and the sub-controls can be expanded and collapsed as necessary.

Known Issues

This release contains the following known issues which are tracked for a future update.

Snapshots getting stuck in uploading stage on RHOSP13 with ISCSI based Volumes

Observation:

Login into the iscsi device is getting rejected for the Trilio service
Result is the Snapshot not moving forward until timing out

Workloadmgr Quota feature does not support RHOSP13 & Canonical Queens Horizon

The workloadmgr Quota feature is still fully supported through CLI.

[Openstack Ansible] snapshot mount failed with error 'Permission denied'

Observation:

It has been observed that on non-kolla, non-rhosp setups, such as openstack ansible, nova user id is not same as we consider as default(162).
Another observation is that, the id which is assigned to nova, was conflicting with id of system user in TVM, this created a situation where we had to redeploy Openstack.

Workaround:

Update permissions of /var/triliovault-mounts to 755

[Volume Backup Exclude] Excluded Ceph Volume after restore not mountable or formattable

Observation:

VM Volumes stored on Ceph are successfully excluded from backup if desired
Restore does create empty Ceph Volume
created empty Ceph Volume is not attachable or formattable

Restored VMs have blank metadata config_drive attached

Observation:

For every restore will the metadata config_drive be set as blank value
No impact on restored VMs known

Workaround

delete metadata config_drive
or set desired value

TVM reconfig fails when adding new Trilio VM to the cluster

Observation:

TVault re-configuration while adding nodes to existing TVM cluster fails at "Configuring Trilio Cluster"
Reason is that the prev mysql password was not working and mysql root access has be reset.

Workaround:

remove /root/.my.cnf file on already configured TVM and reconfigure it

Database does not sync after Trilio cluster gets new nodes

Observation:

After TVault re-configuration post addition of 2 more nodes to existing TVM cluster ("import workloads" was not seleted), the databases do not sync against already existing TVM.
It is expected that while adding the 2 new nodes, the db on node1 should get synced up with 2 new nodes and the existing workloads should be available post the reconfig on the new 3 node TVM cluster.

Workaround:

Run workload import from CLI

[exclude_boot_disk] Data on boot disk gets backed up despite exclusion

Observation:

VM was set with metadata exclude_boot_disk_from_backup set to true
Restored instance showed, that data was backed up and restored

After reinitialize and import Openstack certificates are missing

Observation:

Reinitialize does not keep the already uploaded Openstack Certificates used to communicate with Openstack.

Workaround:

Upload Certificates again

CLI import changes scheduler trust value to disabled

Observation:

When the is used via CLI is the scheduler trust changed from enabled to disabled.

Workaround:

Configure/re-configure T4O with import option from UI after reinitialize.

Unable to get node details after reinitializing the Trilio Appliance

Observation:

After reinitializing was neither the UI nor CLI showing Node information

Workaround:

Restart wlm-workloads and wlm-cron services on Trilio nodes
systemctl restart wlm-workloads
systemctl restart wlm-cron

Snapshots fails with "object is not subscriptable" for many workload jobs at the exact same time

Oberservation:

Running more than 25 workloads at the exact same time leads to error
dmapi service is not responding
Snapshots fail with "object is not subscriptable"

Workaround:

Contact Trilio Support to implement a known workaround.

[Kolla Ansible] dmapi container stuck in restarting if storage backend changes

Observation:

Just changing the backup target in the Kolla Ansible configuration files and redeploying leads to dmapi container stuck in restart

Workaround:

Follow guide to

No operation is permitted in insecure way for ssl enabled Keystone URL

Observation:

SSL enabled Openstack
Backup and Restore jobs fail with with missing TLS CA certificate bundle error

Workaround:

Configure the Trilio appliance with Openstack CA provided
OR Provide Openstack CA to /etc/workloadmgr/ca-chain.pem

Upgrading on RHOSP

0] Pre-requisites

Please ensure the following points are met before starting the upgrade process:

No Snapshot or Restore is running
Global job scheduler is disabled
wlm-cron is disabled on the Trilio Appliance

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

1.] [On Undercloud node] Clone latest Trilio repository and upload Trilio puppet module

All commands need to be run as user 'stack' on undercloud node

The Trilio appliance connected to this installation needs to be of version 4.1 HF10

1.1] Clone Trilio cfg scripts repository

Separate directories are created as per Redhat OpenStack release under 'triliovault-cfg-scripts/redhat-director-scripts/' directory. Use all scripts/templates from respective directory. For ex, if your RHOSP release is 13, then use scripts/templates from 'triliovault-cfg-scripts/redhat-director-scripts/rhosp13' directory only.

Available RHOSP_RELEASE___DIRECTORY values are:

rhosp13 rhosp16.1 rhosp16.2

RHOSP 16.0 is not supported anymore as RedHat has officially stopped supporting it. However, Trilio maintained it for some time and stopped the support from 4.1HF11 onwards. The latest hotfix available for RHOSP16.0 is 41.HF10. Reach out to the Support team for any help.

1.2] If backup target type is 'Ceph based S3' with SSL:

If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into the puppet directory of the right release.

2] Upload Trilio puppet module

3] Update overcloud roles data file to include Trilio services

Trilio has two services as explained below. You need to add these two services to your roles_data.yaml. If you do not have customized roles_data file, you can find your default roles_data.yaml file at /usr/share/openstack-tripleo-heat-templates/roles_data.yaml on undercloud.

You need to find that role_data file and edit it to add the following Trilio services.

i) Trilio Datamover Api Service:

Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamoverApi

This service needs to be co-located with database and keystone services. That said, you need to add this service on the same role as of keystone and database service.

Typically this service should be deployed on controller nodes where keystone and database runs. If you are using RHOSP's pre-defined roles, you need to addOS::TripleO::Services::TrilioDatamoverApiservice to Controller role.

ii) Trilio Datamover Service: Service Entry in roles_data yaml: OS::TripleO::Services::TrilioDatamover This service should be deployed on role where nova-compute service is running.

If you are using RHOSP's pre-defined roles, you need to add our OS::TripleO::Services::TrilioDatamover service to Compute role.

If you have defined your custom roles, then you need to identify the role name where in 'nova-compute' service is running and then you need to add 'OS::TripleO::Services::TrilioDatamover' service to that role.

4] Prepare latest Trilio container images

All commands need to be run as user 'stack'

Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL is ''. The Trilio container URLs are as follows:

4.1] available container images

RHOSP 13

RHOSP 16.1

RHOSP 16.2

There are three registry methods available in RedHat OpenStack Platform.

Remote Registry
Local Registry
Satellite Server

4.2] Remote Registry

Please refer to the to see which containers are available.

Follow this section when 'Remote Registry' is used.

For this method, it is not necessary to pull the containers in advance. It is only necessary to populate the trilio_env.yaml file with the Trilio container URLs from Redhat registry.

Populate the trilio_env.yaml with container URLs for:

Trilio Datamover container
Trilio Datamover api container
Trilio Horizon Plugin

trilio_env.yaml will be available in __triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments

Example

4.2] Local Registry

Please refer to the to see which containers are available.

Follow this section when 'local registry' is used on the undercloud.

In this case it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts which will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and updates the trilio_env.yaml.

RHOSP13

Verify the changes

RHOSP16.1

Verify the changes:

RHOSP16.2

Verify the changes

The changes can be verified using the following commands.

4.3] Red Hat Satellite Server

Please refer to the to see which containers are available.

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given

Populate the trilio_env.yaml with container urls.

RHOSP 13

RHOSP16.1

RHOSP16.2

5. Verify Trilio environment details

It is recommended to re-populate the backup target details in the freshly downloaded trilio_env.yaml file. This will ensure that parameters that have been added since the last update/installation of Trilio are available and will be filled out too.

Locations of the trilio_env.yaml:

For more details about the trilio_env.yaml please check .

6] Update Overcloud Trilio components

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use correct Trilio endpoint map file as per available Keystone endpoint configuration

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

7] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

7.1] On Controller node

Make sure Trilio dmapi and horizon containers are in a running state and no other Trilio container is deployed on controller nodes. When the role for these containers is not "controller" check on respective nodes according to configured roles_data.yaml.

7.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

7.3] On the node with Horizon service

Make sure horizon container is in running state. Please note that 'Horizon' container is replaced with Trilio Horizon container. This container will have latest OpenStack horizon + Trilio's horizon plugin.

Workloads

Definition

A workload is a backup job that protects one or more Virtual Machines according to a configured policy. There can be as many workloads as needed. But each VM can only be part of one Workload.

List of Workloads

Using Horizon

To view all available workloads of a project inside Horizon do:

Login to Horizon
Navigate to Backups
Navigate to Workloads

The overview in Horizon lists all workloads with the following additional information:

Creation time
Workload Name
Workload description
Total amount of Snapshots inside this workload

Using CLI

--all {True,False}➡️List all workloads of all projects (valid for admin user only)
--nfsshare <nfsshare>➡️List all workloads of nfsshare (valid for admin user only)

Workload Create

Using Horizon

To create a workload inside Horizon do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads
Click "Create Workload"

The created Workload will be available after a few seconds and starts to take backups according to the provided schedule and policy.

Using CLI

--display-name➡️Optional workload name. (Default=None)
--display-description➡️Optional workload description. (Default=None)
--workload-type-id➡️

Workload Overview

A workload contains many information, which can be seen in the workload overview.

Using Horizon

To enter the workload overview inside Horizon do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads
Identify the workload to show the details on

Details Tab

The Workload Details tab provides you with the general most important information about the workload:

Name
Description
Availability Zone
List of protected VMs including the information of qemu guest agent availability

The status of the qemu-guest-agent just shows, whether the necessary Openstack configuration has been done for this VM to provide qemu guest agent integration. It does not check, whether the qemu guest agent is installed and configured on the VM.

It is possible to navigate to the protected VM directly from the list of protected VMs.

Snapshots Tab

The Workload Snapshots Tab shows the list of all available Snapshots in the chosen Workload.

From here it is possible to work with the Snapshots, create Snapshots on demand and start Restores.

Please refer to the and User Guide to learn more about those.

Policy Tab

The Workload Policy Tab gives an overview of the current configured scheduler and retention policy. The following elements are shown:

Scheduler Enabled / Disabled
Start Date / Time
End Date / Time
RPO

Filesearch Tab

The Workload Filesearch Tab provides access to the powerful search engine, which allows to find files and folders on Snapshots without the need of a restore.

Please refer to the File Search User Guide to learn more about this feature.

Misc. Tab

The Workload Miscellaneous Tab shows the remaining metadata of the Workload. The following information are provided:

Creation time
last update time
Workload ID
Workload Type

Using CLI

<workload_id> ➡️ ID/name of the workload to show
--verbose➡️option to show additional information about the workload

Edit a Workload

Workloads can be modified in all components to match changing needs.

Editing a Workload will set the User, who edits the Workload, as the new owner.

Using Horizon

To edit a workload in Horizon do the following steps:

Login to the Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload to be modified

Using CLI

--display-name ➡️ Optional workload name. (Default=None)
--display-description➡️Optional workload description. (Default=None)
--instance <instance-id=instance-uuid>➡️

Delete a Workload

Once a workload is no longer needed it can be safely deleted.

All Snapshots need to be deleted before the workload gets deleted. Please refer to the User Guide to learn how to delete Snapshots.

Using Horizon

To delete a workload do the following steps:

Login to Horizon
Navigate to the Backups
Navigate to Workloads
Identify the workload to be deleted

Using CLI

<workload_id> ➡️ ID/name of the workload to delete
--database_only <True/False>➡️Keep True if want to delete from database only.(Default=False)

Unlock a Workload

Workloads that are actively taking backups or restores are locked for further tasks. It is possible to unlock a workload by force if necessary.

It is highly recommend to use this feature only as last resort in case of backups/restores being stuck without failing or a restore is required while a backup is running.

Using Horizon

Login to the Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload to unlock

Using CLI

<workload_id> ➡️ ID of the workload to unlock

Reset a Workload

In rare cases it might be necessary to start a backup chain all over again, to ensure the quality of the created backups. To not recreate a Workload in such cases is it possible to reset a Workload.

The Workload reset will:

Cancel all ongoing tasks
Delete all existing Openstack Trilio Snapshots from the protected VMs
recalculate the next Snapshot time
take a full backup at the next Snapshot

Using Horizon

To reset a Workload do the following steps:

Login to the Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload to reset

Using CLI

<workload_id> ➡️ ID/name of the workload to reset

Upgrading on Kolla OpenStack

Upgrading from Trilio 4.0 to Trilio 4.1

Due to the new installation method of Trilio for Kolla OpenStack, it is required to reinstall the Trilio components running on the Kolla Openstack nodes when upgrading from Trilio 4.0.

The Trilio appliance can be upgraded as documented.

Upgrading from Trilio 4.1 to a higher version

Trilio 4.1 can be upgraded without reinstallation to a higher version of T4O if available.

Refer to the below-mentioned acceptable values for the placeholders in this document as per the Openstack environment: kolla_base_distro : ubuntu / centos triliovault_tag : 4.1.94-hotfix-13-ussuri / 4.1.94-hotfix-12-victoria

Pre-requisites

Please ensure the following points are met before starting the upgrade process:

Either 4.1 GA OR any hotfix patch against 4.1 should be already deployed
No Snapshot OR Restore is running
Global job scheduler should be disabled
wlm-cron is disabled on the primary Trilio Appliance

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

Clone latest configuration scripts

Before the latest configuration script is loaded it is recommended to take a backup of the existing config scripts' folder & Trilio ansible roles. The following command can be used for this purpose:

Clone the latest configuration scripts of the required branch and access the deployment script directory for Kolla Ansible Openstack. Available branches to upgrade T4O 4.1 are:

Copy the downloaded Trilio ansible role into the Kolla-Ansible roles directory.

Append Trilio variables

Clean old Trilio variables and append new Trilio variables

This step is not always required. It is recommended to comparetriliovault_globals.ymlwith the Trilio entries in the/etc/kolla/globals.ymlfile.

In case of no changes, this step can be skipped.

This is required, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_globals.yml they need to be updated in /etc/kolla/globals.yml file.

Clean old Trilio passwords and append new Trilio password variables

This step is not always required. It is recommended to comparetriliovault_passwords.ymlwith the Trilio entries in the/etc/kolla/passwords.ymlfile.

In case of no changes, this step can be skipped.

This step is required, when some password variable names have been added, changed, or removed in the latest triliovault_passwords.yml. In this case, the /etc/kolla/passwords.yml needs to be updated.

Append triliovault_site.yml content to kolla ansible's site.yml

This step is not always required. It is recommended to comparetriliovault_site.ymlwith the Trilio entries in the/usr/local/share/kolla-ansible/ansible/site.ymlfile.

In case of no changes, this step can be skipped.

This is required because, in case of some variable names changed, some new variables have been added, or old variables removed in the latest triliovault_site.yml they need to be updated in /usr/local/share/kolla-ansible/ansible/site.yml file.

Append triliovault_inventory.txt to the kolla-ansible inventory file

This step is not always required. It is recommended to comparetriliovault_inventory.yml ith the Trilio entries in the/root/multinodefile.

In case of no changes, this step can be skipped.

By default, the triliovault-datamover-api service gets installed on ‘control' hosts and the trilio-datamover service gets installed on 'compute’ hosts. You can edit the T4O groups in the inventory file as per your cloud architecture.

T4O group names are ‘triliovault-datamover-api’ and ‘triliovault-datamover’

Edit globals.yml to set T4O parameters

Edit '/etc/kolla/globals.yml' file to fill triliovault backup target and build details. You will find the triliovault related parameters at the end of globals.yml file. User needs to fill in details like triliovault build version, backup target type, backup target details, etc.

Following is the list of parameters that the user needs to edit.

Parameter

Defaults/choices

comments

Enable T4O Snapshot mount feature

This step is already part of the 4.1 GA installation procedure and should only be verified.

To enable Trilio's Snapshot mount feature it is necessary to make the Trilio Backup target available to the nova-compute and nova-libvirt containers.

Edit /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults/main.yml and find nova_libvirt_default_volumes variable. Append the Trilio mount bind /var/trilio:/var/trilio:shared to the list of already existing volumes.

For a default Kolla installation, will the variable look as follows afterward:

Next, find the variable nova_compute_default_volumes in the same file and append the mount bind /var/trilio:/var/trilio:shared to the list.

After the change will the variable look for a default Kolla installation as follows:

In case of using Ironic compute nodes one more entry need to be adjusted in the same file. Find the variable nova_compute_ironic_default_volumes and append trilio mount /var/trilio:/var/trilio:shared to the list.

After the changes the variable will looks like the following:

Pull containers in case of private repository

In case, the user doesn’t want to use the docker hub registry for triliovault containers during cloud deployment, then the user can pull triliovault images before starting cloud deployment and push them to other preferred registries.

Following are the triliovault container image URLs. Replace kolla_base_distro and triliovault_tag variables with their values

Pull T4O container images

Run the below command from the directory with the multinode file tull pull the required images.

Run Kolla-Ansible upgrade command

Run the below command from the directory with the multinode file to start the upgrade process.

Verify Trilio deployment

Verify on the nodes that are supposed to run the Trilio containers, that those are available and healthy.

Advance settings/configuration for Trilio services

Customize HAproxy configuration parameters for Trilio datamover api service

Following are the default haproxy conf parameters set against triliovault datamover api service.

These values work best for triliovault dmapi service. It’s not recommended to change these parameter values. However, in some exceptional cases, If needed to change any of the above parameter values then same can be done on kolla-ansible server in the following file.

After editing, run kolla-ansible deploy command again to push these changes to openstack cloud.

Post kolla-ansible deploy, to verify the changes, please check following file, available on all controller/haproxy nodes.

Backups-Admin Area

Trilio provides Backup-as-a-Service, which allows Openstack Users to manage and control their backups themselves. This doesn't eradicate the need for a Backup Administrator, who has an overview of the complete Backup Solution.

To provide Backup Administrators with the tools they need does Trilio for Openstack provide a Backup-Admin area in Horizon in addition to the API and CLI.

Access the Backups-Admin area

To access the Backups-Admin area follow these steps:

Login to Horizon using admin user.
Click on Admin Tab.
Navigate to Backups-Admin Tab.
Navigate to Trilio page.

The Backups-Admin area provides the following features.

It is possible to reduce the shown information down to a single tenant. That way seeing the exact impact the chosen Tenant has.

Status overview

The status overview is always visible in the Backups-Admin area. It provides the most needed information on a glance, including:

Storage Usage (nfs only)
Number of protected VMs compared to number of existing VMs
Number of currently running Snapshots
Status of TVault Nodes

The status of nodes is filled when the services are running and in good status.

Workloads tab

This tab provides information about all currently existing Workloads. It is the most important overview tab for every Backup Administrator and therefor the default tab shown when opening the Backup-Admins area.

The following information are shown:

User-ID that owns the Workload
Project that contains the Workload
Workload name
Workload Type

Usage tab

Administrators often need to figure out, where a lot of resources are used up, or they need to quickly provide usage information to a billing system. This tab helps in these tasks by providing the following information:

Storage used by a Tenant
VMs protected by a Tenant

It is possible to drill down to see the same information per workload and finally per protected VM.

The Usage tab includes workloads and VMs that are no longer actively used by a Tenant, but exist on the backup target.

Nodes tab

This tab displays information about Trilio cluster nodes. The following information are shown:

Node name
Node ID
Trilio Version of the node
IP Address

The Virtual IP is shown as it's own node. It is typically shown directly below the current active Controller Node.

Data Movers tab (Trilio Data Mover Service)

This tab displays information about Trilio contego service. The following information are shown:

Service-Name
Compute Node the service is running on
Zone
Service Status from Openstack perspective (enabled/disabled)

Storage tab

This tab displays information about the backup target storage. It contains the following information:

Storage Name

Clicking on the Storage name provides an overview of all workloads stored on that storage.

Capacity of the storage
Total utilization of the storage
Status of the storage
Statistic information

Audit tab

Audit logs provide the sequence of workload related activities done by users, like workload creation, snapshot creation, etc. The following information are shown:

Time of the entry
What task has been done
Project the task has performed in
User that performed the task

The Audit log can be searched for strings to find for example only entries down by a specific user.

Additionally, can the shown timeframe be changed as necessary.

License tab

The license tab provides an overview over the current license and allows to upload new licenses, or validate the current license.

A license validation is automatically done, when opening the tab.

The following information about an active license are shown:

Organization (License name)
License ID
Purchase date - when was the license created
License Expiry Date

Trilio will stop all activities once a license is no longer valid or expired.

Policy tab

The policy tab gives Administrators the possibility to work with workload policies.

Please use in the Admin guide to learn more about how to create and use Workload Policies.

Settings tab

This tab manages all global settings for the whole cloud. Trilio has two types of settings:

Email settings
Job scheduler settings.

Email Settings

These settings will be used by Trilio to send email reports of snapshots and restores to users.

Configuring the Email settings is a must-have to provide Email notification to Openstack users.

The following information are required to configure the email settings:

SMTP Server
SMTP username
SMTP password
SMTP port

A test email can be sent directly from the configuration page.

To work with email settings through CLI use the following commands:

To set an email setting for the first time or after deletion use:

--description➡️Optional description (Default=None) ➡️ Not required for email settings
--category➡️Optional setting category (Default=None) ➡️ Not required for email settings

To update an already set email setting through CLI use:

--description➡️Optional description (Default=None) ➡️ Not required for email settings
--category➡️Optional setting category (Default=None) ➡️ Not required for email settings

To show an already set email setting use:

--get_hidden➡️show hidden settings (True) or not (False) ➡️ Not required for email settings, use False if set
<setting_name>➡️name of the setting to show➡️ Take from the list below

To delete a set email setting use:

<setting_name>➡️name of the setting to delete ➡️ Take from the list below

Setting name

Value type

example

Disable/Enable Job Scheduler

The Global Job Scheduler can be used to deactivate all scheduled workloads without modifying each one of them.

To activate/deactivate the Global Job Scheduler through the Backups-Admin area:

Login to Horizon using admin user.
Click on Admin Tab.
Navigate to Backups-Admin Tab.
Navigate to Trilio page.

The Global Job Scheduler can be controlled through CLI as well.

To get the status of the Global Job Scheduler use:

To deactivate the Global Job Scheduler use:

To activate the Global Job Scheduler use:

Installing on RHOSP

The Red Hat Openstack Platform Director is the supported and recommended method to deploy and maintain any RHOSP installation. Trilio integrates natively into the RHOSP Director and manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and also associated configuration needs. The following backup target types are supported by Trilio:

Backup Target Types

Required Configuration

1.2] Clone triliovault-cfg-scripts repository

The overcloud-deploy command must already have been run successfully prior to this point and overcloud should be available. Perform the following steps for 'undercloud' node on an existing RHOSP environment:

All commands need to be run as user 'stack' on undercloud node

Ensure that the Trilio appliance connected to this installation is on the latest Hotfix version. Failure to ensure this may lead to your installation not working as expected.
1. Refer to this doc :
Run the following command to clone the triliovault-cfg-scripts github repository:

If your backup target type is 'Ceph-based S3' with SSL, skip this step. Otherwise, access the Red Hat Director scripts according to the RHOSP version being used:
- RHOSP 13 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp13/
- RHOSP 16.1 - cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

From this point onwards in the documentation, only the following path will be used for examples: cd triliovault-cfg-scripts/redhat-director-scripts/rhosp16.1/

2] Upload Trilio-Puppet module

RHOSP 16.1

The following commands upload the Trilio puppet module to the overcloud registry. The upload only happens upon the next deployment.

Step 1: -

The output of the above command looks like the following.

Trilio puppet module is uploaded to overcloud as a swift deploy artifact with heat resource name 'DeployArtifactURLs'.

Step 2: - Check Trilio's Puppet module artifact file and ensure that it looks like the following:

Step 3: -

Firstly, check to make sure that your overcloud deploy environment files uses deploy artifacts. To do this check string DeployArtifactURLs in your environment files (only those mentioned in the overcloud deploy command with -e option). If you find any environment file with the -e option, then your deploy command is using deploy artifacts.
If your deploy command is using deploy artifact, you must merge all deploy artifacts in a single file. For example, if your artifact file path is /home/stack/templates/user-artifacts.yaml, then perform the following steps to merge both urls in single file, which is passed to the overcloud deploy command with the -e option.

3] Update overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

In the case of uncustomized roles_data.yaml can the default file be found on the undercloud at:

/usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Add the following services to the roles_data.yaml

All commands need to be run as user 'stack'

3.1] Add Trilio Datamover Api Service to role data file

This service needs to share the same role as the keystone and database service. In case of the pre-defined roles will these services run on the role Controller. In case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.

Add the following line to the identified role:

3.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In case of the pre-defined roles will the nova-compute service run on the role Compute. In case of custom-defined roles, it is necessary to use the role that the nova-compute service is using.

Add the following line to the identified role:

4] Prepare Trilio container images

All commands need to be run as user 'stack'

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL: registry.connect.redhat.com Container pull URLs are given below.

Please note that using the hotfix containers requires that the Trilio Appliance is getting upgraded to the desired hotfix level as well.

Refer to the word <HOTFIX-TAG-VERSION> as 4.1.94-hotfix-16 in the below sections

RHOSP 13

RHOSP 16.1

RHOSP 16.2

There are three registry methods available in the RedHat OpenStack Platform.

Remote Registry
Local Registry
Satellite Server

4.1] Remote Registry

Follow this section when 'Remote Registry' is used.

In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution. User can set the remote registry to the RedHat registry or any other private registry that he wants to use. The user needs to provide credentials for the registry in containers-prepare-parameter.yaml file.

Make sure other OpenStack service images are also using the same method to pull container images. If it's not the case you can not use this method.
Populate containers-prepare-parameter.yaml with content like following. Important parameters are 'push_destination: false', ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to the registry registry.connect.redhat.com Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.

Note: This file - containers-prepare-parameter.yaml

Redhat document for remote registry method:

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise, image pull operation will fail.

4. Populate the trilio_env.yaml with Trilio container image URLs:

Trilio Datamover container
Trilio Datamover API container
Trilio Horizon Plugin

trilio_env.yaml will be available in

4.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case, it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts that will pull the containers from 'registry.connect.redhat.com' and push them to the undercloud and update the trilio_env.yaml.

RHOSP13

Verify the changes

RHOSP16.1

Verify the changes:

RHOSP16.2

Verify the changes

The changes can be verified using the following commands.

4.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Pull the Trilio containers on the Red Hat Satellite using the given

Populate the trilio_env.yaml with container urls.

RHOSP 13

RHOSP16.1

RHOSP16.2

5] Provide environment details in trilio-env.yaml

Provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still it is recommended to verify the container URLs.

The following information are required additionally:

Network for the datamover api
datamover password
Backup target type {nfs/s3}
In case of NFS

Use ceph_s3 for any non-aws S3 backup targets.

6. Advanced Settings/Configuration

6.1 Haproxy customized configuration for Trilio dmapi service __

The existing default haproxy configuration works fine with most of the environments. Only when timeout issues with the dmapi are observed or other reasons are known, change the configuration as described here.

Following is the haproxy conf file location on haproxy nodes of the overcloud. Trilio datamover api service haproxy configuration gets added to this file.

Trilio datamover haproxy default configuration from the above file looks as follows:

The user can change the following configuration parameter values.

To change these default values, you need to do the following steps. i) On the undercloud node, open the following file for edit (Edit <RHOSP_RELEASE> with your cloud's release information. Valid values are - rhosp13, rhosp16, rhosp16.1)

For RHOSP13

For RHOSP16.0

For RHOSP16.1

For RHOSP16.2

ii) Search the following entries and edit as required

iii) Save the changes.

7] Deploy overcloud with trilio environment

Use the following heat environment file and roles data file in overcloud deploy command:

trilio_env.yaml
roles_data.yaml
Use the correct Trilio endpoint map file as per available Keystone endpoint configuration

To include new environment files use '-e' option and for roles data file use '-r' option. An example overcloud deploy command is shown below:

8] Verify deployment

If the containers are in restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

8.1] On Controller node

Verify the haproxy configuration under:

8.2] On Compute node

Make sure Trilio datamover container is in running state and no other Trilio container is deployed on compute nodes.

8.3] On the node with Horizon service

9] Additional Steps on Trilio Appliance

9.1] Change the nova user id on the Trilio Nodes

In RHOSP, nova user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes needs to be set the same. Do the following steps on all Trilio nodes:

Download the shell script that will change the user id
Assign executable permissions
Execute the script
Verify that nova user and group id have changed to '42436'

10] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

In case if the overcloud deployment is failing, do the following command to provide the list of errors. The following document also provides valuable insights:

Restores

Definition

A Restore is the workflow to bring back the backed up VMs from a Trilio Snapshot.

List of Restores

Using Horizon

To reach the list of Restores for a Snapshot follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to show

Using CLI

--snapshot_id <snapshot_id> ➡️ ID of the Snapshot to show the restores of

Restores overview

Using Horizon

To reach the detailed Restore overview follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to show

Details Tab

The Restore Details Tab shows the most important information about the Restore.

Name
Description
Restore Type
Status

The Restore Options are the restore.json provided to Trilio.

List of VMs restored
- restored VM Name
- restored VM Status
- restored VM ID

Misc Tab

The Misc tab provides additional Metadata information.

Creation Time
Restore ID
Snapshot ID containing the Restore
Workload

Using CLI

<restore_id> ➡️ ID of the restore to be shown
--output <output> ➡️ Option to get additional restore details, Specify --output metadata for restore metadata,--output networks --output subnets --output routers --output flavors

Delete a Restore

Once a Restore is no longer needed, it can be safely deleted from a Workload.

Deleting a Restore will only delete the Trilio information about this Restore. No Openstack resources are getting deleted.

Using Horizon

There are 2 possibilities to delete a Restore.

Possibility 1: Single Restore deletion through the submenu

To delete a single Restore through the submenu follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to delete

Possibility 2: Multiple Restore deletion through a checkbox in Snapshot overview

To delete one or more Restores through the Restore list do the following:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to show

Using CLI

<restore_id> ➡️ ID of the restore to be deleted

Cancel a Restore

Ongoing Restores can be canceled.

Using Horizon

To cancel a Restore in Horizon follow these steps:

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to delete

Using CLI

<restore_id> ➡️ ID of the restore to be deleted

One Click Restore

The One Click Restore will bring back all VMs from the Snapshot in the same state as they were backed up. They will:

be located in the same cluster in the same datacenter
use the same storage domain
connect to the same network
have the same flavor

The user can't change any Metadata.

The One Click Restore requires, that the original VM's that have been backed up are deleted or otherwise lost. If even one VM is still existing, will the One Click Restore fail.

The One Click Restore will automatically update the Workload to protect the restored VMs.

Using Horizon

There are 2 possibilities to start a One Click Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored

Using CLI

<snapshot_id> ➡️ ID of the snapshot to restore.
--display-name <display-name> ➡️ Optional name for the restore.
--display-description <display-description> ➡️

Selective Restore

The Selective Restore is the most complex restore Trilio has to offer. It allows to adapt the restored VMs to the exact needs of the User.

With the selective restore the following things can be changed:

Which VMs are getting restored
Name of the restored VMs
Which networks to connect with
Which Storage domain to use

The Selective Restore is always available and doesn't have any prerequirements.

The Selective Restore will automatically update the Workload to protect the created instance in the case that the original instance is no longer existing.

Using Horizon

There are 2 possibilities to start a Selective Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored

Using CLI

<snapshot_id> ➡️ ID of the snapshot to restore.
--display-name <display-name> ➡️ Optional name for the restore.
--display-description <display-description> ➡️

Inplace Restore

The Inplace Restore covers those use cases, where the VM and its Volumes are still available, but the data got corrupted or needs to a rollback for other reasons.

It allows the user to restore only the data of a selected Volume, which is part of a backup.

The Inplace Restore only works when the original VM and the original Volume are still available and connected. Trilio is checking this by the saved Object-ID.

The Inplace Restore will not create any new RHV resources. Please use one of the other restore options if new Volumes or VMs are required.

Using Horizon

There are 2 possibilities to start an Inplace Restore.

Possibility 1: From the Snapshot list

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored

Possibility 2: From the Snapshot overview

Login to Horizon
Navigate to Backups
Navigate to Workloads
Identify the workload that contains the Snapshot to be restored

Using CLI

<snapshot_id> ➡️ ID of the snapshot to restore.
--display-name <display-name> ➡️ Optional name for the restore.
--display-description <display-description> ➡️

required restore.json for CLI

The workloadmgr client CLI is using a restore.json file to define the restore parameters for the selective and the inplace restore.

An example for a selective restore of this restore.json is shown below. A detailed analysis and explanation is given afterwards.

The restore.json requires many information about the backed up resources. All required information can be gathered in the .

General required information

Before the exact details of the restore are to be provided it is necessary to provide the general metadata for the restore.

name➡️the name of the restore
description➡️the description of the restore
oneclickrestore <True/False>➡️

Selective Restore required information

The Selective Restore requires a lot of information to be able to execute the restore as desired.

Those information are divided into 3 components:

instances
restore_topology
networks_mapping

Information required in instances

This part contains all information about all instances that are part of the Snapshot to restore and how they are to be restored.

Even when VMs are not to be restored are they required inside the restore.json to allow a clean execution of the restore.

Each instance requires the following information

id ➡️ original id of the instance
include <True/False> ➡️ Set True when the instance shall be restored

All further information are only required, when the instance is part of the restore.

name ➡️ new name of the instance
availability_zone ➡️ Nova Availability Zone the instance shall be restored into. Leave empty for "Any Availability Zone"
Nics ➡️

To use the next free IP available in the set Nics to an empty list [ ]

Using an empty list for Nics combined with the Network Topology Restore, will the restore automatically restore the original IP address of the instance.

vdisks ➡️ List of all Volumes that are part of the instance. Each Volume requires the following information:
- id ➡️ Original ID of the Volume
- new_volume_type

The root disk needs to be at least as big as the root disk of the backed up instance was.

The following example describes a single instance with all values.

Information required in network topology restore or network mapping

Do not mix network topology restore together with network mapping.

To activate a network topology restore set:

To activate network mapping set:

When the network mapping is activated it is used, it is necessary to provide the mapping details, which are part of the networks_mapping block:

networks ➡️ list of snapshot_network and target_network pairs
- snapshot_network ➡️ the network backed up in the snapshot, contains the following:

Full selective restore example

Inplace Restore required information

The Inplace Restore requires less information thana selective restore. It only requires the base file with some information about the Instances and Volumes to be restored.

Information required in instances

id ➡️ ID of the instance inside the Snapshot
restore_boot_disk ➡️ Set to True if the boot disk of that VM shall be restored.

When the boot disk is at the same time a Cinder Disk, both values need to be set true.

include ➡️ Set to True if at least one Volume from this instance shall be restored
vdisks ➡️ List of disks, that are connected to the instance. Each disk contains:
- id

Network mapping information required

There are no network information required, but the field have to exist as empty value for the restore to work.