Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

3. Install the Trilio Horizon plugin into the Horizon service

How these steps look in detail is depending on the Openstack distribution Trilio is installed in. Each supported Openstack distribution has its own deployment tools. Trilio is integrating into these deployment tools to provide a native integration from the beginning to the end.

Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other openstack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

1. Efficient capture and storage of snapshots. Since our full backups only include data that is committed to storage volume and the incremental backups only include changed blocks of data since last backup, our backup processes are efficient and storages backup images efficiently on the backup media

2. Faster and reliable recovery. When your applications become complex that snap multiple VMs and storage volumes, our efficient recovery process will bring your application from zero to operational with just click of button

3. Easy migration of workloads between clouds. Trilio captures all the details of your application and hence our migration includes your entire application stack without leaving any thing for guess work.

4. Through policy and automation lower the Total Cost of Ownership. Our tenant driven backup process and automation eliminates the need for dedicated backup administrators, there by improves your total cost of ownership.

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

AWS S3 eventual consistency¶

AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of them describes how an object will reach its consistent state after an object is created/updated or deleted. None of them provides strong consistency and there is a lag time for an object to reach the consistent state. Though Trilio employed mechanisms to work around the limitations of eventual consistency of AWS S3, when an object reach its consistency state is not deterministic. There is no official statement from AWS on how long it takes for an object to reach consistent state. However read-after-write has a shorter time to reach consistency compared to other IO patterns. Our solution is designed to maximize read-after-write IO pattern. The time in which an object reaches eventual consistency also depends on the AWS region. For example, aws-standard region does not have strong consistency model compared to us-east or us-west. We suggest to use these regions when creating s3 buckets for Trilio. Though read-after-update IO pattern is hard to avoid completely, we employed ample delays in accessing objects to accommodate larger durations for objects to get into consistent state. However in rare occasions, backups may still fail and need to restarted.

Trilio Cluster¶

Trilio can be deployed as a single node or a three node cluster. It is highly recommended that Trilio is deployed as three node cluster for fault tolerance and load balancing. Starting with 3.0 release, Trilio requires additional IP for cluster and is required for both single node and three node deployments. Cluster ip a.k.a virtual ip is used for managing cluster and is used to register Trilio service endpoint in the keystone sevice catalog.

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

The following charms exist:

• ➡️ Installs and manages Trilio Controller services.

• ➡️Installs and manages the Trilio Datamover API service.

• ➡️

The documentation of the charms can be found here:

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

The recommended size of the VM for the Trilio Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

2.3. Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Each Openstack distribution comes with a set of supported operating systems. Please check the to see, which Openstack Distribution is supported with which Operating System.

Additional it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

Trilio 4.0 GA

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reinitialize"

• Verify that you want to reinitialize the Trilio

To change the Trilio GUI password do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reset Password"

• Set the new Trilio password

Like all Openstack services is Trilio using a technical openstack user.

The password for this user can be changed through the Trilio web gui.

To change it use the Service Password option on the main menu.

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

[root@TVM1 ~]# source /home/stack/myansible/bin/activate
(myansible) [root@TVM1 ~]# cd /home/stack/myansible/lib/python3.6/site-packages/tvault_configurator
(myansible) [root@TVM1 tvault_configurator]# python recreate_conf.py
(myansible) [root@TVM1 tvault_configurator]# systemctl restart tvault-config

The dashboard login will be reset to:

Username: admin
Password: password

You may want to decommission Trilio as a service for your OpenStack. In this approach, you may not have the need to keep backups that were created and you would like to do complete uninstall of the product.

1. Delete all instances of Trilio.

2. On nova controller node, uninstall tvault-contego-api module.

3. On each nova compute node, uninstall tvault-contego data mover.

4. Uninstall tvault-horizon-plugin from the horizon dashboard node

5. Login as Admin to horizon dashboard and delete triliovault user.

6. You can also clear all the contents from the NFS share or S3 bucket that is configured as backup target for Trilio.

Uninstall Trilio components in RHOSP16

To uninstall the Trilio components, you need modify the deploy command and update the cloud using the new command.

Remove the trilio_env_osp16.yaml and roles_data.yaml from overcloud deploy command and then run it. After the changes the overcloud deployment command will look like this:

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the guide afterwards.

Trilio allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.

Trilio ships with a ubuntu based file manager image. This image includes a web based file manager application that helps with browsing and download files when snapshot is mounted. Before you create a File Manager instance, File Manager (a.k.a Recovery Manager) image needs to be uploaded to the Glance. The cloud administrator can upload the image to Glance and mark the image public so that every tenant will have access to the image. File Manager image includes a qemu guest agent. Trilio data mover communicates with the qemu agent to map backup images and discover file systems in the backup images. Since the file manager image includes qemu guest agent, the image should be uploaded to glance with property hw_qemu_guest_agent=yes so Nova will create a qmp channel when creating an instance from this image. Trilio horizon plugin uses a special property tvault_recovery_manager, when set to yes it will filter out instances that don’t have this property. If a tenant has a huge number of instances, setting this property on the image will help Trilio horizon plug-in present only the instances whose image has this property. In this release we only support launching File Manager instance with virtio bus so please set hw_disk_bus to virtio. Use the following command to upload the image to Glance and mark it as public image and set the property.

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

Trilio is using the which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

The upgrade process contains upgrading the Trilio appliance and the Openstack components and is dependent on the underlying operating system.

Trilio release 4.0 introduces new features and capabilities including

• Openstack Train support

• Trilio Openstack Quota integration

• Openstack CLI integration

• CLI restore of networks only

• CLI restore of security groups only

• Disk integrity check after Snapshots

• Performance Enhancements for big environments

• Enablement of rolling upgrades

Trilio 4.0.92 is the GA release of Trilio 4.0 Trilio 4.0.115 is the first patch release of Trilio 4.0.

Openstack Train support

Trilio 4.0 continues to enable Openstack versions and Distributions, allowing Trilio customers to stay up to date with Openstack releases.

Trilio 4.0 introduces full support for Openstack Train over the four supported Distributions. In addition, does Trilio 4.0 of course support longterm releases, giving Openstack users of those releases the possibility to use the latest Trilio functionalities.

Trilio Openstack Quota integration

Trilio for Openstack integrates natively with Openstack and we are happy to announce, that Trilio made another important step.

With the possibility to create Project quotas for Workloads, VMs and Storage usage can Openstack administrators now clearly control how Trilio is used by any tenant.

Combining the Quota feature with the already available policies enables Openstack administrators to define exactly what an Openstack Project can do or can't do with Trilio.

Openstack CLI integration

Trilio has been using its very own Python based CLI client, the workloadmgr client.

This client has followed the Openstack standards for CLI clients and is now again following the established standards, which integrated all Python Openstack clients into a single Openstack Python CLI client.

Trilio 4.0 enables the usage of the Openstack client for the most common commands required during the daily work with Trilio.

CLI restore of networks and security groups only

Trilio restore possibilities allowed to restore any VM and its attached resources as required. Still it was always necessary to restore the complete VM to get the networks or security groups tied to that VM or backup.

Trilio 4.0 introduces the possibility to restore only the networks or security groups directly from the CLI.

Disk integrity check after a backup

Trilio always protected the integrity of any Snapshots. However, when Trilio identified a failure in the integrity did the backup job fail and not only was the Snapshot with a failed integrity lost but the following backups as well.

To overcome this limitation did Trilio enhance the already available disk integrity check and implemented it at the end of each Snapshot. When it is detected that a Snapshot disk integrity is not provided, will the next Snapshot automatically be a full Snapshot.

That way ensuring that future backups are again having a reliable disk integrity.

Performance enhancements for big environments

With the success of Trilio around the world was it only a question of time until Trilio would be tested against a real big environment with thousands of API calls per second.

We are happy to say that our solution did keep up to the task, but it still showed potential to improve even more. And that improvement comes with a small change on the Trilio Appliance, which allows the load-balancing of incoming API calls against a complete Trilio cluster, instead of a master node.

Enablement of rolling upgrades and updates

Trilio has been part of some Openstack environments for years, going through the process of updating and upgrading together with the environment and stand alone.

This upgrade process so far was always a complete uninstallation and reinstallation of the Trilio solution, which was very time consuming and frustrating for Openstack Administrators.

Trilio 4.0 changed the way the Trilio Appliance is build to allow to update and upgrade of the complete Trilio solution.

The exact process will be announced together with the first patch release of Trilio 4.0.

Known Issues

This release contains the following known issues which are tracked for a future update.

Workloadmgr Quota feature does not support RHOSP13 & Canonical Queens Horizon

The workloadmgr Quota feature is still fully supported through CLI.

[Openstack Ansible] snapshot mount failed with error 'Permission denied'

Observation:

• It has been observed that on non-kolla, non-rhosp setups, such as openstack ansible, nova user id is not same as we consider as default(162).

• Another observation is that, the id which is assigned to nova, was conflicting with id of system user in TVM, this created a situation where we had to redeploy Openstack.

Workaround:

• Update permissions of /var/triliovault-mounts to 755

Snapshots are getting stuck when wlm-workloads service stops during a backup process

Observation:

• When wlm-workloads service gets stopped on the node with an active snapshot process does the process get stuck without erroring.

Workaround:

• Restart wlm-workloads service on Trilio cluster, snapshot will throw error.

[Volume Backup Exclude] Excluded Ceph Volume after restore not mountable or formatable

Observation:

• VM Volumes stored on Ceph are successfully excluded from backup if desired

• Restore does create empty Ceph Volume

• created empty Ceph Volume is not attachable or formatable

Restored VMs have blank metadata config_drive attached

Observation:

• For every restore will the metadata config_drive be set as blank value

• No impact on restored VMs known

Workaround

• delete metadata config_drive

• or set desired value

Scheduler trust not getting imported together with workload

Observation:

• Workload gets imported successfully

• original User and Project still exist (ownership stays same)

• All schedulers trusts are defined as disabled

Workaround:

• Edit any (just 1) workload, modify the scheduler policy and submit

• trust will be recreated

TVM reconfig fails when adding new Trilio VM to the cluster

Observation:

• TVault re-configuration while adding nodes to existing TVM cluster fails at "Configuring Trilio Cluster"

• Reason is that the prev mysql password was not working and mysql root access has be reset.

Workaround:

• remove /root/.my.cnf file on already configured TVM and reconfigure it

Database does not sync after Trilio cluster gets new nodes

Observation:

• After TVault re-configuration post addition of 2 more nodes to existing TVM cluster ("import workloads" was not seleted), the databases do not sync against already existing TVM.

• It is expected that while adding the 2 new nodes, the db on node1 should get synced up with 2 new nodes and the existing workloads should be available post the reconfig on the new 3 node TVM cluster.

Workaround:

• Run workload import from CLI

Workload scheduler gets disabled after primary node goes down

Observation:

• Deactivating the primary node of a TVM cluster deactivates scheduler of associated workloads.

Workaround:

• restart wlm-cron service on new primary node

[exclude_boot_disk] Data on boot disk gets backed up despite exclusion

Observation:

• VM was set with metadata exclude_boot_disk_from_backup set to true

• Restored instance showed, that data was backed up and restored

Post workload reassignment, scheduler trust gets disabled even if Established before reassignment

Observation:

• Workload with scheduler trust established gets reassigned to new project

• scheduler trust is getting disabled

Workaround:

• Edit the workload to enable scheduler

Additional parameters required in rc file to run wlm commands from openstack cli

Observation:

• using workloadmgr commands with Openstack CLI still requires OS_TENANT_ID and OS_USER_DOMAIN_ID

Workaround:

• edit rc file and add required fields

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log

The next important log is the workloadmgr-api.log, which contains all logs about API calls received by the Trilio Cluster. It can be found at:

/var/log/workloadmgr/workloadmgr-api.log

The log for the remaining service is the workloadmgr-scheduler.log, which contains all logs about the internal job scheduling in the Trilio Cluster.

/var/log/workloadmgr/workloadmgr-scheulder.log

On the Controller node (nova-api extension)

The logs for the nova-api extension can be found here:

/var/log/nova/nova-api.log

On the Compute nodes (datamover)

The logs for the datamover can be found here:

/var/log/nova/tvault-contego.log

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none

vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:[email protected]:5672,openstack:[email protected]:5672,openstack:[email protected]:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi

[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .volumes.keyring
ceph_dir = /etc/ceph/directory1/,/etc/ceph/directory2/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper

[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster.

It shows for each Trilio Appliance the Status of the following Trilio services:

• wlm-workloads

• wlm-scheduler

• wlm-api

To give administrators an overview of the HA status, does the dashboard also show the service status for:

• Pacemaker

• RabbitMQ

• MySQL Galeera Cluster

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio cluster

The Trilio Cluster is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio cluster also responsible to gather the metadata about the backed up VMs and networks from the Openstack environment. It is sending API calls towards the Openstack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received does the Trilio Cluster write it as json files on the Backup Target.

The Trilio cluster is also sending the Cinder Snapshot command.

During a restore process

During restore process is the Trilio cluster reading the VM metadata from its Database and is using the metadata to create the Shell for the restore. It is sending API calls to the Openstack environment to create the necessary resources.

dmapi

The dmapi service is the connector between the Trilio cluster and the datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so is the dmapi service connecting to the nova api database requesting the compute hose of a provided VM.

Once the compute host has been identified is the dmapi forwarding the command from the Trilio Cluster to the datamover running on the identified compute host.

datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover is controlling the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during creation of Workloads, backups or restores.

Openstack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The Openstack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

Change the nova user id on the Trilio Nodes

In Kolla, 'nova' user id on nova-compute docker container is set to '42436'. The 'nova' user id on the Trilio nodes need to be set the same. Do the following steps on all compute nodes:

1. Download the shell script that will change the user id

2. Assign executable permissions

3. Execute the script

4. Verify that 'nova' user and group id has changed to '42436'

Deploy Trilio Datamover API on all Openstack controller nodes

Trilio Datamover Api container should be deployed on all nodes where nova_api container is running. In standard deployment, we can call these nodes as OpenStack controller nodes.

Pull TrilioVaullt Datamover API container image

The very first step is to pull the container image from docker.io.

Login to docker and pull the Trilio Datamover API container.

Create the Trilio Datamover API configuration

In this part of the process is the configuration file for the Trilio Datamover API created.

The following steps need to be done:

1. Create config directory

2. get default config file dmapi.conf

3. edit dmapi.conf

Create config directory for trilio-datamover-api service

Get the trilio-datamover-api default config file

Edit dmapi.conf

The dmapi.conf located in /etc/kolla/trilio-datamover-api/ needs to be edited to adjust to the Openstack environment.

An example dmapi.conf can be seen here:

Copy nova.conf of nova-api service to trilio-datamover-api directory

Create trilio-datamover-api log directory

Add trilio-datamover-api to haproxy.cfg

Run trilio-datamover-api container

Now the trilio-datamover-api container can be deployed and started.

Verify deployment of trilio-datamover-api container

To verify the deployment was successful check the container status using docker ps.

Deploy Trilio Datamover container on all Compute nodes

Trilio Datamover container should be deployed on all nodes where nova_compute container is running. In standard deployment, we can call these nodes as openstack compute nodes.

Pull Trilio Datamover container

The very first step is to pull the container image from docker.io.

Login to docker and pull the Trilio Datamover API container.

Create the Trilio Datamover configuration

In this part of the process is the configuration file for the Trilio Datamover created.

The following steps need to be done:

1. Create config directory

2. copy nova.conf to config directory

3. get default config file tvault-contego.conf

Create service config directory for trilio-datamover service

Copy nova.conf of nova-compute service to trilio-datamover config directory

Get the trilio-datamover default config file

Edit tvault-contego.conf

Edit /etc/kolla/trilio-datamover/tvault-contego.conf config file to provide NFS/S3 details as per backup storage selected.

Create trilio-datamover log directory

If Ceph is used for Nova/Cinder Storage

If ceph is getting used for cinder/nova, the correct permissions for ceph.conf and keyrings files need to be assigned. The trilio_datamover container will be using ceph.conf and keyring files with the 'nova' user.

If nova/cinder backend is ceph, you need to add ceph user and keyring details to /etc/kolla/trilio-datamover/tvault-contego.conf file. Add the following sections to the tvault-contego.conf file. In the provided example is ceph's 'cinder' user configured to use for trilio read/write operations.

Run trilio-datamover container

Verify deployment of trilio-datamover container

To verify the deployment was successful check the container status using docker ps.

Installing Trilio Horizon plugin

Trilio Horizon plugin needs to be installed inside the OpenStack horizon container. Once installed, the Trilio dashboard will be visible in OpenStack Horizon.

The following steps need to be done:

1. Download installation shell script

2. Run the shell script

3. Edit Horizon settings

Download the installation shell script

To download the shell script directly into the Horizon container do:

Run the shell script and restart Horizon container

Run the shell script and restart horizon container. This will restart apache service, which may enforce a log out of the container.

Edit Horizon settings

The following line needs to be aded in 'local_settings' of Openstack's Horizon file to enable workloadmanager quota feature in the Horizon dashboard.

Restart the Horizon container

To enable the done changes restart the Horizon container

Known issues for Horizon plugin installation

If OpenStack is based on 'Centos' platform

Grafana yum repository has an issue on the latest horizon containers of OpenStack (not Trilio). To confirm the issue, you can just run yum repolist, it will fail. Use the following command to disable the grafana repository.

If Openstack is based on Ubuntu platform

Trilio horizon install script will ask for horizon's openstack_dashboard directory path if it's not at the default location - /usr/shar/openstack-dashboard

For train ubuntu bionic, it's : /var/lib/kolla/venv/lib/python2.7/site-packages

If Trilio Horizon tabs are not accessible but Openstack Horizon works

If Trilio Horizon tabs are not accessible but OpenStack Horizon is working fine, make sure that endpoints for service 'TrilioVaultWLM' are created correctly. The root cause of this issue is typically, that SSL is enabled on all three endpoint types of 'TrilioVaultWLM' service.

If SSL is enable only on public 'keystone' service endpoints, then create 'TrilioVaultWLM' service endpoints in the same fashion. Endpoints for service 'TrilioVaultWLM' get created during step. If these endpoints need to be edited reconfigure the Trilio.

Enabling Snapshout Mount

To make 'snapshot mount' functionality work, the cloud administrator needs to complete the following steps.

1. Identify backup target mount point on Trilio VM

2. install nfs-common on nova_compute and nova_libvirt containers

3. Mount backup target nfs share on nova_compute and nova_libvirt containers

Identify backup target mount point in Trilio VM

The following command will provide the active mountpoint on the Trilio VM

This example gives the following information:

Backup target is NFS share: 192.168.1.33:/mnt/tvault Mountpoint is: /var/triliovault-mounts/MTkyLjE2OC4xLjMzOi9tbnQvdHZhdWx0

Install nfs-common on nova_compute and nova_libvirt containers

It is necessary to install nfs-common package on both nova_compute and nova_libvirt containers.

Mount backup target nfs share on nova_compute and nova_libvirt containers

Mount the backup target nfs share on 'nova_compute' and 'nova_libvirt' container at exactly same mount point as done on triliovault VM.

Troubleshoot installation

If any triliovault container is stuck in restarting state the following logs can be checked.

Possible issues for trilio-datamover container failure are for example NFS mount issues or S3 credentials might be wrong. If it's Amazon S3, then network connectivity between compute node and AWS s3 is needed. The docker logs will clearly tell the exact error.

If the above logs do not help OR If containers running well but, backups fail, following service logs will help:

If the Trilio Horizon tabs are not visible on Openstack, verify the following:

• Make sure trilio horizon plugin is installed on OpenStack horizon container

• Trilio configuration step needs to be completed to see the triliovault dashboard on OpenStack

• Make sure correct openstack_dashboard directory got provided and the triliovault horizon plugin files got successfully copied there.

Generic Pre-requisites

• Please ensure to complete the upgrade of all the TVault components on Openstack controller & compute nodes before starting the rolling upgrade of TVM.

• The mentioned gemfury repository should be accessible from TVault VM.

• Please ensure the following points before starting the upgrade process:

  • No snapshot OR restore to be running.

  • Global job-scheduler should be disabled.

• Take a backup of the conf files on all TVM nodes.

• Activate the virtual environment on all TVM nodes.

• Export new TVault version PYPI url:

Upgrade s3fuse/tvault-object-store

• Run the following command on all TVM nodes to upgrade s3fuse and its dependent packages.

Upgrade tvault-configurator

• Run the following command on all TVM nodes to upgrade tvault-configurator and its dependent packages.

Upgrade workloadmgr

• Run the upgrade command on all TVM nodes to upgrade workloadmgr and its dependent packages (workloadmgrclient, contegoclient, etc)

Updating Config Parameters of specific services

• Update wlm-cron service entries

  • If Reconfigure is NOT planned, please perform following steps on all TVM nodes, else skip.

    • Update the following two parameters in wlm-cron.service systemd file (

Post Upgrade Steps

• Restore the backed-up config files

• Restart following services on all node(s) using respective commands

• Enable Global Job Scheduler

Trilio Workloads are designed to allow a Desaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

1. and Trilio for the target cloud

4. Notify users to of Workloads being available

Mount-paths

Trilio incremental Snapshots involve a backing file to the prior backup taken, which makes every Trilio incremental backup a synthetic full backup.

Trilio is using qcow2 backing files for this feature:

As can be seen in the example is the backing file an absolute path, which makes it necessary, that this path exists so the backing files can be accessed.

Trilio is using the base64 hashing algorithm for the NFS mount-paths, to allow the configuration of multiple NFS Volumes at the same time. The hash value is calculated using the provided NFS path.

When the path of the backing file is not available on the Trilio appliance and Compute nodes, will the restores of incremental backups fail.

The tested and recommended method to make the backing files available is creating the required directory path and using mount --bind to make the path available for the backups.

Running the mount --bind command will make the necessary path available until the next reboot. If it is required to have access to the path beyond a reboot is it necessary to edit the fstab.

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

• Status (Enabled/Disabled)

• Start Day/Time

• End Day

• Hrs between 2 snapshots

Disable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using CLI

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Enable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using CLI

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Modify a schedule

To modify a schedule the workload itself needs to be modified.

Please follow this procedure to .

Verify the scheduler trust is working

Trilio is using the which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Using Horizon

As a trust is bound to a specific user for each Workload does the Trilio Horizon plugin show the status of the Scheduler on the Workload list page.

Using CLI

• <workload_id> ➡️ ID of the workload to validate

Each Trilio Workload has a dedicated owner. The ownership of a Workload is defined by:

• Openstack User - The Openstack User-ID is assigned to a Workload

• Openstack Project - The Openstack Project-ID is assigned to a Workload

To use the workloadmgr CLI tool on the Trilio appliance it is only necessary to activate the virtual environment of the workloadmgr

source /home/stack/myansible/bin/activate

Trilio’s tenant driven backup service gives tenants control over backup policies. However, sometimes it may be too much control to tenants and the cloud admins may want to limit what policies are allowed by tenants. For example, a tenant may become overzealous and only uses full backups every 1 hr interval. If every tenant were to pursue this backup policy, it puts a severe strain on cloud infrastructure. Instead, if cloud admin can define predefined backup policies and each tenant is only limited to those policies then cloud administrators can exert better control over backup service.

Workload policy is similar to nova flavor where a tenant cannot create arbitrary instances. Instead, each tenant is only allowed to use the nova flavors published by the admin.

List and showing available Workload policies

Using Horizon

To see all available Workload policies in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

The following information are shown in the policy tab for each available policy:

• Creation time

• name

• description

Using CLI

• <policy_id>➡️ Id of the policy to show

Create a policy

Using Horizon

To create a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

Using CLI

• --policy-fields <key=key-name> ➡️ Specify following key value pairs for policy fields Specify option multiple times to include multiple keys. 'interval' : '1 hr' 'retention_policy_type' : 'Number of Snapshots to Keep' or 'Number of days to retain Snapshots' 'retention_policy_value' : '30' 'fullbackup_interval' : '-1' (Enter Number of incremental snapshots to take Full Backup between 1 to 999, '-1' for 'NEVER' and '0' for 'ALWAYS')For example --policy-fields interval='1 hr' --policy-fields retention_policy_type='Number of Snapshots to Keep '--policy-fields retention_policy_value='30' --policy- fields fullbackup_interval='2'

• --display-description <display_description> ➡️ Optional policy description. (Default=No description)

Edit a policy

Using Horizon

To edit a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

Using CLI

• --display-name <display-name>➡️Name of the policy

• --display-description <display_description> ➡️ Optional policy description. (Default=No description)

Assign/Remove a policy

Using Horizon

To assign or remove a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

Using CLI

• --add_project <project_id> ➡️ ID of the project to assign policy to. Use multiple times to assign multiple projects.

• --remove_project <project_id> ➡️ ID of the project to remove policy from. Use multiple times to remove multiple projects.

Delete a policy

Using Horizon

To delete a policy in Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on Admin Tab.

3. Navigate to Backups-Admin

Using CLI

• <policy_id> ➡️ID of the policy to be deleted

Definition

Trilio allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.

Trilio ships with a ubuntu based file manager image. This image includes a web based file manager application that helps with browsing and download files when snapshot is mounted. Before you create a File Manager instance, File Manager (a.k.a Recovery Manager) image needs to be uploaded to the Glance.

Spin up a File Recovery Manager

To be able to mount a Snapshot a File Recovery Manager needs to be available in the tenant.

The File Recovery Manager is a normal Openstack instance, which requires at least 40GB boot disk.

In addition are the following metadata entries required upon boot:

• hw_qemu_guest_agent=yes

• tvault-recovery-manager=yes

It is recommended to provide the File Recovery Manager instance with a Floating IP and appropriate Security Groups. Appropriate would be to restrict access to ports 80, 443, and 22.

Mounting a Snapshot

Mounting a Snapshot to a File Recovery Manager provides read access to all data that is located on the in the mounted Snapshot.

Using Horizon

There are 2 possibilities to mount a Snapshot in Horizon.

Through the Snapshot list

To mount a Snapshot through the Snapshot list follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Through the File Search results

To mount a Snapshot through the File Search results follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using CLI

• <snapshot_id> ➡️ ID of the Snapshot to be mounted

• <mount_vm_id> ➡️ ID of the File Recovery Manager instance to mount the Snapshot to.

Accessing the File Recovery Manager

The File Recovery Manager is a normal Openstack instance running a webserver. It can be accessed through

• the Openstack VNC console (requires password access)

• SSH

• HTTP on port 80

The Webserver provides a File Commander application, which will show the mounted Snapshot and provides the possibility to navigate through it and download Files as necessary.

When using SSH or the Openstack VNC console can the mounted Snapshot be found under:

/home/ubuntu/tvault-mounts/mounts/

Identifying mounted Snapshots

Sometimes a Snapshot is mounted for a longer time and it needs to be identified, which Snapshots are mounted.

Using Horizon

There are 2 possibilities to identify mounted Snapshots inside Horizon.

From the File Recovery Manager instance Metadata

1. Login to Horizon

2. Navigate to Compute

3. Navigate to Instances

The mounted_snapshot_url contains the Snapshot ID of the Snapshot that has been mounted last.

From the Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using CLI

• --workloadid <workloadid> ➡️ Restrict the list to snapshots in the provided workload

Unmounting a Snapshot

Once a mounted Snapshot is no longer needed it is possible and recommended to unmount the snapshot.

Using Horizon

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using the CLI

• <snapshot_id> ➡️ ID of the snapshot to unmount.

Start File Search

POST https://$(tvm_address):8780/v1/$(tenant_id)/search

Starts a File Search with the given parameters

Clone updated triliovault-cfg-scripts github repo

Note: Run all the commands with 'stack' user.

• Clone updated repo\

• If your backup target is Ceph S3 with self signed certs If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/puppet/trilio/files'

• Upload triliovault puppet module

• Copy old trilio env file to new repository

Pull latest triliovault container images and update triliovault environment yaml file

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL is ''. Following are the triliovault container pull urls.

Trilio container urls for RHOSP13:

Trilio container urls for RHOSP16:

There are three registry methods available in RedHat Openstack Platform.

1. Remote Registry 2. Local Registry 3. Satellite Server

Identify which method you are using. Below we have explained all three methods to pull and configure trilioVault's container images for overcloud deployment.

Remote Registry:

If you are using 'Remote Registry' method follow this section. You don't need to pull anything. You just need to populate following container urls in trilio env yaml.

• If it’s RHOSP13, populate 'trilio_env.yaml' file with triliovault container urls. Changes looks like following.\

• If it’s RHOSP16, populate 'trilio_env_osp16.yaml' file with triliovault container urls. Changes looks like following.\

Registry on Undercloud:

If you are using 'local registry' on undercloud, follow this section.

Note: Run following script from undercloud node with 'stack' user

• If it’s RHOSP13, run following script. Script, pulls the triliovault containers and updates the triliovault environment file with urls.

• If it’s RHOSP13, user needs to set horizon image url to triliovault horizon image url in overcloud_images.yaml. Changes looks like following.

• If it’s RHOS16, run following script. Script, pulls the triliovault containers and updates the triliovault environment file with urls.\

## Above script pushes trilio container images to undercloud registry and sets correct trilio images urls in trilio_env_osp16.yaml. Verify the changes using following command.

Registry on Redhat Satellite Server

If you are using 'Satellite Server' for container registry, follow this section.

Following are the pull urls for Trilio containers. You need to pull these containers on Redhat Satellite that you already have.

Trilio container urls for RHOSP13:

Trilio container urls for RHOSP16:

• Pull above containers as per your RHOSP release and push to redhat satellite server. For this, you can follow process same as other openstack containers.

• After this, if it’s RHOSP13, populate 'trilio_env.yaml' file with the urls. Changes looks like following.

• If it’s RHOSP13, user needs to set horizon image url to triliovault horizon image url in overcloud_images.yaml. Changes looks like following.

• If it’s RHOSP16, populate 'trilio_env_osp16.yaml' file with the triliovault container urls. Changes looks like following.

Update overcloud - This step will update the triliovault containers on overcloud

• Run your overcloud deploy command again. It will update the triliovault containers on overcloud.

• Command looks like following. \

[Optional] If you want to verify that triliovault got upgraded successfully, use following steps.

1. Verify triliovault datamover api container got updated on nova-api nodes

2. Verify triliovault datamover container got updated on nova-compute nodes

3. Verify triliovault horizon got updated on horizon nodes

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

• Public Endpoints

• Internal Endpoints

• Admin Endpoints

Each of these endpoint types is designed for a specific purpose. Public endpoints are meant to be used by the Openstack end-users to work with Openstack. Internal endpoints are meant to be used by the Openstack services to communicate with each other. Admin endpoints are meant to be used by Openstack administrators.

Out of those 3 endpoint types does only the admin endpoint sometimes contain APIs which are not available on any other endpoint type.

To learn more about Openstack endpoints please visit the official Openstack documentation.

Openstack endpoints required by Trilio

Trilio is communicating with all services of Openstack on a defined endpoint type. Which endpoint type Trilio is using to communicate with Openstack is decided during the configuration of the Trilio appliance.

The following network requirement can be identified this way:

• Trilio appliance needs access to the Keystone admin endpoint on the admin endpoint network

• Trilio appliance needs access to all endpoints of one type

Recommendation: Provide access to all Openstack Endpoint types

Trilio is recommending providing full access to all Openstack endpoints to the Trilio appliance to follow the Openstack standards and best practices.

Trilio is generating its own endpoints as well. These endpoints are pointing towards the Trilio Appliance directly. This means that using those endpoints will not send the API calls towards the Openstack Controller nodes first, but directly to the Trilio VM.

Following the Openstack standards and best practices, it is therefore recommended to put the Trilio endpoints on the same networks as the already existing Openstack endpoints. This allows to extend the purpose of each endpoint type to the Trilio service:

• The public endpoint to be used by Openstack users when using Trilio CLI or API

• The internal endpoint to communicate with the Openstack services

• The admin endpoint to use the required admin only APIs of Keystone

Backup target access required by Trilio

The Trilio solution is using backup target storage to securely place the backup data. Trilio is dividing its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio appliance through communicating with the Openstack Endpoints. All metadata that is stored together with a backup is written by the Trilio Appliance to the backup target in the json format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service is reading the Volume Data from the Cinder or Nova storage and transferring this data as qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• The Trilio appliance needs access to the backup target

• Every compute node needs access to the backup target

Example of a typical Trilio network integration

Most Trilio customers are following the Openstack standards and best practices to have the public, internal, and admin endpoints on separate networks. They also typically don't have any network yet, which can access the desired backup target.

The starting network configuration typically looks like this:

Following the Openstack standards and Trilio's recommendation will the Trilio Appliance be placed on all those 3 networks. Further is the access to the backup target required by Trilio Appliance and Compute nodes. Here done by adding a 4th network.

The resulting network configuration would look like this:

It is of course possible to combine networks as necessary. As long as the required network access is available will Trilio work.

Other examples of Trilio network integrations

Each Openstack installation is different and so is the network configuration. There are endless possibilities of how to configure the Openstack network and how to implement the Trilio appliance into this network. The following three examples have been seen in production:

The first example is from a manufacturing company, which wanted to split the networks by function and decided to put the Trilio backup target on the internal network as the backup and recovery function was identified as an Openstack internal solution. This example looks complex but integrates Trilio just as recommended.

The second example is from a financial institute that wanted to be sure that the Openstack Users have no direct uncontrolled network access to the Openstack infrastructure. Following this example requires additional work as the internal HA-Proxy needs to be configured to correctly translates the API calls towards the Trilio appliance.

The third example is from a service company that was forced to treat Trilio as an external 3rd party solution, as we require a virtual machine running outside of Openstack. This kind of network configuration requires good planning on the Trilio endpoints and firewall rules.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Creating the cloud-init image

The Trilio appliance is utilizing cloud-init to provide the initial network and user configuration.

Cloud-init is reading it's information either from a metadata server or from a provided cd image. Trilio is utilizing the cd image.

Needed tools

To create the cloud-init image it is required to have genisoimage available.

Providing the Metadata

Cloud-init is using two files for it's metadata.

The first file is called meta-data and contains the information about the network configuration. Below is an example of this file.

The second file is called user-data and contains little scripts and information to set up for example the user passwords. Below is an example of this file.

creating the image file

The image is getting created using genisoimage follwing this general command:

genisoimage -output <name>.iso -volid cidata -joliet -rock </path/user-data> </path/meta-data>

An example of this command is shown below.

Spining up the Trilio appliance

After the cloud-init image has been created the TriloVault appliance can be spun up on the desired KVM server.

Extract the Trilio QCOW2 tar file using the following command :

See below an example command, how to spin up the Trilio appliance using virsh and the created iso image.

Uninstalling cloud-init after first boot

Once the Trilio appliance is up and running with it's initial configuration is it recommended to uninstall cloud-init.

To uninstall cloud-init, follow the example below.

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Installing Trilio Datamover-API container

The installation of the Datamover-API, short dmapi, requires to create a new container, in which all necessary packages and the Trilio dmapi code are loaded.

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Trilio enables Openstack administrators to set Project Quotas against the usage of Trilio.

The following Quotas can be set:

• Number of Workloads a Project is allowed to have

• Number of VMs a Project is allowed to protect

Every Trilio appliance provides the possibility to download the following elements:

• Shell-Scripts usable to install Trilio components

• Workloadmgr Python client