Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other openstack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

The upgrade process contains upgrading the Trilio appliance and the Openstack components and is dependent on the underlying operating system.

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

The dashboard login will be reset to:

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

To change the Trilio GUI password do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

To use the workloadmgr CLI tool on the Trilio appliance it is only necessary to activate the virtual environment of the workloadmgr

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the guide afterwards.

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log

Like all Openstack services is Trilio using a technical openstack user.

The password for this user can be changed through the Trilio web gui.

To change it use the Service Password option on the main menu.

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

4. Trilio horizon plugin is installed as an add on to horizon servers. This module is installed on every server that runs horizon service.

System requirements Trilio Appliance

The Trilio Appliance gets delivered as a qcow2 image, which gets attached to a virtual machine.

Trilio supports KVM-based hypervisors on x86 architectures, with the following properties:

The recommended size of the VM for the Trilio Appliance is:

The qcow2 image itself defines the 40GB disk size of the VM.

2.3. Software Requirements

In addition to the Trilio Appliance does Trilio contain components, which are installed directly into the Openstack itself.

Each Openstack distribution comes with a set of supported operating systems. Please check the to see, which Openstack Distribution is supported with which Operating System.

Additional it is necessary to have the nfs-common packages installed on the compute nodes in case of using the NFS protocol for the backup target.

Trilio is using the Openstack Keystone Trust system which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

Trusts can only be worked with via CLI

List all trusts

Show a trust

• <trust_id> ➡️ ID of the trust to show

Create a trust

• <role_name> ➡️Name of the role that trust is created for

• --is_cloud_trust {True,False} ➡️ Set to true if creating cloud admin trust. While creating cloud trust use same user and tenant which used to configure Trilio and keep the role admin.

Delete a trust

• <trust_id> ➡️ ID of the trust to be deleted

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

Example E-Mails

The following screenshots show example E-mails send by Trilio.

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Trilio Workloads are designed to allow a Desaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Every Trilio appliance provides the possibility to download the following elements:

• Shell-Scripts usable to install Trilio components

• Workloadmgr Python client

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster.

It shows for each Trilio Appliance the Status of the following Trilio services:

• wlm-workloads

• wlm-scheduler

You may want to decommission Trilio as a service for your OpenStack. In this approach, you may not have the need to keep backups that were created and you would like to do complete uninstall of the product.

1. Delete all instances of Trilio.

2. On nova controller node, uninstall tvault-contego-api module.

Trilio allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.

Trilio ships with a ubuntu based file manager image. This image includes a web based file manager application that helps with browsing and download files when snapshot is mounted. Before you create a File Manager instance, File Manager (a.k.a Recovery Manager) image needs to be uploaded to the Glance. The cloud administrator can upload the image to Glance and mark the image public so that every tenant will have access to the image. File Manager image includes a qemu guest agent. Trilio data mover communicates with the qemu agent to map backup images and discover file systems in the backup images. Since the file manager image includes qemu guest agent, the image should be uploaded to glance with property hw_qemu_guest_agent=yes so Nova will create a qmp channel when creating an instance from this image. Trilio horizon plugin uses a special property tvault_recovery_manager, when set to yes it will filter out instances that don’t have this property. If a tenant has a huge number of instances, setting this property on the image will help Trilio horizon plug-in present only the instances whose image has this property. In this release we only support launching File Manager instance with virtio bus so please set hw_disk_bus to virtio. Use the following command to upload the image to Glance and mark it as public image and set the property.

Trilio release 4.0 introduces new features and capabilities including

• Openstack Train support

• Trilio Openstack Quota integration

• Openstack CLI integration

• CLI restore of networks only

• CLI restore of security groups only

• Disk integrity check after Snapshots

• Performance Enhancements for big environments

• Enablement of rolling upgrades

Trilio 4.0.92 is the GA release of Trilio 4.0 Trilio 4.0.115 is the first patch release of Trilio 4.0.

Openstack Train support

Trilio 4.0 continues to enable Openstack versions and Distributions, allowing Trilio customers to stay up to date with Openstack releases.

Trilio 4.0 introduces full support for Openstack Train over the four supported Distributions. In addition, does Trilio 4.0 of course support longterm releases, giving Openstack users of those releases the possibility to use the latest Trilio functionalities.

Trilio Openstack Quota integration

Trilio for Openstack integrates natively with Openstack and we are happy to announce, that Trilio made another important step.

With the possibility to create Project quotas for Workloads, VMs and Storage usage can Openstack administrators now clearly control how Trilio is used by any tenant.

Combining the Quota feature with the already available policies enables Openstack administrators to define exactly what an Openstack Project can do or can't do with Trilio.

Openstack CLI integration

Trilio has been using its very own Python based CLI client, the workloadmgr client.

This client has followed the Openstack standards for CLI clients and is now again following the established standards, which integrated all Python Openstack clients into a single Openstack Python CLI client.

Trilio 4.0 enables the usage of the Openstack client for the most common commands required during the daily work with Trilio.

CLI restore of networks and security groups only

Trilio restore possibilities allowed to restore any VM and its attached resources as required. Still it was always necessary to restore the complete VM to get the networks or security groups tied to that VM or backup.

Trilio 4.0 introduces the possibility to restore only the networks or security groups directly from the CLI.

Disk integrity check after a backup

Trilio always protected the integrity of any Snapshots. However, when Trilio identified a failure in the integrity did the backup job fail and not only was the Snapshot with a failed integrity lost but the following backups as well.

To overcome this limitation did Trilio enhance the already available disk integrity check and implemented it at the end of each Snapshot. When it is detected that a Snapshot disk integrity is not provided, will the next Snapshot automatically be a full Snapshot.

That way ensuring that future backups are again having a reliable disk integrity.

Performance enhancements for big environments

With the success of Trilio around the world was it only a question of time until Trilio would be tested against a real big environment with thousands of API calls per second.

We are happy to say that our solution did keep up to the task, but it still showed potential to improve even more. And that improvement comes with a small change on the Trilio Appliance, which allows the load-balancing of incoming API calls against a complete Trilio cluster, instead of a master node.

Enablement of rolling upgrades and updates

Trilio has been part of some Openstack environments for years, going through the process of updating and upgrading together with the environment and stand alone.

This upgrade process so far was always a complete uninstallation and reinstallation of the Trilio solution, which was very time consuming and frustrating for Openstack Administrators.

Trilio 4.0 changed the way the Trilio Appliance is build to allow to update and upgrade of the complete Trilio solution.

The exact process will be announced together with the first patch release of Trilio 4.0.

Known Issues

This release contains the following known issues which are tracked for a future update.

Workloadmgr Quota feature does not support RHOSP13 & Canonical Queens Horizon

The workloadmgr Quota feature is still fully supported through CLI.

[Openstack Ansible] snapshot mount failed with error 'Permission denied'

Observation:

• It has been observed that on non-kolla, non-rhosp setups, such as openstack ansible, nova user id is not same as we consider as default(162).

• Another observation is that, the id which is assigned to nova, was conflicting with id of system user in TVM, this created a situation where we had to redeploy Openstack.

Workaround:

• Update permissions of /var/triliovault-mounts to 755

Snapshots are getting stuck when wlm-workloads service stops during a backup process

Observation:

• When wlm-workloads service gets stopped on the node with an active snapshot process does the process get stuck without erroring.

Workaround:

• Restart wlm-workloads service on Trilio cluster, snapshot will throw error.

[Volume Backup Exclude] Excluded Ceph Volume after restore not mountable or formatable

Observation:

• VM Volumes stored on Ceph are successfully excluded from backup if desired

• Restore does create empty Ceph Volume

• created empty Ceph Volume is not attachable or formatable

Restored VMs have blank metadata config_drive attached

Observation:

• For every restore will the metadata config_drive be set as blank value

• No impact on restored VMs known

Workaround

• delete metadata config_drive

• or set desired value

Scheduler trust not getting imported together with workload

Observation:

• Workload gets imported successfully

• original User and Project still exist (ownership stays same)

• All schedulers trusts are defined as disabled

Workaround:

• Edit any (just 1) workload, modify the scheduler policy and submit

• trust will be recreated

TVM reconfig fails when adding new Trilio VM to the cluster

Observation:

• TVault re-configuration while adding nodes to existing TVM cluster fails at "Configuring Trilio Cluster"

• Reason is that the prev mysql password was not working and mysql root access has be reset.

Workaround:

• remove /root/.my.cnf file on already configured TVM and reconfigure it

Database does not sync after Trilio cluster gets new nodes

Observation:

• After TVault re-configuration post addition of 2 more nodes to existing TVM cluster ("import workloads" was not seleted), the databases do not sync against already existing TVM.

• It is expected that while adding the 2 new nodes, the db on node1 should get synced up with 2 new nodes and the existing workloads should be available post the reconfig on the new 3 node TVM cluster.

Workaround:

• Run workload import from CLI

Workload scheduler gets disabled after primary node goes down

Observation:

• Deactivating the primary node of a TVM cluster deactivates scheduler of associated workloads.

Workaround:

• restart wlm-cron service on new primary node

[exclude_boot_disk] Data on boot disk gets backed up despite exclusion

Observation:

• VM was set with metadata exclude_boot_disk_from_backup set to true

• Restored instance showed, that data was backed up and restored

Post workload reassignment, scheduler trust gets disabled even if Established before reassignment

Observation:

• Workload with scheduler trust established gets reassigned to new project

• scheduler trust is getting disabled

Workaround:

• Edit the workload to enable scheduler

Additional parameters required in rc file to run wlm commands from openstack cli

Observation:

• using workloadmgr commands with Openstack CLI still requires OS_TENANT_ID and OS_USER_DOMAIN_ID

Workaround:

• edit rc file and add required fields

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

Generic Pre-requisites

• Please ensure to complete the upgrade of all the TVault components on Openstack controller & compute nodes before starting the rolling upgrade of TVM.

• The mentioned gemfury repository should be accessible from TVault VM.

• Please ensure the following points before starting the upgrade process:

  • No snapshot OR restore to be running.

  • Global job-scheduler should be disabled.

• Take a backup of the conf files on all TVM nodes.

• Activate the virtual environment on all TVM nodes.

• Export new TVault version PYPI url:

Upgrade s3fuse/tvault-object-store

• Run the following command on all TVM nodes to upgrade s3fuse and its dependent packages.

Upgrade tvault-configurator

• Run the following command on all TVM nodes to upgrade tvault-configurator and its dependent packages.

Upgrade workloadmgr

• Run the upgrade command on all TVM nodes to upgrade workloadmgr and its dependent packages (workloadmgrclient, contegoclient, etc)

Updating Config Parameters of specific services

• Update wlm-cron service entries

  • If Reconfigure is NOT planned, please perform following steps on all TVM nodes, else skip.

    • Update the following two parameters in wlm-cron.service systemd file (

Post Upgrade Steps

• Restore the backed-up config files

• Restart following services on all node(s) using respective commands

• Enable Global Job Scheduler

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

• Public Endpoints

• Internal Endpoints

• Admin Endpoints

Each of these endpoint types is designed for a specific purpose. Public endpoints are meant to be used by the Openstack end-users to work with Openstack. Internal endpoints are meant to be used by the Openstack services to communicate with each other. Admin endpoints are meant to be used by Openstack administrators.

Out of those 3 endpoint types does only the admin endpoint sometimes contain APIs which are not available on any other endpoint type.

To learn more about Openstack endpoints please visit the official Openstack documentation.

Openstack endpoints required by Trilio

Trilio is communicating with all services of Openstack on a defined endpoint type. Which endpoint type Trilio is using to communicate with Openstack is decided during the configuration of the Trilio appliance.

The following network requirement can be identified this way:

• Trilio appliance needs access to the Keystone admin endpoint on the admin endpoint network

• Trilio appliance needs access to all endpoints of one type

Recommendation: Provide access to all Openstack Endpoint types

Trilio is recommending providing full access to all Openstack endpoints to the Trilio appliance to follow the Openstack standards and best practices.

Trilio is generating its own endpoints as well. These endpoints are pointing towards the Trilio Appliance directly. This means that using those endpoints will not send the API calls towards the Openstack Controller nodes first, but directly to the Trilio VM.

Following the Openstack standards and best practices, it is therefore recommended to put the Trilio endpoints on the same networks as the already existing Openstack endpoints. This allows to extend the purpose of each endpoint type to the Trilio service:

• The public endpoint to be used by Openstack users when using Trilio CLI or API

• The internal endpoint to communicate with the Openstack services

• The admin endpoint to use the required admin only APIs of Keystone

Backup target access required by Trilio

The Trilio solution is using backup target storage to securely place the backup data. Trilio is dividing its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio appliance through communicating with the Openstack Endpoints. All metadata that is stored together with a backup is written by the Trilio Appliance to the backup target in the json format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service is reading the Volume Data from the Cinder or Nova storage and transferring this data as qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• The Trilio appliance needs access to the backup target

• Every compute node needs access to the backup target

Example of a typical Trilio network integration

Most Trilio customers are following the Openstack standards and best practices to have the public, internal, and admin endpoints on separate networks. They also typically don't have any network yet, which can access the desired backup target.

The starting network configuration typically looks like this:

Following the Openstack standards and Trilio's recommendation will the Trilio Appliance be placed on all those 3 networks. Further is the access to the backup target required by Trilio Appliance and Compute nodes. Here done by adding a 4th network.

The resulting network configuration would look like this:

It is of course possible to combine networks as necessary. As long as the required network access is available will Trilio work.

Other examples of Trilio network integrations

Each Openstack installation is different and so is the network configuration. There are endless possibilities of how to configure the Openstack network and how to implement the Trilio appliance into this network. The following three examples have been seen in production:

The first example is from a manufacturing company, which wanted to split the networks by function and decided to put the Trilio backup target on the internal network as the backup and recovery function was identified as an Openstack internal solution. This example looks complex but integrates Trilio just as recommended.

The second example is from a financial institute that wanted to be sure that the Openstack Users have no direct uncontrolled network access to the Openstack infrastructure. Following this example requires additional work as the internal HA-Proxy needs to be configured to correctly translates the API calls towards the Trilio appliance.

The third example is from a service company that was forced to treat Trilio as an external 3rd party solution, as we require a virtual machine running outside of Openstack. This kind of network configuration requires good planning on the Trilio endpoints and firewall rules.

Each Trilio Workload has a dedicated owner. The ownership of a Workload is defined by:

• Openstack User - The Openstack User-ID is assigned to a Workload

• Openstack Project - The Openstack Project-ID is assigned to a Workload

• Openstack Cloud - The Trilio Serviceuser-ID is assigned to a Workload

This ownership secures, that only the owners of a Workload are able to work with it.

Openstack Administrators can reassign Workloads or reimport Workloads from older Trilio installations.

Import workloads

Workload import allows to import Workloads existing on the Backup Target into the Trilio database.

To get a list of importable Workloads use the following CLI command:

• --project_id <project_id> ➡️ List workloads belongs to given project only.

To import Workloads into the Trilio database use the following CLI command:

• --workloadids <workloadid> ➡️ Specify workload ids to import only specified workloads. Repeat option for multiple workloads.

Orphaned Workloads

The definition of an orphaned Workload is from the perspective of a specific Trilio installation. Any workload that is located on the Backup Target Storage, but not known to the TrilioVualt installation is considered orphaned.

Further is to divide between Workloads that were previously owned by Projects/Users in the same cloud or are migrated from a different cloud.

The following CLI command provides the list of orphaned workloads:

• --migrate_cloud {True,False} ➡️ Set to True if you want to list workloads from other clouds as well. Default is False.

• --generate_yaml {True,False} ➡️ Set to True if want to generate output file in yaml format, which would be further used as input for workload reassign API.

Reassigning Workloads

Openstack administrators are able to reassign a Workload to a new owner. This involves the possibility to migrate a Workload from one cloud to another or between projects.

Use the following CLI command to reassign a Workload:

• --old_tenant_ids <old_tenant_id>➡️ Specify old tenant ids from which workloads need to reassign to new tenant. Specify multiple times to choose Workloads from multiple tenants.

• --new_tenant_id <new_tenant_id> ➡️ Specify new tenant id to which workloads need to reassign from old tenant. Only one target tenant can be specified.

A sample mapping file with explanations is shown below:

Trilio 4.0 GA

Trilio 4.0 SP1

Trilio enables Openstack administrators to set Project Quotas against the usage of Trilio.

The following Quotas can be set:

• Number of Workloads a Project is allowed to have

• Number of VMs a Project is allowed to protect

• Amount of Storage a Project is allowed to use on the Backup Target

Work with Workload Quotas via Horizon

Workload Quotas are managed like any other Project Quotas.

1. Login into Horizon as user with admin role

2. Navigate to Identity

3. Navigate to Projects

Work with Workload Quotas via CLI

List available Quota Types

Trilio is providing several different Quotas. The following command allows listing those.

Show Quota Type Details

The following command will show the details of a provided Quota Type.

• <quota_type_id> ➡️ID of the Quota Type to show

Create a Quota

The following command will create a Quota for a given project and set the provided value.

• <quota_type_id> ➡️ID of the Quota Type to be created

• <allowed_value>➡️ Value to set for this Quota Type

List allowed Quotas

The following command lists all Trilio Quotas set for a given project.

• <project_id>➡️ Project to list the Quotas from

Show allowed Quota

The following command shows the details about a provided allowed Quota.

• <allowed_quota_id> ➡️ID of the allowed Quota to show.

Update allowed Quota

The following command shows how to update the value of an already existing allowed Quota.

• <allowed_value>➡️ Value to set for this Quota Type

• <high_watermark>➡️ Value to set for High Watermark warnings

Delete allowed Quota

The following command will delete an allowed Quota and sets the value of the connected Quota Type back to unlimited for the affected project.

• <allowed_quota_id> ➡️ID of the allowed Quota to delete

Trilio’s tenant driven backup service gives tenants control over backup policies. However, sometimes it may be too much control to tenants and the cloud admins may want to limit what policies are allowed by tenants. For example, a tenant may become overzealous and only uses full backups every 1 hr interval. If every tenant were to pursue this backup policy, it puts a severe strain on cloud infrastructure. Instead, if cloud admin can define predefined backup policies and each tenant is only limited to those policies then cloud administrators can exert better control over backup service.

Workload policy is similar to nova flavor where a tenant cannot create arbitrary instances. Instead, each tenant is only allowed to use the nova flavors published by the admin.

List and showing available Workload policies

Installing Trilio Datamover-API container

The installation of the Datamover-API, short dmapi, requires to create a new container, in which all necessary packages and the Trilio dmapi code are loaded.

Create the dmapi container on Controller nodes

Create lxc container for hosting dmapi service on controller nodes with below commands.

Prepare the dmapi container for installation

Add nova user and required directory on container controller_dmapi.

Add required packages on container controller_dmapi.

Copy nova.conf file from nova-api container to /etc/nova directory in dmapi container. Run the below command on controller nodes:

Create a new interface with specific ip for dmapi container.

Edit /var/lib/lxc/controller_dmapi/config and add below section as per network bridge available on the controller node.

Restart the container with the below commands.

Install the dmapi service

Download and run the tvault-installation script inside the container.

The script to be executed inside dmapi container, after the following changes have been done: Comment the 2 lines below and add a line below NOVA_VERSION = 20, as nova-manage doesn't work in Ansible Openstack.

Run the script

Verify the installation and set necessary configuration

Verify in dmapi.conf domain name for the nova service user under keystone section. Check field values for project_domain_nameand user_domain_name and update those if not in keystone section

If SSL is enabled then add the following section in dmapi.conf:

Verify below entries are there in keystone policy.json file

Once verified above checks, start the dmapi service.

Install Trilio Datamover on Compute nodes

Prepare the installation

Activate the virtual environment on the compute node.

After activating the virtual environment, find out the location of compute.filters file.

Download the installation script.

Modify install script to use the same location for creating trilio filters.

Also comment the 2 lines below and add a line below NOVA_VERSION = 20, as nova-manage doesn't work in Ansible Openstack.

Install the Trilio datamover service

Run install script and if you get prompt while installing, choose the default selection.

Make sure ExecStart value look like below in /etc/systemd/system/tvault-contego.service file.

Use below commands to restart and verify the service.

Use below command and check if nfs/s3 storage is mounted or not.

Install Horizon Plugin into Horizon service

List running containers on controller nodes and login to horizon container using the below command.

Install curl package on the Horizon container if not present.

Activate virtual environment on horizon container

Download script to install horizon plugin on horizon container and run install script

Install script will ask for the dashboard folder, provide below path

Verify installation using below commands

Haproxy configuration on controller nodes

Refer to the keystone haproxy settings for dmapi haproxy.

A sample configuration is shown below.

Check the syntax of the file and restart the service.

Definition

Trilio allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.

Trilio ships with a ubuntu based file manager image. This image includes a web based file manager application that helps with browsing and download files when snapshot is mounted. Before you create a File Manager instance, File Manager (a.k.a Recovery Manager) image needs to be uploaded to the Glance.

Spin up a File Recovery Manager

To be able to mount a Snapshot a File Recovery Manager needs to be available in the tenant.

The File Recovery Manager is a normal Openstack instance, which requires at least 40GB boot disk.

In addition are the following metadata entries required upon boot:

• hw_qemu_guest_agent=yes

• tvault-recovery-manager=yes

It is recommended to provide the File Recovery Manager instance with a Floating IP and appropriate Security Groups. Appropriate would be to restrict access to ports 80, 443, and 22.

Mounting a Snapshot

Mounting a Snapshot to a File Recovery Manager provides read access to all data that is located on the in the mounted Snapshot.

Using Horizon

There are 2 possibilities to mount a Snapshot in Horizon.

Through the Snapshot list

To mount a Snapshot through the Snapshot list follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Through the File Search results

To mount a Snapshot through the File Search results follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using CLI

• <snapshot_id> ➡️ ID of the Snapshot to be mounted

• <mount_vm_id> ➡️ ID of the File Recovery Manager instance to mount the Snapshot to.

Accessing the File Recovery Manager

The File Recovery Manager is a normal Openstack instance running a webserver. It can be accessed through

• the Openstack VNC console (requires password access)

• SSH

• HTTP on port 80

The Webserver provides a File Commander application, which will show the mounted Snapshot and provides the possibility to navigate through it and download Files as necessary.

When using SSH or the Openstack VNC console can the mounted Snapshot be found under:

/home/ubuntu/tvault-mounts/mounts/

Identifying mounted Snapshots

Sometimes a Snapshot is mounted for a longer time and it needs to be identified, which Snapshots are mounted.

Using Horizon

There are 2 possibilities to identify mounted Snapshots inside Horizon.

From the File Recovery Manager instance Metadata

1. Login to Horizon

2. Navigate to Compute

3. Navigate to Instances

The mounted_snapshot_url contains the Snapshot ID of the Snapshot that has been mounted last.

From the Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using CLI

• --workloadid <workloadid> ➡️ Restrict the list to snapshots in the provided workload

Unmounting a Snapshot

Once a mounted Snapshot is no longer needed it is possible and recommended to unmount the snapshot.

Using Horizon

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Using the CLI

• <snapshot_id> ➡️ ID of the snapshot to unmount.

Start File Search

POST https://$(tvm_address):8780/v1/$(tenant_id)/search

Starts a File Search with the given parameters

Trilio configuration process is using ansible scripts. Ansible, in the last few years, has grown in popularity as a preferred configuration management tool and Trilio uses ansible play books extensively to configure the Trilio cluster. To troubleshoot Trilio configuration issues, user should have basic understanding of ansible playbook output.

Ansible modules are inherently idempotent and hence Trilio configuration can run any number of times to change or reconfigure Trilio cluster.

Once the VM is booted, point your browser (Chrome or Firefox) to Trilio node IP address.

This will bring you to the Trilio Dashboard, which contains the Trilio configurator.

The user is: admin The default password is: password

Unlike previous versios of Trilio, the current version only requires you to configure the cluster once and the Trilio dashboard provides cluster wide management capability.

Uploading the OpenStack certificate bundle

OpenStack endpoints can be configured to use TLS. In such a configuration the Trilio appliance needs to trust the certificates provided by the OpenStack endpoints.

To achieve this trust it is required to upload the OpenStack certificate bundle through the OS API certificate tab of the Trilio appliance Dashboard.

The certificate bundle is located on the controller nodes of the OpenStack installation.

The default paths for each distribution are as follows:

The uploaded certificates can be verified on the Trilio appliance at the following location.

Details needed for the Trilio Appliance

Upon login into an unconfigured Trilio Appliance, the shown page is the configurator. The configurator requires some information about the Trilio Appliance, Openstack and Backup Storage.

Trilio Cluster information

The Trilio Cluster needs to be integrated into an existing environment to be able to operate correctly. This block asks for the information about the Trilio Cluster operating details.

• Controller Nodes

  • This is the list of Trilio virtual appliance IP addresses along with their hostnames.

  • Format: comma separated list with pairs combined through '='

• Virtual IP Address

  • This is the Trilio cluster IP address which is mandatory

  • Format: IP/Subnet

• Name Server

  • List of nameservers, primarily used to resolve OpenStack service endpoints.

  • Format: comma separated list

• Domain Search Order

  • The domain the Trilio Cluster will use.

  • Format: comma separated list

Openstack Credentials informations

The Trilio appliance integrates with one RHV environment. This block asks for the information required to access and connect with the RHV Cluster.

• Keystone Admin URL

  • The Keystone admin endpoint mainly used during configuration

  • format: URL

• Administrator

  • Username of an account with the domain admin role

  • format: String

Backup Storage Configuration information

This block is requesting the necessary information about the backup target that the Trilio installation will be used to store and read backups.

The very first field in this block decides the protocol used to connect with Backup Storage, NFS or S3.

Using the NFS protocol

• NFS Export

  • Path under which the NFS Volumes to be used can be found

  • format: comma separated list of NFS Volumes paths

Using the S3 protocol

• S3 Compatible

  • Switch between Amazon and Ceph

  • format: predefined list

• Access Key

  • Access Key necessary to login into the S3 storage

  • format: access key

Workload Import

If you are upgrading either from older versions of Trilio or reinstalling the appliance for maintenance reasons, please check this box during the configuration. Trilio is a stateless appliance and all the state is securely saved on the NFS/S3 storage. So, during the upgrade process, the user will need to import all backup job records back from the NFS/S3 storage to the appliance MySQL database. By checking this box, the configuration automatically imports all backup records.

Advanced settings

At the end of the configuratorformulars is the option to activate the advanced settings. Activating this option does provide the possibility to configure the Keystone endpoints used for the Datamover API and Trilio.

Starting the configurator

Once all entries have been set and all validations are error free the configurator can be started.

• Click Finish

• Reconfirm in the pop-up that you want to start the configuration

• Wait for the configurator to finish

The configurator is using Ansible and a few Trilio internal API calls. After each configuration block or after the configurator finished it is possible to visit the ansible output.

At the end of a successful configuration does the configurator forward to the set VIP.

Generic Pre-requisites

1. Please ensure the following points before starting the upgrade process:

   1. No snapshot OR restore to be running.

   2. Global job scheduler should be disabled.

2. The mentioned gemfury repository should be accessible from TVault VM.

Rolling upgrade on Openstack Ansible

Repo Updates

Add gemfury repo on each dmapi, horizon containers & compute node(s) to get updated packages.

Create a file /etc/apt/sources.list.d/fury.list and add below line in it.

Use the below commands to get a list of updated packages available on the configured repositories.

Upgrade tvault-datamover-api package

• Login to dmapi container from the controller node using the below command.

• Take a backup of the following file on each dmapi container(s).

• Add the gemfury repo and upgrade the dmapi package using the below command. Select the appropriate package depending on the python version used.

• Restore the backed-up config files

• Now restart and check the service tvault-datamover-api on controller.

Upgrade Horizon plugin

• For the Horizon plugin upgrade, we need to upgrade below two packages.

python3-tvault-horizon-plugin python3-workloadmgrclient

• Login to the horizon container from the controller node using the below command.

• Add the gemfury repo and upgrade the tvault-horizon-plugin & workloadmgrclient packages using the below command. Select the appropriate package depending on the python version used.

• Restart the apache2 service and verify the workloadmgrclient version using the below commands.

Upgrade the tvault-contego package

NFS Storage Backend

• Take a backup of following file on each compute node(s) for nfs storage backend.

• Upgrade the tvault-contego package using below command.Select the appropriate package depending on python version used.

• Restore the backed-up config files

• Now restart and check the service tvault-contego on compute node(s).

Please make sure all configuration files are unchanged.

S3 Storage Backend

• Take a backup of following file on each compute node(s) for s3 storage backend.

• Upgrade the tvault-contego package using below command. Select the appropriate package depending on the python version used.

Note*:- If you get prompt while installing, choose the default selection.

• Restore the backed-up config files

• Now restart and check the service tvault-contego on compute node(s).

Please make sure all configuration files are unchanged.

Rolling upgrade on Kolla Openstack

Upgrade tvault-datamover-api package (trilio_datamover_api Container)

Container trilio_datamover_api needs to be redeployed. Follow below steps on all controller nodes :

• Backup Existing conf files/folders

• Stop and Remove existing container trilio_datamover_api

• Pull trilio_datamover_api container image:

• Run datamover api container.

• Verify deployment

Upgrade Horizon Plugin Package

• Login to horizon container (on controller node)

• Add gemfury repo on horizon to get updated packages.

  Create a file /etc/apt/sources.list.d/fury.list and add below line in it.

  Use below commands to get list of updated packages available on repo's.

• For Horizon plugin upgrade, following packages need to be upgraded.

python3-tvault-horizon-plugin python3-workloadmgrclient

• Run following commands to upgrade the tvault-horizon-plugin & workloadmgrclient packages. Select the appropriate package depending on python version used.

• Restart docker container (from controller node) and verify the workloadmgrclient version (inside horizon container).

Upgrade Tvault-Contego (trilio_datamover Container)

Container trilio_datamover needs to be redeployed. Follow below steps on all Compute nodes :

• Backup Existing conf files/folders

• Stop & remove existing trilio_datamover container

• Pull Trilio Datamover container image using the following command:

• Run datamover container.

• Verify Deployment of trilio_datamover

Clone updated triliovault-cfg-scripts github repo

Note: Run all the commands with 'stack' user.

• Clone updated repo\

• If your backup target is Ceph S3 with self signed certs If your backup target is ceph S3 with SSL and SSL certificates are self signed or authorized by private CA, then user needs to provide CA chain certificate to validate the SSL requests. For that, user needs to rename his ca chain cert file to 's3-cert.pem' and copy it into directory - 'triliovault-cfg-scripts/redhat-director-scripts/redhat-director-scripts/puppet/trilio/files'

• Upload triliovault puppet module

• Copy old trilio env file to new repository

Pull latest triliovault container images and update triliovault environment yaml file

Trilio containers are pushed to 'RedHat Container Registry'. Registry URL is ''. Following are the triliovault container pull urls.

Trilio container urls for RHOSP13:

Trilio container urls for RHOSP16:

There are three registry methods available in RedHat Openstack Platform.

1. Remote Registry 2. Local Registry 3. Satellite Server

Identify which method you are using. Below we have explained all three methods to pull and configure trilioVault's container images for overcloud deployment.

Remote Registry:

If you are using 'Remote Registry' method follow this section. You don't need to pull anything. You just need to populate following container urls in trilio env yaml.

• If it’s RHOSP13, populate 'trilio_env.yaml' file with triliovault container urls. Changes looks like following.\

• If it’s RHOSP16, populate 'trilio_env_osp16.yaml' file with triliovault container urls. Changes looks like following.\

Registry on Undercloud:

If you are using 'local registry' on undercloud, follow this section.

Note: Run following script from undercloud node with 'stack' user

• If it’s RHOSP13, run following script. Script, pulls the triliovault containers and updates the triliovault environment file with urls.

• If it’s RHOSP13, user needs to set horizon image url to triliovault horizon image url in overcloud_images.yaml. Changes looks like following.

• If it’s RHOS16, run following script. Script, pulls the triliovault containers and updates the triliovault environment file with urls.\

## Above script pushes trilio container images to undercloud registry and sets correct trilio images urls in trilio_env_osp16.yaml. Verify the changes using following command.

Registry on Redhat Satellite Server