Backup-as-a-Service

Trilio is an add on service to OpenStack cloud infrastructure and provides backup and disaster recovery functions for tenant workloads. Trilio is very similar to other OpenStack services including nova, cinder, glance, etc and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is installed on all OpenStack controller nodes where the nova-api service is running.

3. Trilio Datamover is a python module that is installed on every OpenStack compute nodes

Service Endpoints

Trilio is both a provider and consumer into OpenStack ecosystem. It uses other OpenStack services such as nova, cinder, glance, neutron, and keystone and provides its own service to OpenStack tenants. To accomodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise Trilio provides its own public, internal and admin URLs.

Network Topology

This figure represents a typical network topology. Trilio exposes its public URL endpoint on public network and Trilio virtual appliances and data movers typically use either internal network or dedicated backup network for storing and retrieving backup images from backup store.

Trilio, by TrilioData, is a native OpenStack service that provides policy-based comprehensive backup and recovery for OpenStack workloads. The solution captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS AWS S3 compatible storage. With Trilio and its single click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations and storage data as one unit. It also takes incremental backups that only captures the changes that were made since last backup. Incremental snapshots save time and storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized as below:

The Trilio Appliance can be reinitialized, which will delete all workload related values from the Trilio database.

To reinitialize the Trilio Appliance do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reinitialize"

• Verify that you want to reinitialize the Trilio

In case of the Trilio Dashboard being lost it can be resetted as long as SSH access to the appliance is available.

To reset the password to its default do the following:

[root@TVM1 ~]# source /home/stack/myansible/bin/activate
(myansible) [root@TVM1 ~]# cd /home/stack/myansible/lib/python3.6/site-packages/tvault_configurator
(myansible) [root@TVM1 tvault_configurator]# python recreate_conf.py
(myansible) [root@TVM1 tvault_configurator]# systemctl restart tvault-config

The dashboard login will be reset to:

Username: admin
Password: password

The Trilio database is following an older OpenStack database schema. This schema is not forgetting anything and is instead setting the deleted flag for any data that is no longer valid/deleted.

This leads to ever-growing databases, which can lead to performance degradation.

To counter this a database clean-up and optimization tool has been made available.

To configure the banner shown upon accessing the Trilio Appliance GUI do the following.

1. Login into Trilio Appliance console

2. edit the banner.yaml at /etc/tvault-config/banner.yaml

3. restart tvault-config service

The content of the banner.yaml looks as follows and can be edited as required:

The Trilio solution is using the qcow2 backing file to provide full synthetic backups.

Especially when the NFS backup target is used, there are scenarios at which this backing file needs to be updated to a new mount path.

To make this process easier and streamlined Trilio is providing the following rebase tool.

Once the Trilio VM or the Cluster of Trilio VMs has been spun, the actual installation process can begin. This process contains of the following steps:

1. Install the Trilio dm-api service on the control plane

2. Install the Trilio datamover service on the compute plane

It is recommended to think about the following elements prior to the installation of Trilio for Openstack.

Tenant Quotas

Trilio uses Cinder snapshots for calculating full and incremental backups. For full backups, Trilio creates Cinder snapshots for all the volumes in the backup job. It then leaves these Cinder snapshots behind for calculating the incremental backup image during next backup. During an incremental backup operation it creates new Cinder snapshots, calculates the changed blocks between the new snapshots and the old snapshots that were left behind during full/previous backups. It then deletes the old snapshots but leaves the newly created snapshots behind. So, it is important that each tenant who is availing Trilio backup functionality has sufficient Cinder snapshot quotas to accommodate these additional snapshots. The guideline is to add 2 snapshots for every volume that is added to backups to volume snapshot quotas for that tenant. You may also increase the volume quotas for the tenant by the same amount because Trilio briefly creates a volume from snapshot to read data from the snapshot for backup purposes. During a restore process, Trilio creates additional instances and Cinder volumes. To accommodate restore operations, a tenant should have sufficient quota for Nova instances and Cinder volumes. Otherwise restore operations will result in failures.

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. The dashboard can be accessed using the virtual IP.

The Trilio Appliance Dashboard gives an overview of the running services and their Status inside the Cluster. This dashboard is accessible through the virtual IP.

The uninstallation of Trilio is depending on the Openstack Distribution it is installed in. The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

To use the workloadmgr CLI tool on the Trilio appliance it is only necessary to activate the virtual environment of the workloadmgr

Download Trilio logs

It is possible to download the Trilio logs directly through the Trilio web gui.

To download logs throught the Trilio web gui:

Change Trilio Dashboard password

To change the Trilio GUI password do:

• Login into the Trilio Dashboard

• Click on "admin" in the upper right corner to open the submenu

• Choose "Reset Password"

• Set the new Trilio password

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

It is possible to configure Cinder to have multiple configurations and keyrings for CEPH.

In this case, the Trilio Datamover file needs to be extended with the CEPH information.

For Trilio to be able to work in such an environment it is required to put copies of each of these configurations and keyrings into a separate directory, which is then made known to the Trilio Datamover inside a [ceph] block in the tvault-contego.conf.

A tvault-contego.conf file with the extended [ceph] block would look like this.

[DEFAULT]

vault_storage_type = nfs
vault_storage_nfs_export = 192.168.1.34:/mnt/tvault/tvm5
vault_storage_nfs_options = nolock,soft,timeo=180,intr,lookupcache=none


vault_data_directory_old = /var/triliovault
vault_data_directory = /var/trilio/triliovault-mounts
log_file = /var/log/kolla/triliovault-datamover/tvault-contego.log
debug = False
verbose = True
max_uploads_pending = 3
max_commit_pending = 3

dmapi_transport_url = rabbit://openstack:[email protected]:5672,openstack:[email protected]:5672,openstack:[email protected]:5672//

[dmapi_database]
connection = mysql+pymysql://dmapi:x5nvYXnAn4rXmCHfWTK8h3wwShA4vxMq3gE2jH57@kolla-victoriaR-internal.triliodata.demo:3306/dmapi


[libvirt]
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volumes

[ceph]
keyring_ext = .volumes.keyring
ceph_dir = /etc/ceph/directory1/,/etc/ceph/directory2/

[contego_sys_admin]
helper_command = sudo /usr/bin/privsep-helper


[conductor]
use_local = True

[oslo_messaging_rabbit]
ssl = false

[cinder]
http_retries = 10

For example: RHOSP16.1 to RHOSP16.2 upgrade

1] Finish upgrade process for RHOSP cloud

Without any changes in TrilioVault, customer needs to complete RHOSP upgrade process first.

2] Now we need to upgrade TrilioVault to match with new RHOSP release

2.1] Identify correct TrilioVault release for new RHOSP release

Check if current TrilioVault release which is deployed on the RHOSP cloud supports new RHOSP release If it does not support new RHOSP release, identify the next TrilioVault release which supports this new RHOSP release.

2.2] Install TrilioVault using devops code compatible with new RHOSP release

Install TrilioVault release identified in step 2.1, Refer TrilioVault install for RHOSP Get the correct deployment document as per Triliovault release from Trilio support team.

Follow all the steps.

2.3] Done

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the Openstack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Same cloud - different owner

Migration within the same cloud to a different owner Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project A — User B Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project B — User B Cloud A — Domain A — Project A — User A =>Cloud A — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create encrypted workload in Project A in Domain A via User A. Take snapshot.

3. Reassign workload to new owner

4. Load rc file of User A & provide read only rights through acl to the new owner

   openstack acl user add --user <userB_id> <secret_href> --insecure

Different cloud

Migration between clouds Cloud A — Domain A — Project A — User A => Cloud B — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create an encrypted workload in Project A in Domain A via User A. Trigger snapshot.

3. Reassign workload to Cloud B - Domain B — Project B — User B

The Trilio appliance can be reconfigured at any time to adjust the Trilio cluster to any changes in the Openstack environment or the general backup solution.

To reconfigure the Trilio Cluster go to the "Configure". The configure page shows the current configuration of the TriloVault cluster.

The configuration page also gives access to the ansible playbooks of the last successful configuration.

To start the reconfiguration of the Trilio Cluster click "Reconfigure" at the end of the table.

Follow the Configuring Trilio guide afterwards.

Trilio has four main software components:

1. Trilio ships as a QCOW2 image. User can instantiate one or more VMs from the QCOW2 image on a standalone KVM boxes.

2. Trilio API is a python module that is an extension to nova api service. This module is installed on all OpenStack controller nodes

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to adopt the keyring extension in the tvault-contego.conf inside the Ceph block.

The following example will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

Trilio is using a base64 hash for the mount point of NFS Backup targets. This hash makes sure, that multiple NFS Shares can be used with the same Trilio installation.

This base64 hash is part of the Trilio incremental backups as an absolute path of the backing files. This requires the usage of mount bind during a DR scenario or quick migration scenario.

In the case that there is time for a thorough migration there is another option to change the backing file and make the Trilio backups available on a different NFS Share. This option is updating the backing file to the new NFS Share mount point.

1. Can Trilio for OpenStack restore instance UUIDs?

Answer: NO

Trilio for OpenStack does not restore Instance UUIDs (also known as Instance IDs). The only scenario where we do not modify the Instance UUID is during an Inplace Restore, where we only recover the data without creating new instances.

When Trilio for OpenStack restores virtual machines (VMs), it effectively creates new instances. This means that new Virtual Machine Instance UUIDs are generated for the restored VMs. We achieve this by orchestrating a call to Nova, which creates new VMs with new UUIDs.

After the Trilio VM has been configured and all components are installed can the license be applied.

The license can be applied either through the admin-tab in Horizon or the CLI

Apply license through Horizon

To apply the license through Horizon follow these steps:

Starting Trilio for Openstack 4.0 does Trilio for Openstack allow in-place upgrades.

The following versions can be upgraded to each other:

Troubleshooting inside a complex environment like Openstack can be very time-consuming. The following tipps will help to speed up the troubleshooting process to identify root causes.

What is happening where

Openstack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio Workloads are designed to allow a Desaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

Trilio is using the which enables the Trilio service user to act in the name of another Openstack user.

This system is used during all backup and restore features.

By default is the Trilio GUI available on all NICs on port 443.

To limit this to only one IP the following steps need to be applied.

Network Setup

The Trilio Appliance provides by default the possibility of 4 VIPs.

• A general VIP which can be used for everything

• A public VIP for the public endpoint

• An internal VIP for the internal endpoint

• An admin VIP for the admin endpoint

Should an additional VIP be required to restrict the access of the Trilio Dashboard to this VIP the new VIP needs to be created as a new resource inside the PCS cluster.

Nginx setup

When the new dashboard_ip has been created or decided, then the next step is to set up the proxy forwarding inside Nginx, which will make the Trilio GUI available through port 8000.

1. Create new conf file at /etc/nginx/conf.d/tvault-dashboard.conf. Replace variables dashboard_ip and virtual_ip as configured or decided.

2. edit /etc/nginx/nginx.conf and uncomment line #include /etc/nginx/conf.d/*.conf;

Limit the access of the Dashboard

The configured dashboard_ip will always end on the nginx service on port 8000 and will then be forwarded to the local dashboard service on port 443.

This configuration limits the required access to the local dashboard service to the Trilio appliance cluster itself. All other connections on port 443 can be dropped.

The following commands will set the required iptable rules.

Verify the accessibility as required

At this point is the Trilio GUI only reachable on the dashboard_ip on port 8000. Accessing the Trilio GUI through any other IP or on port 443 is not allowed.

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

Trilio for OpenStack is generating a base64 hash value for every NFS backup target connected to the T4O solution. This enables T4O to mount multiple NFS backup targets to the same T4O installation.

The mountpoints are generated utilizing a hash value inside the mountpoint, providing a unique mount for every NFS backup target.

This mountpoint is then used inside the incremental backups to point to the qcow2 backing files. The backing file path is required as a full path and can not be set as a relative path. This is a limitation of the qcow2 format.

T4O 4.2 has changed how the hash value gets calculated. T4O 4.1 and prior calculated the hash value out of the complete NFS path provided as shown in the example below.

T4O 4.2 is now only considering the NFS directory part for the hash value as shown below.

It is therefore necessary to make older backups taken by T4O 4.1 or prior available for TVO4.2 to restore.

This can be done by one of two methods:

1. Rebase all backups and change the backing file path

2. Make the old mount point available again and point it to the new one using mount bind

Rebase all backups

Trilio is providing a script, that is taking care of the rebase procedure. This script can be downloaded from the following location.

Copy and use the script from the Trilio appliance as the nova user.

The script is requiring the complete path to the workload directory on the backup target.

This needs to be repeated until all workloads have been rebased.

Use mount bind with the old mount point

This method is generating a second mount point based on the old hash value calculation and then mounting the new mount point to the old one. By doing this mount bind both mount points will be available and point to the same backup target.

To use this method the following information needs to be available:

• old mount path

• new mount path

Examples of how to calculate the base64 hash value are shown below.

Old mountpoint hash value:

New mountpoint hash value:

The mount bind needs to be done for all Trilio appliances and Datamover services.

Trilio Appliance

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

An example is given below.

RHOSP/TripleO Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

Kolla Ansible OpenStack Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

Ansible OpenStack Datamover

The following steps need to be done on the overcloud compute node. They don't need to be done inside any container.

To enable the mount bind on the Trilio appliance follow these steps:

1. Create the old mountpoint directory mkdir -p old_mount_path

2. run mount --bind command mount --bind <new_mount_path> <old_mount_path>

3. set permissions for mountpoint chmod 777 <old_mount_path>

Canonical Openstack WLM & Datamover containers

Canonical OpenStack does the creation of the mountpoint and the mount bind through JuJu using the following commands.

To create the mountpoint, if it doesn't already exist:

To create the mount bind

Uninstall Trilio Services

The Trilio Ansible OpenStack playbook can be run to uninstall the Trilio services.

Destroy Trilio Datamover API container

To cleanly remove the Trilio Datamover API container run the following Ansible playbook.

Clean openstack_user_config.yml

Remove the tvault-dmapi_hosts and tvault_compute_hosts entries from /etc/openstack_deploy/openstack_user_config.yml

Remove Trilio haproxy settings in user_variables.yml

Remove Trilio Datamover API settings from /etc/openstack_deploy/user_variables.yml

Remove Trilio Datamover API inventory file

Remove Trilio Datamover API service endpoints

Delete Trilio Datamover API database and user

• Go inside galera container.

• Login as root user in mysql database engine.

• Drop dmapi database.

Remove dmapi rabbitmq user from rabbitmq container

• Go inside rabbitmq container.

• Delete dmapi user.

• Delete dmapi vhost.

Clean haproxy

Remove /etc/haproxy/conf.d/datamover_service file.

Remove HAproxy configuration entry from /etc/haproxy/haproxy.cfg file.

Restart the HAproxy service.

Remove certificates from Compute nodes

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

Destroy the Trilio VMs

Undefine the Trilio VMs

Delete the TrlioVault VM disk from KVM Host storage

Trilio integrates natively with Openstack. This includes that Trilio communicates completely through APIs using the Openstack Endpoints. Trilio is also generating its own Openstack endpoints. In addition, is the Trilio appliance and the compute nodes writing to and reading from the backup target. These points affect the network planning for the Trilio installation.

Existing endpoints in Openstack

Openstack knows 3 types of endpoints:

• Public Endpoints

• Internal Endpoints

• Admin Endpoints

Each of these endpoint types is designed for a specific purpose. Public endpoints are meant to be used by the Openstack end-users to work with Openstack. Internal endpoints are meant to be used by the Openstack services to communicate with each other. Admin endpoints are meant to be used by Openstack administrators.

Out of those 3 endpoint types does only the admin endpoint sometimes contain APIs which are not available on any other endpoint type.

To learn more about Openstack endpoints please visit the official Openstack documentation.

Openstack endpoints required by Trilio

Trilio is communicating with all services of Openstack on a defined endpoint type. Which endpoint type Trilio is using to communicate with Openstack is decided during the configuration of the Trilio appliance.

The following network requirement can be identified this way:

• Trilio appliance needs access to the Keystone admin endpoint on the admin endpoint network

• Trilio appliance needs access to all endpoints of one type

Recommendation: Provide access to all Openstack Endpoint types

Trilio is recommending providing full access to all Openstack endpoints to the Trilio appliance to follow the Openstack standards and best practices.

Trilio is generating its own endpoints as well. These endpoints are pointing towards the Trilio Appliance directly. This means that using those endpoints will not send the API calls towards the Openstack Controller nodes first, but directly to the Trilio VM.

Following the Openstack standards and best practices, it is therefore recommended to put the Trilio endpoints on the same networks as the already existing Openstack endpoints. This allows to extend the purpose of each endpoint type to the Trilio service:

• The public endpoint to be used by Openstack users when using Trilio CLI or API

• The internal endpoint to communicate with the Openstack services

• The admin endpoint to use the required admin only APIs of Keystone

Backup target access required by Trilio

The Trilio solution is using backup target storage to securely place the backup data. Trilio is dividing its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio appliance through communicating with the Openstack Endpoints. All metadata that is stored together with a backup is written by the Trilio Appliance to the backup target in the json format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service is reading the Volume Data from the Cinder or Nova storage and transferring this data as qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• The Trilio appliance needs access to the backup target

• Every compute node needs access to the backup target

Example of a typical Trilio network integration

Most Trilio customers are following the Openstack standards and best practices to have the public, internal, and admin endpoints on separate networks. They also typically don't have any network yet, which can access the desired backup target.

The starting network configuration typically looks like this:

Following the Openstack standards and Trilio's recommendation will the Trilio Appliance be placed on all those 3 networks. Further is the access to the backup target required by Trilio Appliance and Compute nodes. Here done by adding a 4th network.

The resulting network configuration would look like this:

It is of course possible to combine networks as necessary. As long as the required network access is available will Trilio work.

Other examples of Trilio network integrations

Each Openstack installation is different and so is the network configuration. There are endless possibilities of how to configure the Openstack network and how to implement the Trilio appliance into this network. The following three examples have been seen in production:

The first example is from a manufacturing company, which wanted to split the networks by function and decided to put the Trilio backup target on the internal network as the backup and recovery function was identified as an Openstack internal solution. This example looks complex but integrates Trilio just as recommended.

The second example is from a financial institute that wanted to be sure that the Openstack Users have no direct uncontrolled network access to the Openstack infrastructure. Following this example requires additional work as the internal HA-Proxy needs to be configured to correctly translates the API calls towards the Trilio

The third example is from a service company that was forced to treat Trilio as an external 3rd party solution, as we require a virtual machine running outside of Openstack. This kind of network configuration requires good planning on the Trilio endpoints and firewall rules.

Introduction

Filename and location:

This file is used only when the user wants to configure 'multiple IP/endpoints based NFS share' as a backup target for Trilio. In all other cases like single IP NFS, S3 this file does not get used. Follow regular install doc.

If the user is using multiple-IP/endpoints based NFS share as a backup target for triliovault then triliovault mounts any one IP/endpoint on a given compute node for a given NFS share. Users can distribute NFS share IPs/endpoints across compute nodes.

'triliovault_nfs_map_input.yml' file is a tool that users can use to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Examples

Using hostnames

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

The remaining 2 node hostnames do not follow any format/pattern.

Now the mapping file will look like this

Using IPs

Compute node IP range used here: 172.30.3.11-40 and 172.30.4.40, 172.30.4.50 Total of 32 compute nodes

Getting the correct compute hostnames/IPs

RHOSP or TripleO

Use the following command to get compute hostnames. Check the ‘Name' column. Use these exact hostnames in 'triliovault_nfs_map_input.yml' file.

In the following command output, ‘overcloudtrain1-novacompute-0' and ‘overcloudtrain1-novacompute-1' are correct hostnames.

Kolla-Ansible OpenStack

Compute hostnames/IPs should match in kolla-ansible inventory file and triliovault_nfs_map_input.yml.

If IP addresses are sued in kolla-ansible inventory file then the same IP addresses should be used in ‘triliovault_nfs_map_input.yml' file too. If the kolla-ansible deploy command looks like the following,

kolla-ansible -i multinode deploy

then, the inventory file is 'multinode'. Generally, it is available at “/root/multinode“.

OpenStack Ansible

Compute hostnames/IPs should match in the Openstack Ansible inventory file and triliovault_nfs_map_input.yml

If IP addresses have been used in the Openstack-ansible inventory file then you should use the same IP addresses in the ‘triliovault_nfs_map_input.yml' file too.

Generally inventory file available at /etc/openstack_deploy/openstack_user_config.yml

Openstack Ansible deploy command looks like following,

openstack-ansible os-tvault-install.yml

Openstack Ansible automatically picks up values that the user has set in the file openstack_user_config.yml

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab in Horizon

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

Configuring and starting a file search Horizon

A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.

To run a file search the following elements need to be decided and configured

Choose the VM the file search shall run against

Under VM Name/ID choose the VM that the search is done upon. The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.

Set the File Path

The File Path defines the search string that is run against the chosen VM and Snapshots. This search string does support basic RegEx.

Example File Path for all files inside /etc : /etc/*

Define the Snapshots to search in

"Filter Snapshots by" is the third and last component that needs to be set. This defines which Snapshots are going to be searched.

There are 3 possibilities for a pre-filtering:

1. All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots

2. Last Snapshots - Choose between the last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.

3. Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.

After the pre-filtering is done all matching Snapshots are automatic prechosen. Uncheck any Snapshot that shall not be searched.

Start the File Search and retrieve the results in Horizon

To start a File Search the following elements need to be set:

• A VM to search in has to be chosen

• A valid File Path provided

• At least one Snapshot to search in selected

Once those have been set click "Search" to start the file search.

After a short time the results will be presented. The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.

For each found file or folder the following information are provided:

• POSIX permissions

• Amount of links pointing to the file or folder

• User ID who owns the file or folder

Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option at the top of the table. It is also possible to directly mount the Snapshot using the "Mound Snapshot" Button at the end of the table.

Doing a CLI File Search

• <vm_id> ➡️ ID of the VM to be searched

• <file_path> ➡️ Path of the file to search for

To gracefully shutdown/restart the Trilio cluster the following steps are recommended.

Verify no snapshots or restores are running

It is recommended to verify that no snapshots or restores are running on the Trilio Cluster.

This can be verified using the following two commands:

Identify the master node for the VIP(s) and wlm-cron service

The Trilio cluster is using the pacemaker service for setting the VIP(s) of the cluster and controlling the active node for the wlm-cron service. The identified node will be the last to shut down in case that the whole cluster gets shut down.

This can be checked using the following command:

In the following example is the master node the tvm1

Shutdown/Restart of a single node in the cluster

A single node in the cluster can be shut down or restarted without issues. All services will come up and the RabbitMQ and Galeera service will rejoin the remaining cluster.

When the master node gets shutdown or restarted the VIP(s) and the wlm-cron service will switch to one of the remaining cluster nodes.

Stop the services on the node

To speed up the shutdown/restart process it is recommended to stop the Trilio services, the RabbitMQ service, and the MariaDB service on the node.

Shutdown/Restart the node

After the services have been stopped the node can be restarted or shut down using standard Linux commands.

Restarting the complete cluster node by node

Restarting the whole cluster node by node follows the same procedure as restarting a single node, with the difference that each restarted node needs to be fully started again before the next node can be restarted.

Shutdown/Restart the complete cluster as a whole

When the complete cluster needs to get stopped and restarted at the same time the following procedure needs to be completed.

The procedure on a high level is:

• Shutdown the two slave nodes

• Shutdown the master node

• Start the master node

Shutdown the two slave nodes

Before shutting down the two slave nodes it is recommended to stop running Trilio services, the RabbitMQ server, and the MariaDB on the nodes.

Afterward, the nodes can be shut down.

Shutdown the master node

Before shutting down the master node it is recommended to stop running Trilio services, the RabbitmQ server, the MariaDB, the wlm-cron and the VIP(s) resource in Pacemaker.

Afterward, the node can be shut down.

Start the master node

The first server that is getting booted will be the master node. It is highly recommended that the old master node will be booted first again.

Enable the Galeera cluster

Login into the freshly started master node and run the following command. This will restart the Galeera cluster with this node as master.

Start the slave nodes

After the master node has been booted and the Galeera cluster started the remaining nodes can be started and will automatically rejoin the Trilio cluster.

The following Trilio services are providing certificates for secured access to the Trilio solution.

Changing the certificate of TVault-Config and Nginx for Grafana Service

The TVault-Config service and the Nginx Resource for the Grafana Dashboard are using the same certificate.

The certificate used is a symlink to a host-specific certificate. Each Trilio VM has its own self-signed certificate by default which is getting recreated every time the TVault-Config service is restarted.

When the certificate for the TVault-Config and Nginx (Grafana) is to be changed to a customer chosen certificate it is required to deactivate the recreation of the certificates upon service restart.

1. Login into the Trilio VM via SSH

2. Edit the following file: /home/stack/myansible/lib/python3.6/site-packages/tvault_configurator/tvault_config_bottle.py

3. Look for create_ssl_certificates() in the main function

The resulting main function will look like this:

Afterward, the certificates can be replaced manually by overwriting the files.

Once the certificates have been replaced by the desired ones restart the TVault-Config service and the Nginx pcs resource.

Changing the certificate used by Nginx for wlm-api service

The certificate provided by the Nginx for the wlm-api service is set during configuration when HTTPS endpoints are configured for the Trilio appliance. This certificate is provided to the end-user or Openstack every time an API call to the Trilio solution is sent.

The certificate and its related private key can be changed through the OS API certificate tab.

In this tab is the section to Upload Server certificate | Private key. Use this section to update the wlm-api certificate as required.

Each Trilio Workload has a dedicated owner. The ownership of a Workload is defined by:

• OpenStack User - The OpenStack User-ID is assigned to a Workload

• OpenStack Project - The OpenStack Project-ID is assigned to a Workload

The Trilio Appliance is delivered as qcow2 image and runs as VM on top of a KVM Hypervisor.

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

The documentation of the charms can be found here:

Juju charms for OpenStack Yoga release onwards

Juju charms for other supported OpenStack releases upto wallaby

Prerequisite

Have a canonical OpenStack base setup deployed for a required release like Jammy Zed/Yoga, Focal yoga/Wallaby/Victoria/Ussuri, or Bionic Ussuri/Queens.

Steps to install the Trilio charms

1. Export the OpenStack base bundle

2. Create a Trilio overlay bundle as per the OpenStack setup release using the charms given above.

Some sample Trilio overlay bundles can be found .

3. If file search functionality is required, provision any additional node(s) that will be required for deploying the Trilio Workload manager (trilio-wlm) as a VM instead of lxd container(s).

4. Commission the additional node from MAAS UI.

5. Do a dry run to check if the Trilio bundle is working

6. Do the deployment

7. Wait till all the Trilio units are deployed successfully. Check the status via juju status command.

8. Once the deployment is complete, perform the below operations:

1. Create cloud admin trust

1. Add license

Note: Reach out to the Trilio support team for the license file.

For multipath enabled environments, perform the following actions

1. log into each nova compute node

2. add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf

3. restart tvault-contego service

Sample Trilio overlay bundles

A few Sample overlay bundles for different OpenStack versions can be found .

Clean Trilio Datamover API service

The following steps need to be run on all nodes, which have the Trilio Datamover API service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamoverApi.

Once the role that runs the Trilio Datamover API service has been identified will the following commands clean the nodes from the service.

Stop trilio_dmapi container.

Remove trilio_dmapi container.

Clean Trilio Datamover API service conf directory.

Clean Trilio Datamover API service log directory.

Clean Trilio Datamover Service

The following steps need to be run on all nodes, which have the Trilio Datamover service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamover.

Once the role that runs the Trilio Datamover service has been identified will the following commands clean the nodes from the service.

Stop trilio_datamover container.

Remove trilio_datamover container.

Unmount Trilio Backup Target on compute host.

Clean Trilio Datamover service conf directory.

Clean log directory of Trilio Datamover service.

Clean Trilio haproxy resources

The following steps need to be run on all nodes, which have the haproxy service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::HAproxy.

Once the role that runs the haproxy service has been identified will the following commands clean the nodes from all Trilio resources.

Edit the following file inside the haproxy container and remove all Trilio entries.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

An example of these entries is given below.

Restart the haproxy container once all edits have been done.

Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

Clean Trilio database resources

Trilio creates a database for the dmapi service. This database needs to be cleaned.

Login into the database cluster

Run the following SQL statements to clean the database.

Revert overcloud deploy command

Remove the following entries from roles_data.yaml used in the overcloud deploy command.

• OS::TripleO::Services::TrilioDatamoverApi

• OS::TripleO::Services::TrilioDatamover

Follow these steps to clean the overcloud deploy command from all Trilio entries.

1. Remove trilio_env.yaml entry

2. Remove trilio endpoint map file Replace with original map file if existing

Revert back to original RHOSP Horizon container

Run the cleaned overcloud deploy command.

Destroy the Trilio VM Cluster

List all VMs running on the KVM node

Destroy the Trilio VMs

Undefine the Trilio VMs

Delete the TrlioVault VM disk from KVM Host storage

Generic Prerequisites

• Please ensure to complete the upgrade of all the Trilio components on the Openstack controller & compute nodes before starting the rolling upgrade of Trilio.

• The mentioned Gemfury repository should be accessible from Trilio VM.

• Either 4.2 GA OR any hotfix patch against 4.2 should be already deployed for performing upgrades mentioned in the current document.

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completly shut-down.

Steps to upgrade Trilio having Internet access

1. Download the hf_upgrade.sh script on all T4O nodes where the upgrade is to be done.

You can check the usage of this script .

1. Run the following command to download and Install the upgraded packages

1. Run the following command to enable the wlm-cron service

Steps to upgrade Trilio having No Internet access

Here the packages need to be downloaded on a separate host which has internet access. The script ./hf_upgrade.sh can be used with the below-mentioned option to download the required package

1. Download the hf_upgrade.sh script and use the script to download the upgraded packages. You can check the usage of this script .\

2. Copy the downloaded 4.3-offlinePkgs.tar.gz package and hf_upgrade.sh script to all the Trilio nodes\

Post upgrade steps

1. Verify all wlm services on all Trilio nodes

2. Make sure all wlm services are up and running from python version 3.8.x

3. Check the mount point using “df -h” command

Additional check for wlm-cron on the primary node

[RHOSP, TripleO and Kolla only] Verify nova UID/GID for nova user on the Appliance

Red Hat OpenStack, TripleO, and Kolla Ansible Openstack are using the nova UID/GID of 42436 inside their containers instead of 162:162 which is the standard in other Openstack environments.

Please verify that the nova UID/GID on the Trilio Appliance is still 42436, to do so follow the below document provided here:

Introduction

Integrating Trilio for OpenStack 4.2 with Barbican facilitates encryption support for the qcow2-data segment of Trilio backups. However, the JSON files housing the backed-up OpenStack metadata remain unencrypted.

This capability necessitates the presence of the OpenStack Barbican service. Absence of the Barbican service will result in the omission of encryption options for Workloads within the Horizon interface.

Trilio for OpenStack (T4O) 4.2 exclusively retrieves secrets from Barbican, without generating, modifying, or deleting any secrets within the Barbican platform.

Barbican secrets are indispensable for executing backups or restorations in encryption-enabled Workloads. The onus lies on the OpenStack project user to supply these secrets and guarantee their accurate availability.

In order to employ encrypted Workloads, the Trilio trustee role must be capable of interacting with the Barbican service to access and retrieve secrets from Barbican. By default, only the 'admin' and 'creator' roles are endowed with these permissions.

Encryption availability for a Workload is confined to the Workload creation stage. Post-creation, the encryption status of a Workload is irreversible; it cannot be altered or toggled.

The following secret configurations are supported for AES-256

By default Barbican will generate secrets of the following type:

• Alghorithm: AES-256 (All supported types utilize this algorithm)

• Mode: cbc

• content type: application/octet-stream

Prerequisite

For encrypted workload Barbican should enabled on Openstack, then while configuring TVO-4.2, trustee role should be kept as creator .

Additionally every user who’ll be interacting with TrilioVault for any operator, should also have creator role assigned.

1. Secret creation

Secret can be created via OpenStack cli with following the below steps

1. Source the rc file of the user who is going to create the encrypted workload.

2. Create any supported type of secret and fetch the secret UUID using OpenStack secret cli.

   1. Refer to this guide for more information about

()[root@overcloudtrain1-controller-0 /]# openstack secret order create --name secret2 --algorithm aes --mode ctr --bit-length 256 --payload-content-type=application/octet-stream key +----------------+-------------------------------------------------------------------------------------------+ | Field | Value | +----------------+-------------------------------------------------------------------------------------------+ | Order href | https://overcloudtrain1.trilio.local:13311/v1/orders/641bac2c-b5b2-4a7a-9172-8fe0cf55425d | | Type | Key | | Container href | N/A | | Secret href | None | | Created | None | | Status | None | | Error code | None |

b. View the details of the order to identify the location of the generated key, shown here as the Secret href value: ()[root@overcloudtrain1-controller-0 /]# openstack secret order get https://overcloudtrain1.trilio.local:13311/v1/orders/641bac2c-b5b2-4a7a-9172-8fe0cf55425d +----------------+--------------------------------------------------------------------------------------------+ | Field | Value | +----------------+--------------------------------------------------------------------------------------------+ | Order href | https://overcloudtrain1.trilio.local:13311/v1/orders/641bac2c-b5b2-4a7a-9172-8fe0cf55425d | | Type | Key | | Container href | N/A | | Secret href | https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c | | Created | 2023-07-12T11:38:17+00:00 |

c. Fetch the secret UUID via below command (use Secret href)

()[root@overcloudtrain1-controller-0 /]# openstack secret get https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c +---------------+--------------------------------------------------------------------------------------------+ | Field | Value | +---------------+--------------------------------------------------------------------------------------------+ | Secret href | https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c | | Name | secret2 | | Created | 2023-07-12T11:38:17+00:00 | | Status | ACTIVE | | Content types | {'default': 'application/octet-stream'} | | Algorithm | aes | | Bit length | 256 |

Note down the last value from Secret href URL which is UUID (8-4-4-4-12 format).

For example secret UUID for the Secret href URL https://overcloudtrain1.trilio.local:13311/v1/secrets/f2c96fe2-6ae7-4985-b98c-e571ba05403c will be f2c96fe2-6ae7-4985-b98c-e571ba05403c

This UUID will be used further for creating the encrypted workload.

2. Workload creation

1. Login to openstack horizon dashboard with user who has created the secret & go to Project->Backup->workloads

2. click on create workload.

3. select the checkbox for Enable Encryption

1. Follow the usual procedure for further tabs (Workload member, Schedule, policy & Option) & click on create.

2. Workload will be created and value for Encryption field will be True.

Encrypted workload migration

For migration of encrypted workload please follow:

Upgrade from older release to 4.2

For upgrade from older release to 4.2 please follow:

For Trilio configuration please follow:

Start File Search

POST https://$(tvm_address):8780/v1/$(tenant_id)/search

Starts a File Search with the given parameters

On the Trilio Nodes

The Trilio Cluster contains multiple log files.

The main log is workloadmgr-workloads.log, which contains all logs about ongoing and past Trilio backup and restore tasks. It can be found at:

/var/log/workloadmgr/workloadmgr-workloads.log