6.0.0-beta

What's New?

1. Deployment of Trilio on RHOCP (with RHOSP17.1).

2. Support for multiple target backends.

3. Support for advanced scheduling of workloads' snapshots.

Known Issues

1. Workload import from non-default BTT gets stuck

   • If non-default BTT is provided as source btt in import command, then the same gets stuck and no workload gets imported. However same works well if source btt is the default one.

2. workload-create CLI expects jobschedule as mandatory parameter

   • Workaround:

     • While creating workload from CLI, parameter jobschedule to be provided with True/False values. Eg. --jobschedule enabled=False

3. get-importworkload-list and get-orphaned-workloads-list show incorrect workloads

   • The commands get-importworkload-list and get-orphaned-workloads-list show incorrect workloads in the output.

   • Workaround:

     • Use --project option with get-importworkload-list CLI command to get the list of workloads that can be imported on any particular project of the OpenStack. workloadmgr workload-get-importworkloads-list --project_id <project_id>

Trilio seamlessly integrates with OpenStack, functioning exclusively through APIs utilizing the OpenStack Endpoints. Furthermore, Trilio establishes its own set of OpenStack endpoints. Additionally, both the Trilio appliance and compute nodes interact with the backup target, impacting the network strategy for a Trilio installation.

Existing endpoints in OpenStack

OpenStack comprises three endpoint groupings:

• Public Endpoints

Public endpoints are meant to be used by the OpenStack end-users to work with OpenStack.

• Internal Endpoints

Internal endpoints are intended to be used by the OpenStack services to communicate with each

• Admin Endpoints

Admin endpoints are meant to be used by OpenStack administrators.

Among these three endpoint categories, it's important to note that the admin endpoint occasionally hosts APIs not accessible through any other type of endpoint.

To learn more about OpenStack endpoints please visit the official OpenStack documentation.

OpenStack endpoints required by Trilio

Trilio communicates with all OpenStack services through a designated endpoint type, determined and configured during the deployment of Trilio's services.

The following network requirements can be identified this way:

• Trilio services need access to the Keystone admin endpoint on the admin endpoint network if it is available.

• Trilio services need access to all endpoints of the set endpoint type during deployment.

Recommendation: Provide access to all OpenStack Endpoint types

Trilio recommends granting comprehensive access to all OpenStack endpoints for all Trilio services, aligning with OpenStack's established standards and best practices.

Additionally, Trilio generates its own endpoints, which are integrated within the same network as other OpenStack API services.

To adhere to OpenStack's prescribed standards and best practices, it's advisable that Trilio containers operate on the same network as other OpenStack containers.

• The public endpoint to be used by OpenStack users when using Trilio CLI or API

• The internal endpoint to communicate with the OpenStack services

• The admin endpoint to use the required admin-only APIs of Keystone

Backup target access required by Trilio

The Trilio solution uses backup target storage to place the backup data securely. Trilio divides its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio Workloadmgr services through communication with the OpenStack Endpoints. All metadata that is stored together with a backup is written by the Trilio Workloadmgr services to the backup target in the JSON format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service reads the Volume Data from the Cinder or Nova storage and transfers this data as a qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• Every Trilio Workloadmgr service containers need access to the backup target

• Every Trilio Datamover service containers need access to the backup target

Before embarking on the installation process for Trilio in your OpenStack environment, it is highly advisable to carefully consider several key elements. These considerations will not only streamline the installation procedure but also ensure the optimal setup and functionality of Trilio's solutions within your OpenStack infrastructure.

Tenant Quotas

Trilio leverages Cinder snapshots to facilitate the computation of both full and incremental backups.

When executing full backups, Trilio orchestrates the generation of Cinder snapshots for all volumes included in the backup job. These Cinder snapshots remain intact for subsequent incremental backup image calculations.

During incremental backup operations, Trilio generates fresh Cinder snapshots and computes the altered blocks between these new snapshots and the earlier retained snapshots from full or previous backups. The old snapshots are subsequently deleted, while the newly generated snapshots are preserved.

Consequently, it becomes imperative for each tenant benefiting from Trilio's backup functionality to possess adequate Cinder snapshot quotas capable of accommodating these supplementary snapshots. As a guiding principle, it is recommended to append 2 snapshots for each volume incorporated into the backup quotas for the respective tenant. Additionally, a commensurate increase in volume quotas for the tenant is advisable, as Trilio briefly materializes a volume from the snapshot to access data for backup purposes.

During the restoration process, Trilio generates supplementary instances and Cinder volumes. To facilitate seamless restore operations, tenants should maintain adequate quota levels for Nova instances and Cinder volumes. Failure to meet these quota requirements may lead to disruptions in restoration procedures.

AWS S3 Eventual Consistency

The AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of these models explains how an object becomes consistent after being created, updated, or deleted. None of these methods ensures strong consistency, leading to a delay before an object becomes fully consistent.

Although Trilio has introduced measures to address AWS S3's eventual consistency limitations, the exact time an object achieves consistency cannot be predicted through deterministic means.

There is no official statement from AWS on how long it takes for an object to reach a consistent state. However, read-after-write has a shorter time to reach consistency compared to other IO patterns. Therefore, our solution is designed to maximize the read-after-write IO pattern.

AWS Region Considerations

The time in which an object reaches eventual consistency also depends on the AWS region.

For instance, the AWS-standard region doesn't offer the same level of strong consistency as regions like us-east or us-west. Opting for these regions when setting up S3 buckets for Trilio is advisable. While fully avoiding the read-after-update IO pattern is complex, we've introduced significant access delays for objects to achieve consistency over longer periods. On rare occasions when this does happen, it will cause a backup failure and require a retry.

Trilio Cluster

Trilio can be deployed as a single node or a three-node cluster. It is highly recommended that Trilio is deployed as a three-node cluster for fault tolerance and load balancing. Starting with the 3.0 release, Trilio requires additional IP or FQDN for the cluster and is required for both single-node and three-node deployments. Cluster IP a.k.a virtual IP is used for managing clusters and is used to register the Trilio service endpoint in the keystone service catalog.

Trilio Data, Inc

Trilio Data, Inc. is the leader in providing backup and recovery solutions for cloud-native applications. Established in 2013, its flagship products, Trilio for OpenStack(T4O) and Trilio for Kubernetes(T4K) are used by a wide number of large corporations around the world.

About Trilio for OpenStack

T4O, by Trilio Data, is a native OpenStack service-based solution that provides policy-based comprehensive backup and recovery for OpenStack workloads. It captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS, AWS S3, and other S3-compatible storages. With Trilio and its one-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection, and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations, and storage data as one unit. It also takes incremental backups that capture only the changes that were made since the last backup. Incremental snapshots save a considerable amount of storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized below:

About this Documentation

This documentation serves as the end-user technical resource to accompany Trilio for OpenStack. You will learn about the architecture, installation, and vast number of operations of this product. The intended audience is anyone who wants to understand the value, operations, and nuances of protecting their cloud-native applications with Trilio for OpenStack.

Deployment Scripts

git clone -b 6.0.0-beta-2 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

Container Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-beta-6-0/

deb [trusted=yes] https://apt.fury.io/trilio-beta-6-0/ /

https://yum.fury.io/trilio-beta-6-0/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-beta-6-0/
enabled=1
gpgcheck=0

Cloud-Native and Containerized

T4O represents a comprehensively containerized deployment model, eliminating the necessity for any KVM-based appliances for service deployment. This marks a departure from earlier T4O releases, where such appliances were required.

Summary of Requirements

Trilio requires its containers to be deployed on the same plane as OpenStack, utilizing existing cluster resources.

As described in the , Trilio requires sufficient cluster resources to deploy its components on both the Controller Plane and Compute Planes.

1. Valid Trilio License & Acceptance of the

2. Sufficient resources available on the target OpenShift cluster nodes

3. Sufficient storage capacity and connectivity on Cinder for snapshotting operations

4. Sufficient network capabilities for efficient data transfer of workloads

5. User and Role permissions for access to required cluster objects

6. Optional features may have specific requirements such as encryption, file search, snapshot mount, FRM instance, etc

7. Set hw_qemu_guest_agent=True property on the image and install qemu-guest-agent on the VM, in order to avoid any file system inconsistencies post restore.

8. For the VMware to OpenStack migration feature, please refer to the and pages.

Trilio for OpenStack Compatibility Matrix

NFS & S3 Support:

All versions of T4O-6.x releases support NFSv3 and S3 as backup targets on all the compatible distributions.

Encryption Support:

All versions of T4O-6.x releases support encryption using Barbican service on all the compatible distributions.

QEMU Guest Agent

QEMU guest agent is highly recommended to be running inside the VM that being backed up to avoid data corruption during backup process. QEMU Guest Agent is a daemon that runs inside a virtual machine (VM) and communicates with the host system (the hypervisor) to provide enhanced management and control of the VM. It is an essential component in virtualized environments specially OpenStack.

The is the supported and recommended method to deploy and maintain any RHOSP installation.

Trilio is integrating natively into the RHOSP Director. Manual deployment methods are not supported for RHOSP.

1. Prepare for deployment

Refer to the below-mentioned acceptable values for the placeholders triliovault_tag, trilio_branch, RHOSP_version and CONTAINER-TAG-VERSION in this document as per the Openstack environment:

1.1] Select 'backup target' type

Backup target storage is used to store backup images taken by Trilio and details needed for configuration:

The following backup target types are supported by Trilio

a) NFS

Need NFS share path

b) Amazon S3

- S3 Access Key - Secret Key - Region - Bucket name

c) Other S3 compatible storage (Like Ceph based S3)

- S3 Access Key - Secret Key - Region - Endpoint URL (Valid for S3 other than Amazon S3) - Bucket name

1.2] Clone triliovault-cfg-scripts repository

The following steps are to be done on the undercloud node on an already installed RHOSP environment. The overcloud-deploy command has to be run successfully already and the overcloud should be available.

The following command clones the triliovault-cfg-scripts github repository.

1.3] Set executable permissions for all shell scripts

1.4] If the backup target type is 'Ceph based S3' with SSL:

If your backup target is CEPH S3 with SSL and SSL certificates are self-signed or authorized by a private CA, then the user needs to provide a CA chain certificate to validate the SSL requests. For all s3 backup target with self signed TLS certificates, user need to copy ca chain files in following location and in given file name format in trilio puppet module. Edit <S3_BACKUP_TARGET_NAME>, <S3_SELF_SIGNED_CERT_CA_CHAIN_FILE> parameters in following command.

For example if S3_BACKUP_TARGET_NAME = BT2_S3 and your S3_SELF_SIGNED_CERT_CA_CHAIN_FILE='s3-ca.pem' then command to copy this ca chain file to trilio puppet module would be

1.5] If the VM migration feature needs to be enabled:

1.5.1] Copy the VMware vSphere Virtual Disk Development Kit 7.0

Rename the file as vddk.tar.gz and place it at

1.5.2] For secure connection with the vCenter

Copy the vCenter SSL cert file to

2] Update the overcloud roles data file to include Trilio services

Trilio contains multiple services. Add these services to your roles_data.yaml.

Add the following services to the roles_data.yaml

2.1] Add Trilio Datamover Api and Trilio Workload Manager services to role data file

This service needs to share the same role as the keystone and database service. In the case of the pre-defined roles, these services will run on the role Controller. In the case of custom-defined roles, it is necessary to use the same role where OS::TripleO::Services::Keystone service installed.

Add the following line to the identified role:

2.2] Add Trilio Datamover Service to role data file

This service needs to share the same role as the nova-compute service. In the case of the pre-defined roles will the nova-compute service run on the role Compute. In the case of custom-defined roles, it is necessary to use the role that the nova-compute service uses.

Add the following line to the identified role:

3] Prepare Trilio container images

Trilio containers are pushed to the RedHat Container Registry. Registry URL: 'registry.connect.redhat.com'. Container pull URLs are given below.

There are three registry methods available in the RedHat OpenStack Platform.

1. Remote Registry

2. Local Registry

3. Satellite Server

3.1] Remote Registry

Follow this section when 'Remote Registry' is used.

In this method, container images get downloaded directly on overcloud nodes during overcloud deploy/update command execution. Users can set the remote registry to a redhat registry or any other private registry that they want to use. The user needs to provide credentials for the registry in containers-prepare-parameter.yaml file.

1. Make sure other OpenStack service images are also using the same method to pull container images. If it's not the case you can not use this method.

2. Populate containers-prepare-parameter.yaml with content like the following. Important parameters are push_destination: false, ContainerImageRegistryLogin: true and registry credentials. Trilio container images are published to registry registry.connect.redhat.com. Credentials of registry 'registry.redhat.io' will work for registry.connect.redhat.com registry too.

Note: This file containers-prepare-parameter.yaml

Note: File 'containers-prepare-parameter.yaml' gets created as output of command 'openstack tripleo container image prepare'. Refer above document by RedHat

3. Make sure you have network connectivity to the above registries from all overcloud nodes. Otherwise image pull operation will fail.

4. The user needs to manually populate the trilio_env.yaml file with Trilio container image URLs as given below:

At this step, you have configured Trilio image URLs in the necessary environment file.

3.2] Local Registry

Follow this section when 'local registry' is used on the undercloud.

In this case, it is necessary to push the Trilio containers to the undercloud registry. Trilio provides shell scripts that will pull the containers from registry.connect.redhat.com and push them to the undercloud and update the trilio_env.yaml.

At this step, you have downloaded Trilio container images and configured Trilio image URLs in the necessary environment file.

3.3] Red Hat Satellite Server

Follow this section when a Satellite Server is used for the container registry.

Populate the trilio_env.yaml with container URLs.

At this step, you have downloaded Trilio container images into the RedHat satellite server and configured Trilio image URLs in the necessary environment file.

4] Provide environment details in trilio-env.yaml

Edit /home/stack/triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env.yaml file and provide backup target details and other necessary details in the provided environment file. This environment file will be used in the overcloud deployment to configure Trilio components. Container image names have already been populated in the preparation of the container images. Still, it is recommended to verify the container URLs.

You don't need to provide anything for resource_registry, keep it as it is.

4.1] TrilioBackupTargets Details

T4O supports setting up multiple target backend for storing snapshots. User can define any number of storage backends as required. At a high level, NFS and S3 are supported.

Following table provides the details of parameters to be set in trilio_env.yaml file against all the S3 target backends.

Following table provides the details of parameters to be set in trilio_env.yaml file against all the NFS target backends.

4.2] Update triliovault object store yaml

After you fill in details of backup targets in trilio_env.yaml, user needs to run following script from ‘scripts' directory on undercloud node. This script will update ‘services/triliovault-object-store.yaml' file. User don’t need to verify that.

4.3] For enabling the VM migration feature, populate the below mentioned parameters

5] Generate random passwords for TrilioVault OpenStack resources

The user needs to generate random passwords for Trilio resources using the following script.

This script will generate random passwords for all Trilio resources that are going to get created in OpenStack cloud.

5.1] Change the directory and run the script

5.2] Output will be written to the below file.

6] Fetch ids of required OpenStack resources

6.1] User needs to source 'overcloudrc' file of cloud admin user. This is needed to run OpenStack CLI.

6.2] User must have filled in the cloud admin user details of overcloud in 'trilio_env.yaml' file in the 'Provide environment details in trilio-env.yaml' section. If not please do so.

6.3] Cloud admin user should have admin role on cloud admin domain

6.4] After this, user can run the following script.

The output will be written to

7] Load necessary Linux drivers on all Controller and Compute nodes

For TrilioVault functionality to work, we need the following Linux kernel modules to be loaded on all controllers and compute nodes(Where Trilio WLM and Datamover services are going to be installed).

7.1] Install nbd module

7.2] Install fuse module

8] Upload Trilio puppet module

8.1] Source the stackrc

8.2] The following commands upload the Trilio puppet module to the overcloud registry. The actual upload happens upon the next deployment.

9] Deploy overcloud with Trilio environment

9.1] Include the environment file defaults.yaml in overcloud deploy command with `-e` option as shown below.

This YAML file holds the default values, like default Trustee Role is creator and Keystone endpoint interface is Internal. There are some other parameters as well those User can update as per their requirements.

9.2] Additionally include the following heat environment files and roles data file we mentioned in the above sections in an overcloud deploy command:

1. trilio_env.yaml

2. roles_data.yaml

3. passwords.yaml

4. defaults.yaml

5. Use the correct Trilio endpoint map file as per the available Keystone endpoint configuration. You have to remove your OpenStack's endpoint map file from overcloud deploy command and instead of that use Trilio endpoint map file.

   1. Instead of tls-endpoints-public-dns.yaml file, use triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_endpoints_public_dns.yaml

   2. Instead of tls-endpoints-public-ip.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_endpoints_public_ip.yaml

   3. Instead of tls-everywhere-endpoints-dns.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_tls_everywhere_dns.yaml

   4. Instead of no-tls-endpoints-public-ip.yaml file, usetriliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_env_non_tls_endpoints_ip.yaml

To include new environment files use -e option and for roles data files use -r option.\

Below is an example of an overcloud deploy command with Trilio environment:

10] Verify deployment

11] Troubleshooting for overcloud deployment failures

Trilio components will be deployed using puppet scripts.

=> If any Trilio containers do not start well or are in a restarting state on the Controller/Compute node, use the following logs to debug.

12] Advanced Settings/Configuration

12.1] Configure Multi-IP NFS

i] On Undercloud node, change the directory

ii] Edit file triliovault_nfs_map_input.yml in the current directory and provide compute host and NFS share/IP map.

Get the overcloud Controller and Compute hostnames from the following command. Check Name column. Use exact host names in the triliovault_nfs_map_input.yml file.

Below is an example of how you can set the multi-IP NFS details:

iii] Update pyyaml on the undercloud node only

Expand the map file to create a one-to-one mapping of the compute nodes and the NFS shares.

iv] Validate output map file

The result will be stored in the triliovault_nfs_map_output.yml file

Open file triliovault_nfs_map_output.yml available in the current directory and validate that all compute nodes are covered with all the necessary NFS shares.

v] Append this output map file to triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

Validate the changes in the file triliovault-cfg-scripts/redhat-director-scripts/<RHOSP_RELEASE_DIRECTORY>/environments/trilio_nfs_map.yaml

vi] Include the environment file (trilio_nfs_map.yaml) in overcloud deploy command with '-e' option as shown below.

vii] Ensure that MultiIPNfsEnabled is set to true in the trilio_env.yaml file and that NFS is used as a backup target.

viii] After this you need to run the overcloud deployment.

12.2] Haproxy customized configuration for Trilio dmapi service

Following is the HAproxy conf file location on HAproxy nodes of the overcloud. Trilio Datamover API service HAproxy configuration gets added to this file.

Trilio Datamover HAproxy default configuration from the above file looks as follows:

The user can change the following configuration parameter values.

To change these default values, you need to do the following steps. i) On the undercloud node, open the following file for editing.

ii) Search the following entries and edit as required

iii) Save the changes and do the overcloud deployment again to reflect these changes for overcloud nodes.

12.3] Configure Custom Volume/Directory Mounts for the Trilio Datamover Service

i) If the user wants to add one or more extra volume/directory mounts to the Trilio Datamover Service container, they can use a variable named 'TrilioDatamoverOptVolumes' is available in the below file.

ii) The variable 'TrilioDatamoverOptVolumes' accepts a list of volume/bind mounts. User needs to edit the file and add their volume mounts in the below format.

iii) Lastly you need to do overcloud deploy/update.

After successful deployment, you will see that volume/directory mount will be mounted inside the Trilio Datamover Service container.

12.4] Advanced Ceph Configration (Optional)

We are using cinder's ceph user for interacting with Ceph cinder storage. This user name is defined using parameter - 'ceph_cinder_user'.

Each T4O release includes a set of artifacts such as version-tagged containers, package repositories, and distribution packages.

To help users quickly identify the resources associated with each release, we have added dedicated sub-pages corresponding to a specific release version.

After the installation and configuration of Trilio for Openstack did succeed the following steps can be done to verify that the Trilio installation is healthy.

On the Controller node

Make sure below containers are in a running state, triliovault-wlm-cron would be running on only one of the controllers in case of multi-controller setup.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

If the containers are in a restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

After successful deployment, triliovault-wlm-cron service would get added in pcs cluster as a cluster resource, you can verify through pcs status command

Verify the HAproxy configuration under:

On Compute node

Make sure the Trilio Datamover container is in running state and no other Trilio container is deployed on compute nodes.

Check if provided backup target is mounted well on Compute host.

On the node with Horizon service

Make sure the horizon container is in a running state. Please note that the Horizon container is replaced with Trilio's Horizon container. This container will have the latest OpenStack horizon + Trilio horizon plugin.

About the workloadmgr CLI client

The workloadmgr CLI client is provided as rpm and deb packages.

The following operating systems have been verified with the installation:

• CentOS8

• Ubuntu 18.04, Ubuntu 20.04

Installing the workloadmgr client will automatically install all required OpenStack clients as well.

Further, the installation of the workloadmgr client will integrate the client into the global OpenStack Python client, if available.

Install workloadmgr client rpm package on CentOS8

The Trilio workload manager CLI client has several requirements that need to be met before the client can be installed without dependency issues.

Preparing the workloadmgr client installation

The following steps need to be done to prepare the installation of the workloadmgr client:

1. Add required repositories

   1. epel-release

   2. centos-release-openstack-train

2. install base packages

   1. yum -y install epel-release

   2. yum -y install centos-release-openstack-train

Installing the workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/yum.repos.d/trilio.repo Enter the following details into the repository file:

[trilio]
name=Trilio Repository
baseurl=https://yum.fury.io/trilio-5-0/
enabled=1
gpgcheck=0

Install the workloadmgr client issuing the following command:

yum install python-3-workloadmgrclient-el8

Install workloadmgr client deb packages on Ubuntu

The Trilio workloadmgr client packages for Ubuntu are only available from the online repository.

Preparing the workloadmgr client installation

There is no preparation required. All dependencies are automatically resolved by the standard repositories provided by Ubuntu.

Installing the Workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/apt/sources.list.d/fury.list Enter the following details into the repository file:

deb [trusted=yes] https://apt.fury.io/trilio-5-0/ /

run apt update to make the new repository available.

The apt package manager is used to install the workloadmgr client package:

apt-get install python3-workloadmgrclient

Clean Trilio Datamover API and Workloadmanager services

The following steps need to be run on all nodes, which have the Trilio Datamover API & Workloadmanager services running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the below entries

OS::TripleO::Services::TrilioDatamoverApi
OS::TripleO::Services::TrilioWlmApi
OS::TripleO::Services::TrilioWlmWorkloads
OS::TripleO::Services::TrilioWlmScheduler
OS::TripleO::Services::TrilioWlmCron

Once the role that runs the Trilio Datamover API & Workloadmanager services has been identified will the following commands clean the nodes from the service.

Remove triliovault_datamover_api container.

podman rm -f triliovault_datamover_api
podman rm -f triliovault_datamover_api_db_sync
podman rm -f triliovault_datamover_api_init_log

Clean Trilio Datamover API service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultdmapi
rm /var/lib/config-data/puppet-generated/triliovaultdmapi.md5sum
rm -rf /var/lib/config-data/triliovaultdmapi*
rm -f /var/lib/config-data/triliovault_datamover_api*

Clean Trilio Datamover API service log directory.

rm -rf /var/log/containers/triliovault-datamover-api/

Remove triliovault_wlm_api container.

podman rm -f triliovault_wlm_api
podman rm -f triliovault_wlm_api_cloud_trust_init
podman rm -f triliovault_wlm_api_db_sync
podman rm -f triliovault_wlm_api_config_dynamic
podman rm -f triliovault_wlm_api_init_log

Clean Trilio Workloadmanager API service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmapi
rm /var/lib/config-data/puppet-generated/triliovaultwlmapi.md5sum
rm -rf /var/lib/config-data/triliovaultwlmapi*
rm -f /var/lib/config-data/triliovault_wlm_api*

Clean Trilio Workloadmanager API service log directory.

rm -rf /var/log/containers/triliovault-wlm-api/

Remove triliovault_wlm_workloads container.

podman rm -f triliovault_wlm_workloads
podman rm -f triliovault_wlm_workloads_config_dynamic
podman rm -f triliovault_wlm_workloads_init_log

Clean Trilio Workloadmanager Workloads service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmworkloads
rm /var/lib/config-data/puppet-generated/triliovaultwlmworkloads.md5sum
rm -rf /var/lib/config-data/triliovaultwlmworkloads*

Clean Trilio Workloadmanager Workloads service log directory.

rm -rf /var/log/containers/triliovault-wlm-api/

Remove triliovault_wlm_scheduler container.

podman rm -f triliovault_wlm_scheduler
podman rm -f triliovault_wlm_scheduler_config_dynamic
podman rm -f triliovault_wlm_scheduler_init_log

Clean Trilio Workloadmanager Scheduler service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmscheduler
rm /var/lib/config-data/puppet-generated/triliovaultwlmscheduler.md5sum
rm -rf /var/lib/config-data/triliovaultwlmscheduler*

Clean Trilio Workloadmanager Scheduler service log directory.

rm -rf /var/log/containers/triliovault-wlm-scheduler/

Remove triliovault-wlm-cron-podman-0 container from controller.

podman rm -f triliovault-wlm-cron-podman-0
podman rm -f triliovault_wlm_cron_config_dynamic
podman rm -f triliovault_wlm_cron_init_log

Clean Trilio Workloadmanager Cron service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultwlmcron
rm /var/lib/config-data/puppet-generated/triliovaultwlmcron.md5sum
rm -rf /var/lib/config-data//triliovaultwlmcron*

Clean Trilio Workloadmanager Cron service log directory.

rm -rf /var/log/containers/triliovault-wlm-cron/

Clean Trilio Datamover Service

The following steps need to be run on all nodes, which have the Trilio Datamover service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamover.

Once the role that runs the Trilio Datamover service has been identified will the following commands clean the nodes from the service.

Remove triliovault_datamover container.

podman rm -f triliovault_datamover

Unmount the Trilio Backup Target on the compute host.

## Following steps are applicable for all supported RHOSP releases.

# Check triliovault backup target mount point
mount | grep trilio

# Unmount it
-- If it's NFS	(COPY UUID_DIR from your compute host using above command)
umount /var/lib/nova/triliovault-mounts/<UUID_DIR>

-- If it's S3
umount /var/lib/nova/triliovault-mounts

# Verify that it's unmounted		
mount | grep trilio
	
df -h  | grep trilio

# Remove mount point directory after verifying that backup target unmounted successfully.
# Otherwise actual data from backup target may get cleaned.	

rm -rf /var/lib/nova/triliovault-mounts

Clean Trilio Datamover service conf directory.

rm -rf /var/lib/config-data/puppet-generated/triliovaultdm/
rm /var/lib/config-data/puppet-generated/triliovaultdm.md5sum
rm -rf /var/lib/config-data/triliovaultdm*

Clean log directory of Trilio Datamover service.

rm -rf /var/log/containers/triliovault-datamover/

Clean wlm-cron resource from pcs cluster

Remove wlm cron resource from pcs cluster on the controller node.

pcs resource delete triliovault-wlm-cron

Clean Trilio HAproxy resources

The following steps need to be run on all nodes, which have the HAproxy service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::HAproxy.

Once the role that runs the HAproxy service has been identified will the following commands clean the nodes from all the Trilio resources.

Edit the following file inside the HAproxy container and remove all Trilio entries.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

An example of these entries is given below.

listen triliovault_datamover_api
  bind 172.30.5.23:13784 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.30.5.23:8784 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloudtrain1-controller-0.internalapi.trilio.local 172.30.5.28:8784 check fall 5 inter 2000 rise 2 verifyhost overcloudtrain1-controller-0.internalapi.trilio.local

listen triliovault_wlm_api
  bind 172.30.5.23:13781 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.30.5.23:8781 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem
  balance roundrobin
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  maxconn 50000
  option httpchk
  option httplog
  retries 5
  timeout check 10m
  timeout client 10m
  timeout connect 10m
  timeout http-request 10m
  timeout queue 10m
  timeout server 10m
  server overcloudtrain1-controller-0.internalapi.trilio.local 172.30.5.28:8780 check fall 5 inter 2000 rise 2 verifyhost overcloudtrain1-controller-0.internalapi.trilio.local

Restart the HAproxy container once all edits have been done.

podman restart haproxy-bundle-podman-0

Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

openstack service delete dmapi
openstack user delete dmapi
openstack service delete TrilioVaultWLM
openstack user delete triliovault

Clean Trilio database resources

Trilio creates databases for dmapi and workloadmgr services. These databases need to be cleaned.

Login into the database cluster

podman exec -it galera-bundle-podman-0 mysql -u root

Run the following SQL statements to clean the database.

## Clean database
DROP DATABASE dmapi;

## Clean dmapi user
=> List 'dmapi' user accounts
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
+-------+-----------------------------------------+
| user  | host                                    |
+-------+-----------------------------------------+
| dmapi | %                                       |
| dmapi | 172.30.5.28                             |
| dmapi | overcloudtrain1internalapi.trilio.local |
+-------+-----------------------------------------+
3 rows in set (0.000 sec)

=> Delete those user accounts
MariaDB [(none)]> DROP USER dmapi@'%';
Query OK, 0 rows affected (0.005 sec)

MariaDB [(none)]> DROP USER dmapi@172.30.5.28;
Query OK, 0 rows affected (0.006 sec)

MariaDB [(none)]> DROP USER dmapi@overcloudtrain1internalapi.trilio.local;
Query OK, 0 rows affected (0.005 sec)

=> Verify that dmapi user got cleaned
MariaDB [mysql]> select user, host from mysql.user where user='dmapi';
Empty set (0.00 sec)

## Clean database
DROP DATABASE workloadmgr;

## Clean workloadmgr user
=> List 'workloadmgr' user accounts
MariaDB [(none)]> select user, host from mysql.user where user='workloadmgr';
+-------------+-----------------------------------------+
| user        | host                                    |
+-------------+-----------------------------------------+
| workloadmgr | %                                       |
| workloadmgr | 172.30.5.28                             |
| workloadmgr | overcloudtrain1internalapi.trilio.local |
+-------------+-----------------------------------------+
3 rows in set (0.000 sec)

=> Delete those user accounts
MariaDB [(none)]> DROP USER workloadmgr@'%';
Query OK, 0 rows affected (0.012 sec)

MariaDB [(none)]> DROP USER workloadmgr@172.30.5.28;
Query OK, 0 rows affected (0.006 sec)

MariaDB [(none)]> DROP USER workloadmgr@overcloudtrain1internalapi.trilio.local;
Query OK, 0 rows affected (0.005 sec)

=> Verify that workloadmgr user got cleaned
MariaDB [(none)]> select user, host from mysql.user where user='workloadmgr';
Empty set (0.000 sec)

Revert overcloud deploy command

Remove the following entries from roles_data.yaml used in the overcloud deploy command.

• OS::TripleO::Services::TrilioDatamoverApi

• OS::TripleO::Services::TrilioWlmApi

• OS::TripleO::Services::TrilioWlmWorkloads

• OS::TripleO::Services::TrilioWlmScheduler

• OS::TripleO::Services::TrilioWlmCron

• OS::TripleO::Services::TrilioDatamover

Follow these steps to clean the overcloud deploy command from all Trilio entries.

1. Remove trilio_env.yaml entry

2. Remove trilio endpoint map file Replace with original map file if existing

Revert back to the original RHOSP Horizon container

Run the cleaned overcloud deploy command.

Trilio employs a base64 hash to establish the mount point for NFS Backup targets, ensuring compatibility across multiple NFS Shares within a single Trilio installation. This hash is an integral component of Trilio's incremental backups, functioning as an absolute path for backing files.

Consequently, during a disaster recovery or rapid migration situation, the utilization of a mount bind becomes necessary.

In scenarios that allow for a comprehensive migration period, an alternative approach comes into play. This involves modifying the backing file, thereby enabling the accessibility of Trilio backups from a different NFS Share. The backing file is updated to correspond with the mount point of the new NFS Share.

Backing file change script

Trilio provides a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Downloading the shell script

Prerequisites

The following requirements need to be met before the change of the backing file can be attempted.

• The Trilio Appliance has been reconfigured with the new NFS Share

• The Openstack environment has been reconfigured with the new NFS Share

  • Please check here for Red Hat Openstack Platform

• The workloads are available on the new NFS Share

• The workloads are owned by nova:nova user

Usage

The shell script is changing one workload at a time.

Run the following command:

./backing_file_update.sh /var/triliovault-mounts/<base64>/workload_<workload_id>

with

• /var/triliovault-mounts/<base64>/ being the new NFS mount path

• workload_<workload_id> being the workload to rebase

Logging of the procedure

The shell script is generating the following log file at the following location:

/tmp/backing_file_update.log

The log file will not get overwritten when the script is run multiple times. Each run of the script will append the available log file.

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to delete the keyring extension from the triliovault-datamover.conf inside the Ceph block by following below mentioned steps:

1. Deploy Trilio as per the documented steps.

2. Post successful deployment, please modify the triliovault-datamover.conf file present at following locations on all compute nodes.

   For RHOSP : /var/lib/config-data/puppet-generated/triliovaultdm/etc/triliovault-datamover/

   For Kolla : /etc/kolla/triliovault-datamover/

3. Modify keyring_ext value with valid keyring extension (eg. .keyring). This extension is expected to be same for all the keyring files. It will be present under [ceph] block in triliovault-datamover.conf file.

Sample conf entry below. This will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

...

[ceph]
keyring_ext = .keyring

...

1. Restart triliovault_datamover container on all compute nodes.

1] Prerequisites

Please ensure the following requirements are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

• wlm-cron is disabled on the Trilio Appliance

Deactivating the wlm-cron service

The following sets of commands will disable the wlm-cron service and verify that it is has been completely shutdown.

[root@TVM2 ~]# pcs resource disable wlm-cron
[root@TVM2 ~]# systemctl status wlm-cron
● wlm-cron.service - workload's scheduler cron service
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; vendor preset                                                                                                                                                                                                                                                                                                                                                           : disabled)
   Active: inactive (dead)

Jun 11 08:27:06 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:06 - INFO - 1...t
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 140686268624368 Child 11389 ki...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - 1...5
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Shutting down thread pool
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...l
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: Stopping the threads
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - S...s
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: All threads are stopped succes...y
Jun 11 08:27:07 TVM2 workloadmgr-cron[11115]: 11-06-2021 08:27:07 - INFO - A...y
Jun 11 08:27:09 TVM2 systemd[1]: Stopped workload's scheduler cron service.
Hint: Some lines were ellipsized, use -l to show in full.
[root@TVM2 ~]# pcs resource show wlm-cron
 Resource: wlm-cron (class=systemd type=wlm-cron)
  Meta Attrs: target-role=Stopped
  Operations: monitor interval=30s on-fail=restart timeout=300s (wlm-cron-monito                                                                                                                                                                                                                                                                                                                                                           r-interval-30s)
              start interval=0s on-fail=restart timeout=300s (wlm-cron-start-int                                                                                                                                                                                                                                                                                                                                                           erval-0s)
              stop interval=0s timeout=300s (wlm-cron-stop-interval-0s)
[root@TVM2 ~]# ps -ef | grep -i workloadmgr-cron
root     15379 14383  0 08:27 pts/0    00:00:00 grep --color=auto -i workloadmgr                                                                                                                                                                                                                                                                                                                                                           -cron

2] Install latest Trilio release

Please follow this documentation to install the latest Trilio release.

3] Enable mount-bind for NFS

T4O has changed the calculation of the mount point from 4.2 release onwards. It is necessary to set up the mount-bind to make T4O 4.1 or older backups available for T4O 4.2

Please follow this documentation to set up the mount bind for RHOSP.

4] Enable Global Job Scheduler

After the upgrade, global job scheduler would be disabled. You can enable that either through UI or CLI.

4.1] Through UI

Login to the Dashboard and go to Admin -> Backups-Admin -> Settings tab and check the 'Job Scheduler Enabled' checkbox. By clicking the 'Change' button you can enable the Global Job Scheduler.

4.2] Through CLI

Login to the any WLM container, create and source the admin rc file. Execute the enable the global job scheduler cli.

$ podman exec -itu root triliovault_wlm_api /bin/bash
$ source admin.rc
$ workloadmgr enable-global-job-scheduler
Global job scheduler is successfully enabled

5] Import old workloads existing on the Backup Target

Workload import allows importing Workloads existing on the Backup Target into the Trilio database.

Please follow this documentation to import the Workloads.

The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover container

4. Uninstall the workloadmgr container

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

Introduction

Filename and location:

triliovault-cfg-scripts/common/triliovault_nfs_map_input.yml

This file exclusively comes into play when users aim to configure Trilio with a NFS backup target that employs multiple network endpoints. For all other scenarios, such as single IP NFS or S3, this file remains inactive, and in such instances, please consult the standard installation documentation.

When using an NFS backup target with multiple network endpoints, T4O will mount a single IP/endpoint on a designated compute node for a specific NFS share. This approach enables users to distribute NFS share IPs/endpoints across various compute nodes.

The 'triliovault_nfs_map_input.yml' file allows users to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Examples

Using hostnames

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

192.168.1.33:/var/share1 
192.168.1.34:/var/share1 
192.168.1.35:/var/share1

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

prod-compute-1.trilio.demo 
prod-compute-2.trilio.demo 
prod-compute-3.trilio.demo 
. 
. 
. 
prod-compute-30.trilio.demo

The remaining 2 node hostnames do not follow any format/pattern.

compute_bare.trilio.demo 
compute_virtual

Now the mapping file will look like this

multi_ip_nfs_shares: 
 - "192.168.1.34:/var/share1": ['prod-compute-[1:10].trilio.demo', 'compute_bare.trilio.demo'] 
   "192.168.1.35:/var/share1": ['prod-compute-[11:20].trilio.demo', 'compute_virtual'] 
   "192.168.1.33:/var/share1": ['prod-compute-[21:30].trilio.demo'] 

single_ip_nfs_shares: []

Using IPs

Compute node IP range used here: 172.30.3.11-40 and 172.30.4.40, 172.30.4.50 Total of 32 compute nodes

multi_ip_nfs_shares: 
 - "192.168.1.34:/var/share1": ['172.30.3.[11:20]', '172.30.4.40'] 
   "192.168.1.35:/var/share1": ['172.30.3.[21:30]', '172.30.4.50'] 
   "192.168.1.33:/var/share1": ['172.30.3.[31:40]'] 

single_ip_nfs_shares: []

Getting the correct compute hostnames/IPs

RHOSP

Use the following command to get compute hostnames. Check the ‘Name' column. Use these exact hostnames in 'triliovault_nfs_map_input.yml' file.

In the following command output, ‘overcloudtrain1-novacompute-0' and ‘overcloudtrain1-novacompute-1' are correct hostnames.

(undercloud) [stack@ucqa161 ~]$ openstack server list
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| ID                                   | Name                          | Status | Networks             | Image          | Flavor  |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+
| 8c3d04ae-fcdd-431c-afa6-9a50f3cb2c0d | overcloudtrain1-controller-2  | ACTIVE | ctlplane=172.30.5.18 | overcloud-full | control |
| 103dfd3e-d073-4123-9223-b8cf8c7398fe | overcloudtrain1-controller-0  | ACTIVE | ctlplane=172.30.5.11 | overcloud-full | control |
| a3541849-2e9b-4aa0-9fa9-91e7d24f0149 | overcloudtrain1-controller-1  | ACTIVE | ctlplane=172.30.5.25 | overcloud-full | control |
| 74a9f530-0c7b-49c4-9a1f-87e7eeda91c0 | overcloudtrain1-novacompute-0 | ACTIVE | ctlplane=172.30.5.30 | overcloud-full | compute |
| c1664ac3-7d9c-4a36-b375-0e4ee19e93e4 | overcloudtrain1-novacompute-1 | ACTIVE | ctlplane=172.30.5.15 | overcloud-full | compute |
+--------------------------------------+-------------------------------+--------+----------------------+----------------+---------+

Trilio services logs on RHOSP

On the Controller Node

Below Trilio containers gets deployed on the Contoller node.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

The log files for above Trilio services can be found here:

/var/log/containers/triliovault-datamover-api/triliovault-datamover-api.log

/var/log/containers/triliovault-wlm-api/triliovault-wlm-api.log

/var/log/containers/triliovault-wlm-cron/triliovault-wlm-cron.log

/var/log/containers/triliovault-wlm-scheduler/triliovault-wlm-scheduler.log

/var/log/containers/triliovault-wlm-workloads/triliovault-wlm-workloads.log

In the case of using S3 as a backup target is there also a log file that keeps track of the S3-Fuse plugin used to connect with the S3 storage.

/var/log/containers/triliovault-wlm-workloads/triliovault-object-store.log

For file serach operation, logs can be found on Controller node at below location:

/var/log/containers/triliovault-wlm-workloads/workloadmgr-filesearch.log

On the Compute Node

Trilio Datamover container gets deployed on the Compute node.

The log file for the Trilio Datamover service can be found at:

/var/log/containers/triliovault-datamover/triliovault-datamover.log

Troubleshooting inside a complex environment like OpenStack can be very time-consuming.

The following tips will help to speed up the troubleshooting process to identify root causes.

What is happening where

OpenStack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio Workloadmgr

The Trilio Workloadmgr is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio Workloadmgr also responsible for gathering the metadata about the backed-up VMs and networks from the OpenStack environment. It sends API calls to the OpenStack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received, the Trilio Workloadmgr writes it as JSON files on the Backup Target.

The Trilio Workloadmgr is also sending the Cinder Snapshot command.

During a restore process

During the restore process, the Trilio Workloadmgr reads the VM metadata from its Database and uses the metadata to create the Shell for the restore. It sends API calls to the OpenStack environment to create the necessary resources.

Datamover API

The dmapi service is the connector between the Trilio cluster and the Datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so, the dmapi service connects to the nova API requesting the compute hose of a provided VM.

Once the compute host has been identified, the dmapi forwards the command from the Trilio Workloadmgr to the datamover running on the identified compute host.

Datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover controls the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise, backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Input/Output Error with Cohesity NFS

On Cohesity NFS if an Input/Output error is observed, then increase the timeo and retrans parameter values in your NFS options.

Timeout receiving packet error in multipath ISCSi environment

Logging inside all datamover containers and add uxsock_timeout with value as 60000 which is equal to 60 sec inside /etc/multipath.conf.Restart datamover container

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during the creation of Workloads, backups, or restores.

OpenStack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The OpenStack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

After all Trilio components are installed, the license can be applied.

The license can be applied either through the Admin tab in Horizon or the CLI

Apply license through Horizon

To apply for the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on the Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Read the license agreement

8. Click on "I accept the terms in the License Agreement"

9. Click on "Next"

10. Click "Choose File"

11. Choose license file on the client system

12. Click "Apply"

Apply license through CLI

workloadmgr license-create <license_file>

Read and accept the End User License Agreement to complete the license application.

Users can preview the latest EULA at our main site: https://trilio.io/eula/

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab in Horizon

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload a file search shall be done in

5. Click the workload name to enter the Workload overview

6. Click File Search to enter the file search tab

Configuring and starting a file search Horizon

A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.

To run a file search the following elements need to be decided and configured

Choose the VM the file search shall run against

Under VM Name/ID choose the VM that the search is done upon. The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.

Set the File Path

The File Path defines the search string that is run against the chosen VM and Snapshots. This search string does support basic RegEx.

Example File Path for all files inside /etc : /etc/*

Define the Snapshots to search in

"Filter Snapshots by" is the third and last component that needs to be set. This defines which Snapshots are going to be searched.

There are 3 possibilities for a pre-filtering:

1. All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots

2. Last Snapshots - Choose between the last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.

3. Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.

After the pre-filtering is done all matching Snapshots are automatic prechosen. Uncheck any Snapshot that shall not be searched.

Start the File Search and retrieve the results in Horizon

To start a File Search the following elements need to be set:

• A VM to search in has to be chosen

• A valid File Path provided

• At least one Snapshot to search in selected

Once those have been set click "Search" to start the file search.

After a short time the results will be presented. The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.

For each found file or folder the following information are provided:

• POSIX permissions

• Amount of links pointing to the file or folder

• User ID who owns the file or folder

• Group ID assigned to the file or folder

• Actual size in Bytes of the file or folder

• Time of creation

• Time of last modification

• Time of last access

• Full path to the found file or folder

Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option at the top of the table. It is also possible to directly mount the Snapshot using the "Mound Snapshot" Button at the end of the table.

Doing a CLI File Search

workloadmgr filepath-search [--snapshotids <snapshotid>]
                            [--end_filter <end_filter>]
                            [--start_filter <start_filter>]
                            [--date_from <date_from>]
                            [--date_to <date_to>]
                            <vm_id> <file_path>

Definition

A Restore is the workflow to bring back the backed up VMs from a Trilio Snapshot.

List of Restores

Using Horizon

To reach the list of Restores for a Snapshot follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the Snapshot Name

9. Navigate to the Restores tab

Using CLI

workloadmgr restore-list [--snapshot_id <snapshot_id>]

Restores overview

Using Horizon

To reach the detailed Restore overview follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the Snapshot Name

9. Navigate to the Restores tab

10. Identify the restore to show

11. Click the restore name

Details Tab

The Restore Details Tab shows the most important information about the Restore.

• Name

• Description

• Restore Type

• Status

• Time taken

• Size

• Progress Message

• Progress

• Host

• Restore Options

• List of VMs restored

  • restored VM Name

  • restored VM Status

  • restored VM ID

Misc Tab

The Misc tab provides additional Metadata information.

• Creation Time

• Restore ID

• Snapshot ID containing the Restore

• Workload

Using CLI

workloadmgr restore-show [--output <output>] <restore_id>

Delete a Restore

Once a Restore is no longer needed, it can be safely deleted from a Workload.

Using Horizon

There are 2 possibilities to delete a Restore.

Possibility 1: Single Restore deletion through the submenu

To delete a single Restore through the submenu follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to delete

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the Snapshot Name

9. Navigate to the Restore tab

10. Click "Delete Restore" in the line of the restore in question

11. Confirm by clicking "Delete Restore"

Possibility 2: Multiple Restore deletion through a checkbox in Snapshot overview

To delete one or more Restores through the Restore list do the following:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshots in the Snapshot list

8. Enter the Snapshot by clicking the Snapshot name

9. Navigate to the Restore tab

10. Check the checkbox for each Restore that shall be deleted

11. Click "Delete Restore" in the menu above

12. Confirm by clicking "Delete Restore"

Using CLI

workloadmgr restore-delete <restores_id>

Cancel a Restore

Ongoing Restores can be canceled.

Using Horizon

To cancel a Restore in Horizon follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to delete

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the Snapshot Name

9. Navigate to the Restore tab

10. Identify the ongoing Restore

11. Click "Cancel Restore" in the line of the restore in question

12. Confirm by clicking "Cancel Restore"

Using CLI

workloadmgr restore-cancel <restore_id>

One Click Restore

The One Click Restore will bring back all VMs from the Snapshot in the same state as they were backed up. They will:

• be located in the same cluster in the same datacenter

• use the same storage domain

• connect to the same network

• have the same flavor

The user can't change any Metadata.

Using Horizon

There are 2 possibilities to start a One Click Restore.

Possibility 1: From the Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to be restored

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the Snapshot to be restored

8. Click "One Click Restore" in the same line as the identified Snapshot

9. (Optional) Provide a name / description

10. Click "Create"

Possibility 2: From the Snapshot overview

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to be restored

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the Snapshot to be restored

8. Click the Snapshot Name

9. Navigate to the "Restores" tab

10. Click "One Click Restore"

11. (Optional) Provide a name / description

12. Click "Create"

Using CLI

workloadmgr snapshot-oneclick-restore [--display-name <display-name>]
                                      [--display-description <display-description>]
                                      <snapshot_id>

Selective Restore

The Selective Restore is the most complex restore Trilio has to offer. It allows to adapt the restored VMs to the exact needs of the User.

With the selective restore the following things can be changed:

• Which VMs are getting restored

• Name of the restored VMs

• Which networks to connect with

• Which Storage domain to use

• Which DataCenter / Cluster to restore into

• Which flavor the restored VMs will use

Using Horizon

There are 2 possibilities to start a Selective Restore.

Possibility 1: From the Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to be restored

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the Snapshot to be restored

8. Click on the small arrow next to "One Click Restore" in the same line as the identified Snapshot

9. Click on "Selective Restore"

10. Configure the Selective Restore as desired

11. Click "Restore"

Possibility 2: From the Snapshot overview

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to be restored

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the Snapshot to be restored

8. Click the Snapshot Name

9. Navigate to the "Restores" tab

10. Click "Selective Restore"

11. Configure the Selective Restore as desired

12. Click "Restore"

Using CLI

workloadmgr snapshot-selective-restore [--display-name <display-name>]
                                       [--display-description <display-description>]
                                       [--filename <filename>]
                                       <snapshot_id>

Inplace Restore

The Inplace Restore covers those use cases, where the VM and its Volumes are still available, but the data got corrupted or needs to a rollback for other reasons.

It allows the user to restore only the data of a selected Volume, which is part of a backup.

Using Horizon

There are 2 possibilities to start an Inplace Restore.

Possibility 1: From the Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to be restored

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the Snapshot to be restored

8. Click on the small arrow next to "One Click Restore" in the same line as the identified Snapshot

9. Click on "Inplace Restore"

10. Configure the Inplace Restore as desired

11. Click "Restore"

Possibility 2: From the Snapshot overview

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to be restored

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the Snapshot to be restored

8. Click the Snapshot Name

9. Navigate to the "Restores" tab

10. Click "Inplace Restore"

11. Configure the Inplace Restore as desired

12. Click "Restore"

Using CLI

workloadmgr snapshot-inplace-restore [--display-name <display-name>]
                                     [--display-description <display-description>]
                                     [--filename <filename>]
                                     <snapshot_id>

required restore.json for CLI

The workloadmgr client CLI is using a restore.json file to define the restore parameters for the selective and the inplace restore.

An example for a selective restore of this restore.json is shown below. A detailed analysis and explanation is given afterwards.

{
    oneclickrestore: False,
    restore_type: selective, 
    type: openstack, 
    openstack: 
        {
            instances: 
                [
                    {
                        include: True, 
                        id: 890888bc-a001-4b62-a25b-484b34ac6e7e,                        
                        name: cdcentOS-1, 
                        availability_zone:, 
                        nics: [], 
                        vdisks: 
                            [
                                {
                                    id: 4cc2b474-1f1b-4054-a922-497ef5564624, 
                                    new_volume_type:, 
                                    availability_zone: nova
                                }
                            ], 
                        flavor: 
                            {
                                ram: 512, 
                                ephemeral: 0, 
                                vcpus: 1,
                                swap:,
                                disk: 1, 
                                id: 1
                            }                         
                    }
                ], 
            restore_topology: True, 
            networks_mapping: 
                {
                    networks: []
                }
        }
}

General required information

Before the exact details of the restore are to be provided it is necessary to provide the general metadata for the restore.

• openstackstarts the exact definition of the restore

Selective Restore required information

The Selective Restore requires a lot of information to be able to execute the restore as desired.

Those information are divided into 3 components:

• instances

• restore_topology

• networks_mapping

Information required in instances

This part contains all information about all instances that are part of the Snapshot to restore and how they are to be restored.

Each instance requires the following information

The following example describes a single instance with all values.

'instances':[
  {
     'name':'cdcentOS-1-selective',
     'availability_zone':'US-East',
     'nics':[
       {
          'mac_address':'fa:16:3e:00:bd:60',
          'ip_address':'192.168.0.100',
          'id':'8b871820-f92e-41f6-80b4-00555a649b4c',
          'network':{
             'subnet':{
                'id':'2b1506f4-2a7a-4602-a8b9-b7e8a49f95b8'
             },
             'id':'d5047e84-077e-4b38-bc43-e3360b0ad174'
          }
       }
     ],
     'vdisks':[
       {
          'id':'4cc2b474-1f1b-4054-a922-497ef5564624',
          'new_volume_type':'ceph',
          'availability_zone':'nova'
       }
     ],
     'flavor':{
        'ram':2048,
        'ephemeral':0,
        'vcpus':1,
        'swap':'',
        'disk':20,
        'id':'2'
     },
     'include':True,
     'id':'890888bc-a001-4b62-a25b-484b34ac6e7e'
  }
]

Information required in network topology restore or network mapping

To activate a network topology restore set:

restore_topology:True

To activate network mapping set:

restore_topology:False

When the network mapping is activated it is used, it is necessary to provide the mapping details, which are part of the networks_mapping block:

Full selective restore example

{
   'oneclickrestore':False,
   'openstack':{
      'instances':[
         {
            'name':'cdcentOS-1-selective',
            'availability_zone':'US-East',
            'nics':[
               {
                  'mac_address':'fa:16:3e:00:bd:60',
                  'ip_address':'192.168.0.100',
                  'id':'8b871820-f92e-41f6-80b4-00555a649b4c',
                  'network':{
                     'subnet':{
                        'id':'2b1506f4-2a7a-4602-a8b9-b7e8a49f95b8'
                     },
                     'id':'d5047e84-077e-4b38-bc43-e3360b0ad174'
                  }
               }
            ],
            'vdisks':[
               {
                  'id':'4cc2b474-1f1b-4054-a922-497ef5564624',
                  'new_volume_type':'ceph',
                  'availability_zone':'nova'
               }
            ],
            'flavor':{
               'ram':2048,
               'ephemeral':0,
               'vcpus':1,
               'swap':'',
               'disk':20,
               'id':'2'
            },
            'include':True,
            'id':'890888bc-a001-4b62-a25b-484b34ac6e7e'
         }
      ],
      'restore_topology':False,
      'networks_mapping':{
         'networks':[
            {
               'snapshot_network':{
                  'subnet':{
                     'id':'8b609440-4abf-4acf-a36b-9a0fa70c383c'
                  },
                  'id':'8b871820-f92e-41f6-80b4-00555a649b4c'
               },
               'target_network':{
                  'subnet':{
                     'id':'2b1506f4-2a7a-4602-a8b9-b7e8a49f95b8'
                  },
                  'id':'d5047e84-077e-4b38-bc43-e3360b0ad174',
                  'name':'internal'
               }
            }
         ]
      }
   },
   'restore_type':'selective',
   'type':'openstack'
}

Inplace Restore required information

The Inplace Restore requires less information thana selective restore. It only requires the base file with some information about the Instances and Volumes to be restored.

Information required in instances

Network mapping information required

There are no network information required, but the field have to exist as empty value for the restore to work.

Full Inplace restore example

{
   'oneclickrestore':False,
   'restore_type':'inplace',
   'type':'openstack',   
   'openstack':{
      'instances':[
         {
            'restore_boot_disk':True,
            'include':True,
            'id':'ba8c27ab-06ed-4451-9922-d919171078de',
            'vdisks':[
               {
                  'restore_cinder_volume':True,
                  'id':'04d66b70-6d7c-4d1b-98e0-11059b89cba6',
               }
            ]
         }
      ]
   }
}

Definition

A Snapshot is a single Trilio backup of a workload including all data and metadata. It contains the information of all VM's that are protected by the workload.

List of Snapshots

Using Horizon

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Workloads

4. Identify the workload to show the details on

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

The List of Snapshots for the chosen Workload contains the following additional information:

• Creation Time

• Name of the Snapshot

• Description of the Snapshot

• Total amount of Restores from this Snapshot

  • Total amount of succeeded Restores

  • Total amount of failed Restores

• Snapshot Type

• Snapshot Size

• Snapshot Status

Using CLI

 workloadmgr snapshot-list [--workload_id <workload_id>]
                           [--tvault_node <host>]
                           [--date_from <date_from>]
                           [--date_to <date_to>]
                           [--all {True,False}]

Creating a Snapshot

Snapshots are automatically created by the Trilio scheduler. If necessary or in case of deactivated scheduler is it possible to create a Snapshot on demand.

Using Horizon

There are 2 possibilities to create a snapshot on demand.

Possibility 1: From the Workloads overview

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that shall create a Snapshot

5. Click "Create Snapshot"

6. Provide a name and description for the Snapshot

7. Decide between Full and Incremental Snapshot

8. Click "Create"

Possibility 2: From the Workload Snapshot list

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that shall create a Snapshot

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Click "Create Snapshot"

8. Provide a name and description for the Snapshot

9. Decide between Full and Incremental Snapshot

10. Click "Create"

Using CLI

workloadmgr workload-snapshot [--full] [--display-name <display-name>]
                              [--display-description <display-description>]
                              <workload_id>

Snapshot overview

Each Snapshot contains a lot of information about the backup. These information can be seen in the Snapshot overview.

Using Horizon

To reach the Snapshot Overview follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload that contains the Snapshot to show

5. Click the workload name to enter the Workload overview

6. Navigate to the Snapshots tab

7. Identify the searched Snapshot in the Snapshot list

8. Click the Snapshot Name

Details Tab

The Snapshot Details Tab shows the most important information about the Snapshot.

• Snapshot Name / Description

• Snapshot Type

• Time Taken

• Size

• Which VMs are part of the Snapshot

• for each VM in the Snapshot

  • Instance Info - Name & Status

  • Security Group(s) - Name & Type

  • Flavor - vCPUs, Disk & RAM

  • Networks - IP, Networkname & Mac Address

  • Attached Volumes - Name, Type, size (GB), Mount Point & Restore Size

  • Misc - Original ID of the VM

Restores Tab

The Snapshot Restores Tab shows the list of Restores that have been started from the chosen Snapshot. It is possible to start Restores from here.

Misc. Tab

The Snapshot Miscellaneous Tab provides the remaining metadata information about the Snapshot.

• Creation Time

• Last Update time

• Snapshot ID

• Workload ID of the Workload containing the Snapshot

Using CLI

workloadmgr snapshot-show [--output <output>] <snapshot_id>

Delete Snapshots

Once a Snapshot is no longer needed, it can be safely deleted from a Workload.

Using Horizon

There are 2 possibilities to delete a Snapshot.