Trilio Data, Inc. is the leader in providing backup and recovery solutions for cloud-native applications. Established in 2013, its flagship products, Trilio for OpenStack(T4O) and Trilio for Kubernetes(T4K) are used by a wide number of large corporations around the world.

T4O, by Trilio Data, is a native OpenStack service-based solution that provides policy-based comprehensive backup and recovery for OpenStack workloads. It captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS, AWS S3, and other S3-compatible storages. With Trilio and its one-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection, and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations, and storage data as one unit. It also takes incremental backups that capture only the changes that were made since the last backup. Incremental snapshots save a considerable amount of storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized below:

This documentation serves as the end-user technical resource to accompany Trilio for OpenStack. You will learn about the architecture, installation, and vast number of operations of this product. The intended audience is anyone who wants to understand the value, operations, and nuances of protecting their cloud-native applications with Trilio for OpenStack.

Backup-as-a-Service

Trilio is an add-on service to OpenStack cloud infrastructure and provides backup and disaster recovery solutions for tenant workloads. Trilio is very similar to other OpenStack services including Nova, Cinder, Glance, etc., and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has four main software components which are again segregated into multiple services.

WorkloadManager

This component is registered as a keystone service of type workloads which manages all the workloads being created for utilizing the snapshot and restore functionalities. It has 4 services responsible for managing these workloads, their snapshots, and restores.

1. workloadmgr-api

2. workloadmgr-scheduler

3. workloadmgr-workloads

4. workloadmgr-cron

Similar to WorkloadManager, this component registers a keystone service of the type datamover which manages the transfer of extensive data to/from the backup targets. It has 2 services that are responsible for taking care of the data transfer and communication with WorkloadManager

1. datamover-api

2. datamover

For ease of access and better user experience, T4O provides an integrated UI with the OpenStack dashboard service Horizon,

1. Trilio API is a Python module that is installed on all OpenStack controller nodes where the nova-api service is running.

2. Trilio Datamover is a Python module that is installed on every OpenStack compute nodes

3. Trilio Horizon plugin is installed as an add-on to Horizon servers. This module is installed on every server that runs Horizon service.

Introduced in 6.2, DMS is the component responsible for mounting and unmounting backup targets on the nodes that need them. It runs as a single container per host on the controller and on every compute node, and replaces the per-backup-target object-store containers that Trilio used in earlier releases.

1. trilio-dms-server: receives mount/unmount requests over RabbitMQ, fetches S3 credentials from Barbican using the requesting job's Keystone token, and performs the actual NFS mount or spawns an s3vaultfuse process for S3 targets. One server runs per host.

2. trilio-dms-client: invoked in-process by workloadmgr (controller-side) and dmapi (compute-side) to send mount/unmount requests, and maintains the per-job mount ledger in the database so concurrent jobs can share a mount and the last release triggers the actual umount.

Mounts are performed on demand, scoped to a backup job, and reference-counted; targets are unmounted automatically when the last job releases them. Stale mounts (dead FUSE processes, broken NFS sessions) are detected before every mount and recovered automatically.

Trilio is both a provider and consumer in the OpenStack ecosystem. It uses other OpenStack services such as Nova, Cinder, Glance, Neutron, and Keystone and provides its own services to OpenStack tenants. To accommodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise, Trilio provides its own public, internal, and admin URLs for two of its services WorkloadManager API and Datamover API.

Unlike the previous versions of Trilio for OpenStack, now it utilizes the existing network of the OpenStack deployed environment. The networks for the Trilio services can be configured as per the user's desire in the same way the user configures any other OpenStack service. Additionally, a dedicated network can be provided to Trilio services on both control and compute planes for storing and retrieving backup data from the backup target store.

Before embarking on the installation process for Trilio in your OpenStack environment, it is highly advisable to carefully consider several key elements. These considerations will not only streamline the installation procedure but also ensure the optimal setup and functionality of Trilio's solutions within your OpenStack infrastructure.

Tenant Quotas

Trilio leverages Cinder snapshots to facilitate the computation of both full and incremental backups.

When executing full backups, Trilio orchestrates the generation of Cinder snapshots for all volumes included in the backup job. These Cinder snapshots remain intact for subsequent incremental backup image calculations.

During incremental backup operations, Trilio generates fresh Cinder snapshots and computes the altered blocks between these new snapshots and the earlier retained snapshots from full or previous backups. The old snapshots are subsequently deleted, while the newly generated snapshots are preserved.

Consequently, it becomes imperative for each tenant benefiting from Trilio's backup functionality to possess adequate Cinder snapshot quotas capable of accommodating these supplementary snapshots. As a guiding principle, it is recommended to append 2 snapshots for each volume incorporated into the backup quotas for the respective tenant. Additionally, a commensurate increase in volume quotas for the tenant is advisable, as Trilio briefly materializes a volume from the snapshot to access data for backup purposes.

During the restoration process, Trilio generates supplementary instances and Cinder volumes. To facilitate seamless restore operations, tenants should maintain adequate quota levels for Nova instances and Cinder volumes. Failure to meet these quota requirements may lead to disruptions in restoration procedures.

The AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of these models explains how an object becomes consistent after being created, updated, or deleted. None of these methods ensures strong consistency, leading to a delay before an object becomes fully consistent.

Although Trilio has introduced measures to address AWS S3's eventual consistency limitations, the exact time an object achieves consistency cannot be predicted through deterministic means.

There is no official statement from AWS on how long it takes for an object to reach a consistent state. However, read-after-write has a shorter time to reach consistency compared to other IO patterns. Therefore, our solution is designed to maximize the read-after-write IO pattern.

The time in which an object reaches eventual consistency also depends on the AWS region.

For instance, the AWS-standard region doesn't offer the same level of strong consistency as regions like us-east or us-west. Opting for these regions when setting up S3 buckets for Trilio is advisable. While fully avoiding the read-after-update IO pattern is complex, we've introduced significant access delays for objects to achieve consistency over longer periods. On rare occasions when this does happen, it will cause a backup failure and require a retry.

Trilio can be deployed as a single node or a three-node cluster. It is highly recommended that Trilio is deployed as a three-node cluster for fault tolerance and load balancing. Starting with the 3.0 release, Trilio requires additional IP or FQDN for the cluster and is required for both single-node and three-node deployments. Cluster IP a.k.a virtual IP is used for managing clusters and is used to register the Trilio service endpoint in the keystone service catalog.

Each T4O release includes a set of artifacts such as version-tagged containers, package repositories, and distribution packages.

To help users quickly identify the resources associated with each release, we have added dedicated sub-pages corresponding to a specific release version.

The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover container

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

Overview

It provides the procedure for installing and configuring Grafana to enable monitoring and visualization of metrics in the OpenStack environment.

It covers the integration of Grafana with the required data sources (such as Prometheus) and explains how to configure dashboards for effective monitoring of infrastructure and services.

1] Uninstall trilio-openstack helm chart

Run following commands to uninstall trilio services from MOSK cloud.

2] Uninstall Dynamic Backup Targets (Optional)

If you have installed multiple backup targets other than the default one, you can uninstall them using:

3] Verify uninstallation

Verify if the trilio-openstack chart and other components got uninstalled. No objects should be listed in following command output:

4] Clean TrilioVault’s OpenStack resources

5] Unmount the targets from All Worker and Master Nodes

To unmount the backup target mount points from each node, run:

Repeat this on each worker and master node where the target was mounted.

Cloud-Native and Containerized

T4O represents a comprehensively containerized deployment model, eliminating the necessity for any KVM-based appliances for service deployment. This marks a departure from earlier T4O releases, where such appliances were required.

Summary of Requirements

Trilio requires its containers to be deployed on the same plane as OpenStack, utilizing existing cluster resources.

As described in the architecture overview, Trilio requires sufficient cluster resources to deploy its components on both the Controller Plane and Compute Planes.

1. Valid Trilio License & Acceptance of the EULA

2. When deploying the Control Plane to OpenShift, ensure sufficient resources are available on Worker Nodes.

3. Sufficient storage capacity and connectivity on Cinder for snapshotting operations

4. Sufficient network capabilities for efficient data transfer of workloads

5. User and Role permissions for access to required cluster objects

6. Optional features may have specific requirements such as encryption, file search, snapshot mount, FRM instance, etc

7. Set hw_qemu_guest_agent=True property on the image and install qemu-guest-agent on the VM, in order to avoid any file system inconsistencies post restore.

8. For the VMware to OpenStack migration feature, please refer to the and pages.

Trilio seamlessly integrates with OpenStack, functioning exclusively through APIs utilizing the OpenStack Endpoints. Furthermore, Trilio establishes its own set of OpenStack endpoints. Additionally, both the Trilio appliance and compute nodes interact with the backup target, impacting the network strategy for a Trilio installation.

Existing endpoints in OpenStack

OpenStack comprises three endpoint groupings:

• Public Endpoints

Public endpoints are meant to be used by the OpenStack end-users to work with OpenStack.

• Internal Endpoints

Internal endpoints are intended to be used by the OpenStack services to communicate with each

• Admin Endpoints

Admin endpoints are meant to be used by OpenStack administrators.

Among these three endpoint categories, it's important to note that the admin endpoint occasionally hosts APIs not accessible through any other type of endpoint.

To learn more about OpenStack endpoints please visit the official OpenStack documentation.

Trilio communicates with all OpenStack services through a designated endpoint type, determined and configured during the deployment of Trilio's services.

The following network requirements can be identified this way:

• Trilio services need access to the Keystone admin endpoint on the admin endpoint network if it is available.

• Trilio services need access to all endpoints of the set endpoint type during deployment.

Trilio recommends granting comprehensive access to all OpenStack endpoints for all Trilio services, aligning with OpenStack's established standards and best practices.

Additionally, Trilio generates its own endpoints, which are integrated within the same network as other OpenStack API services.

To adhere to OpenStack's prescribed standards and best practices, it's advisable that Trilio containers operate on the same network as other OpenStack containers.

• The public endpoint to be used by OpenStack users when using Trilio CLI or API

• The internal endpoint to communicate with the OpenStack services

• The admin endpoint to use the required admin-only APIs of Keystone

The Trilio solution uses backup target storage to place the backup data securely. Trilio divides its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio Workloadmgr services through communication with the OpenStack Endpoints. All metadata that is stored together with a backup is written by the Trilio Workloadmgr services to the backup target in the JSON format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service reads the Volume Data from the Cinder or Nova storage and transfers this data as a qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• Every Trilio Workloadmgr service containers need access to the backup target

• Every Trilio Datamover service containers need access to the backup target

The following versions can be upgraded to each other:

Trilio for OpenStack 6.2 introduces the , which changes how backup targets are mounted on controller and compute nodes. Upgrading from 6.1.x to 6.2.x therefore has a one-time migration beyond the usual package upgrade:

1. The trilio-dms service is deployed alongside the existing Trilio services on every controller and compute node that needs backup-target access. Deployment scripts in triliovault-cfg-scripts (branch 6.2.x) handle this automatically.

2. The /etc/triliovault-dms/ configuration tree — server.conf, client.conf, rootwrap.conf

Detailed platform-specific steps are in the platform upgrade guides.

Platform availability at 6.2.0 GA: the upgrade path from 6.1.x to 6.2.x is supported on RHOSO 18.0 at GA. Support for other platforms will land in subsequent 6.2.x releases.

After all Trilio components are installed, the license can be applied.

The license can be applied either through the Admin tab in Horizon or the CLI

Apply license through Horizon

To apply for the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on the Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Read the license agreement

8. Click on "I accept the terms in the License Agreement"

9. Click on "Next"

10. Click "Choose File"

11. Choose license file on the client system

12. Click "Apply"

• <license_file> ➡️ path to the license file

Read and accept the End User License Agreement to complete the license application.

Users can preview the latest EULA at our main site:

Trilio employs a base64 hash to establish the mount point for NFS Backup targets, ensuring compatibility across multiple NFS Shares within a single Trilio installation. This hash is an integral component of Trilio's incremental backups, functioning as an absolute path for backing files.

Consequently, during a disaster recovery or rapid migration situation, the utilization of a mount bind becomes necessary.

In scenarios that allow for a comprehensive migration period, an alternative approach comes into play. This involves modifying the backing file, thereby enabling the accessibility of Trilio backups from a different NFS Share. The backing file is updated to correspond with the mount point of the new NFS Share.

Backing file change script

Trilio provides a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Downloading the shell script

Prerequisites

The following requirements need to be met before the change of the backing file can be attempted.

• The Trilio Appliance has been reconfigured with the new NFS Share

• The OpenStack environment has been reconfigured with the new NFS Share

  • Please check for Red Hat OpenStack Platform

• The workloads are available on the new NFS Share

• The workloads are owned by nova:nova user

The shell script is changing one workload at a time.

Run the following command:

with

• /var/trilio/triliovault-mounts/<base64>/ being the new NFS mount path

• workload_<workload_id> being the workload to rebase

The shell script is generating the following log file at the following location:

The log file will not get overwritten when the script is run multiple times. Each run of the script will append the available log file.

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the OpenStack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

4. Check/Uncheck the box for "Enable Email Alerts"

The following screenshots show example E-mails send by Trilio.

The Trilio solution is using the qcow2 backing file to provide full synthetic backups.

Especially when the NFS backup target is used, there are scenarios at which this backing file needs to be updated to a new mount path.

To make this process easier and streamlined Trilio is providing the following rebase tool.

By default, during a snapshot, Trilio will consider all the instances that are members of a workload and starts uploading the resources of each instance simulateously. In some cases, it may be required that the snapshots must be done in serial per VM such that backup of all the disks of one VM/instance must be finished before taking backup of the other VMs that are part of that workload.

Trilio has a provision to update the default behaviour of considering all the instances of a workload during backup process, by introducing a configrable parameter in the on of its services.

Follow the below mentioned steps to set the serial upload as the default behavior.

Finding the required configuration file

As per the OpenStack distribution, look for the configuration file of workloadmgr-api(wlm-api) and workloadmgr-workloads(wlm-workloads) services or containers on the controller plane.

The default location of this configration file in the container or VM where it is running is /etc/triliovault/triliovault-wlm.conf

Once the file is located, take a backup of the file in-case of future reference.

Once the files are located, add the below mentioned parameter with its value to the [DEFAULT] section of the configuration file. serial_vm_backup=true

After successfully updating the file, restart the services or containers and these steps need to be followed on all the controller plane.

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to delete the keyring extension from the triliovault-datamover.conf inside the Ceph block by following below mentioned steps:

1. Deploy Trilio as per the documented steps.

2. Post successful deployment, please modify the triliovault-datamover.conf file present at following locations on all compute nodes.

   For RHOSP : /var/lib/config-data/puppet-generated/triliovaultdm/etc/triliovault-datamover/

   For Kolla : /etc/kolla/triliovault-datamover/

3. Modify keyring_ext value with valid keyring extension (eg. .keyring). This extension is expected to be same for all the keyring files. It will be present under [ceph] block in triliovault-datamover.conf file.

Sample conf entry below. This will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

1. Restart triliovault_datamover container on all compute nodes.

Migration within the same cloud to a different owner Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project A — User B Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project B — User B Cloud A — Domain A — Project A — User A =>Cloud A — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create encrypted workload in Project A in Domain A via User A. Take snapshot.

Trilio is using the which enables the Trilio service user to act in the name of another OpenStack user.

This system is used during all backup and restore features.

Trusts can only be worked with via CLI

• <trust_id> ➡️ ID of the trust to show

• <role_name> ➡️Name of the role that trust is created for

Trilio for OpenStack Compatibility Matrix

All versions of T4O-6.x releases support NFSv3, NFSv4 and S3 as backup targets on all the compatible distributions.

All versions of T4O-6.x releases support encryption using Barbican service on all the compatible distributions.

Installing the QEMU guest agent is an optional but recommended service to run inside the VM to ensure backup data consistency with Trilio. QEMU Guest Agent is the standard OpenSource QEMU/KVM hypervisor agent that runs inside a virtual machine (VM) and communicates with the host system (the hypervisor) to provide enhanced management and control of the VM. It is an essential component of any hypervisor platform that OpenStack can use to provide enhanced functionality when required.

To enable this feature, you must set hw_qemu_guest_agent=yes as a metadata parameter on the image you wish to use to create the guest-agent-capable instances from. Reference: https://docs.openstack.org/nova/pike/admin/configuration/hypervisor-kvm.html

If the Virtual Machine is already created from the glance image which doesn't have the hw_qemu_guest_agent=yes property set, guest agent can be enabled using following process:

The process is stopping the VM, manually adding the lines below to the virsh definition using "virsh edit ", and starting it again:

The VM the needs to be running the qemu-guest agent software:

Ubuntu/Debian: sudo apt install -y qemu-guest-agent && sudo systemctl enable qemu-guest-agent && sudo systemctl restart qemu-guest-agent && sudo systemctl status qemu-guest-agent --no-pager

RHEL/CentOS VMs: sudo yum install -y qemu-guest-agent && sudo systemctl enable qemu-guest-agent && sudo systemctl restart qemu-guest-agent && sudo systemctl status qemu-guest-agent --no-pager

Windows VMs: Inside the guest OS, install the guest agent tools from: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win-guest-tools.exe. Finally, make sure the service is running and set to start on boot automatically, in PowerShell, as Administrator: Set-Service QEMU-GA -StartupType Automatic; Restart-Service QEMU-GA; Get-Service QEMU-GA

You can confirm the qemu-guest-agent is running properly after installed, from the compute host, like this:

virsh qemu-agent-command $INSTANCE '{"execute":"guest-ping"}'

The output must be: {"return":{}}

Software Driven Migration is supported in T4O 5.X release. Please refer

1] Prerequisites

Please ensure the following requirements are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

• Disable the wlm services on all controller nodes

sudo systemctl stop tripleo_triliovault_wlm_api.service
sudo systemctl disable tripleo_triliovault_wlm_api.service
sudo systemctl stop tripleo_triliovault_wlm_scheduler.service
sudo systemctl disable tripleo_triliovault_wlm_scheduler.service
sudo systemctl stop tripleo_triliovault_wlm_workloads.service
sudo systemctl disable tripleo_triliovault_wlm_workloads.service

• Disable the datamover service on all compute nodes

Take a backup of existing triliovault-cfg-scripts directory

Clone triliovault-cfg-scripts github repository of latest release.

Copy the passwords and triliovault wlm ids files from old cfg-scripts directory.

Please follow from section 1.4 to section 10. Please note that sections 5 and 6 needs to be skipped during this process.

While providing the backup target details in section 4.1, please ensure to provide the existing backup target (of 5.x release) as default.

From any one of the controller nodes, run the below command.

From any one of the controller nodes, login to wlm api container and run the below commands to fetch the backup target details.

This command will return the backup_target_id and backup_target_type_id.

To import workloads after upgrade, run the following command from any controller node by logging into the WLM API container:

Above --inplace-upgrade will update all the workload and policy records in database as per the current release version.

The above command gives you the list of all the workloads that can be imported from the specified backup target.

The above command imports all workloads from the specified backup target in the current cloud.

Please follow to update the backing file path of existing snapshots.

After the installation and configuration of Trilio for OpenStack did succeed the following steps can be done to verify that the Trilio installation is healthy.

On the Controller node

Make sure below containers are in a running state, triliovault-wlm-cron would be running on only one of the controllers in case of multi-controller setup.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

If the containers are in a restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

After successful deployment, triliovault-wlm-cron service would get added in pcs cluster as a cluster resource, you can verify through pcs status command

Verify the HAproxy configuration under:

Make sure the Trilio Datamover container is in running state and no other Trilio container is deployed on compute nodes.

Check if provided backup target is mounted well on Compute host.

Make sure the horizon container is in a running state. Please note that the Horizon container is replaced with Trilio's Horizon container. This container will have the latest OpenStack horizon + Trilio horizon plugin.

Trilio Workloads are designed to allow a Disaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

1. Install and Configure Trilio for the target cloud

2. Verify required mount-paths and create if necessary

3. Reassign Workloads

4. Notify users to of Workloads being available

Trilio incremental Snapshots involve a backing file to the prior backup taken, which makes every Trilio incremental backup a synthetic full backup.

Trilio is using qcow2 backing files for this feature:

As can be seen in the example is the backing file an absolute path, which makes it necessary, that this path exists so the backing files can be accessed.

Trilio is using the base64 hashing algorithm for the NFS mount-paths, to allow the configuration of multiple NFS Volumes at the same time. The hash value is calculated using the provided NFS path.

When the path of the backing file is not available on the Trilio appliance and Compute nodes, will the restores of incremental backups fail.

The tested and recommended method to make the backing files available is creating the required directory path and using mount --bind to make the path available for the backups.

Running the mount --bind command will make the necessary path available until the next reboot. If it is required to have access to the path beyond a reboot is it necessary to edit the fstab.

Trilio services logs on RHOSP

On the Controller Node

Below Trilio containers gets deployed on the Contoller node.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

The log files for above Trilio services can be found here:

/var/log/containers/triliovault-datamover-api/triliovault-datamover-api.log

/var/log/containers/triliovault-wlm-api/triliovault-wlm-api.log

/var/log/containers/triliovault-wlm-cron/triliovault-wlm-cron.log

/var/log/containers/triliovault-wlm-scheduler/triliovault-wlm-scheduler.log

/var/log/containers/triliovault-wlm-workloads/triliovault-wlm-workloads.log

In the case of using S3 as a backup target is there also a log file that keeps track of the S3-Fuse plugin used to connect with the S3 storage.

For Trilio for OpenStack 6.2 and later, the s3vaultfuse processes are spawned and managed by the Dynamic Mount Service. Per-target log paths are emitted by the trilio-dms-server container; see .

For 6.1 and earlier, a single per-node S3-Fuse plugin log is written at:

/var/log/containers/triliovault-wlm-workloads/triliovault-object-store.log

For file serach operation, logs can be found on Controller node at below location:

/var/log/containers/triliovault-wlm-workloads/workloadmgr-filesearch.log

Trilio Datamover container gets deployed on the Compute node.

The log file for the Trilio Datamover service can be found at:

/var/log/containers/triliovault-datamover/triliovault-datamover.log

1. Can Trilio for OpenStack restore instance UUIDs?

Answer: NO

Trilio for OpenStack does not restore Instance UUIDs (also known as Instance IDs). The only scenario where we do not modify the Instance UUID is during an Inplace Restore, where we only recover the data without creating new instances.

When Trilio for OpenStack restores virtual machines (VMs), it effectively creates new instances. This means that new Virtual Machine Instance UUIDs are generated for the restored VMs. We achieve this by orchestrating a call to Nova, which creates new VMs with new UUIDs.

By following this approach, we maintain the principles of OpenStack and auditing. We do not update or modify existing database entries when objects are deleted and subsequently recovered. Instead, all deletions are marked as such, and new instances, including the recovered ones, are created as new objects in the Nova tables. This ensures compliance and preserves the integrity of the OpenStack environment.

2. Can Trilio for OpenStack restore MAC addresses?

Answer: YES

Trilio can restore the VMs MAC address, however, there is a caveat when restoring a virtual machine (VM) to a different IP address: a new MAC address will be assigned to the VM.

In the case of a One-Click Restore, the original MAC addresses and IP addresses will be recovered, but the VM will be created with a new UUID, as mentioned in question #1.

When performing a Selective Restore, you have the option to recover the original MAC address. To do so, you need to select the original IP address from the available dropdown menu during the recovery process.

By choosing the original IP address, Trilio for OpenStack will ensure that the VM is restored with its original MAC address, providing more flexibility and customization in the restoration process.

Example of Selective Restore with original MAC (and IP address):

1. In this example, we have taken a Trilio backup of a VM called prod-1.

1. The VM is deleted and we perform a Selective Restore of a VM called prod-1, selecting the IP address it was originally assigned from the drop-down menu:

1. Trilio then restores the VM with the original MAC address:

1. If you left the option as "Choose next available IP address", it will assign a new MAC to the VM instead as Neutron maps all MAC addresses to IP addresses on the Subnet - so logically a new IP will result in a new MAC address.

1. New Qualifcation

   • Qualification support for RHOSO18.0-FR6 (18.0.21).

   • Qualification support for Canonical Jammy.

Filename and location:

This file exclusively comes into play when users aim to configure Trilio with a NFS backup target that employs multiple network endpoints. For all other scenarios, such as single IP NFS or S3, this file remains inactive, and in such instances, please consult the standard installation documentation.

When using an NFS backup target with multiple network endpoints, T4O will mount a single IP/endpoint on a designated compute node for a specific NFS share. This approach enables users to distribute NFS share IPs/endpoints across various compute nodes.

The 'triliovault_nfs_map_input.yml' file allows users to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

The workloadmgr CLI client is provided as rpm and deb packages.

The following operating systems have been verified with the installation:

• Rocky9, RHEL9

• CentOS8

• Ubuntu 18.04, Ubuntu 20.04

Installing the workloadmgr client will automatically install all required OpenStack clients as well.

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

• Status (Enabled/Disabled)

• Start Day/Time

• End Day

Before creating a Slack Webhook URL, ensure the following requirements are met:

• A Slack workspace where you have admin privileges.

• Basic understanding of enterprise messaging platforms.

• Access to the Slack API portal.

Delete tvocontrolplane resource

Use the following command

List openshift trilio control plane nodes

SSH Login to all these nodes and unmount the backup targets used by Trilio control plane services

Delete dataplane trilio resources from openshift

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

Trilio creates databases for dmapi and workloadmgr services. These databases need to be cleaned.

Login into the database cluster

Run the following SQL statements to clean the database.

Troubleshooting inside a complex environment like OpenStack can be very time-consuming.

The following tips will help to speed up the troubleshooting process to identify root causes.

OpenStack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

The Trilio Workloadmgr is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process is the Trilio Workloadmgr also responsible for gathering the metadata about the backed-up VMs and networks from the OpenStack environment. It sends API calls to the OpenStack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received, the Trilio Workloadmgr writes it as JSON files on the Backup Target.

The Trilio Workloadmgr is also sending the Cinder Snapshot command.

During the restore process, the Trilio Workloadmgr reads the VM metadata from its Database and uses the metadata to create the Shell for the restore. It sends API calls to the OpenStack environment to create the necessary resources.

1] Procedure to Access the Grafana Dashboard for Trilio

The following command provides the URL for the Grafana dashboard:

oc -n openshift-user-workload-monitoring get routes -n openshift-user-workload-monitoring

The above command shows the Grafana URL. Log in to Grafana using this URL.

Use the credentials mentioned in the grafana-secret-creds.yaml file to log in to the Grafana UI.

Alternatively, you can fetch the credentials using the following commands:

kubectl get secret -n openshift-user-workload-monitoring credentials -o "jsonpath={.data.GF_SECURITY_ADMIN_USER}" | base64 --decode

kubectl get secret -n openshift-user-workload-monitoring credentials -o "jsonpath={.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode

1.1] Import Grafana Dashboard File

Download or navigate to the Grafana dashboard JSON file using the following location:

cd triliovault-cfg-scripts/redhat-director-scripts/monitoring-openshift/grafana/

Grafana dashboard JSON file:

vi TrilioGrafanaDashboard.json

2] Steps to Import a Grafana Dashboard

After successful login to Grafana, follow these steps.

2.1] Go to Dashboards

a) On the left sidebar, click the Dashboards icon (four-square icon).

b) Click New → Import.

(In some Grafana versions, you may see an Import option directly in the Dashboards section.)

You will see the following options:

• Import dashboard from file or import via dashboard JSON model

• Upload or paste JSON file

Perform one of the following actions:

• Click Upload JSON file, then select your dashboard JSON file (for example, TrilioGrafanaDashboard.json)

OR

• Paste the JSON content directly into the text box

• Grafana will create the dashboard from the JSON file

• You will be redirected to the dashboard view

Select the Prometheus datasource when prompted.

The dashboard import is now complete.

The Grafana dashboard has two levels:

1. Project level

2. Cloud level

This panel shows the successful, failed, and running backups.

• Green indicates successful backups

• Red indicates failed backups

• Yellow indicates running backups

The panel also shows the percentage ratio of each status.

This panel shows the successful, failed, and running restore operations.

• Green indicates successful restores

• Red indicates failed restores

• Yellow indicates running restores

The panel also shows the percentage ratio of each status.

This panel shows the number of protected and unprotected instances.

This panel shows the quota usage details for:

• Backups

• Virtual machines

• Snapshots

• Workloads

This panel shows the number of workloads available in the selected project.

This panel shows the same details as the project level panels, but it displays the information for the entire cloud instead of a specific project.

Trilio enables OpenStack administrators to set Project Quotas against the usage of Trilio.

The following Quotas can be set:

• Number of Workloads a Project is allowed to have

• Number of Snapshots a Project is allowed to have

• Number of VMs a Project is allowed to protect

Trilio Branch

Branch of the repository to be used for DevOps scripts.

trilio_branch : 6.2.0

Trilio Branch for Grafana setup

Branch of the repository to enable grafana monitoring on RHOSO18.0.

trilio_grafana_branch : 6.2.0

Deployment Scripts

git clone -b 6.2.0 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/

git clone -b 6.2.0 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/kolla-ansible/ansible

Container/Revision Information

All packages are Python 3 (py3.6 - py3.10) compatible only.

1. Open Grafana Dashboard

• Log in to Grafana.

• Navigate to Alerting → Contact Points.

2. Create a Webhook Contact Point

• Click New Contact Point.

• Enter a name (e.g., custom-webhook-alert).

• Select Webhook as the integration type.

• URL → This is the endpoint where Grafana will send alerts. You must provide a reachable API URL (for example, your backend service or webhook receiver).

Example:

• HTTP Method → POST (default).

• Authentication → Configure if your API requires authentication.

• Custom Headers → Add headers if required by your API.

Example header:

Grafana allows you to customize the JSON payload sent to your webhook.

• Click Save Contact Point.

• Your webhook notification channel is now ready to use.

• Go to Alerting → Alert Rules.

• Click New Alert Rule.

• Select a data source (e.g., Prometheus).

• Write a query for the metric you want to monitor.

Example:

Example condition:

This means the alert will trigger if the condition remains true for 5 minutes.

• Under Configure Notifications, select the contact point you created (custom-webhook-alert).

• Save the alert rule.

• Go to Contact Points.

• Select your webhook.

• Click Test.

Grafana will send a sample JSON payload to your configured endpoint URL.

• Python must be installed on the system.

• The pip package manager must be available.

• Install the Flask Python module using the following command: pip3 install flask.

• All commands must be executed on the system where the webhook server will run.

1. Save the above Python code in a file (e.g., server1.py).

2. Test the webhook:

• A file named test.json will be created.

• It will contain the received webhook payload.

1. Run the following command on your host machine:

1. Once the server starts, it will be available at:

Use this URL in Grafana as your webhook endpoint.

This confirms that Grafana successfully sent an alert to your webhook.

Note: Similarly, if you follow the above steps and create a Grafana alert rule (for example, using the MySQL snapshot query), the webhook will receive the actual alert payload. The details will be stored in test.json, confirming that Grafana successfully sent the alert to your webhook.

Follow the steps below to remove a backup target.

1. Clean Workloads and Backups (Optional)

• All data related to workloads and the cloud resides on the backup target. The administrator can clean this data from the backup target before deletion, but this step is optional and should not prevent the user from deleting the backup target.

• Before cleaning up workloads or snapshots, log in to the correct OpenStack project, as each backup target and its workloads belong to that specific project. Select the right project you need before proceeding.

• Delete all backups stored on the required dynamically added S3 backup target using the UI (Horizon) or Trilio CLI.

Refer to for detailed steps.

Refer to for detailed steps.

Note: Repeat this procedure for each backup and workload that needs to be deleted from the S3 backup target created earlier.

Deleting a backup target removes it from the TrilioVault database. Make sure you do not need this backup target anymore before you delete it.

• Log in to the OpenStack Horizon Dashboard as an Admin user.

• Navigate to Admin → Backups-Admin → Backup Targets.

• Click the button next to the target, and confirm the deletion when prompted.

• Multiple targets can be deleted by selecting their checkboxes and clicking the

1. Identify the WLM API pod in the trilio-openstack namespace:

2. Exec into the pod and source the admin RC file:

1. Exec into the wlm-api container and source the admin RC file:

1. List the backup targets to find the ID of the one you want to delete:

   Note the Backup Target ID and the Secret Ref URL (for S3 targets) — you will need the secret ref to clean up the Barbican secret in step 3.

2. Delete the backup target:

3. Verify deletion:

In 6.2, the S3 access/secret keys for each backup target are stored in an OpenStack Barbican secret. Once the backup target record is gone, the corresponding Barbican secret is no longer referenced and can be removed.

From inside the WLM API pod/container:

Or use the Secret Ref URL captured from workloadmgr backup-target-list in section 2 directly:

DMS unmounts the backup target automatically when the last in-flight job releases it; no manual unmount is required. You can confirm that:

• The deleted backup target no longer appears in workloadmgr backup-target-list.

• The DMS server pods on the controller and compute nodes are still Running:

• No per-BT triliovault-object-store-<bt-name>-* pods remain in the trilio-openstack namespace (in 6.2 they should never be present):

• The DMS server containers on the controller and compute nodes are still Running:

Introduction

Integrating Trilio for OpenStack with Barbican facilitates encryption support for the qcow2-data segment of Trilio backups. However, the JSON files housing the backed-up OpenStack metadata remain unencrypted.

This capability necessitates the presence of the OpenStack Barbican service. Absence of the Barbican service will result in the omission of encryption options for Workloads within the Horizon interface.

Trilio for OpenStack (T4O) exclusively retrieves secrets from Barbican, without generating, modifying, or deleting any secrets within the Barbican platform.

Barbican secrets are indispensable for executing backups or restorations in encryption-enabled Workloads. The onus lies on the OpenStack project user to supply these secrets and guarantee their accurate availability.

In order to employ encrypted Workloads, the Trilio trustee role must be capable of interacting with the Barbican service to access and retrieve secrets from Barbican. By default, only the 'admin' and 'creator' roles are endowed with these permissions.

Encryption availability for a Workload is confined to the Workload creation stage. Post-creation, the encryption status of a Workload is irreversible; it cannot be altered or toggled.

The following secret configurations are supported for AES-256

• Alghorithm: AES-256 (All supported types utilize this algorithm)

• Mode: cbc

• content type: application/octet-stream

• payload-content-encoding: base64

For encrypted workload Barbican should enabled on Openstack, then while configuring T4O, trustee role should be kept as creator .

Additionally every user who’ll be interacting with TrilioVault for any operator, should also have creator role assigned.

Secret can be created via OpenStack cli with following the below steps

1. Source the rc file of the user who is going to create the encrypted workload.

2. Create any supported type of secret and fetch the secret UUID using OpenStack secret cli.

   1. Refer to this guide for more information about

   2. Below is the example of symmetric secret type creation & fetch the secret UUID.

b. View the details of the order to identify the location of the generated key, shown here as the Secret href value:

c. Fetch the secret UUID via below command (use Secret href)

Note down the last value from Secret href URL which is UUID (8-4-4-4-12 format).

For example secret UUID for the Secret href URL http://172.30.0.75:9311/v1/secrets/b14bc1ae-9251-4dd5-9fd4-18ef50bbb664 will be b14bc1ae-9251-4dd5-9fd4-18ef50bbb664

This UUID will be used further for creating the encrypted workload.

1. Login to openstack horizon dashboard with user who has created the secret & go to Project->Backup->workloads

2. click on create workload.

3. select the checkbox for Enable Encryption

4. Provide the UUID noted in above steps in Secret UUID text box

1. Follow the usual procedure for further tabs (Workload member, Schedule, policy & Option) & click on create.

2. Workload will be created and value for Encryption field will be True.

For migration of encrypted workload please follow:

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

• A Canonical (Charmed) OpenStack base setup deployed for a required release, with all units in active/idle state. Refer to the .

• Juju controller connected to the OpenStack model. Verify with juju status.

The following steps need to be run on all nodes, which have the Trilio Datamover API & Workloadmanager services running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the below entries

Once the role that runs the Trilio Datamover API & Workloadmanager services has been identified will the following commands clean the nodes from the service.

Remove triliovault_datamover_api container.

Clean Trilio Datamover API service conf directory.

Clean Trilio Datamover API service log directory.

Remove triliovault_wlm_api container.

Clean Trilio Workloadmanager API service conf directory.

OpenStack is designed to be scalable. To manage scale, OpenStack supports various resource segregation constructs: Regions, cells, and availability zones to manage OpenStack resources. Resource segregation is essential to define fault domains and localize network traffic.

From an end user's perspective, OpenStack regions are equivalent to regions in Amazon Web Services. Regions live in separate data centers, often named after their location. If your organization has a data center in Chicago and one in Boston, you'll have at least a CHG and a BOS region. Users who want to disperse their workloads geographically will place some in CHG and some in BOS. Regions have separate API endpoints for all services except for Keystone. Users, Tenants, and Domains are shaped across regions through a single Keystone deployment.

Availability Zones are an end-user visible logical abstraction for partitioning a cloud without knowing the physical infrastructure. Availability zones can partition a cloud on arbitrary factors, such as location (country, data center, rack), network layout, and power source. Because of the flexibility, the names and purposes of availability zones can vary massively between clouds.

In addition, other services, such as the and the , also provide an availability zone feature. However, the implementation of these features differs vastly between these different services. Please look at the documentation for these other services for more information on their implementation of this feature.

Cells

POST https://$(tvm_address):8780/v1/$(tenant_id)/search

Starts a File Search with the given parameters